The　Video　Quality　Assessment　(VQA)　has　been　attracting　attentions　of　image　processing　community　due　to　its　applications　in　evaluation　and　optimization　of　user　experience　during　video　playback.　Until　now,　the　researches　on　VQA　mainly　focus　on　short　video　sequences　less　than　10 s　that　have　similar　quality　representations　to　images.　In　this　work,　we　propose　a　VQA　model　for　the　medium-length　video　sequences　by　considering　the　user　perception,　expectation　and　memory.　Firstly,　we　aggregate　the　popular　perception-based　VQA　metrics　to　develop　a　hybrid　metric　with　high　consistency　to　subjective　scores.　Secondly,　we　exploit　the　impact　of　user　expectation　on　quality　evaluation　and　further　utilize　it　to　refine　our　model.　Thirdly,　the　exponential　decay　law　of　user　memory　is　introduced　to　generate　the　final　model,　namely,　Expectation　and　Memory-based　VQA　(EM-VQA).　Experimental　results　show　that　the　proposed　model　achieves　superior　performance　than　the　state-of-the-art　VQA　models.　©　2021,　Springer　Nature　Switzerland　AG.