With　the　rapid　development　of　manipulation　technologies,　the　generation　of　Deep　Fake　videos　is　more　accessible　than　ever.　As　a　result,　face　forgery　detection　becomes　a　challenging　task,　attracting　a　significant　amount　of　attention　from　researchers　worldwide.　However,　most　previous　work,　consisting　of　convolutional　neural　networks　(CNN),　is　not　sufficiently　discriminative　and　cannot　fully　utilise　subtle　clues　and　similar　textures　during　the　process　of　facial　forgery　detection.　Moreover,　these　methods　cannot　simultaneously　consider　accuracy　and　time　efficiency.　To　address　such　problems,　we　propose　a　novel　framework　named　FPC-Net　to　extract　some　meaningful　and　unnatural　expressions　in　local　regions.　This　framework　utilises　CNN,　long　short-term　memory　(LSTM),　channel　groups　loss　(CG-Loss)　and　adaptive　feature　fusion　to　detect　face　forgery　videos.　First,　the　proposed　method　exploits　spatial　features　by　CNN,　and　a　channel-wise　attention　mechanism　is　employed　to　separate　channels.　Specifically,　with　the　help　of　channel　groups　loss,　the　channels　are　divided　into　two　groups,　each　representing　a　specific　class.　Second,　LSTM　is　applied　to　learn　the　correlation　of　spatial　features.　Finally,　the　correlation　of　features　is　mapped　into　other　latent　spaces.　Through　a　lot　of　experiments,　the　results　are　that　the　detection　speed　of　the　proposed　method　reaches　420　FPS　and　the　auc　scores　achieve　best　performance　of　99.7%,　99.9%,　94.7%,　and　82.0%　on　Raw　Celeb-DF,　Raw　Face　Forensics++,　F2F　and　NT　datasets　respectively.　The　experimental　results　demonstrate　that　the　proposed　framework　has　great　time　efficiency　performance　while　improving　the　detection　performance　compared　with　other　frame-level　methods　in　most　cases.　©　2022　The　Authors.　IET　Computer　Vision　published　by　John　Wiley　&　Sons　Ltd　on　behalf　of　The　Institution　of　Engineering　and　Technology.