Face　tracking　in　surveillance　videos　is　one　of　the　important　issues　in　the　field　of　computer　vision　and　has　realistic　significance.　In　this　paper,　a　new　face　tracking　framework　in　videos　based　on　convolutional　neural　networks　(CNNs)　and　Kalman　filter　algorithm　is　proposed.　The　framework　uses　a　rough-to-fine　CNN　to　detect　faces　in　each　frame　of　the　video.　The　rough-to-fine　CNN　method　has　a　higher　accuracy　in　complex　scenes　such　as　face　rotation,　light　change　and　occlusion.　When　face　tracking　fails　due　to　severe　occlusion　or　significant　rotation,　the　framework　uses　Kalman　filter　to　predict　face　position.　The　experimental　results　show　that　the　proposed　method　has　high　precision　and　fast　processing　speed.