Federated　Learning　(FL)　is　widely　used　in　various　industries　because　it　effectively　addresses　the　predicament　of　isolated　data　island.　However,　eavesdroppers　is　capable　of　inferring　user　privacy　from　the　gradients　or　models　transmitted　in　FL.　Homomorphic　Encryption　(HE)　can　be　applied　in　FL　to　protect　sensitive　data　owing　to　its　computability　over　ciphertexts.　However,　traditional　HE　as　a　single-key　system　cannot　prevent　dishonest　users　from　intercepting　and　decrypting　the　ciphertexts　from　cooperative　users　in　FL.　Guaranteeing　privacy　and　efficiency　in　this　multi-user　scenario　is　still　a　challenging　target.　In　this　article,　we　propose　a　secure　and　efficient　Federated　Learning　scheme　(SecFed)　based　on　multi-key　HE　to　preserve　user　privacy　and　delegate　some　operations　to　TEE　to　improve　efficiency　while　ensuring　security.　Specifically,　we　design　the　first　TEE-based　multi-key　HE　cryptosystem　(EMK-BFV)　to　support　privacy-preserving　FL　and　optimize　operation　efficiency.　Furthermore,　we　provide　an　offline　protection　mechanism　to　ensure　the　normal　operation　of　system　with　disconnected　participants.　Finally,　we　give　their　security　proofs　and　show　their　efficiency　and　superiority　through　comprehensive　simulations　and　comparisons　with　existing　schemes.　SecFed　offers　a　3x　performance　improvement　over　TEE-based　scheme　and　a　2x　performance　improvement　over　HE-based　solution.