Semantic　information　from　images　can　be　used　to　improve　the　performance　of　deep　learning　methods　in　recognizing　human　emotions.　In　this　paper,　we　propose　a　novel　framework　based　on　the　graph　convolutional　network　for　emotion　recognition　by　utilizing　the　semantic　relationships　of　different　regions.　First,　we　extract　the　salient　image　regions　within　video　frame　clips　by　using　the　bottom-up　attention　module　to　construct　the　node　features　of　a　graph.　Then,　we　build　the　graphs　containing　the　node　features　and　the　semantic　correlations　of　nodes　by　using　the　graph　convolutional　network.　For　refinement,　each　node　feature　of　graph　vectors　is　enhanced　via　a　gated　recurrent　unit　consisting　of　gate　and　memory　units　to　remove　redundant　feature　information.　Experimental　results　show　that　our　proposed　method　achieves　superior　performance　over　state-of-the-art　approaches　for　the　emotion　recognition　on　the　CEAR　and　AFEW　datasets.