In　this　paper,　we　propose　a　novel　multi-scale　attention　based　network　(called　MSA-Net)　for　feature　matching　problems.　Current　deep　networks　based　feature　matching　methods　suffer　from　limited　effectiveness　and　robustness　when　applied　to　different　scenarios,　due　to　random　distributions　of　outliers　and　insufficient　information　learning.　To　address　this　issue,　we　propose　a　multi-scale　attention　block　to　enhance　the　robustness　to　outliers,　for　improving　the　representational　ability　of　the　feature　map.　In　addition,　we　also　design　a　novel　context　channel　refine　block　and　a　context　spatial　refine　block　to　mine　the　information　context　with　less　parameters　along　channel　and　spatial　dimensions,　respectively.　The　proposed　MSA-Net　is　able　to　effectively　infer　the　probability　of　correspondences　being　inliers　with　less　parameters.　Extensive　experiments　on　outlier　removal　and　relative　pose　estimation　have　shown　the　performance　improvements　of　our　network　over　current　state-of-the-art　methods　with　less　parameters　on　both　outdoor　and　indoor　datasets.　Notably,　our　proposed　network　achieves　an　11.7%　improvement　at　error　threshold　5　degrees　without　RANSAC　than　the　state-of-the-art　method　on　relative　pose　estimation　task　when　trained　on　YFCC100M　dataset.