There　are　huge　differences　in　data　distribution　and　feature　representation　of　different　modalities.　How　to　flexibly　and　accurately　retrieve　data　from　different　modalities　is　a　challenging　problem.　The　mainstream　common　subspace　method　only　focus　on　the　heterogeneity　gap　between　modalities,　and　use　a　unified　method　to　jointly　learn　the　common　representation　of　different　modalities,　which　can　easily　lead　to　the　difficulty　of　multi-modal　unified　fitting.　In　this　work,　we　innovatively　propose　the　concept　of　multi-modal　information　density　discrepancy,　and　propose　a　modality-specific　adaptive　scaling　method　incorporating　prior　knowledge,　which　can　adaptively　learn　the　most　suitable　network　for　different　modalities.　Comprehensive　experimental　results　on　three　widely　used　cross-modal　retrieval　datasets　show　the　proposed　MASM　achieves　the　state-of-the-art　results　and　significantly　outperforms　other　existing　methods.