Neoadjuvant　systemic　treatment　before　surgery　is　a　prevalent　regimen　in　the　patients　with　advanced-stage　or　high-risk　tumor,　which　has　shaped　the　treatment　strategies　and　cancer　survival　in　the　past　decades.　However,　some　patients　present　with　poor　response　to　the　neoadjuvant　treatment.　Therefore,　it　is　of　great　significance　to　develop　tools　to　help　distinguish　the　patients　that　could　achieve　pathological　complete　response　before　surgery　to　avoid　inappropriate　treat-ment.　Here,　this　study　demonstrated　a　multi-task　deep　learning　tool　called　DeepInteg.　In　the　DeepInteg　framework,　the　segmentation　module　was　constructed　based　on　the　CE-Net　with　a　context　extractor　to　achieve　end-to-end　delineation　of　region　of　interest　(ROI)　from　radiological　images,　then　the　features　of　segmented　Magnetic　Resonance　Imaging　(MRI)　and　Computed　To-mography　(CT)　images　of　each　case　were　fused　and　input　to　the　classification　module　based　on　a　convolution　neural　network　for　treatment　outcome　prediction.　The　dataset　with　1700　MRI　and　CT　slices　collected　from　the　prospectively　randomized　clinical　trial　(NCT01211210)　on　systemic　treatment　for　rectal　cancer　was　used　to　develop　and　systematically　optimize　DeepInteg.　As　a　result,　DeepInteg　achieved　automatic　segmentation　of　tumoral　ROI　with　Dices　of　0.766　and　0.719　and　mIoUs　of　0.788　and　0.756　in　CT　and　MRI　images,　respectively.　In　addition,　DeepInteg　ach-ieved　AUC　of　0.833,　accuracy　of　0.826　and　specificity　of　0.856　in　the　prediction　for　pathological　complete　response　after　treatment,　which　showed　better　performance　compared　with　the　model　based　on　CT　or　MRI　alone.　This　study　provide　a　robust　framework　to　develop　disease-specific　tools　for　automatic　delineation　of　ROI　and　clinical　outcome　prediction.　The　well-trained　Deep-Integ　could　be　readily　applied　in　clinic　to　predict　pathological　complete　response　after　neo-adjuvant　therapy　in　rectal　cancer　patients.