Ultra-high　resolution　image　segmentation　poses　a　formidable　challenge　for　UAVs　with　limited　computation　resources.　Moreover,　with　multiple　deployed　tasks　(e.g.,　mapping,　localization,　and　decision　making),　the　demand　for　a　memory　efficient　model　becomes　more　urgent.　This　letter　delves　into　the　intricate　problem　of　achieving　efficient　and　effective　segmentation　of　ultra-high　resolution　UAV　imagery,　while　operating　under　stringent　GPU　memory　limitation.　To　address　this　problem,　we　propose　a　GPU　memory-efficient　and　effective　framework.　Specifically,　we　introduce　a　novel　and　efficient　spatial-guided　high-resolution　query　module,　which　enables　our　model　to　effectively　infer　pixel-wise　segmentation　results　by　querying　nearest　latent　embeddings　from　low-resolution　features.　Additionally,　we　present　a　memory-based　interaction　scheme　with　linear　complexity　to　rectify　semantic　bias　beneath　the　high-resolution　spatial　guidance　via　associating　cross-image　contextual　semantics.　For　evaluation,　we　perform　comprehensive　experiments　over　public　benchmarks　under　both　conditions　of　small　and　large　GPU　memory　usage　limitations.　Notably,　our　model　gains　around　3%　advantage　against　SOTA　in　mIoU　using　comparable　memory.　Furthermore,　we　show　that　our　model　can　be　deployed　on　the　embedded　platform　with　less　than　8　G　memory　like　Jetson　TX2.