In　reinforcement,　exploration　and　utilization　of　agents＇　action　selection　has　always　been　the　key　problem.　Agents　should　not　only　make　full　use　of　maximum　action,　but　also　explore　potential　optimal　action.　Inspired　by　the　exploration　and　utilization　of　actions　selection,　a　novel　value　function　exploration　algorithm　based　on　an　error　Gaussian　mixture　model　(EGMM)　is　proposed　in　this　paper.　First,　appropriate　variables　are　chosen　from　error　data,　and　the　number　of　Gaussian　components　are　obtained　by　optimizing　a　Bayesian　information　criterion　via　the　EGMM.　Then,　the　EGMM　is　used　for　the　fitting　and　calculation　of　error　data　to　obtain　the　conditional　error　mean　to　compensate　for　the　output,　thus　obtaining　more　accurate　results.　We　test　the　performance　of　the　designed　algorithm　via　a　virtual　experimental　platform　in　a　cloud　computing　environment.　Experiments　demonstrate　the　proposed　algorithm　eliminate　the　influence　of　non-Gaussian　noise　on　model　prediction　performance.