An　intelligent　energy　management　strategy　(EMS)　based　on　an　improved　Reinforcement　Learning　(RL)　algorithm　is　developed　to　enhance　the　adaptability　of　the　EMS　and　to　further　improve　the　fuel　efficiency　of　a　Plug-in　Parallel　Hybrid　Electric　Vehicle　(PHEV).　Both　the　numerical　model　and　the　energy　management　strategy　of　a　plug-in　PHEV　are　described.　The　improved　RL　with　Q-learning　algorithm　is　implemented　to　acquire　the　optimal　control　strategies　for　improving　fuel　economy.　The　Markov　Chain　is　employed　to　calculate　the　Transition　Probability　Matrix　of　the　required　power.　A　Kullback-Leibler　(KL)　divergence　rate　is　designed　to　activate　the　update　of　EMS,　when　a　new　corresponding　driving　cycle　is　expected.　An　Exploration　Factor　(EF)　is　proposed　to　overcome　the　disadvantages　of　the　normal　RL　algorithm　in　convergence　rate　and　reward　cost　evaluation.　The　diverse　KL　divergence　rates　are　examined　to　seek　optimal　solutions.　The　normal-RL　strategy,　rule-based　strategy,　and　dynamic　programming　strategy　are　implemented　as　benchmark　strategies　to　verify　the　effectiveness　of　the　proposed　strategy.　The　validation　results　indicate　that　the　improved　RL　algorithm　with　EF　makes　it　possible　to　promote　the　EMS　capable　of　significantly　improving　the　energy　efficiency　of　a　plug-in　PHEV.