Data-Free　Knowledge　Distillation　(DFKD)　has　shown　great　potential　in　creating　a　compact　student　model　while　alleviating　the　dependency　on　real　training　data　by　synthesizing　surrogate　data.　However,　prior　arts　are　seldom　discussed　under　distribution　shifts,　which　may　be　vulnerable　in　real-world　applications.　Recent　Vision-Language　Foundation　Models,　e.g.,　CLIP,　have　demonstrated　remarkable　performance　in　zero-shot　out-of-distribution　generalization,　yet　consuming　heavy　computation　resources.　In　this　paper,　we　discuss　the　extension　of　DFKD　to　Vision-Language　Foundation　Models　without　access　to　the　billion-level　image-text　datasets.　The　objective　is　to　customize　a　student　model　for　distribution-agnostic　downstream　tasks　with　given　category　concepts,　inheriting　the　out-of-distribution　generalization　capability　from　the　pre-trained　foundation　models.　In　order　to　avoid　generalization　degradation,　the　primary　challenge　of　this　task　lies　in　synthesizing　diverse　surrogate　images　driven　by　text　prompts.　Since　not　only　category　concepts　but　also　style　information　are　encoded　in　text　prompts,　we　propose　three　novel　Prompt　Diversification　methods　to　encourage　image　synthesis　with　diverse　styles,　namely　Mix-Prompt,　Random-Prompt,　and　Contrastive-Prompt.　Experiments　on　out-of-distribution　generalization　datasets　demonstrate　the　effectiveness　of　the　proposed　methods,　with　Contrastive-Prompt　performing　the　best.　©　2023　ACM.