To　overcome　the　numerical　singularity　of　the　yield　surface　based　on　the　Hoek-Brown　strength　criterion,　a　new　approach　was　presented　to　smooth　the　edges　of　the　yield　surface.　The　ideal　elasto-plastic　constitutive　model　was　implemented　in　ABAQUS　software　with　the　fully　implicit　backward　Euler　integral　algorithm　employed　in　the　stress　updating.　The　strength　reduction　method　in　finite　element　method　based　on　the　Hoek-Brown　criterion　was　implemented　with　the　smoothed　yield　function.　The　similarities　and　differences　of　the　strength　reduction　techniques　applied　with　the　Mohr-Coulomb　strength　criterion　and　the　Hoek-Brown　nonlinear　strength　criterion　were　discussed.　The　modeling　on　a　circular　tunnel　shows　that　the　numerical　results　of　stresses　and　plastic　zones　in　the　surrounding　rock　agree　well　with　the　analytical　results.　The　modeling　on　a　rock　slope　shows　that　in　comparison　with　the　linear　strength　reduction　technique　of　the　equivalent　Mohr-Coulomb　model,　the　nonlinear　strength　reduction　technique　based　on　the　Hoek-Brown　model　produced　the　slip　surface　which　agree　better　with　the　characteristics　of　the　slip　surface　of　a　rock　slope,　which　are　shallower　in　location　and　steeper　in　shape.　©　2017,　Science　Press.　All　right　reserved.