This　paper　proposes　a　robust　fabric　defect　detection　method,　based　on　the　improved　RefineDet.　This　is　done　using　the　strong　object　localization　ability　and　good　generalization　of　the　object　detection　model.　Firstly,　the　method　uses　RefineDet　as　the　base　model,　inheriting　the　advantages　of　the　two-stage　and　one-stage　detectors　and　can　efficiently　and　quickly　detect　defect　objects.　Secondly,　we　design　an　improved　head　structure　based　on　the　Full　Convolutional　Channel　Attention　(FCCA)　block　and　the　Bottom-up　Path　Augmentation　Transfer　Connection　Block　(BATCB),　which　can　improve　the　defect　localization　accuracy　of　the　method.　Finally,　the　proposed　method　applies　many　general　optimization　methods,　such　as　attention　mechanism,　DIoU-NMS,　and　cosine　annealing　scheduler,　and　verifies　the　effectiveness　of　these　optimization　methods　in　the　fabric　defect　localization　task.　Experimental　results　show　that　the　proposed　method　is　suitable　for　the　defect　detection　of　fabric　images　with　unpattern　background,　regular　patterns,　and　irregular　patterns.　©　2020　by　the　authors.　Licensee　MDPI,　Basel,　Switzerland.