Crack　detection　is　a　critical　task　in　routine　inspection　of　building　structures.　Most　of　the　traditional　crack　detection　methodologies　are　conducted　by　human　inspectors　that　may　submit　inaccurate　damage　assessments.　In　recent　years,　deep　learning　has　produced　extremely　promising　results　for　various　tasks.　In　this　work,　a　lightweight　end-to-end　pixel-wise　classification　architecture　called　SegNet　is　employed　to　segment　the　structure　surface　cracks.　Compared　with　other　semantic　segmentation　architectures,　SegNet　uses　pooling　indices　calculated　in　the　pooling　step　of　the　encoder　to　perform　non-linear　upsampling　in　the　corresponding　decoder,　which　doesn’t　require　to　learn　in　the　upsample.　In　this　paper,　a　crack　image　dataset　collected　under　a　variety　of　complex　environment　are　utilized　to　train　and　test　the　SegNet　model,　i.e.　a　self-labeled　dataset　with　2068　bridge　cracks　images　at　the　size　of.　In　order　to　improve　the　generalization　ability　of　network　data　augmentation　is　used.　The　experimental　results　show　that　the　SegNet　outperforms　the　traditional　edge　detection　algorithm,　such　as　Canny　and　Sobel,　in　the　dataset.　The　trained　SegNet　model　can　be　used　to　segment　the　cracks　in　images　at　any　size　with　the　assistant　of　sliding　window　scanning　technique.　©　Springer　Nature　Switzerland　AG　2019.