Research on weld surface defect detection method based on RGB-D feature fusion

XIE Yuxin; GONG Yefei; GU Xinhao; CHEN Xiaobin; WANG Meng; XU Huigang

doi:10.12073/j.hjxb.20230712002

TRANSACTIONS OF THE CHINA WELDING INSTITUTION > 2024 > 45(12): 72-78. > DOI: 10.12073/j.hjxb.20230712002

XIE Yuxin, GONG Yefei, GU Xinhao, CHEN Xiaobin, WANG Meng, XU Huigang. Research on weld surface defect detection method based on RGB-D feature fusion[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2024, 45(12): 72-78. DOI: 10.12073/j.hjxb.20230712002

Citation:

PDF (10337 KB)

Research on weld surface defect detection method based on RGB-D feature fusion

1.
School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, 221116, China
2.
School of Electrical and Automation Engineering, Changshu Institute of Technology, Changshu, 215500, China
3.
School of Physics and Electronic Engineering, Northeast Petroleum University, Daqing, 163318, China

More Information

Received Date: July 11, 2023
Available Online: May 14, 2024

Graphical Abstract

Abstract

Abstract

The correct detection of surface defects of metal welds is the prerequisite to ensure the safe use of workpieces. Because the defects are similar in color to the base material and the images are not clear, it is difficult to completely detect all defect categories using conventional 2D RGB vision, and depth information needs to be added to assist detection. Therefore, a weld surface defect detection method based on RGB-D data feature fusion is proposed. Based on YOLOv8 network model, this method extracts effective feature layers of RGB and depth features by using improved symmetric backbone network structure, and introduces RGB-D data feature fusion module to realize the fusion of RGB and depth features in space and channel position. The CIOU-NMS non-maximum suppression module was added to YOLOv8 model to improve the accuracy of the check box. In this paper, the experiments were carried out in four categories: pore, weld beading, splash and burnthrough. The results showed that the missed detection rate of the improved YOLOv8 was reduced by 17.84% and the false detection rate by 19.46% compared with YOLOv8. The effectiveness and accuracy of the proposed method are proved.
- target detection,
- weld defect detection,
- feature fusion,
- YOLOv8

FullText(HTML)

References (17)

References

[1]	Viola P, Jones M J. Robust real-time face detection[J]. International Journal of Computer Vision, 2004, 57(2): 137 − 154. doi: 10.1023/B:VISI.0000013087.49260.fb
[2]	胡丹, 吕波, 王静静, 等. 焊缝表面缺陷激光视觉传感HOG-SVM的检测方法[J]. 焊接学报, 2023, 44(1): 56 − 62. doi: 10.12073/j.hjxb.20211231001 Hu Dan, Lü Bo, Wang Jingjing, et al. Weld surface defect detection method based on laser vision sensor HOG-SVM[J]. Transactions of the Chinese Welding Institution, 2023, 44(1): 56 − 62. doi: 10.12073/j.hjxb.20211231001
[3]	Girshick R, Donahue J, Darrell T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 2014: 580 − 587.
[4]	Girshick R. Fast R-CNN[C]//2015 IEEE International Conference on Computer Vision, Santiago, Chile, USA, 2015: 1440 − 1448.
[5]	Ren S, He K, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137 − 1149. doi: 10.1109/TPAMI.2016.2577031
[6]	He K, Gkioxari G, Dollár P, et al. Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 386 − 397. doi: 10.1109/TPAMI.2018.2844175
[7]	Redmon J, Divvala S, Girshick R, et al. You only look once: unified, real-time object detection[C]//2016 IEEE Conference on Computer Vision and Pattern Recognitio, Las Vegas, NV, USA, 2016: 779 − 788.
[8]	吴天成, 王晓荃, 蔡艺军, 等. 基于特征融合的轻量级SSD目标检测方法[J]. 液晶与显示, 2021, 36(10): 1437 − 1444. doi: 10.37188/CJLCD.2021-0007 Wu Tiancheng, Wang Xiaoquan, Cai Yijun, et al. Lightweight SSD target detection method based on feature fusion[J]. Chinese Journal of Liquid Crystals and Displays, 2021, 36(10): 1437 − 1444. doi: 10.37188/CJLCD.2021-0007
[9]	Bocllkovskiy A, Wang C Y, Liao H. YOLOv4: Optimal speed and accuracy of object detection[EB/OL]. arXiv preprint arXiv: 2004.10934. Available at: https://arxiv.org/abs/2004.10934.
[10]	Wang C Y, Bochkovskiy A, Liao H Y M. YOLOv7 trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 7464 − 7475.
[11]	Zuo Y, Wang J, Song J. Application of YOLO object detection network in weld surface defect detection[C]//2021 IEEE 11th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems, Jiaxing, China. 2021: 704-710.
[12]	王睿, 胡云雷, 刘卫朋, 等. 基于边缘AI的焊缝X射线图像缺陷检测[J]. 焊接学报, 2022, 43(1): 79 − 84. doi: 10.12073/j.hjxb.20210516001 Wang Rui, Hu Yunlei, Liu Weipeng, et al. Weld X-ray image defect detection based on edge AI[J]. Transactions of the China Welding Institution, 2022, 43(1): 79 − 84. doi: 10.12073/j.hjxb.20210516001
[13]	Pablo R, Manuel R. Weld bead detection based on 3D geometric features and machine learning approaches[J]. IEEE Access, 2019, 7: 14714 − 14727. doi: 10.1109/ACCESS.2019.2891367
[14]	张文铠, 刘佳. 基于改进YOLOv8s的钢材表面缺陷检测[J]. 北京信息科技大学学报(自然科学版), 2023, 38(6): 33 − 40. Zhang Wenkai, Liu Jia. Steel surface defect detection based on improved YOLOv8s[J]. Journal of Beijing University of Information Science and Technology (Natural Science Edition), 2023, 38(6): 33 − 40.
[15]	Li X, Lü C, Wang W, et al. Generalized focal loss: towards efficient representation learning for dense object detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(3): 3139 − 3153.
[16]	Oro D, Fernández C, Martorell X, et al. Work-efficient parallel non-maximum suppression kernels[J]. The Computer Journal, 2022, 65(4): 773 − 787.
[17]	Lin T, Maire M, Belongie S, et al. Microsoft COCO: common objects in context[C]//European Conference on Computer Vision, Cham, Switzerland, 2014: 740 − 755.

[1]	WANG Tianqi, MENG Kaiquan, WANG Chuanrui. Prediction and optimization of multi-layer and multi-pass welding process parameters based on GA-BP neural network[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2024, 45(5): 29-37. DOI: 10.12073/j.hjxb.20230523001
[2]	LI Chengwen, JI Haibiao, YAN Zhaohui, LIU Zhihong, MA Jianguo, WANG Rui, WU Jiefeng. Prediction of residual stress and deformation of 316L multi-layer multi-pass welding based on GA-BP neural network[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2024, 45(5): 20-28. DOI: 10.12073/j.hjxb.20230520002
[3]	DONG Jianwei, HU Jianming, LUO Zhen. Quality prediction of aluminum alloy resistance spot welding based on correlation analysis and SSA-BP neural network[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2024, 45(2): 13-18, 32. DOI: 10.12073/j.hjxb.20230226001
[4]	FAN Ding, HU Ande, HUANG Jiankang, XU Zhenya, XU Xu. X-ray image defect recognition method for pipe weld based on improved convolutional neural network[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2020, 41(1): 7-11. DOI: 10.12073/j.hjxb.20190703002
[5]	YU Guo, YIN Yuhuan, GAO Jiashuang, GUO Lijie. Orthogonal experiment method and BP neural networks in optimization of microbeam TIG welded GH4169[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2018, 39(11): 119-123. DOI: 10.12073/j.hjxb.2018390285
[6]	ZHANG Aihua, GAO Folai, NIU Xiaoge, LUO Huan. Prediction of gray-spot area in rail flash butt welded joint based on BP neural network[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2016, 37(11): 11-14.
[7]	GAO Xiangdong, LIN Jun, XIAO Zhenlin, CHEN Xiaohui. Recognition model of arc welding penetration using ICA-BP neural network[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2016, 37(5): 33-36.
[8]	ZOU Yirong, WU Zheming, GUO Guilin, DU Dong. Image processing algorithm for weld seam recognition based on color analyzing[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2009, (10): 37-40.
[9]	ZHANG Peng-xian, CHEN Jian-hong, DU Wen-jiang. Quality monitoring of resistance spot welding based on image processing of welding spot surface[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2006, (12): 57-60,64.
[10]	WANG Qing-xiang, SUN Bing-da, LI Di. Image processing method for recognizing position of welding seam[J]. TRANSACTIONS OF THE CHINA WELDING INSTITUTION, 2005, (2): 59-63.

Cited By

Cited by

Periodical cited type(1)

王振民，吴健文，范文艳，叶春显. 基于SiC MOSFET的谐振软开关等离子体电源. 华南理工大学学报(自然科学版). 2019(01): 1-6 .

Other cited types(8)

Get Citation

PDF

XML

Article views (220) PDF downloads (58) Cited by(9)

Turn off MathJax

Article Contents

Abstract

References

Research on weld surface defect detection method based on RGB-D feature fusion