Automated defect inspection system for metal surfaces based on deep learning and data augmentation

基于深度学习和数据增强的金属表面缺陷自动检测系统

简述：卷积变分自动编码器(CVAE)生成特定的图像，再使用基于深度CNN的缺陷分类算法进行分类。在生成足够的数据来训练基于深度学习的分类模型之后，使用生成的数据来训练分类模型。

解决：由于金属制造业的性质，由于一些缺陷很少发生，较难收集

ABSTRACT

owing to the nature of the metal manufacturing industry, it is diﬃcult to obtain enough data because some defects occur rarely.

由于金属制造业的性质，由于一些缺陷很少发生，因此获取足够的数据是一种狂热的做法。

In this study, we propose a new convolutional variational autoencoder (CVAE) and deep CNN-based defect classiﬁcation al- gorithm to solve this problem.A conditional CVAE (CCVAE) is proposed to generate images for each defect type in a single CVAE model.

在这项研究中，我们提出了一种新的卷积变分自动编码器(CVAE)和基于深度CNN的缺陷分类算法来解决这一问题。提出了一种条件CVAE（CCVAE）来为单个CVAE模型中的每种缺陷类型生成图像。

1. Introduction

In general, a smart factory can be deﬁned as a factory that operates base on advanced information and communications technologies.

一般而言，智能工厂可以被定义为基于先进信息和通信技术(信通技术)运作工厂

In order to optimize production conditions and to develop the technology needed to create products using minimal cost and time, it is important to measure the product quality in real time and store the measurement in a data-base.many manufacturers still rely on visual and sampling inspection by an operator, as opposed to total in- spection in real-time.

为了优化生产条件和开发以最小成本和时间创造产品所需的技术，重要的是实时测量产品质量并将测量结果存储在数据库中。许多制造商仍然依赖操作员的目测和抽样检查，而不是实时的全面检查。

For this reason, vision-based systems that automatically detect defects in products have been studied in various industries such as image sensors [1], bottles [2], IC chips [3], fabrics [4]-[5], LCDs [6], car body surfaces [7], magnetic tiles [8], concrete bridges [9], fruits [10], ceramic tiles [11], machines [12]- [13], metals [14] and railways [15].

由于这个原因，已经在各种行业研究了自动检测产品缺陷的基于视觉的系统，如图像传感器[1]，瓶子[2]，IC芯片[3]，织物[4]-[5] LCD[6]]，汽车表面[7]，磁性瓷砖[8]，混凝土桥梁[9]，水果[10]，陶瓷瓷砖[11]，机器[12]-[13]，金属[14]和铁路[15]。

This paper focuses on defect inspection of products in the product quality control process.In this study, we propose a defect classiﬁcation algorithm for metal surfaces using a CNN model.a data augmentation method using the conditional convolu- tional variational autoencoder (CCVAE) in place of a general data augmentation method such as rotation, translation, or noise addition.

本文主要研究产品质量控制过程中的产品缺陷检测问题。在这项研究中，我们提出了一种基于CNN模型的金属表面缺陷分类算法，一种使用条件卷积变分自动编码器(CCVAE)代替诸如旋转、平移或噪声添加的一般数据增强方法的数据增强方法。

2. System conﬁguration

Fig. 1. To inspect the surface of moving metal products, a line scan camera suitable for moving objects is used. The collimated illu- mination is used to increase the signal to noise ratio of the surface defects.Furthermore, the result is displayed on the human-machine interface (HMI) screen, and defect information is provided to the operator.

图1.为了检查移动的金属产品的表面，使用了适合移动物体的线扫描相机。采用准直照明来提高表面缺陷的信噪比。此外，结果显示在人机界面(HMI)屏幕上，并将缺陷信息提供给操作员。

Fig. 2 shows the defect images for the metal surfaces that were obtained from the proposed system.

图2显示了从所提出的系统获得的金属表面的缺陷图像。

The size of the obtained defect images is 120 × 120 (width × height) pixels and consists of six types of defects, as shown in Fig. 2. The brightness of the background also varies. The status of the defect images obtained is shown in Table 1.

得到的缺陷图像大小为120×120(宽×高)像素，由六种缺陷组成，如图2所示。背景的亮度也不同。所获得的缺陷图像的状态如表1所示。

We propose a new defect classiﬁcation algorithm using CCVAE and DCNN in order to solve the problem of im- balanced data between classes and to improve defect classiﬁcation performance.

我们提出了一种新的基于CCVAE和DCNN的缺陷分类算法，以解决类别间数据不平衡的问题，提高缺陷分类性能。

3. Defect classiﬁcation algorithm

The proposed defect classiﬁcation algorithm consists of a data generation module and a defect classiﬁcation module.After generating enough data to train a deep learning- based classiﬁcation model, the classiﬁcation model is trained by using the generated data.

这个提出的缺陷分类算法由数据生成模块和缺陷分类模块组成。在生成足够的数据来训练基于深度学习的分类模型之后，使用生成的数据来训练分类模型。

In an actual test, overﬁtting occurs and the performance drops sharply whenever a slight change in data occurs.

在实际测试中，每当数据发生轻微变化时，就会发生过拟合，性能会急剧下降。

In general, the augmentation method for data classiﬁcation uses ﬂipping, rotation, shifting, addition of noise, blurring, sharpening, and so on. However, there are restrictions when using this method in actual manufacturing.

一般而言，数据分类的增强方法使用翻转、旋转、平移、添加噪声、模糊、锐化等。然而，在实际生产中使用这种方法时有一些限制。

if data augmentation of these defects by the rotation method is used, diﬀerent types of defects will be generated.Eventually, training with such data degrades defect classiﬁcation performance.

如果采用旋转法对这些缺陷进行数据增强，则会产生不同类型的缺陷。最终，使用这样的数据进行训练会降低缺陷分类性能。

In this study, augmentation of images is per- formed by a variational autoencoder method which can generate images by learning the distribution of the given data itself.Based on the generated data, we propose a CNN based classiﬁcation model with high generalization performance.

在这项研究中，图像的增强是通过一种变分自动编码方法来实现的，该方法可以通过学习给定数据本身的分布来生成图像。基于生成的数据，我们提出了一种具有较高泛化性能的基于CNN的分类模型。

3.1. CCVAE

The variational autoencoder (VAE) is a directed graphical model with certain types of latent variables,The VAE consists of an encoder that extracts features from input x and expresses latent variable z, which is the compressed information of the input.The encoder and decoder of the VAE can be composed of deep neural net- works.

变分自动编码器(VAE)是具有某些类型的潜在变量的有向图形模型，VAE由从输入x中提取特征并表示潜在变量z的编码器组成，该潜在变量z是输入的压缩信息。VAE的编解码器可以由深度神经网络组成。

In this study, we propose a CCVAE structure to generate image data for each defect.The CCVAE uses the same convolutional VAE (CVAE) structure based on an encoder and decoder, but here is a diﬀerence, in that it uses information from each class as conditions.

在这项研究中，我们提出了一种CCVAE结构来生成每个缺陷的图像数据。CCVAE使用基于编码器和解码器的相同卷积VAE（CVAE）结构，但这里有一个不同之处，因为它使用来自每个类的信息作为条件。

The diﬀerence is that to generate image data for each type of defect, a defect label composed of one-hot encoding is concatenated with latent variable z to be input to the decoder.

不同之处在于，为了生成每种类型缺陷的图像数据，将由一次热编码组成的缺陷标签与潜在变量z连接以输入到解码器。

The encoder consists of one basic block consisting of a convolution layer, which has 3 × 3 ﬁlters, a batch normalization layer, a rectiﬁed linear unit (ReLU) activation function, and a 2 × 2 max pooling layer, as shown in Fig. 4.

该编码器由一个基本块组成，如图4所示，该基本块由具有3×3个滤波器的卷积层、批归一化层、纠错线性单元(RELU)激活函数和2×2最大池层组成。

Figs. 4 and 5 show the structure of the data generation model based on the CCVAE, which is proposed in this study, to solve the defect data imbalance.

图4和图5显示了基于CCVAE的数据生成模型的结构，该模型是为了解决缺陷数据不平衡而提出的。

As shown in Fig. 6, various images related to the original image are generated using CCVAE, and the generated images are used as the training data of classiﬁer.

如图6所示，使用CCVAE生成与原始图像相关的各种图像，并将生成的图像用作分类器的训练数据。

Fig. 6. Sample images: (a) original image (b) generated image by CCVAE with condition 1.

3.2. Classiﬁcation

We propose a CNN-based deep learning model that can classify defects using the training data generated by CCVAE, as shown in Fig. 8.

我们提出了一个基于CNN的深度学习模型，该模型可以使用CCVAE生成的训练数据对缺陷进行分类，如图8所示。

4. Experimental results

Precision and Recall are complementary, and the higher the two indicators, the better.The F1 score is a harmonic average of precision and recall rate .

查准率和查全率是相辅相成的，两个指标越高越好。F1的分数是准确率和召回率的调和平均值。

Data augmentation converges faster than when raw data is used. The performance index for the trained model's average precision, average recall rate, accuracy, and F1- score is shown in Fig. 10.

数据扩充的收敛速度比使用原始数据时更快。训练模型的平均准确率、平均召回率、准确率和F1得分的性能指标如图10所示。

in the case of data augmentation with CCVAE, performance degradation owing to the amount of data was resolved, and the performance was excellent for all classes.

在使用CCVAE进行数据增强的情况下，解决了由于数据量而导致的性能下降问题，并且所有类别的性能都很优秀。

Therefore, the defect classiﬁcation system proposed in this study showed good performance when tested on actual metal surface defects with imbalanced data.

因此，本文提出的缺陷分类系统在对非平衡数据的实际金属表面缺陷进行测试时表现出了良好的性能。