This article is part of a series where I will be documenting my journey on the development of a social distancing feedback system for the blind as part of the OpenCV Spatial Competition. Check out the full series: Part 1, Part 2.

本文是系列文章的一部分，在该系列文章中，我将记录我在OpenCV空间竞赛中为盲人开发社交疏散反馈系统的过程。 查看完整的系列文章： 第1 部分 ， 第2部分 。

为什么要选择面罩？ (Why Face Mask Detection?)

As mentioned in Part 1 of the series, the objective of this project is to develop a feedback system for the blind to help them maintain the social distance with the people around using the OAK-D (by the way, congratulations to Luxonis LLC and Satya Mallick for the successful Kickstarter campaign!). In this article, I will focus on the detection of the people around using Deep Learning models in order to keep the distance to those people.

如该系列的第1部分所述，该项目的目的是为盲人开发一个反馈系统，以帮助他们使用OAK-D保持与周围人的社交距离 (顺便恭喜， Luxonis LLC和Satya Mallick为成功的Kickstarter广告系列！ )。 在本文中，我将重点介绍使用深度学习模型对周围的人进行检测，以使其与这些人保持距离。

The detection of the people around the user could be done in multiple ways, one option could be to train a model for pedestrian detection as in this repository. Another possible option could be to detect only faces instead of detecting the whole body. The benefit of face detection models is that due to the unique features of faces, it is easier to detect faces even without the need of Deep Learning models. For example, in this OpenCV tutorial a Haar feature-based cascade classifiers method is used to detect faces in real time even in low computation devices.

可以通过多种方式对用户周围的人进行检测，一种选择可以是像在该存储库中那样为行人检测训练模型。 另一个可能的选择是仅检测面部而不是检测整个身体。 人脸检测模型的优势在于，由于人脸的独特功能，即使不需要深度学习模型，也更易于检测人脸。 例如，在此OpenCV教程中 基于Haar特征的级联分类器方法甚至可以在低计算设备中实时检测人脸。

However, in this particular application, we also want to know whether the people around are wearing face mask or not. Then, the best option will be to use only one model that detect where the people are and if they are using face mask, i.e. a face mask detector.

但是，在此特定应用程序中，我们还想知道周围的人是否戴着口罩。 然后，最好的选择是仅使用一种模型来检测人员所在的位置以及是否使用了面罩，即面罩检测器。

面罩检测器过多 (Overabundance of Face Mask Detectors)

Due to the increased use of face masks in the current pandemic, a high number of people have develop systems for face mask detection. A quick search in Github for “face mask detection” returns around 700 repositories on the topic. Similarly, a search for the same term in Youtube returns an endless list of videos showing the implementation of face detection models.

由于当前大流行中越来越多地使用面罩，因此许多人已经开发出用于面罩检测的系统。 在Github中快速搜索“面罩检测”将返回大约700个关于该主题的存储库。 同样， 在Youtube中搜索同一词会返回无尽的视频列表，这些视频显示了人脸检测模型的实现。

So, with such a high number of available examples, I expected that it should be quite easy to find an example that is fast enough for the OpenCV AI Kit with Depth (OAK-D) and that has good accuracy even in real life environments.

因此，由于有大量可用示例，我希望找到一个足够快的带深度的OpenCV AI套件 ( OAK-D )且即使在现实环境中也具有良好准确性的示例应该非常容易。

However, which face mask detection example should we choose?

但是， 我们应该选择哪个面罩检测示例？

First, I decided to look at the top starred face mask detection repositories in Github. In the table below, a list of datasets and models used in some of those repositories are summarized.

首先，我决定查看Github中最受关注的加星标口罩检测存储库。 在下表中，汇总了其中一些存储库中使用的数据集和模型的列表。

As it can be observed, there are two main ways to perform face mask detection: 1. Face detection + face mask classification on each detected face or 2. Directly perform face mask detection. The first method could have better accuracy because already available face detection models have been trained in thousands of face images. In comparison, as ashown in the table, the face mask detection datasets have fewer images for training, where the AIZOOTech dataset has the higher number of images.

可以看出，有两种主要的方法来执行面罩检测： 1.面部检测+在每个检测到的面部上进行面罩分类或2.直接执行面罩检测 。 第一种方法可能具有更好的准确性，因为已经在数千张面部图像中训练了已有的面部检测模型。 相比之下，如表中所示，面罩检测数据集具有较少的训练图像，其中AIZOOTech数据集具有较高的图像数目。

However, most of previous face detection models are trained on faces that were not covered in most of the cases. For this reason, in cases where the face is covered with a mask, the face detection model could miss the detection of the faces (as explained in the very well detailed article in pyimagesearch.com).

但是，大多数以前的面部检测模型都是针对大多数情况下未涵盖的面部进行训练的。 因此，在人脸被面具覆盖的情况下，人脸检测模型可能会错过人脸的检测(如pyimagesearch.com中非常详细的文章所述 )。

使用OAK-D进行口罩检测 (Face mask detection using OAK-D)

Before analyzing the previously mentioned examples, the people at Luxonis already provide a demo for face mask detection using the OAK-D. The model in the demo is a MobileNetV2 (SSD) trained in Google Colab.

在分析前面提到的示例之前， Luxonis的人员已经提供了使用OAK-D进行面罩检测的演示 。 该演示中的模型是经过Google Colab培训的MobileNetV2(SSD)。

Note: Even though it is not included in the demo, they also provide another Google Colab script for training a YOLOv3-tiny model for face mask detection.

注意：尽管演示中未包含该工具，但他们还提供了另一个Google Colab脚本，用于训练YOLOv3-tiny模型以检测面罩 。

To run the demo, it is necessary to install the DepthAI Python module. As mentioned in Part 1, the Windows version is still experimental. However, recently the process has been updated so that it is much easier to install the library following the steps in the last comment in the Luxonis Dicussion.

要运行该演示，必须安装DepthAI Python模块。 如第1部分所述，Windows版本仍处于试验阶段。 但是，最近该过程已更新，因此按照Luxonis Dicussion中 最后一个注释中 的 步骤安装该库要容易得多。

However, as of today, the face mask detection demo is configured to work with an older version of the DepthAI library. So, I modified the demo to work with the current version of DepthAI and can be found in my Github repository for this project. To run the demo, it is necessary to install the DepthAI library as explained before and add the depthai folder to the PYTHONPATH of your system. Next open the command and run the following commands:

但是，截至今天，面罩检测演示已配置为可与DepthAI库的旧版本一起使用。 因此，我修改了演示以与DepthAI的当前版本一起使用 ，可以在我的 这个项目的Github仓库 。 要运行该演示，必须按照前面的说明安装DepthAI库，并将depthai文件夹添加到系统的PYTHONPATH中。 接下来打开命令并运行以下命令：

git clone https://github.com/ibaiGorordo/Social-Distance-Feedback.git
cd "Social-Distance-Feedback\Part 2 - Mask Detection"
python demo_mask_detector.py

The demo_mask_detector.py is a script that configures the OAK-D to perform face mask detection on RGB camera and shows the image and detection from the OAK-D.

demo_mask_detector.py是一个脚本，该脚本将OAK-D配置为在RGB相机上执行面部遮罩检测，并显示图像和来自OAK-D的检测。

import numpy as np
import cv2
from demo_helpers import config, capture_image, get_detection, calculate_frame_speed, decode_mobilenet_ssd, show_mobilenet_ssd
from time import time, sleep, monotonic
import os
import depthai
print('Using depthai module from: ', depthai.__file__)# Create a list of enabled streams ()
stream_names = ['metaout', 'previewout']device = depthai.Device('', False)# create the pipeline, here is the first connection with the device
p = device.create_pipeline(config=config)if p is None:print('Pipeline is not created.')exit(3)while True:# retreive data from the device# data is stored in packets, there are nnet (Neural NETwork) packets which have additional functions for NNet result interpretationnnet_packets, data_packets = p.get_available_nnet_and_data_packets(True)ret, frame = capture_image(data_packets)nnet_prev = get_detection(nnet_packets)if ret:frame_count = calculate_frame_speed()nn_frame = show_mobilenet_ssd(nnet_prev["entries_prev"]['rgb'], frame, is_depth=0)cv2.putText(nn_frame, "fps: " + str(frame_count), (25, 50), cv2.FONT_HERSHEY_SIMPLEX, 1.0, (0, 0, 0))cv2.imshow("Mask detection", nn_frame)key = cv2.waitKey(1)if key == ord('q'):breakdel p  # in order to stop the pipeline object should be deleted, otherwise device will continue working. This is required if you are going to add code after the main loop, otherwise you can ommit it.
device.deinit_device()
print('py: DONE.')

The video below shows the inference output from the OAK-D for the MobileNetV2 (SSD) face mask detection model (trained using DepthAI’s Google Colab script).

以下视频显示了OAK-D对MobileNetV2(SSD) 面罩检测模型的推理输出(使用DepthAI的Google Colab密码进行了训练)。

面罩检测模型在野外如何表现？ (How do face mask detection models behave in the wild?)

The previous example is similar to the many tutorials that end up with an example of the inference using the webcam. However, the objective of my system is to be used in daily life, particularly outdoors. Therefore, the system should be robust in different light conditions and even when there are multiple people around.

前面的示例与许多教程相似，最后以使用网络摄像头进行推理为例。 但是，我系统的目标是在日常生活中，尤其是在户外使用。 因此，即使在周围有多个人的情况下，该系统也应在不同的光照条件下都非常坚固。

For that reason, I decided to see how different face mask detection models behaved in a more realistic environment. For that purpose, I used this public video from pexels.com of people walking in a night market. The video below shows a comparison of the face mask detection for the SSD-MobileNetV2 Vs. the YOLOv3-tiny models trained using DepthAI’s Google Colab scripts. The code for the inference program can be found in my Github repository here.

因此，我决定看看在更现实的环境中不同的面罩检测模型如何表现。 为此，我使用了来自pexels.com的公开视频，内容是在夜市里散步的人们 。 以下视频比较了SSD-MobileNetV2 Vs的面罩检测。 使用DepthAI的Google Colab脚本训练的YOLOv3-tiny模型。 推理程序的代码可以在我的Github存储库中找到 。

As it can be observed, the SSD-MobilenetV2 model had a higher number of detections but as a result, a higher number of those detections were wrong detections. Even increasing the confidence threshold to 0.7 (as in the video above), the SSD-MobilenetV2 model still had a high number of false detections.

可以观察到， SSD-MobilenetV2模型具有较高的检测数量，但结果是，这些检测中的较高数量是错误检测。 即使将置信度阈值提高到0.7(如上面的视频中所示)， SSD-MobilenetV2模型仍然存在大量错误检测。

On the other hand, the YOLOv3-tiny model missed some of the faces (particularly the faces that were far away) but had a more stable detections with a confidence threshold of 0.5. Since our application only needs to detect people that are close to the user (at a distance of 3 meters or closer), the YOLOv3-tiny model seems to be the most promising model of the two.

另一方面， YOLOv3-tiny模型遗漏了一些面部(尤其是距离较远的面部)，但是检测更为稳定，置信度阈值为0.5。 由于我们的应用程序仅需要检测靠近用户(3米或更近的距离)的人员，因此YOLOv3-tiny模型似乎是两者中最有希望的模型。

YOLOv4呢？ (What about YOLOv4?)

Recently, Alexey Bochkovskiy presented a new YOLO version (YOLOv4) which provides higher performance than the previous versions. In the video below, there is an example from Mladen Zamanov where it can be seen that YOLOv4 can perform face mask detection even with many people around.

最近，Alexey Bochkovskiy提出了 YOLO新版本(YOLOv4) 比以前的版本提供更高的性能。 在下面的视频中，有一个来自Mladen Zamanov的示例，可以看出YOLOv4即使在周围很多人的情况下也可以执行面部遮罩检测。

The problem of using YOLOv4 in our application is that in order to pass the model to the OAK-D, the model needs to be converted to a .blob file to be run by the Myriad X inside the OAK-D. However, in order to convert the model, it is necessary to use OpenVINO toolkit’s model optimizer which currently officially supports until YOLOv3.

在我们的应用程序中使用YOLOv4的问题是，为了将模型传递给OAK-D ，需要将模型转换为.blob文件 ，以由OAK-D内部的Myriad X运行。 但是，为了转换模型，必须使用OpenVINO工具箱的模型优化器 ，该模型优化器目前在YOLOv3之前一直受官方支持。

Even though that thanks to the repository by TNTWEN it is possible to use the model optimizer with the latest OpenVINO (2020.4) version, the depthAI module still does not support it. As a result, as of today, it seems that YOLOv4 cannot be used in the OAK-D with the built-in cameras.

即使由于TNTWEN的存储库，仍可以将模型优化器与最新的OpenVINO(2020.4)版本一起使用 ， depthAI模块仍不支持它。 结果，从今天开始，似乎YOLOv4无法在内置相机的OAK-D中使用。

Anyway, I decided to give a try to YOLOv4 in case in a near future it becomes available for the OAK-D. Particularly, I focused on the new YOLOv4-tiny version because it should be more suitable for real time inference in devices with lower computation power (actually the author of YOLOv4 has been able to run a modified version of the full YOLOv4 in the same chip inside the OAK-D as shown here).

无论如何，我决定尝试一下YOLOv4 ，以防它在不久的将来可用于OAK-D 。 特别是，我专注于新版YOLOv4-tiny版本，因为它应该更适合于具有较低计算能力的设备中的实时推断(实际上YOLOv4的作者已经能够在同一芯片内部运行完整版YOLOv4的修改版) OAK-D ，如此处所示 )。

For that purpose, I trained the YOLOv4-tiny model in Google Colab using this script (code below), which was based on DepthAI’s original script. In the same script, I also added the code to detect face masks on the same video I used to compare SSD-MobileNetV2 with YOLOv3.

为此，我使用了该脚本 (以下代码) 在Google Colab中训练了YOLOv4-tiny模型， 该脚本基于DepthAI的原始脚本。 在同一脚本中，我还添加了用于检测同一视频的面罩的代码，该视频用于比较SSD-MobileNetV2和YOLOv3 。

In the video below, I show the comparison of the results for YOLOv3-tiny Vs. YOLOv4-tiny for face mask detection.

在下面的视频中，我显示了YOLOv3-tiny Vs结果的比较。 YOLOv4-tiny用于面罩检测 。

As it can be observed, the results very similar between both models. In some cases, YOLOv4-tiny is capable of detecting faces that YOLOv3-tiny is not capable, but overall the results are almost identical.

可以看出，两个模型之间的结果非常相似。 在某些情况下， YOLOv4-tiny能够检测到YOLOv3-tiny无法检测到的面部，但是总体结果几乎是相同的。

In summary, I will probably continue using YOLOv3-tiny for face mask detection unless new support becomes available for YOLOv4. In the next part, I will focus on the combination of depth and object detection data.

总之，除非可以为YOLOv4提供新的支持，否则我可能会继续使用YOLOv3-tiny进行面罩检测。 在下一部分中，我将重点介绍深度和对象检测数据的组合。

All the code from this part and for the next part can be found in my following repository: https://github.com/ibaiGorordo/Social-Distance-Feedback.

该部分以及下一部分的所有代码都可以在我的以下存储库中找到： https : //github.com/ibaiGorordo/Social-Distance-Feedback 。