Authors: Jenny Huang, Ian Hunt-Isaak, William Palmer

作者： 黄珍妮 ， 伊恩·亨特·伊萨克 ， 威廉·帕尔默

GitHub Repo

GitHub回购

介绍 (Introduction)

Training an image segmentation model on new images can be daunting, especially when you need to label your own data. To make this task easier and faster, we built a user-friendly tool that lets you build this entire process in a single Jupyter notebook. In the sections below, we will show you how our tool lets you:

在新图像上训练图像分割模型可能会令人生畏，尤其是当您需要标记自己的数据时。 为了使此任务更容易，更快捷，我们构建了一个用户友好的工具，可让您在单个Jupyter笔记本中构建整个过程。 在以下各节中，我们将向您展示我们的工具如何使您：

1. Manually label your own images
   手动标记自己的图像
2. Build an effective segmentation model through transfer learning
   通过转移学习建立有效的细分模型
3. Visualize the model and its results
   可视化模型及其结果
4. Share your project as a Docker image
   以Docker映像形式共享您的项目

The main benefits of this tool are that it is easy-to-use, all in one platform, and well-integrated with existing data science workflows. Through interactive widgets and command prompts, we built a user-friendly way to label images and train the model. On top of that, everything can run in a single Jupyter notebook, making it quick and easy to spin up a model, without much overhead. Lastly, by working in a Python environment and using standard libraries like Tensorflow and Matplotlib, this tool can be well-integrated into existing data science workflows, making it ideal for uses like scientific research.

该工具的主要优点是易于使用 ， 在一个平台上全部 使用 ，并且与现有的数据科学工作流程很好地集成在一起 。 通过交互式小部件和命令提示符，我们构建了一种用户友好的方式来标记图像和训练模型。 最重要的是，所有内容都可以在单个Jupyter笔记本中运行，从而快速，轻松地建立模型，而没有太多开销。 最后，通过在Python环境中工作并使用Tensorflow和Matplotlib等标准库，可以将该工具很好地集成到现有的数据科学工作流程中，使其非常适合科学研究等用途。

For instance, in microbiology, it can be very useful to segment microscopy images of cells. However, tracking cells over time can easily result in the need to segment hundreds of images, which can be very difficult to do manually. In this article, we will use microscopy images of yeast cells as our dataset and show how we built our tool to differentiate between the background, mother cells, and daughter cells.

例如，在微生物学中，分割细胞的显微镜图像可能非常有用。 但是，随着时间的推移跟踪单元很容易导致需要分割成百上千个图像，这可能很难手动完成。 在本文中，我们将使用酵母细胞的显微图像作为数据集，并展示如何构建用于区分背景，母细胞和子细胞的工具。

1.标签 (1. Labelling)

There are many existing tools to create labelled masks for images, including Labelme, ImageJ, and even the graphics editor GIMP. While these are all great tools, they can’t be integrated within a Jupyter notebook, making them harder to use with many existing workflows. Fortunately, Jupyter Widgets make it easy for us to make interactive components and connect them with the rest of our Python code.

现有许多工具可以为图像创建带标签的蒙版，包括Labelme ， ImageJ甚至是图形编辑器GIMP 。 这些都是很棒的工具，但是它们无法集成到Jupyter笔记本中，这使得它们很难与许多现有工作流程一起使用。 幸运的是， Jupyter Widgets使我们能够轻松制作交互式组件并将其与我们的其余Python代码连接。

To create training masks in the notebook, we have two problems to solve:

要在笔记本中创建训练口罩，我们要解决两个问题：

1. Select parts of an image with a mouse
   用鼠标选择图像的一部分
2. Easily switch between images and select the class to label
   轻松在图像之间切换并选择要标记的类别

To solve the first problem, we used the Matplotlib widget backend and the built-in LassoSelector. The LassoSelector handles drawing a line to show what you are selecting, but we need a little bit of custom code to draw the masks as an overlay:

为了解决第一个问题，我们使用了Matplotlib小部件后端和内置的LassoSelector 。 LassoSelector会处理一条线以显示您所选择的内容，但是我们需要一些自定义代码来将蒙版绘制为覆盖层：

For the second problem, we added nice looking buttons and other controls using ipywidgets:

对于第二个问题，我们使用ipywidgets添加了漂亮的按钮和其他控件：

We combined these elements (along with improvements like scroll to zoom) to make a single labelling controller object. Now we can take microscopy images of yeast and segment the mother cells and daughter cells:

我们结合了这些元素(以及滚动缩放等改进功能)来制作了一个标签控制器对象。 现在，我们可以对酵母进行显微镜检查，并对母细胞和子细胞进行分段：

You can check out the full object, which lets you scroll to zoom, right click to pan, and select multiple classes here.

您可以签出完整的对象，使您可以滚动以缩放，右键单击以平移并在此处选择多个类。

Now we can label a small number of images in the notebook, save them into the correct folder structure, and start to train CNN!

现在，我们可以在笔记本中标记少量图像，将它们保存到正确的文件夹结构中，然后开始训练CNN！

2.模型训练 (2. Model Training)

该模型 (The Model)

U-net is a convolutional neural network that was initially designed to segment biomedical images but has been successful for many other types of images. It builds upon existing convolutional networks to work better with very few training images and make more precise segmentations. It is a state-of-the-art model that is also easy to implement using the segmentation_models library.

U-net是一个卷积神经网络，最初设计用于分割生物医学图像，但已成功用于许多其他类型的图像。 它以现有的卷积网络为基础，可以在很少的训练图像的情况下更好地工作，并进行更精确的分割。 这是一个最新的模型，也可以使用segmentation_models库轻松实现。

U-net is unique because it combines an encoder and a decoder using cross-connections (the gray arrows in the figure above). These skip connections cross from the same sized part in the downsampling path to the upsampling path. This creates awareness of the original pixels inputted into the model when you upsample, which has been shown to improve performance on segmentation tasks.

U-net的独特之处在于它通过交叉连接将编码器和解码器结合在一起(上图中的灰色箭头)。 这些跳过连接从下采样路径中的相同大小的部分跨到上采样路径。 这样可以提高您对上采样时输入到模型中的原始像素的了解，这已经显示出可以提高分割任务的性能。

As great as U-net is, it won’t work well if we don’t give it enough training examples. And given how tedious it is to manually segment images, we only manually labelled 13 images. With so few training examples, it seems impossible to train a neural network with millions of parameters. To overcome this, we need both Data Augmentation and Transfer Learning.

尽管U-net很棒，但是如果我们没有给它足够的培训示例，它将无法正常工作。 考虑到手动分割图像的繁琐工作，我们仅手动标记了13张图像。 仅用很少的训练示例，就不可能训练具有数百万个参数的神经网络。 为了克服这个问题，我们既需要数据扩充又需要转移学习 。

数据扩充 (Data Augmentation)

Naturally, if your model has a lot of parameters, you would need a proportional amount of training examples to get good performance. Using our small dataset of images and masks, we can create new images that will be as insightful and useful to our model as our original images.

自然，如果您的模型具有很多参数，则需要成比例的训练示例才能获得良好的性能。 使用我们小的图像和蒙版数据集，我们可以创建新图像，这些图像对于模型和原始图像一样具有洞察力和实用性。

How do we do that? We can flip the image, rotate it at an angle, scale it inward or outward, crop it, translate it, or even blur the image by adding noise, but most importantly, we can do a combination of those operations to create many new training examples.

我们该怎么做？ 我们可以翻转图像，旋转角度，向内或向外缩放，裁剪，平移图像，甚至可以通过添加噪点来模糊图像，但最重要的是，我们可以将这些操作结合起来以创建许多新的训练例子。

Image data augmentation has one more complication in segmentation compared to classification. For classification, you just need to augment the image as the label will remain the same (0 or 1 or 2…). However, for segmentation, the label (which is a mask) needs to also be transformed in sync with the image. To do this, we used the albumentations library with a custom data generator since, to our knowledge, the Keras ImageDataGenerator does not currently support the combination “Image + mask”.

与分类相比，图像数据增强在分割方面具有更多的复杂性。 对于分类，您只需要放大图像，因为标签将保持不变(0或1或2…)。 但是，对于分割，还需要与图像同步转换标签(作为蒙版)。 为此，我们使用了带有自定义数据生成器的albumentations库，因为据我们所知， Keras ImageDataGenerator目前不支持“ Image + Mask”组合。

转移学习 (Transfer Learning)

Even though we have now created 100 or more images, this still isn’t enough as the U-net model has more than 6 million parameters. This is where transfer learning comes into play.

即使我们现在已经创建了100个或更多的图像，但这仍然不够，因为U-net模型具有超过600万个参数。 这是转移学习发挥作用的地方。

Transfer Learning lets you take a model trained on one task and reuse it for another similar task. It reduces your training time drastically and more importantly, it can lead to effective models even with a small training set like ours. For example, neural networks like MobileNet, Inception, and DeepNet, learn a feature space, shapes, colors, texture, and more, by training on a great number of images. We can then transfer what was learned by taking these model weights and modifying them slightly to activate for patterns in our own training images.

转移学习使您可以采用在一项任务上受过训练的模型，并将其重用于另一项类似任务。 它极大地减少了您的培训时间，更重要的是，即使像我们这样的小型培训，它也可以产生有效的模型。 例如，诸如MobileNet，Inception和DeepNet之类的神经网络通过训练大量图像来学习特征空间，形状，颜色，纹理等。 然后，我们可以通过获取这些模型权重并对其进行稍微修改以激活我们自己的训练图像中的模式来转移所学的内容。

Now how do we use transfer learning with U-net? We used the segmentation_models library to do this. We use the layers of a deep neural network of your choosing (MobileNet, Inception, ResNet) and the parameters found training on image classification (ImageNet) and use them as the first half (encoder) of your U-net. Then, you train the decoder layers with your own augmented dataset.

现在我们如何在U-net中使用转移学习？ 我们使用segmentation_models库执行此操作。 我们使用您选择的深度神经网络(MobileNet，Inception，ResNet)的各层以及针对图像分类训练的找到的参数(ImageNet)，并将它们用作U-net的前半部分(编码器)。 然后，您可以使用自己的扩充数据集训练解码器层。

Putting it Together

把它放在一起

We put this all together in a Segmentation model class that you can find here. When creating your model object, you get an interactive command prompt where you can customize aspects of your U-net like the loss function, backbone, and more:

我们将所有这些放到了Segmentation模型类中，您可以在此处找到。 创建模型对象时，您会得到一个交互式命令提示符，您可以在其中自定义U-net的各个方面，例如损失函数，主干等。

After 30 epochs of training, we achieved 95% accuracy. Note that it is important to choose a good loss function. We first tried cross-entropy loss, but the model was unable to distinguish between the similar looking mother and daughter cells and had poor performance due to the class imbalance of seeing many more non-yeast pixels than yeast pixels. We found that using dice loss gave us much better results. The dice loss is linked to the Intersection over Union Score (IOU) and is usually better adapted to segmentation tasks as it gives incentive to maximize the overlap between the predicted and ground truth masks.

经过30个星期的培训，我们达到了95％的准确性。 请注意，选择良好的损耗函数很重要。 我们首先尝试了交叉熵损失，但是该模型无法区分相貌相似的母细胞和子细胞，并且由于看到的非酵母像素多于酵母像素的类不平衡，因此该模型的性能不佳。 我们发现使用骰子损失可以获得更好的结果。 骰子损失与联盟分数交叉点(IOU)相关联，通常会更好地适应分段任务，因为它可以最大程度地促进预测的和真实面罩之间的重叠。

3.可视化 (3. Visualization)

Now that our model is trained, let’s use some visualization techniques to see how it works. We follow Ankit Paliwal’s tutorial to do so. You can find the implementation in his corresponding GitHub repository. In this section, we will visualize two of his techniques, Intermediate Layer Activations and Heatmaps of Class Activations, on our yeast cell segmentation model.

现在我们的模型已经训练完毕，让我们使用一些可视化技术来查看其工作原理。 我们按照Ankit Paliwal的教程进行操作。 您可以在他相应的GitHub存储库中找到实现。 在本节中，我们将在酵母细胞细分模型上可视化他的两种技术，即中间层激活和类激活的热图。

中间层激活 (Intermediate Layer Activations)

This first technique shows the output of intermediate layers in a forward pass of the network on a test image. This lets us see what features of the input image are highlighted at each layer. After inputting a test image, we visualized the first few outputs for some convolutional layers in our network:

第一项技术在测试图像上显示了网络的前向传递中中间层的输出。 这使我们可以看到输入图像的哪些功能在每一层都突出显示。 输入测试图像后，我们将网络中一些卷积层的前几个输出可视化：

In the encoder layers, filters close to the input detect more detail and those close to the output of the model detect more general features, which is to be expected. In the decoder layers, we see the opposite pattern, of going from abstract to more specific details, which is also to be expected.

在编码器层中，靠近输入的过滤器可检测更多细节，而靠近模型输出的过滤器可检测更一般的特征，这是可以预期的。 在解码器层中，我们看到了相反的模式，即从抽象到更具体的细节，这也是可以预期的。

类激活的热图 (Heatmaps of Class Activations)

Next, we look at class activation maps. These heat maps let you see how important each location of the image is for predicting an output class. Here, we visualize the final layer of our yeast cell model, since the class prediction label will largely depend on it.

接下来，我们看一下类激活图。 这些热图让您了解图像的每个位置对于预测输出类别的重要性。 在这里，我们可视化酵母细胞模型的最后一层，因为类别预测标签将在很大程度上取决于它。

We see from the heat maps that the cell locations are correctly activated, along with parts of the image border, which is somewhat surprising.

从热图可以看出，单元位置以及部分图像边界已被正确激活，这有些令人惊讶。

We also looked at the last technique in the tutorial, which shows what images each convolutional filter maximally responds to, but the visualizations were not very informative for our specific yeast cell model.

我们还研究了本教程中的最后一项技术，该技术显示了每个卷积滤波器最大程度地响应哪些图像，但是可视化对于我们的特定酵母细胞模型不是很有帮助。

4.制作和共享Docker映像 (4. Making and Sharing a Docker Image)

Finding an awesome model and trying to run it, only to find that it doesn’t work in your environment due to mysterious dependency issues, is very frustrating. We addressed this by creating a Docker image for our tool. This allows us to completely define the environment that the code is run in, all the way down to the operating system. For this project, we based our Docker image off of the jupyter/tensorflow-notebook image from Jupyter Docker Stacks. Then we just added a few lines to install the libraries we needed and to copy the contents of our GitHub repository into the Docker image. If you’re curious, you can see our final Dockerfile here. Finally, we pushed this image to Docker Hub for easy distribution. You can try it out by running:

找到令人敬畏的模型并尝试运行它，却发现它由于神秘的依赖关系问题而在您的环境中不起作用，这非常令人沮丧。 我们通过为我们的工具创建一个Docker镜像来解决这个问题。 这使我们可以完全定义运行代码的环境，一直到操作系统。 对于此项目，我们基于Jupyter Docker Stacks的jupyter/tensorflow-notebook映像构建Docker映像。 然后，我们仅添加了几行内容来安装所需的库，并将GitHub存储库的内容复制到Docker映像中。 如果您好奇，可以在此处查看我们的最终Dockerfile。 最后，我们将此映像推送到Docker Hub以便于分发。 您可以通过运行以下命令进行尝试：

sudo docker run -p 8888:8888 ianhuntisaak/ac295-final-project:v3 \
-e JUPYTER_LAB_ENABLE=yes

结论与未来工作 (Conclusion and Future Work)

This tool lets you easily train a segmentation model on new images in a user-friendly way. While it works, there is still room for improvement in usability, customization, and model performance. In the future, we hope to:

使用此工具，您可以以用户友好的方式轻松地在新图像上训练分割模型。 虽然有效，但在可用性，自定义和模型性能方面仍有改进的余地。 将来，我们希望：

1. Improve the lasso tool by building a custom Jupyter Widget using the html5 canvas to reduce lag when manually segmenting
   通过使用html5 canvas构建自定义的Jupyter小部件来改善套索工具，以减少手动分段时的滞后

2. Explore new loss functions and models (like this U-net pre-trained on broad nucleus dataset) as a basis for transfer learning

   探索新的损失函数和模型(例如在宽核数据集上预先训练的U-net )作为转移学习的基础

3. Make it easier to interpret visualizations and suggest methods of improving the results to the user
   使解释可视化更加容易，并向用户建议改善结果的方法

致谢 (Acknowledgements)

We would like to thank our professor Pavlos Protopapas and the Harvard Applied Computation 295 course teaching staff for their guidance and support.

我们要感谢我们的教授Pavlos Protopapas和哈佛应用计算295课程的教学人员的指导和支持。