This is the Part 2 of the MediaPipe Series I am writing.

这是我正在编写的MediaPipe系列的第2部分。

Previously, we saw how to get started with MediaPipe and use it with your own tflite model. If you haven’t read it yet, check it out here.

以前，我们了解了如何开始使用MediaPipe并将其与您自己的tflite模型一起使用。 如果您尚未阅读，请在此处查看 。

We had tried using the portrait segmentation tflite model in the existing segmentation pipeline with the calculators already present in MediaPipe

我们尝试在现有的细分管道中使用人像细分tflite模型，并在MediaPipe中使用计算器

After getting bored with this Blue Background, I decided to have some fun with it by having some Zoom like Virtual Backgrounds like beautiful stars or some crazy clouds instead :)

在厌倦了这种蓝色背景后，我决定通过一些缩放(例如虚拟背景(例如美丽的星星或一些疯狂的云))来玩一些它：)

For this, I wrote a custom calculator and used it with the existing pipeline.

为此，我编写了一个自定义计算器，并将其与现有管道一起使用。

So today I’ll show how did I go about making this App

所以今天我将展示如何制作此应用程序

Before we get started, I would suggest you to go through this part of the documentation which explains the Flow of a basic calculator.

在我们开始之前，我建议您仔细阅读本文档的这一部分，其中介绍了基本计算器的流程。

https://google.github.io/mediapipe/framework_concepts/calculators.html

https://google.github.io/mediapipe/framework_concepts/calculators.html

Now, let’s get started with the code

现在，让我们开始使用代码

1.从Part1克隆Portrait Segmentation存储库 (1. Clone the Portrait Segmentation repository from Part1)

$ git clone https://github.com/SwatiModi/portrait-segmentation-mediapipe.git

2.管道的新流程(在图.pbtxt文件中进行更改) (2. New Flow of the Pipeline (making changes in the graph .pbtxt file))

So earlier, the rendering/coloring was done by the RecolorCalculator , it used to take image and mask gpu buffer as input and returned gpu buffer rendered output (rendered using opengl)

因此，较早之前，渲染/着色是由RecolorCalculator完成的，它用于将图像和蒙版gpu缓冲区作为输入，并返回gpu缓冲区渲染的输出(使用opengl渲染)

Here, for replacing the Background with an Image(jpg/png), I have used OpenCV operations.

在这里，为了用Image(jpg / png)替换Background，我使用了OpenCV操作。

NOTE : OpenCV operations are performed on CPU — ImageFrame datatype where as opengl operations are performed on GPU — Image-buffer datatype

注意 ：OpenCV操作在CPU — ImageFrame数据类型上执行，而opengl操作在GPU —图像缓冲区数据类型上执行

portrait_segmentation.pbtxt

portrait_segmentation.pbtxt

We will replace the RecolorCalculator with the BackgroundMaskingCalculator

我们将用 BackgroundMaskingCalculator 替换 RecolorCalculator

node {
  calculator: "BackgroundMaskingCalculator"
  input_stream: "IMAGE_CPU:mask_embedded_input_video_cpu"
  input_stream: "MASK_CPU:portrait_mask_cpu"
  output_stream: "OUTPUT_VIDEO:output_video_cpu"
  }

This calculator takes transformed Image and the Mask (Both ImageFrame Datatype) as input and changes the background with given image.

该计算器将转换后的图像和蒙版(两个ImageFrame数据类型)作为输入，并使用给定图像更改背景。

For converting the Image and Mask GpuBuffer to ImageFrame, I used the GpuBufferToImageFrameCalculator.

为了将Image和Mask GpuBuffer转换为ImageFrame，我使用了GpuBufferToImageFrameCalculator。

3.写入计算器文件(background_masking_calculator.cc文件) (3. Write the Calculator File (background_masking_calculator.cc file))

In this we will write the logic of processing and creating the background masking effect

在本文中，我们将编写处理和创建背景遮罩效果的逻辑

background_masking_calculator.cc

background_masking_calculator.cc

place this background_masking_calculator.cc in mediapipe/calculators/image/

放置这个background_masking_calculator.cc 在mediapipe /计算器/图像/

a. Extending the Base Calculator

一个。 扩展基础计算器

b. GetContract()

b。 GetContract()

Here we are just verifying our inputs and their data types

在这里，我们只是在验证我们的输入及其数据类型

c. Open()

C。 打开()

Prepares the calculator’s per-graph-run state.

准备计算器的每图运行状态。

d. Process()

d。 处理()

Taking the inputs, converting to OpenCV Mat and further processing for desired output

接收输入，转换为OpenCV Mat并进一步处理以获得所需的输出

4.将background_masking_calculator.cc添加到具有依赖项的图像文件夹的BUILD文件中 (4. Add the background_masking_calculator.cc to BUILD file of image folder with the dependencies)

BUILD file in mediapipe/calculators/image

Mediapipe / calculators / image中的 BUILD文件

We need to do this so the calculator is available and accessible while compilation and execution

我们需要这样做，以便计算器在编译和执行时可用并且可访问

the deps (dependencies) you see here are the imports you are using in your calculator

您在此处看到的Deps(依赖项)是您在计算器中使用的导入

The name used here is referred in graph BUILD file as well

图BUILD文件中也引用了此处使用的名称

5.将background_masking_calculator.cc添加到图形BUILD文件中 (5. Add the background_masking_calculator.cc to the graph BUILD file)

BUILD file in graphs/portrait_segmentation

图形/ portrait_segmentation中的BUILD文件

Add background_masking_calculator to deps and remove recolor_calculator which we are not using now

将background_masking_calculator添加到deps并删除我们现在不使用的recolor_calculator

Now we are ready to build, we just have to add the asset(image to be masked on the background)

现在我们准备好构建，我们只需要添加资产(要在背景上遮盖的图像)

NOTE: This asset reading method is straight forward when trying on Desktop but quite different for the Android Build.

注意 ：在桌面上尝试时，这种资产读取方法很简单，但对于Android Build则大不相同。

So we will look at the Desktop and Android build one by one

因此，我们将逐一研究台式机和Android

桌面版本： (Desktop Build:)

In the background_masking_calculator.cc , read the asset file as

在background_masking_calculator.cc中，将资产文件读取为

Now, we are ready to build

现在，我们准备建立

bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11  mediapipe/examples/desktop/portrait_segmentation:portrait_segmentation_gpu

The build will finish successfully in around 10–15 minutes

构建将在大约10-15分钟内成功完成

INFO: Build completed successfully, 635 total actions

Now, you can run the Pipeline using the following

现在，您可以使用以下命令运行管道

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/portrait_segmentation/portrait_segmentation_gpu   --calculator_graph_config_file=mediapipe/graphs/portrait_segmentation/portrait_segmentation.pbtxt

the webcam starts and you’ll be able to see the output window

网络摄像头启动，您将能够看到输出窗口

Android版本： (Android Build:)

1.导出背景遮罩图像 (1. Exporting the background masking images)

Add the following code to the BUILD file in testdata folder (folder containing the background asset images)

将以下代码添加到testdata文件夹(包含背景资产图像的文件夹)中的BUILD文件中

exports_files(
    srcs = glob(["**"]),
)

2.将资产添加到Android应用程序使用的android_library (2. Adding the assets to android_library used by the Android App)

Adding the files to android_library along with the tflite model

将文件与tflite模型一起添加到android_library

Modify in android project BUILD file in examples/android/…./portraitsegmentationgpu/

在examples / android /…。/ portraitsegmentationgpu /中的android项目BUILD文件中进行修改

3.在background_masking_calculator中读取此图像文件 (3. Reading this image file in background_masking_calculator)

Here, first using the MediaPipe’s PathToResourceAsFile method, we try to look if the asset is available and if available we read it using it’s value as path to file

在这里，首先使用MediaPipe的PathToResourceAsFile方法，尝试查看资产是否可用，如果可用，我们将其值用作文件路径来读取资产

On a side note, this method was a part of resource_util.h present in mediapipe/util/ folder. So whenever you import a new file into your calculator, make sure to add it to BUILD file mentioned in pt. 4

另外，此方法是存在于mediapipe / util /文件夹中的resource_util.h的一部分。 所以每当你导入一个新的文件到您的计算器，请务必将其添加到 PT提到的BUILD文件 。 4

现在我们准备构建APK (Now we are ready to Build the APK)

# BUILD 
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/portraitsegmentationgpu# On successfully building the APK, it printsINFO: Elapsed time: 1499.898s, Critical Path: 753.09s
INFO: 2002 processes: 1849 linux-sandbox, 1 local, 152 worker.
INFO: Build completed successfully, 2140 total actions

This would take around 20–25 minutes when building for the first time because it downloads all the external dependencies for the Build. Next time it uses the cached dependencies, so it builds much faster.

首次构建时大约需要20-25分钟，因为它会下载Build的所有外部依赖项。 下次使用缓存的依赖项时，它的构建速度要快得多。

# INSTALL
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/portraitsegmentationgpu/portraitsegmentationgpu.apk

Now, you are ready to run the APK and test it.

现在，您可以运行APK并进行测试了。

Here we successfully added our own calculator and modified the pipeline and graph according by first testing it on Desktop and then finally on Android with minimal changes.

在这里，我们成功地添加了自己的计算器，并通过首先在桌面上对其进行测试，然后以最小的更改最终在Android上对其进行了修改，从而修改了管道和图形。

Feel free to ask any questions in the comments or you can reach me out personally.

随时问评论中的任何问题，或者您可以亲自与我联系。

You can find more about me on swatimodi.com

您可以在swatimodi.com上找到有关我的更多信息