「daily updating」k3s + openfaas serverless bench 踩坑指南持续更新中

OpenFaas从入门到实战 – 踩坑指南｜ k3d+OpenFaas | deploy your first python function

https://blog.alexellis.io/first-faas-python-function/

https://docs.openfaas.com/deployment/kubernetes/

搭建环境：第一种方法失败，第二种方法亲测有效嘻嘻嘻，其实我大概知道原因，但先不细究了

1. VM: K3s + OpenFaas on Mac

参考教程：https://midnightprogrammer.net/post/installing-openfaas-on-k3s-single-node/

MacBook Pro

使用 Multipass 来创建一个 VM：需要 4GB 内存和 8GB 磁盘，记得要分配多一点 - 我暂定的

beatles@biantongshusMBP ~ % multipass launch --name k3s --memory 4G --disk 8G 
Launched: k3s

等待 VM 创建，然后为 VM 启动一个 shell

multipass shell k3s

安装 k3s

curl -sfL https://get.k3s.io | sh -

After the installation is completed, you can check if the k3s service is running by executing the below command.

sudo systemctl status k3s

亲测无效，遂脚本自动安装：

curl -SLsf https://dl.get.arkade.dev/ | sudo sh

谁说用着个很简单，总之我还自己凑了两个命名空间

arkade install openfaas

然后就是这样

Release "openfaas" has been upgraded. Happy Helming!
NAME: openfaas
LAST DEPLOYED: Thu Jan  4 19:51:37 2024
NAMESPACE: openfaas
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
To verify that openfaas has started, run:kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas"To retrieve the admin password, run:echo $(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode)
2024/01/04 19:51:42 stderr: WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/ubuntu/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/ubuntu/.kube/config=======================================================================
= OpenFaaS has been installed.                                        =
=======================================================================# Get the faas-cli
curl -SLsf https://cli.openfaas.com | sudo sh# Forward the gateway to your machine
kubectl rollout status -n openfaas deploy/gateway
kubectl port-forward -n openfaas svc/gateway 8080:8080 &# If basic auth is enabled, you can now log into your gateway:
PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
echo -n $PASSWORD | faas-cli login --username admin --password-stdinfaas-cli store deploy figlet
faas-cli list# For Raspberry Pi
faas-cli store list \--platform armhffaas-cli store deploy figlet \--platform armhf# Find out more at:
# https://github.com/openfaas/faas🚀 Speed up GitHub Actions/GitLab CI + reduce costs: https://actuated.dev

check the rollout status of the gateway, After this we can forward the gateway to the machine.

ubuntu@k3s:/etc/rancher/k3s$ kubectl get pod -n openfaas
NAME                            READY   STATUS    RESTARTS      AGE
nats-5c48bc8b46-zksff           1/1     Running   2 (12m ago)   73m
alertmanager-795bbdc56c-6qwpn   1/1     Running   2 (12m ago)   73m
prometheus-78d4c9f748-smjfr     1/1     Running   2 (12m ago)   73m
queue-worker-b9965cc56-bn47b    1/1     Running   6 (12m ago)   73m
gateway-67df8c4d4-jfkbf         2/2     Running   0             73m
ubuntu@k3s:/etc/rancher/k3s$ kubectl rollout status -n openfaas deploy/gateway
deployment "gateway" successfully rolled out
ubuntu@k3s:/etc/rancher/k3s$ kubectl port-forward -n openfaas svc/gateway 8080:8080 &
[1] 4737
ubuntu@k3s:/etc/rancher/k3s$ Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
jobs
[1]+  Running                 kubectl port-forward -n openfaas svc/gateway 8080:8080 &
ubuntu@k3s:/etc/rancher/k3s$

You can then view the password by printing the value of the PASSWORD variable using the echo command. 反正报错需要等一会儿就好了

ubuntu@k3s:/etc/rancher/k3s$ echo $PASSWORD
oJPdijCZF2Dd

192.168.64.1

http://localhost:31112/ui
http://127.0.0.1:31112/ui

2. Docker: K3d + OpenFaas

参考教程（版本较老，于是我自己做了一些记录）：https://mickey.dev/posts/getting-started-with-openfaas/#:~:text=Configure%20faas%2Dcli%20%26%20Login%20to%20the%20OpenFaaS%20Dashboard&text=Open%20a%20browser%20and%20navigate,for%20a%20username%20and%20password.

brew install k3d

Create a cluster named mycluster with just a single server node

k3d cluster create mycluster

You can now use it like this:

kubectl cluster-info

事实上，这句话并没有什么用

export KUBECONFIG="$(k3d kubeconfig get --name='k3s-default')"export KUBECONFIG="$(k3d kubeconfig get --all)"

To install OpenFaaS to your k3d cluster, start by cloning the https://github.com/openfaas/faas-netesrepo:

git clone https://github.com/openfaas/faas-netes.git
cd faas-netes

Install the namespaces:

kubectl apply -f namespaces.yml

This will create two namespaces in your cluster:

openfaas - Which will hold the OpenFaaS cluster services (AKA. ‘Control Plane”).
openfaas-fn - Stores the functions you deploy to OpenFaaS.

Create a password for the OpenFaaS Gateway and add it as a secret into the cluster. The below commands will generate a random password and add the secret:

beatles@biantongshus-MacBook-Pro faas-netes % export PASSWORD=$(head -c 12 /dev/urandom | shasum| cut -d' ' -f1)beatles@biantongshus-MacBook-Pro faas-netes % kubectl -n openfaas create secret generic basic-auth \
--from-literal=basic-auth-user=admin \
--from-literal=basic-auth-password="$PASSWORD"
secret/basic-auth created
beatles@biantongshus-MacBook-Pro faas-netes %

Now deploy the OpenFaaS stack:

kubectl apply -f ./yaml

Install the `faas-cli`

The next step is installing the faas-cli. If you’re on MacOS and already have homebrew install then installation is as simple as:

brew install faas-cli

Configure `faas-cli` & Login to the OpenFaaS Dashboard

Configure the faas-cli to use your local OpenFaaS cluster by using the faas-cli login command. If running k3d you’ll need to forward the gateway service port and set the OPENFAAS_URL environment variable:

kubectl port-forward svc/gateway -n openfaas 31112:8080 &
export OPENFAAS_URL=http://127.0.0.1:31112
echo $PASSWORD | faas-cli login --password-stdin

Open a browser and navigate to http://localhost:31112 to load the UI. The UI will prompt for a username and password. The default username is admin and the password is the one you specified in the deployment instructions above, you can print this to your console using echo $PASSWORD.

echo $PASSWORD
681d0e42c1506c260eba6cc73ad61b3330276302

Cheers!!

再次登陆

http://localhost:31112/

export PASSWORD=681d0e42c1506c260eba6cc73ad61b3330276302

别忘了启动docker engine

kubectl port-forward svc/gateway -n openfaas 31112:8080 &

export OPENFAAS_URL=http://127.0.0.1:31112

echo $PASSWORD | faas-cli login --password-stdin

就好了

3. Deploy New Functions

3.1 bug一览｜排错完整过程（发疯中）

faas-cli template store list
Error while getting templates info: error while requesting template list: Get "https://raw.githubusercontent.com/openfaas/store/master/templates.json": dial tcp [::]:443: connect: connection refused

祈祷：http://grayblog.cn/2020/07/18/解决Mac中Terminal无法访问github/

IP Lookup : 140.82.113.3 (github.com)

sudo vi /etc/hosts
140.82.113.3 www.github.com

他的ipaddress有好多，我就放了第一个https://www.ipaddress.com/ip-lookup

185.199.109.133 raw.githubusercontent.com

终于！！！不行的话再来一遍

beatles@biantongshusMBP ~ % faas-cli template store listNAME                     RECOMMENDED DESCRIPTION        SOURCE
bash-streaming           [x]         openfaas-incubator Bash Streaming template
dockerfile               [x]         openfaas           Classic Dockerfile template
golang-middleware        [x]         openfaas           HTTP middleware interface in Go
java11-vert-x            [x]         openfaas           Java 11 Vert.x template
node18                   [x]         openfaas           HTTP-based Node 18 template
php8                     [x]         openfaas           Classic PHP 8 template
python3-http             [x]         openfaas           Python 3 with Flask and HTTP
python3-http-debian      [x]         openfaas           Python 3 with Flask and HTTP based on Debian
ruby-http                [x]         openfaas           Ruby 2.4 HTTP template
cobol                    [ ]         devries            COBOL Template
crystal                  [ ]         tpei               Crystal template
crystal-http             [ ]         koffeinfrei        Crystal HTTP template
csharp-httprequest       [ ]         distantcam         C# HTTP template
csharp-kestrel           [ ]         burtonr            C# Kestrel HTTP template
lua53                    [ ]         affix              Lua 5.3 Template
perl-alpine              [ ]         tmiklas            Perl language template based on Alpine image
quarkus-native           [ ]         pmlopes            Quarkus.io native image template
rust                     [ ]         openfaas-incubator Community Rust template
rust-http                [ ]         openfaas-incubator Community Rust template with HTTP bindings
swift                    [ ]         affix              Swift 4.2 Template
vala                     [ ]         affix              Vala Template
vala-http                [ ]         affix              Non-Forking Vala Template
vertx-native             [ ]         pmlopes            Eclipse Vert.x native image template
bun-express              [ ]         openfaas           HTTP-based template using bun
csharp                   [ ]         openfaas           Classic C# template
go                       [ ]         openfaas           Legacy Golang template
golang-http              [ ]         openfaas           Request/response style HTTP template
java11                   [ ]         openfaas           Java 11 template
node                     [ ]         openfaas           Legacy Node 12 template
node12                   [ ]         openfaas           HTTP-based Node 12 template
node14                   [ ]         openfaas           HTTP-based Node 14 template
node16                   [ ]         openfaas           HTTP-based Node 16 template
node17                   [ ]         openfaas           HTTP-based Node 17 template
php7                     [ ]         openfaas           Classic PHP 7 template
powershell-http-template [ ]         openfaas-incubator Powershell Core HTTP Ubuntu:16.04 template
powershell-template      [ ]         openfaas-incubator Powershell Core Ubuntu:16.04 template
puppeteer-nodelts        [ ]         alexellis          A puppeteer template for headless Chrome
python                   [ ]         openfaas           Classic Python 2.7 template
python27-flask           [ ]         openfaas           Python 2.7 Flask template
python3                  [ ]         openfaas           Classic Python 3 template
python3-debian           [ ]         openfaas           Python 3 Debian template
python3-flask            [ ]         openfaas           Python 3 Flask template
python3-flask-debian     [ ]         openfaas           Python 3 Flask template based on Debian
ruby                     [ ]         openfaas           Classic Ruby 2.5 template

faas-cli up --gateway=http://localhost:31112

或

faas-cli build -f ./cpu.yml
faas-cli deploy -f ./cpu.yml --gateway=http://localhost:31112

当然了，探索出过程很痛苦，我把我提的github issue挂在这里 https://github.com/openfaas/faas/issues/1831

事到如今，踩坑真得靠自己了

[0] < Building float-operation done in 104.36s.
[0] Worker done.Total build time: 104.36s
Deploying: float-operation.Is OpenFaaS deployed? Do you need to specify the --gateway flag?
Put "http://127.0.0.1:8080/system/functions": dial tcp 127.0.0.1:8080: connect: connection refusedFunction 'float-operation' failed to deploy with status code: 500

beatles@biantongshusMBP openfaas % faas-cli logs float-operation
Cannot connect to OpenFaaS on URL: http://127.0.0.1:8080

排错：

kubectl get service -n openfaas

faas-cli deploy -f ./cpu.yml --gateway=http://localhost:31112

最终成功

beatles@biantongshusMBP openfaas % faas-cli login --username admin --password=681d0e42c1506c260eba6cc73ad61b3330276302 --gateway=http://localhost:31112WARNING! Using --password is insecure, consider using: cat ~/faas_pass.txt | faas-cli login -u user --password-stdin
Calling the OpenFaaS server to validate the credentials...
credentials saved for admin http://localhost:31112
beatles@biantongshusMBP openfaas % faas-cli deploy -f ./cpu.yml --gateway=http://localhost:31112
Deploying: float-operation.Deployed. 202 Accepted.
URL: http://localhost:31112/function/float-operationbeatles@biantongshusMBP openfaas %

然而遇到not ready

Not Ready

beatles@biantongshusMBP openfaas % kubectl get deploy -n openfaas-fn
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
float-operation   0/1     1            0           5h36m
beatles@biantongshusMBP openfaas % kubectl describe -n openfaas-fn deploy/float-operation
Name:                   float-operation
Namespace:              openfaas-fn
CreationTimestamp:      Thu, 25 Jan 2024 12:34:41 +0800
Labels:                 faas_function=float-operation
Annotations:            deployment.kubernetes.io/revision: 1prometheus.io.scrape: false
Selector:               faas_function=float-operation
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 1 max surge
Pod Template:Labels:       faas_function=float-operationAnnotations:  prometheus.io.scrape: falseContainers:float-operation:Image:      float-operation:latestPort:       8080/TCPHost Port:  0/TCPLiveness:   http-get http://:8080/_/health delay=2s timeout=1s period=2s #success=1 #failure=3Readiness:  http-get http://:8080/_/health delay=2s timeout=1s period=2s #success=1 #failure=3Environment:fprocess:  python index.pyMounts:      <none>Volumes:       <none>
Conditions:Type           Status  Reason----           ------  ------Available      False   MinimumReplicasUnavailableProgressing    False   ProgressDeadlineExceeded
OldReplicaSets:  <none>
NewReplicaSet:   float-operation-bfd748bd6 (1/1 replicas created)
Events:          <none>
beatles@biantongshusMBP openfaas %

排错过程：

beatles@biantongshusMBP openfaas % kubectl get deploy -n openfaas-fn
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
float-operation   0/1     1            0           5h36mbeatles@biantongshusMBP openfaas % kubectl logs -n openfaas-fn deploy/float-operation
Error from server (BadRequest): container "float-operation" in pod "float-operation-bfd748bd6-4r67p" is waiting to start: trying and failing to pull imagebeatles@biantongshusMBP openfaas % kubectl get events -n openfaas-fn \--sort-by=.metadata.creationTimestamp
LAST SEEN   TYPE      REASON    OBJECT                                MESSAGE
31m         Normal    Pulling   pod/float-operation-bfd748bd6-p885q   Pulling image "float-operation:latest"
37m         Warning   Failed    pod/float-operation-bfd748bd6-p885q   Failed to pull image "float-operation:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/float-operation:latest": failed to resolve reference "docker.io/library/float-operation:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
37m         Warning   Failed    pod/float-operation-bfd748bd6-p885q   Error: ErrImagePull
3m28s       Normal    BackOff   pod/float-operation-bfd748bd6-p885q   Back-off pulling image "float-operation:latest"beatles@biantongshusMBP ~ % docker pull float-operation:latest
Error response from daemon: pull access denied for float-operation, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

https://docs.openfaas.com/deployment/kubernetes/

于是我准备先学一下docker: https://www.youtube.com/watch?v=pg19Z8LL06w

现在我学完了，我又来解决问题来了

3.2 First Python Function

faas-cli template store pull python3-http

faas-cli new float-operation --lang python3-http

部署函数

faas-cli up --gateway=http://localhost:31112

或如下几步

faas-cli build -f ./cpu.yml

build失败可能是网络问题更换节点可以解决

Here’s how to upload the function to a remote registry (if needed):

docker login -u beatlesbian -p Bts210717! docker.io

faas-cli push -f ./cpu.yml

faas-cli deploy -f ./cpu.yml --gateway=http://localhost:31112

出现了这个问题，找到了 https://skemman.is/bitstream/1946/44391/2/OperatingManual.pdf

beatles@biantongshusMBP openfaas % faas-cli push -f ./cpu.yml Unable to push one or more of your functions to Docker Hub:
- float-operationYou must provide a username or registry prefix to the Function's image such as user1/function1

我需要修改yaml文件，啊！我才懂Alex是什么意思image: beatlesbian/float-operation:latest

终于Push成功

latest: digest: sha256:4f6429e67aea67b421804366a4c89ff39be1dc705cd9a58807fcf656399b9cc2 size: 4695
[0] < Pushing float-operation [beatlesbian/float-operation:latest] done.
[0] Worker done.

然而不是200 ok

beatles@biantongshusMBP openfaas % faas-cli deploy -f ./cpu.yml --gateway=http://localhost:31112
Deploying: float-operation.Deployed. 202 Accepted.
URL: http://localhost:31112/function/float-operation

这回终于ready了，然而，我谢谢嘞，我能把所有坑踩遍是吧，我怀疑是 an error in the application，毕竟这也是我第一次来写OpenFaas函数，好吧让我去刷牙洗脸，曙光就在前方，我预感明天我能把所有函数都部署完，然后再考虑auto-scaling的问题，然后上hey插件测试性能，年前就做这些吧，每天保证两小时改论文，画一些简单轻松的小画就可以了

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

这是我修改前的 hanlder.py,这种直接返回估计肯定是不对的

import math
from time import timedef float_operations(n):start = time()for i in range(0, n):sin_i = math.sin(i)cos_i = math.cos(i)sqrt_i = math.sqrt(i)latency = time() - startreturn latencydef handle(event, context):n = int(event['n'])result = float_operations(n)print(result)return result# return {#     "statusCode": 200,#     "body": "Hello from OpenFaaS!"# }

于是我改成了gpt给我写的，还是不行

这当然就要查看日志了！Your function is crashing due to an error in your code, check the logs. 感谢troubleshooting手册，我们可以很轻松的猜测问题定位在最后一个，当然只是猜测

beatles@biantongshusMBP openfaas % kubectl logs -n openfaas-fn deploy/float-operation
2024/02/07 02:23:31 Version: 0.9.15     SHA: bb7b23f61a2251aa158dac6d409f2fdff383f4f9
2024/02/07 02:23:31 Forking: python, arguments: [index.py]
2024/02/07 02:23:31 Started logging: stderr from function.
.....^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024/02/07 02:24:48 stderr:   File "/home/app/index.py", line 66, in call_handler
2024/02/07 02:24:48 stderr:     response_data = handler.handle(event, context)
2024/02/07 02:24:48 stderr:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024/02/07 02:24:48 stderr:   File "/home/app/function/handler.py", line 16, in handle
2024/02/07 02:24:48 stderr:     n = int(event['n'])
2024/02/07 02:24:48 stderr:             ~~~~~^^^^^
2024/02/07 02:24:48 stderr: TypeError: 'Event' object is not subscriptable
2024/02/07 02:24:48 POST / - 500 INTERNAL SERVER ERROR - ContentLength: 265B (0.0043s)
beatles@biantongshusMBP openfaas %

The function handler is passed two arguments, event and context.

event contains data about the request, including: - body - headers - method - query - path
context contains basic information about the function, including: - hostname

问题就出现在event上

n = int(event.body)

I want to test my function without deploying it，尝试一下吧！Then invoke your function via http://127.0.0.1:8081

faas-cli build -f ./cpu.yml
docker run -p 8081:8080 \--rm \--name float-operation \-ti beatlesbian/float-operation:latest

终于成功了！

至此我认为我已经踩完大部分坑了。

3.3 Add Second Function

faas-cli new pyaes --lang python3-http --append cpu.yml

2024/02/07 08:38:07 stderr:     import numpy as np
2024/02/07 08:38:07 stderr: ModuleNotFoundError: No module named 'numpy'

缺包可能需要修改dockerfile, 添加RUN ....，再手动build

docker build -t beatlesbian/pyaes:latest ./build/pyaes

然后继续，成功了

faas-cli push -f ./cpu.yml

faas-cli deploy -f ./cpu.yml --gateway=http://localhost:31112

排错常用

kubectl logs -n openfaas-fn deploy/pyaes

对于缺包我做了如下补充：PIL

RUN pip install Pillow
RUN pip install requests

https://blog.csdn.net/qq_28304687/article/details/76551196

根据论文，我现在要把这些都添加完，明天测试openFaas可扩展性，并安装插件

pyaes

RUN pip install pyaes# test
{'length_of_message': 4,'num_of_iterations': 5
}event_body = json.loads(event.body.decode("utf-8"))
length_of_message = event_body['length_of_message']
num_of_iterations = event_body['num_of_iterations']{'message': '69wc', 'ciphertext': b'\xb6 \xbf"', 'latency': 0.0022983551025390625}

TypeError: Object of type bytes is not JSON serializable OpenFaas简单的事例，并没有演示出来event.body是字节码类型，还要靠自己反复摸索，解决方案最终如上，好在可以举一反三了

FunctionBench workloads

1. CPU & Memory

⭐️Float Operations(sin, cos, sqrt) done
⭐️MatMul(square matrix multiplication) done
⭐️Linpack(solve linear equations Ax = b) done
⭐️Image Processing
⭐️Video Processing
MapReduce
Chameleon
⭐️pyaes done
Feature Generation
⭐️Model Training
Model Serving
- Video Face Detection - Cascade Classifier
- Classification Image - CNN
- Generating Names- RNN
- Prediction Reviews - LR

2. Disk

⭐️dd
⭐️gzip-compression

3. Network

iPerf3
Cloud storage service download-upload
json serialization deserialization

# from google.cloud import storage
from PIL import Image, ImageFilter
from time import time
import json
import osTMP = "/tmp/"
#输入的image, file_name, 输出的filename需要斟酌# left-right + top-bottom
def flip(image, file_name):path_list = []path = TMP + "flip-left-right-" + file_nameimg = image.transpose(Image.FLIP_LEFT_RIGHT)img.save(path)path_list.append(path)path = TMP + "flip-top-bottom-" + file_nameimg = image.transpose(Image.FLIP_TOP_BOTTOM)img.save(path)path_list.append(path)return path_list# rotate 90 180 270
def rotate(image, file_name):path_list = []path = TMP + "rotate-90-" + file_nameimg = image.transpose(Image.ROTATE_90)img.save(path)path_list.append(path)path = TMP + "rotate-180-" + file_nameimg = image.transpose(Image.ROTATE_180)img.save(path)path_list.append(path)path = TMP + "rotate-270-" + file_nameimg = image.transpose(Image.ROTATE_270)img.save(path)path_list.append(path)return path_list# filter
def filter(image, file_name):path_list = []path = TMP + "blur-" + file_nameimg = image.filter(ImageFilter.BLUR)img.save(path)path_list.append(path)path = TMP + "contour-" + file_nameimg = image.filter(ImageFilter.CONTOUR)img.save(path)path_list.append(path)path = TMP + "sharpen-" + file_nameimg = image.filter(ImageFilter.SHARPEN)img.save(path)path_list.append(path)return path_listdef gray_scale(image, file_name):path = TMP + "gray-scale-" + file_nameimg = image.convert('L')img.save(path)return [path]def resize(image, file_name):path = TMP + "resized-" + file_nameimage.thumbnail((128, 128))image.save(path)return [path]# upon 5 image processing actions
def image_processing(file_name, image_path):path_list = []start = time()with Image.open(image_path) as image:tmp = imagepath_list += flip(image, file_name)path_list += rotate(image, file_name)path_list += filter(image, file_name)path_list += gray_scale(image, file_name)path_list += resize(image, file_name)latency = time() - startresult = {'path_list': path_list,'latency': latency}return result# file_name, image_path
def handle(event, context):image_path = str(event.body)file_name = os.path.basename(image_path)result = image_processing(file_name,image_path)return {"statusCode": 200,"body": json.dumps(result)}

Kubeedge

multipass launch --name kubeedge1 --memory 4G --disk 8G