香橙派5 RK3588 yolov5模型转换rknn及部署踩坑全记录 orangepi 5

零、写在前面

由于距离写这篇文章过去很久，有的部分，官方已更新，请多结合其他人的看，并多琢磨、讨论~
另外打个小广告：博客 https://blog.vrxiaojie.top/ 欢迎大家前来做客玩耍，提出问题~~
以后的文章都会在博客发布了，CSDN这边可能这是最后一篇文章。

一、烧录系统

1.准备工作：

(1) 使用官方提供的Ubuntu镜像：

Orange Pi 5系统下载（我使用Orangepi5_1.1.4_ubuntu_jammy_desktop_gnome_linux5.10.110）

(2) 下载 balenaEtcher 用于烧录系统

(3) 准备一个大于等于32GB的TF卡、一个读卡器。

2.打开软件，根据软件步骤烧录即可

在这里插入图片描述
完成后，插上TF卡、键盘、鼠标，上电开机。

二、在开发板上的准备工作

1. 在终端查看ip，并用ssh连接

香橙派接显示器，与电脑连接同一个网络，
使用 ip address查看ip
使用任意一个ssh软件，连接
默认的用户名有: root、orangepi，它们的默认密码都是orangepi
在这里插入图片描述

2. 安装ftp服务器，并连接

安装

sudo apt-get install vsftpd

修改配置文件

sudo vim /etc/vsftpd.conf

local_enable=YES
write_enable=YES
并确认它们前面没有井号(vim操作：按i键进入编辑模式，esc后输入:wq退出保存)
重启ftp服务

sudo /etc/init.d/vsftpd restart

filezilla设置

主机：香橙派的ip
用户名:orangepi
密码: orangepi

三、pc端的准备工作

1.准备一台装有Windows10及以上系统的电脑，并用WSL装Ubuntu22.04系统

WSl2安装过程参考微软官方教程
请自己手动配置好Ubuntu的运行环境以及conda，这里不多赘述。

2.下载rknn_toolkit2的所有源码 github页面并解压

3.下载 yolov5官方源码，并训练模型

为节省时间，在这里先不自己训练了，就使用预训练权重yolov5n.pt

4.将pt转换成onnx格式

首先，根据官方文档的要求，在训练完成模型之后，再修改models/yolo.py Detect类下的forward函数
在这里插入图片描述
也就是修改为

def forward(self, x):z = []  # inference outputfor i in range(self.nl):x[i] = self.m[i](x[i])  # convreturn x

然后，在yolov5目录下，打开终端输入命令

python export.py --weights yolov5n.pt --data data/coco128.yaml --include onnx --opset 12 --batch-size 1

注意注意！opset一定要为12，不然后面onnx转rknn会报错。weights自己选你训练完成的best.pt，data选你自己设置的，我这里只是做一个最简单情况的演示。

这里我遇到一个错误:

line 715, in run
shape = tuple((y[0] if isinstance(y, tuple) else y).shape) # model output shape
AttributeError: ‘list’ object has no attribute ‘shape’

解决方法
找到这一行，修改为

shape = tuple(y[0].shape)  # model output shape

程序运行结束后，会在当前文件夹下生成一个yolov5n.onnx文件，待会要用。

5. onnx转化为rknn格式

进入你的Ubuntu系统，用conda创建一个新的环境

conda create -name rknn python=3.8

激活这个环境

conda activate rknn

将整个rknn_toolkit2源码解压后的文件夹复制进Ubuntu的 /home/用户名目录下，我将它重命名为rknn。
进入doc目录，使用 pip 安装依赖:

pip install -r requirements-cpu-ubuntu20.04_py38.txt -i https://mirror.baidu.com/pypi/simple

返回上一级目录，然后进入packages目录，安装rknn_toolkit2

pip install rknn_toolkit2-1.4.0_22dcfef4-cp38-cp38-linux_x86_64.whl

完成后，输入命令 python

from rknn.api import RKNN

如图所示，如果没有报错，说明安装成功，使用键盘Ctrl+Z退出此模式。

接下来，把(三、4)步所得到的onnx文件放入rknn/examples/onnx/yolov5文件夹下，终端里进入该文件夹。
用你喜欢的编辑器修改 test.py里面的一些内容，具体位置如图所示

onnx和要生成的rknn的文件名
classes改成你模型的类型，我这里是官方的所以默认
增加target_platform=‘rk3588’，根据你的板子型号自行调整

执行

python test.py

成功运行后的内容大概是这样的
在这里插入图片描述
此时这个目录下也会生成一个yolov5n.rknn文件，待会要用。

四、在香橙派上的部署

1.创建一个名为deploy.py的文件

内容如下

import numpy as np
import cv2
from rknnlite.api import RKNNLiteRKNN_MODEL = 'yolov5n.rknn'QUANTIZE_ON = TrueOBJ_THRESH = 0.25
NMS_THRESH = 0.45
IMG_SIZE = 640CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light","fire hydrant", "stop sign ", "parking meter", "bench", "bird", "cat", "dog ", "horse ", "sheep", "cow", "elephant","bear", "zebra ", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite","baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife ","spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza ", "donut", "cake", "chair", "sofa","pottedplant", "bed", "diningtable", "toilet ", "tvmonitor", "laptop	", "mouse	", "remote ", "keyboard ", "cell phone", "microwave ","oven ", "toaster", "sink", "refrigerator ", "book", "clock", "vase", "scissors ", "teddy bear ", "hair drier", "toothbrush ")def sigmoid(x):return 1 / (1 + np.exp(-x))def xywh2xyxy(x):# Convert [x, y, w, h] to [x1, y1, x2, y2]y = np.copy(x)y[:, 0] = x[:, 0] - x[:, 2] / 2  # top left xy[:, 1] = x[:, 1] - x[:, 3] / 2  # top left yy[:, 2] = x[:, 0] + x[:, 2] / 2  # bottom right xy[:, 3] = x[:, 1] + x[:, 3] / 2  # bottom right yreturn ydef process(input, mask, anchors):anchors = [anchors[i] for i in mask]grid_h, grid_w = map(int, input.shape[0:2])box_confidence = sigmoid(input[..., 4])box_confidence = np.expand_dims(box_confidence, axis=-1)box_class_probs = sigmoid(input[..., 5:])box_xy = sigmoid(input[..., :2])*2 - 0.5col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)grid = np.concatenate((col, row), axis=-1)box_xy += gridbox_xy *= int(IMG_SIZE/grid_h)box_wh = pow(sigmoid(input[..., 2:4])*2, 2)box_wh = box_wh * anchorsbox = np.concatenate((box_xy, box_wh), axis=-1)return box, box_confidence, box_class_probsdef filter_boxes(boxes, box_confidences, box_class_probs):"""Filter boxes with box threshold. It's a bit different with origin yolov5 post process!# Argumentsboxes: ndarray, boxes of objects.box_confidences: ndarray, confidences of objects.box_class_probs: ndarray, class_probs of objects.# Returnsboxes: ndarray, filtered boxes.classes: ndarray, classes for boxes.scores: ndarray, scores for boxes."""boxes = boxes.reshape(-1, 4)box_confidences = box_confidences.reshape(-1)box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])_box_pos = np.where(box_confidences >= OBJ_THRESH)boxes = boxes[_box_pos]box_confidences = box_confidences[_box_pos]box_class_probs = box_class_probs[_box_pos]class_max_score = np.max(box_class_probs, axis=-1)classes = np.argmax(box_class_probs, axis=-1)_class_pos = np.where(class_max_score >= OBJ_THRESH)boxes = boxes[_class_pos]classes = classes[_class_pos]scores = (class_max_score* box_confidences)[_class_pos]return boxes, classes, scoresdef nms_boxes(boxes, scores):"""Suppress non-maximal boxes.# Argumentsboxes: ndarray, boxes of objects.scores: ndarray, scores of objects.# Returnskeep: ndarray, index of effective boxes."""x = boxes[:, 0]y = boxes[:, 1]w = boxes[:, 2] - boxes[:, 0]h = boxes[:, 3] - boxes[:, 1]areas = w * horder = scores.argsort()[::-1]keep = []while order.size > 0:i = order[0]keep.append(i)xx1 = np.maximum(x[i], x[order[1:]])yy1 = np.maximum(y[i], y[order[1:]])xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)inter = w1 * h1ovr = inter / (areas[i] + areas[order[1:]] - inter)inds = np.where(ovr <= NMS_THRESH)[0]order = order[inds + 1]keep = np.array(keep)return keepdef yolov5_post_process(input_data):masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],[59, 119], [116, 90], [156, 198], [373, 326]]boxes, classes, scores = [], [], []for input, mask in zip(input_data, masks):b, c, s = process(input, mask, anchors)b, c, s = filter_boxes(b, c, s)boxes.append(b)classes.append(c)scores.append(s)boxes = np.concatenate(boxes)boxes = xywh2xyxy(boxes)classes = np.concatenate(classes)scores = np.concatenate(scores)nboxes, nclasses, nscores = [], [], []for c in set(classes):inds = np.where(classes == c)b = boxes[inds]c = classes[inds]s = scores[inds]keep = nms_boxes(b, s)nboxes.append(b[keep])nclasses.append(c[keep])nscores.append(s[keep])if not nclasses and not nscores:return None, None, Noneboxes = np.concatenate(nboxes)classes = np.concatenate(nclasses)scores = np.concatenate(nscores)return boxes, classes, scoresdef draw(image, boxes, scores, classes):"""Draw the boxes on the image.# Argument:image: original image.boxes: ndarray, boxes of objects.classes: ndarray, classes of objects.scores: ndarray, scores of objects.all_classes: all classes name."""for box, score, cl in zip(boxes, scores, classes):top, left, right, bottom = boxprint('class: {}, score: {}'.format(CLASSES[cl], score))print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))top = int(top)left = int(left)right = int(right)bottom = int(bottom)cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),(top, left - 6),cv2.FONT_HERSHEY_SIMPLEX,0.6, (0, 0, 255), 2)def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):# Resize and pad image while meeting stride-multiple constraintsshape = im.shape[:2]  # current shape [height, width]if isinstance(new_shape, int):new_shape = (new_shape, new_shape)# Scale ratio (new / old)r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])# Compute paddingratio = r, r  # width, height ratiosnew_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  # wh paddingdw /= 2  # divide padding into 2 sidesdh /= 2if shape[::-1] != new_unpad:  # resizeim = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))left, right = int(round(dw - 0.1)), int(round(dw + 0.1))im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # add borderreturn im, ratio, (dw, dh)if __name__ == '__main__':# Create RKNN objectrknn = RKNNLite()# load RKNN modelprint('--> Load RKNN model')ret = rknn.load_rknn(RKNN_MODEL)# Init runtime environmentprint('--> Init runtime environment')ret = rknn.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)  #使用0 1 2三个NPU核心# ret = rknn.init_runtime('rk3566')if ret != 0:print('Init runtime environment failed!')exit(ret)print('done')# Set inputsimg = cv2.imread(IMG_PATH)# img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE))img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))# Inferenceoutputs = rknn.inference(inputs=[img])# post processinput0_data = outputs[0]input1_data = outputs[1]input2_data = outputs[2]input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:]))input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:]))input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:]))input_data = list()input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))boxes, classes, scores = yolov5_post_process(input_data)img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)if boxes is not None:draw(img_1, boxes, scores, classes)# show outputcv2.imshow("post process result", img_1)cv2.waitKey(0)cv2.destroyAllWindows()rknn.release()

这是基于原来test.py做的一些修改，删除了导出模型的部分，只保留了加载模型和推理部分。
接下来把这几个文件放在同一个文件夹，传到香橙派的 /home/orangepi/Desktop 目录下
在这里插入图片描述

2.在香橙派上安装miniconda并创建环境

建议去清华源下载 miniconda
我下载的是Miniconda3-py39_4.9.2-Linux-aarch64.sh
下载完后传到香橙派，在终端输入

sudo bash Miniconda3-py39_4.9.2-Linux-aarch64.sh

根据提示安装即可。
安装完后，可能会遇到找不到conda命令的问题，解决方法参考：
https://blog.csdn.net/weixin_42314494/article/details/114264103
先换清华源

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --set show_channel_urls yes

然后创建环境

conda create --name rknn python=3.9

激活环境

conda activate rknn

pip换清华源

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

把你PC上的rknn_toolkit2/rknn_toolkit_lite2/packages中的rknn_toolkit_lite2-1.4.0-cp39-cp39-linux_aarch64.whl传输到香橙派上
安装上步的包

pip install rknn_toolkit_lite2-1.4.0-cp39-cp39-linux_aarch64.whl

安装剩下的依赖包

pip install opencv-python

然后就是运行deploy.py

python deploy.py

在ssh终端运行的话，会显示如下界面，可以看到蓝色框内是表示识别出来的类别、置信度以及检测框的四个顶点位置
在这里插入图片描述
如果有外接显示器，你就能看到它处理后的图像了：

五、后记

基础的部署到这里就结束了，我在社区里看到几篇非常不错的文章，这里附上作者的链接，可供参考。

1.如果你要做视频检测，又发现帧数提不上来，请使用多线程：

yolov5实现目标检测系统（使用线程池）

2.模型转换时出现问题，可参考：

yolov5和rknn模型的问题

3.将视频推理完的结果进行推流：

RK3588实战：调用npu加速，yolov5识别图像、ffmpeg发送到rtmp服务器

本文部分参考、致谢：

香橙派5使用RK3588S内置NPU加速yolov5推理，实时识别数字达到50fps