RK3568上电启动流程详解 [十四]

由于 QEMU 在设备仿真方面的能力欠缺，比如我们无法让 QEMU 模拟一个 IIC 设备（除非对 QEMU 的代码动刀子），所以我们需要一个真实的物理环境，这里我使用了烂大街的 RK3568 开发板，我们需要让 X-Hyper 在 RK3568 的真实硬件环境中运行起来，然后再运行两个 VM，一个作为前端，配置前端 Virtio IIC 设备，并处理和真实 IIC 设备的通信，一个 VM 作为后端，通过 Virtio IIC 后端驱动和前端通信，实现 IIC 的半虚拟化。

但是在这些工作之前，我们首先需要搞清楚 RK3568 的整个启动流程，这样对于我们移植 X-Hyper 是十分有帮助的。

在 RK3568 中我们不考虑重映射的场景，那么 CPU 上电后，PC 指向了 0xFFFF0000，而这个位置所指向的就是片内的 BootRom（20KB），BootRom 中在出厂时就存储了一段固化的启动代码，这块实现RK 是不开源的，所以我们并不知道其中的细节。

这段固化的引导程序再从片外的设备中加载后续代码，RK 的 BootRom 支持 SDMMC、EMMC、Serial Nand Flash、Serila Nor Flash 等设备。

从官方文档我们可以看到整个 RK3568 的启动流程如下：

CPU 从 PC 为 0xFFFFF0000 处运行第一条指令，也就是 BootRom；
设备检查顺序 SP Nor-Flash -> SPI Nand Flash -> External Nand Flash -> External eMMC Flash -> External SDMMC;
如果上述设备都没有，或者发生故障，则最终使用 USB OTG；
检测到设备后加载 SDRAM 初始化代码到 SRAM，然后初始化 DDR；
将 uboot 代码搬运到 DDR 中；
然后运行 uboot code；
最后通过 uboot 引导 kernel 启动；

我使用的 RK 板中是有 eMMC 的，但是假设 eMMC 中的分区被破坏，这时候如何恢复呢？这时候需要进入所谓的 MaskRom 模式。在 RK 板的设计中，通过按键将 eMMC_D0 管脚接地造成 BootRom 引导代码无法识别 eMMC，从而进入 USB OTG 的加载模式：

当找到 eMMC 后，BootRom 从中加载 U-boot SPL 到 DRAM 运行，然后 SPL 加载 eMMC 中的 u-boot 代码到 DDR 中，并运行 uboot 代码，最后 uboot 再从 eMMC 中加载 kernel 以及 ramdisk 并运行。

从 eMMC 的视角，整体流程如下：

从输出日志分析如下：

首先是片内 SRAM 的初始化，由 BootRom 负责完成：

DDR 03ea844c5d typ 24/09/03-10:42:57,fwver: v1.23
In
wdqs_if: 0x1010100
LP4/4x derate en, other dram:1x trefi
SRX
ddrconfig:0
MID:0x13
LPDDR4X, 324MHz
BW=32 Col=10 Bk=8 CS0 Row=16 CS=1 Die BW=16 Size=2048MB
tdqss_lf: cs0 dqs0: 24ps, dqs1: -96ps, dqs2: -48ps, dqs3: -144ps,
tdqss_hf: cs0 dqs0: 24ps, dqs1: -96ps, dqs2: -48ps, dqs3: -144ps,change to: 324MHz
PHY drv:clk:36,ca:36,DQ:29,odt:240
vrefinner:16%, vrefout:41%
dram drv:40,odt:0
clk skew:0x62change to: 528MHz
PHY drv:clk:36,ca:36,DQ:29,odt:240
vrefinner:16%, vrefout:41%
dram drv:40,odt:0
clk skew:0x58change to: 780MHz
PHY drv:clk:36,ca:36,DQ:29,odt:60
vrefinner:16%, vrefout:41%
dram drv:40,odt:0
clk skew:0x58
rx vref: 15.6%
tx vref: 36.0%change to: 1560MHz(final freq)
PHY drv:clk:36,ca:36,DQ:29,odt:60
vrefinner:16%, vrefout:22%
dram drv:40,odt:80
vref_ca:00000071
clk skew:0x2b
rx vref: 15.6%
tx vref: 21.8%

然后将 DDR 初始化代码加载到上述初始化的 SRAM 中运行，用于初始化片外 DRAM：

cs 0:
rdtrn RS:
DQS0:0x37, DQS1:0x2b, DQS2:0x36, DQS3:0x37,
min  : 0xf 0x11 0x11 0x10  0x1  0x6  0x9  0x7 , 0x2  0x2  0x4  0x1  0x6  0x5  0x2  0x2 ,0x15 0x11  0xc  0xb  0x3  0x4  0x2  0x5 , 0xf  0x8  0xc  0x2 0x15 0x18  0xe 0x15 ,
mid  :0x2a 0x2b 0x2c 0x2a 0x1a 0x20 0x22 0x21 ,0x1e 0x1c 0x1f 0x1b 0x20 0x1e 0x1d 0x1d ,0x2d 0x2b 0x26 0x26 0x1e 0x1d 0x1b 0x1f ,0x29 0x22 0x25 0x1b 0x30 0x31 0x28 0x30 ,
max  :0x45 0x45 0x47 0x44 0x34 0x3a 0x3c 0x3c ,0x3a 0x36 0x3a 0x36 0x3a 0x38 0x39 0x39 ,0x46 0x45 0x41 0x41 0x39 0x37 0x34 0x3a ,0x43 0x3c 0x3f 0x34 0x4b 0x4a 0x42 0x4b ,
range:0x36 0x34 0x36 0x34 0x33 0x34 0x33 0x35 ,0x38 0x34 0x36 0x35 0x34 0x33 0x37 0x37 ,0x31 0x34 0x35 0x36 0x36 0x33 0x32 0x35 ,0x34 0x34 0x33 0x32 0x36 0x32 0x34 0x36 ,
wrtrn RS:
DQS0:0x2f, DQS1:0x18, DQS2:0x22, DQS3:0xf,
min  :0x6c 0x6f 0x72 0x6e 0x5d 0x61 0x66 0x67 0x6a ,0x58 0x57 0x59 0x54 0x58 0x57 0x58 0x58 0x57 ,0x60 0x60 0x5b 0x59 0x53 0x51 0x51 0x54 0x58 ,0x4c 0x47 0x49 0x3e 0x52 0x54 0x4a 0x53 0x47 ,
mid  :0x86 0x89 0x8c 0x89 0x78 0x7c 0x80 0x81 0x84 ,0x72 0x71 0x72 0x6e 0x72 0x72 0x72 0x72 0x72 ,0x7b 0x7c 0x75 0x74 0x6e 0x6d 0x6b 0x6e 0x71 ,0x66 0x60 0x62 0x58 0x6e 0x6e 0x65 0x6d 0x61 ,
max  :0xa1 0xa3 0xa7 0xa4 0x93 0x98 0x9a 0x9c 0x9e ,0x8d 0x8b 0x8c 0x89 0x8d 0x8d 0x8d 0x8c 0x8d ,0x96 0x98 0x90 0x90 0x89 0x89 0x86 0x89 0x8b ,0x81 0x79 0x7b 0x72 0x8a 0x89 0x81 0x88 0x7c ,
range:0x35 0x34 0x35 0x36 0x36 0x37 0x34 0x35 0x34 ,0x35 0x34 0x33 0x35 0x35 0x36 0x35 0x34 0x36 ,0x36 0x38 0x35 0x37 0x36 0x38 0x35 0x35 0x33 ,0x35 0x32 0x32 0x34 0x38 0x35 0x37 0x35 0x35 ,
CBT RS:
cs:0 min  :0x48 0x45 0x35 0x36 0x33 0x34 0x36 ,0x43 0x40 0x36 0x34 0x37 0x30 0x38 ,
cs:0 mid  :0x84 0x84 0x75 0x73 0x71 0x70 0x67 ,0x84 0x80 0x74 0x73 0x75 0x6d 0x68 ,
cs:0 max  :0xc1 0xc3 0xb6 0xb1 0xb0 0xad 0x99 ,0xc5 0xc1 0xb2 0xb2 0xb3 0xab 0x99 ,
cs:0 range:0x79 0x7e 0x81 0x7b 0x7d 0x79 0x63 ,0x82 0x81 0x7c 0x7e 0x7c 0x7b 0x61 ,
out

片外 DRAM 初始化完成后，从 eMMC 加载 U-Boot SPL，SPL 会进行一些板级初始化，然后继续从 eMMC 中加载 U-Boot 到 DRAM，U-Boot 运行：

## Verified-boot: 0
## Checking atf-1 0x00040000 (gzip @0x00240000) ... sha256(268e1cb8c1...) + sha256(b5946ac63d...) + OK
## Checking uboot 0x00a00000 (gzip @0x00c00000) ... sha256(db02c0cfd4...) + sha256(a9813f24e3...) + OK
## Checking fdt 0x00b55610 ... sha256(a474c6bb61...) + OK
## Checking atf-2 0xfdcc1000 ... sha256(b8dca786b4...) + OK
## Checking atf-3 0x0006b000 ... sha256(2f91089eb7...) + OK
## Checking atf-4 0xfdcce000 ... sha256(86ef885748...) + OK
## Checking atf-5 0xfdcd0000 ... sha256(0b2b146c60...) + OK
## Checking atf-6 0x00069000 ... sha256(a9a1e63bef...) + OK
## Checking optee 0x08400000 (gzip @0x08600000) ... sha256(de867c77df...) + sha256(9ffc99f1a2...) + OK
Jumping to U-Boot(0x00a00000) via ARM Trusted Firmware(0x00040000)
Total: 165.343/235.342 msINFO:    Preloader serial: 2
NOTICE:  BL31: v2.3():v2.3-645-g8cea6ab0b:cl, fwver: v1.44
NOTICE:  BL31: Built : 16:36:43, Sep 19 2023
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    pmu v1 is valid 220114
INFO:    l3 cache partition cfg-0
INFO:    dfs DDR fsp_param[0].freq_mhz= 1560MHz
INFO:    dfs DDR fsp_param[1].freq_mhz= 324MHz
INFO:    dfs DDR fsp_param[2].freq_mhz= 528MHz
INFO:    dfs DDR fsp_param[3].freq_mhz= 780MHz
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 0
INFO:    BL31: Initializing runtime services
INFO:    BL31: Initializing BL32
I/TC:
I/TC: OP-TEE version: 3.13.0-891-g9f2aca7d1 #hisping.lin (gcc version 10.2.1 20201103 (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16))) #2 Thu Oct 31 10:26:19 CST 2024 aarch64, fwver: v2.15
I/TC: OP-TEE memory: TEEOS 0x200000 TA 0xc00000 SHM 0x200000
I/TC: Primary CPU initializing
I/TC: CRYPTO_CRYPTO_VERSION_NEW no support. Skip all algo mode check.
I/TC: Primary CPU switching to normal world boot
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0xa00000
INFO:    SPSR = 0x3c9
usb dr_mode not found
usb dr_mode not foundU-Boot 2017.09-dirty #cuixujia.cxj (Jan 21 2025 - 14:26:47 +0800)Model: Rockchip RK3568 Evaluation Board
MPIDR: 0x0
PreSerial: 2, raw, 0xfe660000
DRAM:  2 GiB
Sysmem: init
Relocation Offset: 7d20a000
Relocation fdt: 7b9f82e8 - 7b9fecd8
CR: M/C/I
usb dr_mode not found
usb dr_mode not found
Using default environmentHotkey: ctrl+h
optee api revision: 2.0
dwmmc@fe2b0000: 1, dwmmc@fe2c0000: 2, sdhci@fe310000: 0
MMC: no card present
mmc_init: -123, time 0
switch to partitions #0, OK
mmc0(part 0) is current device
Bootdev(scan): mmc 0
MMC0: HS200, 200Mhz
PartType: EFI
TEEC: Waring: Could not find security partition
E/TC:? 0 storage_check_security_level_flag:561 Not support security level!
DM: v1
186864 bytes read in 14 ms (12.7 MiB/s)
DTB(Distro): rk-kernel.dtb
No misc partition
boot mode: normal
usb dr_mode not found
usb dr_mode not found
I2c0 speed: 100000Hz
vsel-gpios- not found! Error: -2
vdd_cpu 1025000 uV
can't find shutdown-sequence prop
can't find vb-shutdown-sequence prop
PMIC:  RK8090 (on=0x40, off=0x00)
vdd_logic init 900000 uV
vdd_gpu init 900000 uV
vdd_npu init 900000 uV
io-domain: OK
INFO:    ddr dmc_fsp already initialized in loader.
get vp0 plane mask:0x15, primary id:4, cursor_plane:-1, from dts
get vp1 plane mask:0x2a, primary id:5, cursor_plane:-1, from dts
Could not find baseparameter partition
Model: EmbedFire LubanCat-2
No resource file: logo.bmp
1215954 bytes read in 60 ms (19.3 MiB/s)
logo(Distro): logo.bmp
Rockchip UBOOT DRM driver version: v1.0.1
vp0 have layer nr:3[0 2 4 ], primary plane: 4
vp1 have layer nr:3[1 3 5 ], primary plane: 5
vp2 have layer nr:0[], primary plane: 0
hdmi@fe0a0000 disconnected
CLK: (sync kernel. arm: enter 816000 KHz, init 816000 KHz, kernel 0N/A)apll 1104000 KHzdpll 780000 KHzgpll 1188000 KHzcpll 1000000 KHznpll 1200000 KHzvpll 24000 KHzhpll 24000 KHzppll 200000 KHzarmclk 1104000 KHzaclk_bus 150000 KHzpclk_bus 100000 KHzaclk_top_high 500000 KHzaclk_top_low 400000 KHzhclk_top 150000 KHzpclk_top 100000 KHzaclk_perimid 300000 KHzhclk_perimid 150000 KHzpclk_pmu 100000 KHz
Net:   eth0: ethernet@fe2a0000, eth1: ethernet@fe010000
Hit key to stop autoboot('CTRL+C'):  0
MMC: no card present
mmc_init: -123, time 0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:2...
Found U-Boot script /boot.scr
1775 bytes read in 3 ms (577.1 KiB/s)

U-Boot 再次加载 initrd，kernel 到指定的内存位置，最终通过 booti 跳转到 kernel 处运行：

## Executing script at 00c00000
[boot.cmd] run boot.cmd scripts ...
[boot.cmd] load mmc 0:2 0x0a080000 /uEnv/uEnv.txt ...
3491 bytes read in 17 ms (200.2 KiB/s)
[boot.cmd] Importing environment from mmc ...
bootargs=storagemedia=emmc androidboot.storagemedia=emmc androidboot.mode=normal root=/dev/mmcblk0p3 boot_part=2 earlyprintk console=ttyFIQ0 console=tty1 consoleblank=0 loglevel=7 rootwait rw rootfstype=ext4
[boot.cmd] load mmc 0:2 0x0a200000 /initrd-5.10.209-rk356x ...
7072427 bytes read in 79 ms (85.4 MiB/s)
[boot.cmd] loading mmc 0:2 0x00280000 /Image-5.10.209-rk356x ...
40907264 bytes read in 460 ms (84.8 MiB/s)
[boot.cmd] loading default rk-kernel.dtb
186864 bytes read in 13 ms (13.7 MiB/s)
[boot.cmd] dtoverlay from /uEnv/uEnv.txt
DTOVERLAY[debug]: param no:5  0x0a100000  0x0a000000 /uEnv/uEnv.txt 0x0a080000[boot.cmd] [mmc 0:2] ...
[boot.cmd] [booti] ...
Fdt Ramdisk skip relocation
No misc partition
## Loading init Ramdisk from Legacy Image at 0a200000 ...Image Name:Image Type:   ARM Linux RAMDisk Image (gzip compressed)Data Size:    7072363 Bytes = 6.7 MiBLoad Address: 00000000Entry Point:  00000000Verifying Checksum ... OK
## Flattened Device Tree blob at 0x0a100000Booting using the fdt blob at 0x0a100000reserving fdt memory region: addr=a100000 size=5e000Using Device Tree in place at 000000000a100000, end 000000000a160fff
No resource file: logo_kernel.bmp
1215954 bytes read in 57 ms (20.3 MiB/s)
logo(Distro): logo_kernel.bmp
## reserved-memory:

后面就是 Linux Kernel 的运行了。到此整个 RK3568 的启动流程应该就讲清楚了。下一篇幅我们尝试将 X-Hyper 运行到 RK 板子上。