RISC Zerod cargo-risczero相关模块代码解析

1. 引言

前序博客有：

RISC Zero zk-STARK证明系统代码解析
RISC Zero各功能模块代码解析

cargo-risczero模块开源代码见：

https://github.com/risc0/risc0/tree/main/risc0/cargo-risczero（Rust）

cargo-risczero模块：

用于帮助创建、管理和测试RISC Zero项目的Cargo extension。

## Installing from local source. Note: this can be very slow.
cargo install --path risc0/cargo-risczero
# 查看所安装的risczero版本
cargo risczero --version# 从源码编译安装toolchain，要等待的时间有点长。可使用cargo risczero install
cargo risczero build-toolchain# 确认toolchain安装成功
# rustup toolchain list --verbose | grep risc0
risc0   /root/.risc0/rust/build/host/stage2

RISC Zero toolchain，用于将guest程序，编译为供zkVM执行的ELF二进制文件。

cargo risczero支持的指令有：

# cargo risczero --help
The `risczero` commandUsage: cargo risczero <COMMAND>Commands:build            Build guest codebuild-toolchain  Build the riscv32im-risc0-zkvm-elf toolchaininstall          Install the riscv32im-risc0-zkvm-elf toolchainnew              Creates a new risczero starter projecthelp             Print this message or the help of the given subcommand(s)Options:-h, --help     Print help-V, --version  Print version

使用cargo risczero new来生成默认模板，可：

支持本地proving
和远程proving

如：
cargo risczero new --template risc0/templates/rust-starter --templ-subdir="" hello-world

/data/test# cargo risczero new --template risc0/templates/rust-starter --templ-subdir="" --path $(pwd)/risc0  --dest $(pwd)  --guest-name zyd             hello-world
[ 1/20]   Done: .gitignore                                                                                                                  
[ 2/20]   Done: .vscode/settings.json                                                                                                       
[ 3/20]   Done: .vscode                                                                                                                     
[ 4/20]   Done: Cargo.toml                                                                                                                  
[ 5/20]   Done: LICENSE                                                                                                                     [ 6/20]   Done: README.md                                                                                                                   [ 7/20]   Done: host/Cargo.toml                                                                                                             [ 8/20]   Done: host/src/main.rs                                                                                                            [ 9/20]   Done: host/src                                                                                                                    [10/20]   Done: host                                                                                                                        [11/20]   Done: methods/Cargo.toml                                                                                                          [12/20]   Done: methods/build.rs                                                                                                            [13/20]   Done: methods/guest/Cargo.toml                                                                                                    [14/20]   Done: methods/guest/src/main.rs                                                                                                   [15/20]   Done: methods/guest/src                                                                                                           [16/20]   Done: methods/guest                                                                                                               [17/20]   Done: methods/src/lib.rs                                                                                                          [18/20]   Done: methods/src                                                                                                                 [19/20]   Done: methods                                                                                                                     [20/20]   Done: rust-toolchain.toml                                                                                                         /data/test# ls hello-world/
Cargo.toml  LICENSE  README.md  host  methods  rust-toolchain.toml

所创建的hello-world项目基本结构为：

project_name
├── Cargo.toml
├── host
│   ├── Cargo.toml
│   └── src
│       └── main.rs                        <-- [Host code goes here]
└── methods├── Cargo.toml├── build.rs├── guest│   ├── Cargo.toml│   └── src│       └── bin│           └── method_name.rs         <-- [Guest code goes here]└── src└── lib.rs

可参考Tutorial: Building your first zkVM application 来做个简单的zkVM应用。

hello-world# cargo run --releaseCompiling hello-world-methods v0.1.0 (/data/test/hello-world/methods)
warning: unused import: `valid_control_ids`--> /data/test/risc0/risc0/zkvm/src/host/recursion/mod.rs:33:25|
33 | pub use self::receipt::{valid_control_ids, SuccinctReceipt};|                         ^^^^^^^^^^^^^^^^^|= note: `#[warn(unused_imports)]` on by defaultwarning: `risc0-zkvm` (lib) generated 1 warning (run `cargo fix --lib -p risc0-zkvm` to apply 1 suggestion)
multiply: Starting build for riscv32im-risc0-zkvm-elflo-world-methods(build)                                                              
multiply:     Finished release [optimized] target(s) in 0.05sCompiling host v0.1.0 (/data/test/hello-world/host)Finished release [optimized + debuginfo] target(s) in 1m 31sRunning `target/release/host`
Hello, world! I know the factors of 391, and I can prove it!

# ps -aux|grep risc
root     3500042  102  0.0 1647908 1026484 pts/2 Sl+  10:39   1:03 /data/rust/.rust_up/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name host --edition=2021 host/src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=140 --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C debuginfo=1 -C metadata=09ebe51e3bb70af4 -C extra-filename=-09ebe51e3bb70af4 --out-dir /data/test/hello-world/target/release/deps -L dependency=/data/test/hello-world/target/release/deps --extern hello_world_methods=/data/test/hello-world/target/release/deps/libhello_world_methods-2e425fabd1623d4e.rlib --extern risc0_zkvm=/data/test/hello-world/target/release/deps/librisc0_zkvm-b02ce46c1b1af0a7.rlib --extern serde=/data/test/hello-world/target/release/deps/libserde-9d092d72c197278a.rlib --extern tracing_subscriber=/data/test/hello-world/target/release/deps/libtracing_subscriber-c36e69b81a0b7d84.rlib -L native=/data/test/hello-world/target/release/build/ring-84695c43e581a53c/out
root     3500112  0.0  0.0   7004  2048 pts/5    S+   10:40   0:00 grep --color=auto risc

对guest程序的编译结果见：vim target/release/build/hello-world-methods-085a759ae9cd777d/out/methods.rs

pub const MULTIPLY_ELF: &[u8] = &[.......]
pub const MULTIPLY_ID: [u32; 8] = [3017128806, 1930636717, 3416458147, 548912590, 3161934873, 2600167522, 1958757367, 1384218650];
pub const MULTIPLY_PATH: &str = r#"/data/test/hello-world/target/riscv-guest/riscv32im-risc0-zkvm-elf/release/multiply"#;

let prover = default_prover();对应支持的prover类型有：【默认为ipc类型，用ps -aux查看有root 622607 1212 0.0 8809300 144324 pts/2 Sl+ 02:58 0:24 r0vm --port 46581进程信息。】

/// Return a default [Prover] based on environment variables and feature flags.
///
/// The `RISC0_PROVER` environment variable, if specified, will select the
/// following [Prover] implementation:
/// * `bonsai`: [BonsaiProver] to prove on Bonsai.
/// * `local`: [local::LocalProver] to prove locally in-process. Note: this
///   requires the `prove` feature flag.
/// * `ipc`: [ExternalProver] to prove using an `r0vm` sub-process. Note: `r0vm`
///   must be installed. To specify the path to `r0vm`, use `RISC0_SERVER_PATH`.
///
/// If `RISC0_PROVER` is not specified, the following rules are used to select a
/// [Prover]:
/// * [BonsaiProver] if the `BONSAI_API_URL` and `BONSAI_API_KEY` environment
///   variables are set unless `RISC0_DEV_MODE` is enabled.
/// * [local::LocalProver] if the `prove` feature flag is enabled.
/// * [ExternalProver] otherwise.
pub fn default_prover() -> Rc<dyn Prover> {
.....
}

2. zkvm的serde模块

zkvm的serde模块，代码见：

zkvm/src/serde

serde模块：

为RISC Zero zkVM的序列化和反序列化工具。zkVM host和guest之间所传输的数据需序列化。
包含了相应的序列化和反序列化工具。

host端：

使用类似to_vec这样的序列化函数来对数据序列化后，传输给guest。
当从guest读取数据时，使用类似from_slice这样的反序列化函数。

use risc0_zkvm::serde::{from_slice, to_vec};
let input = 42_u32;
let encoded = to_vec(&[input]).unwrap();
let output: u32 = from_slice(&encoded).unwrap();
assert_eq!(input, output);

guest端：其所需的序列化和反序列化函数，包含在env模块内，如env::read和env::commit。guest端很少使用本serde模块。
详情可参看 RISC Zero zkVM guest程序优化技巧及其与物理CPU的关键差异的“2. guest程序优化技巧及建议”。

见zkvm/src/serde/serializer.rs 序列化模块：

负责将各种结构体转换为Vec<u32>，即a vector of u32 words。

见zkvm/src/serde/deserializer.rs 反序列化模块：

负责将Vec<u32>，即已序列化的word-based data，反序列化为各结构体。

2.1 l2iterative的更紧凑的zkVM序列化模块

l2iterative的更紧凑的zkVM序列化模块，开源代码见：

https://github.com/l2iterative/alternative-serde0

RISC Zero zkVM 官方的 zkvm/src/serde/serializer.rs 序列化模块，只将提供给zkVM的输入序列化为Vec<u32>，这就意味着会有Rust中存在的一个主要公开问题：

当serde为Rust数据结构实现Serialize和Deserialize时，其遵循如下实现：

impl<T> Serialize for Vec<T> where T: Serialize
{#[inline]fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error> where S: Serializer,{serializer.collect_seq(self)}
}

且collect_seq函数具有如下默认实现，即意味着元素需串联逐个序列化：

pub trait Serializer: Sized {fn collect_seq<I>(self, iter: I) -> Result<Self::Ok, Self::Error>where I: IntoIterator, <I as IntoIterator>::Item: Serialize,{let mut iter = iter.into_iter();let mut serializer = tri!(self.serialize_seq(iterator_len_hint(&iter)));tri!(iter.try_for_each(|item| serializer.serialize_element(&item)));serializer.end()}
}

这意味着，当对字节数组Vec<u8>进行序列化时，每个元素将转换为u32类型，然后对Vec<u32>进行序列化，这将导致4倍的存储开销。

为提升效率，应对不同的T定义不同的规则，特别是当T = u8的情况。

解决方案一为：

使用serde_bytes库，通过定制化serde函数，可绕过该限制：
```
use serde::{Deserialize, Serialize};#[derive(Deserialize, Serialize)]
struct Efficient<'a> {#[serde(with = "serde_bytes")]bytes: &'a [u8],#[serde(with = "serde_bytes")]byte_buf: Vec<u8>,#[serde(with = "serde_bytes")]byte_array: [u8; 314],
}
```
serde_bytes库的思想为：不对Vec<T>应用某通用策略，而是由开发者来在2种策略间切换，从而会存在如下问题：
- 1）要求开发者对layers和抽象layers进行修改。若某开发者使用了4个库：A、B、C、D，A依赖B，B依赖C，C依赖D，且D使用Vec<u8>，则开发者需直接到D来修改其数据结构定义。从而会引入与系统其它部分的兼容性问题。
- 2）由于Vec<u8>在Rust数据结构中非常常见，而开发者需深入代码来解决该问题，这将难以理解。
- 3）若需要大量的补丁，将难以将代码与其原始库同步。这不仅增加了维护开销，而且在实践中经常会导致安全问题。

人们寄希望与specialization来解决该问题，但其尽快stable的概率很低。强烈不建议在生产环境中使用nightly版本，且不适合RISC Zero zkVM中使用nightly版本，因为它正在成为Rust的一部分。

l2iterative的解决方案为：

不同于以上，自下而上的解决方案，采用的自上而下的解决方案，从而无需修改现有Rust中所实现的数据结构，展示了将RISC Zero输入替换为Vec<u32>的序列化及反序列化实现。
其核心思想是引入finite-state automata（名为ByteBufAutomata），当其观察到有多个连续u8待序列化时，会临时修改序列化方式，类似于RISC Zero所推荐的 padded-to-word 方式。
- 激活：当遇到u8类型时，则激活。
  - 序列化：若前一u8未占满整个word，则会将新的u8添入该word中。否则，会创建新word。
  - 反序列化：为读取单个u8，该automata会读取整个word，将其转换为4个自己，并提供这4个字节。该流程会一直重复，直到该automata不激活了，此时要么没有剩余的bytes，或者，所有剩余的bytes均为0。【使用activate_byte_buf_automata_and_take!(self)宏】
- 不激活：当遇到其它类型时，则不激活。【使用deactivate_byte_buf_automata!(self)宏】

#[derive(Default)]
struct ByteBufAutomata(pub u8);impl ByteBufAutomata {#[inline(always)]fn deactivate(&mut self) {self.0 = 0;}fn activate_and_take<W: WordWrite>(&mut self, stream: &mut W, v: u8) {if self.0 == 0 {stream.write_word(v as u32);self.0 = 1;} else {let w = stream.get_last_word();stream.set_last_word(w | ((v as u32) << (self.0 as usize * 8)));self.0 = (self.0 + 1) % 4;}}
}macro_rules! activate_byte_buf_automata_and_take {($self_name:ident, $v: expr) => {$self_name.byte_buf_automata.borrow_mut().activate_and_take(&mut $self_name.stream, $v)};
}macro_rules! deactivate_byte_buf_automata {($self_name:ident) => {$self_name.byte_buf_automata.borrow_mut().deactivate();};
}

对应的序列化为：

// Old
impl<'a, W: WordWrite> serde::ser::Serializer for &'a mut Serializer<W> {fn serialize_u8(self, v: u8) -> Result<()> {self.serialize_u32(v as u32)}fn serialize_u32(self, v: u32) -> Result<()> {self.stream.write_words(&[v]);Ok(())}
}// New
impl<'a, W: WordWrite> serde::ser::Serializer for &'a mut Serializer<W> {fn serialize_u8(self, v: u8) -> Result<()> {activate_byte_buf_automata_and_take!(self, v);Ok(())}fn serialize_u32(self, v: u32) -> Result<()> {deactivate_byte_buf_automata!(self);self.stream.write_word(v);Ok(())}
}

对应的反序列化为：

// old
impl<'de, 'a, R: WordRead + 'de> serde::Deserializer<'de> for &'a mut Deserializer<'de, R> {fn deserialize_u8<V>(self, visitor: V) -> Result<V::Value>whereV: Visitor<'de>,{visitor.visit_u32(self.try_take_word()?)}fn deserialize_u128<V>(self, visitor: V) -> Result<V::Value>whereV: Visitor<'de>,{let mut bytes = [0u8; 16];self.reader.read_padded_bytes(&mut bytes)?;visitor.visit_u128(u128::from_le_bytes(bytes))}
}// new
impl<'de, 'a, R: WordRead + 'de> serde::Deserializer<'de> for &'a mut Deserializer<'de, R> {fn deserialize_u8<V>(self, visitor: V) -> Result<V::Value>whereV: Visitor<'de>,{visitor.visit_u8(activate_byte_buf_automata_and_take!(self))}fn deserialize_u128<V>(self, visitor: V) -> Result<V::Value>whereV: Visitor<'de>,{deactivate_byte_buf_automata!(self);let mut bytes = [0u8; 16];self.reader.read_padded_bytes(&mut bytes)?;visitor.visit_u128(u128::from_le_bytes(bytes))}
}

测试用例见：

#[derive(Debug, Serialize, PartialEq, Eq, Deserialize)]
pub struct WrappedU8(pub u8);#[derive(Default, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct Struct1 {pub u8v: Vec<WrappedU8>,pub u16v: Vec<u16>,pub u32v: Vec<u32>,pub u64v: Vec<u64>,pub i8v: Vec<i8>,pub i16v: Vec<i16>,pub i32v: Vec<i32>,pub i64v: Vec<i64>,pub u8s: u8,pub bs: bool,pub some_s: Option<u16>,pub none_s: Option<u32>,pub strings: Vec<u8>,pub stringv: Vec<Vec<u8>>,
}#[test]
fn test_struct_1() {let mut test_s = Struct1::default();test_s.u8v = vec![WrappedU8(1u8), WrappedU8(231u8), WrappedU8(123u8)];test_s.u16v = vec![124u16, 41374u16];test_s.u32v = vec![14710471u32, 3590275702u32, 1u32, 2u32];test_s.u64v = vec![352905235952532u64, 2147102974910410u64];test_s.i8v = vec![-1i8, 120i8, -22i8];test_s.i16v = vec![-7932i16];test_s.i32v = vec![-4327i32, 35207277i32];test_s.i64v = vec![-1i64, 1i64];test_s.u8s = 3u8;test_s.bs = true;test_s.some_s = Some(5u16);test_s.none_s = None;test_s.strings = b"Here is a string.".to_vec();test_s.stringv = vec![b"string a".to_vec(), b"34720471290497230".to_vec()];let mut res = Vec::<u32>::new();let mut serializer = crate::Serializer::new(&mut res);let _ = test_s.serialize(&mut serializer);let answer = vec![3u32, 8120065, 2, 124, 41374, 4, 14710471, 3590275702, 1, 2, 2, 658142100, 82167,1578999754, 499911, 3, 4294967295, 120, 4294967274, 1, 4294959364, 2, 4294962969, 35207277,2, 4294967295, 4294967295, 1, 0, 259, 1, 5, 0, 17, 1701995848, 544434464, 1953701985,1735289202, 46, 2, 8, 1769108595, 1629513582, 17, 842478643, 825701424, 875575602,858928953, 48,];assert_eq!(answer, res);let recovered: Struct1 = crate::deserializer::from_slice(&res).unwrap();assert_eq!(test_s, recovered);
}

3. zkvm的env模块

zkvm的env模块见：

zkvm/src/host/client/env.rs

env模块内：

定义了ExecutorEnv和ExecutorEnvBuilder
可用于host和guest。
详情可参看 RISC Zero zkVM guest程序优化技巧及其与物理CPU的关键差异的“2. guest程序优化技巧及建议”。

/// A builder pattern used to construct an [ExecutorEnv].
#[derive(Default)]
pub struct ExecutorEnvBuilder<'a> {inner: ExecutorEnv<'a>,
}/// The [crate::Executor] is configured from this object.
///
/// The executor environment holds configuration details that inform how the
/// guest environment is set up prior to guest program execution.
#[derive(Default)]
pub struct ExecutorEnv<'a> {pub(crate) env_vars: HashMap<String, String>,pub(crate) args: Vec<String>,pub(crate) segment_limit_po2: Option<u32>,pub(crate) session_limit: Option<u64>,pub(crate) posix_io: Rc<RefCell<PosixIo<'a>>>,pub(crate) slice_io: Rc<RefCell<SliceIoTable<'a>>>,pub(crate) input: Vec<u8>,pub(crate) trace: Vec<Rc<RefCell<dyn TraceCallback + 'a>>>,pub(crate) assumptions: Rc<RefCell<Assumptions>>,pub(crate) segment_path: Option<PathBuf>,pub(crate) pprof_out: Option<PathBuf>,
}