最后在Ubuntu22.04使用Ubuntu提供的安装命令完成了安装。这里是记录尝试在Ubuntu18.04和Ubuntu22.04使用源码安装未果的过程。

---

22版本安装bcc

准备工具

sudo apt install -y zip bison build-essential cmake flex git libedit-dev \libllvm14 llvm-14-dev libclang-14-dev python3 zlib1g-dev libelf-dev libfl-dev python3-setuptools \liblzma-dev libdebuginfod-dev arping netperf iperf

安装命令

wget https://github.com/iovisor/bcc/releases/download/v0.28.0/bcc-src-with-submodule.tar.gz
mkdir bcc/build; cd bcc/build
cmake ..
make
sudo make install
cmake -DPYTHON_CMD=python3 .. # build python3 binding
pushd src/python/
make
sudo make install
popd

注意，上面那段安装命令不是直接使用官网提供的git clone命令，因为那不是release版本缺少头文件，会产生如下报错：

CMake Warning at CMakeLists.txt:73 (message):Failed to update submodule libbpfbcc/src/cc/frontends/clang/b_frontend_action.cc:37:
/home/xmu3/bcc/src/cc/bcc_libbpf_inc.h:8:11: fatal error: libbpf/src/bpf.h: No such file or directory8 | # include "libbpf/src/bpf.h"|           ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [src/cc/frontends/clang/CMakeFiles/clang_frontend-objects.dir/build.make:90: src/cc/frontends/clang/CMakeFiles/clang_frontend-objects.dir/b_frontend_action.cc.o] Error 1

使用报错：

sudo ./execsnoop.py
Traceback (most recent call last):File "/home/bcc/tools/./execsnoop.py", line 23, in <module>from bcc import BPFFile "/usr/lib/python3/dist-packages/bcc-EAD_HASH_NOTFOUND_GITDIR_N-py3.10.egg/bcc/__init__.py", line 26, in <module>File "/usr/lib/python3/dist-packages/bcc-EAD_HASH_NOTFOUND_GITDIR_N-py3.10.egg/bcc/libbcc.py", line 17, in <module>File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__self._handle = _dlopen(self._name, mode)
OSError: libbcc.so.0: cannot open shared object file: No such file or directory

可以查到很多关于libbcc.so的报错，但是后面的错误各有差异，一直解决未果。（谁找到了希望滴滴我）

iovisor封装的安装方式

没有尝试过，但是听所也有坑。

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

ubuntu的安装方式

这种方式安装的目录是在/usr/sbin，那些文件都是*-bpfcc的样式。和源码安装相比有的功能这里是没有的。以及Ubuntu18.04之后的版本才能使用这个命令，之前以及18.04都没有。

sudo apt-get install bpfcc-tools linux-headers-$(uname -r)

For Bionic (18.04 LTS)官方提供的源码安装

准备工具安装

sudo apt-get -y install zip bison build-essential cmake flex git libedit-dev \libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev libfl-dev python3-setuptools \liblzma-dev arping netperf iperf

下载和安装

git clone https://github.com/iovisor/bcc.git
mkdir bcc/build; cd bcc/build
cmake ..
make
sudo make install
cmake -DPYTHON_CMD=python3 .. # build python3 binding
pushd src/python/
make
sudo make install
popd

使用报错：

yi@PC:~/bcc/build$ sudo /usr/share/bcc/tools/opensnoop
Traceback (most recent call last):File "/usr/share/bcc/tools/opensnoop", line 24, in <module>from bcc import ArgString, BPFFile "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 27, in <module>from .libbcc import lib, _CB_TYPE, bcc_symbol, bcc_symbol_option, _SYM_CB_TYPEFile "/usr/lib/python2.7/dist-packages/bcc/libbcc.py", line 20, in <module>lib.bpf_module_create_b.restype = ct.c_void_pFile "/usr/lib/python2.7/ctypes/__init__.py", line 379, in __getattr__func = self.__getitem__(name)File "/usr/lib/python2.7/ctypes/__init__.py", line 384, in __getitem__func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib/x86_64-linux-gnu/libbcc.so.0: undefined symbol: bpf_module_create_b

修改脚本开头，把python换成python3，或者使用命令时加上python3
好的，接下来这个bug成为每一次源码安装都跨不过的坎了。

sudo /usr/share/bcc/tools/execsnoop
LLVM ERROR: 

最开始以为LLVM9以下bcc好像不支持了。其实和LLVM没关系，官方也是用的LLVM-7。安装完LLVM15之后，源码安装过程中另一个问题，关于python

[ 36%] Building sdist for python3
Traceback (most recent call last):File "/home/yi/下载/bcc/build/src/python/bcc-python3/setup.py", line 3, in <module>from setuptools import setupFile "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 23, in <module>from setuptools.dist import DistributionFile "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 34, in <module>from setuptools import windows_supportFile "/usr/local/lib/python3.9/site-packages/setuptools/windows_support.py", line 2, in <module>import ctypesFile "/usr/local/lib/python3.9/ctypes/__init__.py", line 8, in <module>from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
src/python/CMakeFiles/bcc_py_python3.dir/build.make:71: recipe for target 'src/python/bcc-python3/dist/bcc-0.28.0+9596edb9.tar.gz' failed
make[2]: *** [src/python/bcc-python3/dist/bcc-0.28.0+9596edb9.tar.gz] Error 1
CMakeFiles/Makefile2:616: recipe for target 'src/python/CMakeFiles/bcc_py_python3.dir/all' failed
make[1]: *** [src/python/CMakeFiles/bcc_py_python3.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

查了查大家都说是要重装python3，结果系统给我重装没了。
python2运行以下报错，让用python3即可

sudo ./execsnoop
Traceback (most recent call last):File "./execsnoop", line 23, in <module>from bcc import BPF
ImportError: No module named bcc

python3运行又是LLVM ERROR
单步调试在b = BPF(text=bpf_text)处

sudo python3 ./hello_world.py
bpf: Argument list too long. Program  too large (0 insns), at most 4096 insnsTraceback (most recent call last):File "./hello_world.py", line 12, in <module>BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()File "/usr/lib/python3/dist-packages/bcc-0.28.0+9596edb9-py3.6.egg/bcc/__init__.py", line 487, in __init__File "/usr/lib/python3/dist-packages/bcc-0.28.0+9596edb9-py3.6.egg/bcc/__init__.py", line 1465, in _trace_autoloadFile "/usr/lib/python3/dist-packages/bcc-0.28.0+9596edb9-py3.6.egg/bcc/__init__.py", line 527, in load_func
Exception: Failed to load BPF program b'kprobe__sys_clone': Argument list too long

再debug下去感觉没完没了，干了一天后就换Ubuntu22了，两个系统的使用感像从农村到了城里。

安装bpftrace

Ubuntu22自带安装

官方安装说明
安装的目录是在/usr/sbin，那些文件都是*-bt的样式

sudo apt-get install -y bpftrace

但是也会报错：

ERROR: Could not resolve symbol: /proc/self/exe:BEGIN_trigger

这个报错在github上有很长的讨论，不过解决方案就一句命令：

sudo apt-get install bpftrace-dbgsym

然而，安装说无法定位，也是报错。这时候去Launchpad上面有发布的软件包安装bpftrace-dbgsym_0.14.0-1_amd64.ddeb这个调试符号包。下载后

sudo dpkg -i bpftrace-dbgsym_0.14.0-1_amd64.ddeb

在Ubuntu下查找某个软件包的PPA源的方法主要有以下几种:
1、在软件包的官网或者GitHub仓库的说明中查找,维护者通常会提供建议的PPA源。
2、在Launchpad网站上搜索软件包名称,查看是否存在相关的PPA源。Launchpad托管了Ubuntu的大量PPA源。
3、使用Google搜索"软件包名 + ppa",看是否有相关讨论帖子推荐的PPA源。
4、使用命令行工具apt-cache search,看搜索结果中是否有类似“ppa.launchpad.net”的源,这表示系统已知的PPA。
5、使用命令行工具apt-add-repository,它可以直接通过ppa:user/ppa-name的格式添加某PPA源。
6、检查/etc/apt/sources.list.d目录下的配置文件,是否已有该软件包相关的PPA源,文件名称通常是ppa的名称。
7、使用诸如aptly、repoman等第三方工具也可以搜索本地已知的所有PPA源列表。
一般来说,维护者提供的PPA源是首选,查找时注意区分体系架构、Ubuntu版本、软件包版本等差异。

Ubuntu22源码安装

从源码安装的话：bpftrace 二进制文件将安装在 /usr/local/bin/bpftrace 中，工具将安装在 /usr/local/share/bpftrace/tools 中。可以使用 cmake 的参数更改安装位置，默认位置为 -DCMAKE_INSTALL_PREFIX=/usr/local。

sudo apt-get update
sudo apt-get install -y libbpfcc-dev
sudo apt-get install -y \bison \cmake \flex \g++ \git \libelf-dev \zlib1g-dev \libfl-dev \systemtap-sdt-dev \binutils-dev \libcereal-dev \llvm-dev \llvm-runtime \libclang-dev \clang \libpcap-dev \libgtest-dev \libgmock-dev \asciidoctor \libdw-dev \pahole
git clone https://github.com/iovisor/bpftrace --recurse-submodules
mkdir bpftrace/build; cd bpftrace/build;
../build-libs.sh
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j8
sudo make install

也是有报错，但是忘记记录了。

Ubuntu18源码安装

bpftrace将llvm降到7，并且源码安装0.11.1之后，make报错：

bpftrace/src/attached_probe.cpp:801:51: error: too few arguments to function ‘void* bpf_attach_uprobe(int, bpf_probe_attach_type, const char*, const char*, uint64_t, pid_t, int, int, perf_reader_cb, void*)’probe_.pid);^
In file included from /home/xmu2/bpftrace/src/attached_probe.h:10:0,from /home/xmu2/bpftrace/src/attached_probe.cpp:16:
/usr/include/bcc/libbpf.h:80:8: note: declared herevoid * bpf_attach_uprobe(int progfd, enum bpf_probe_attach_type attach_type,^~~~~~~~~~~~~~~~~
/home/xmu2/bpftrace/src/attached_probe.cpp: In member function ‘void bpftrace::AttachedProbe::attach_usdt(int)’:
/home/xmu2/bpftrace/src/attached_probe.cpp:911:77: error: too few arguments to function ‘void* bpf_attach_uprobe(int, bpf_probe_attach_type, const char*, const char*, uint64_t, pid_t, int, int, perf_reader_cb, void*)’eventname().c_str(), probe_.path.c_str(), offset_, pid == 0 ? -1 : pid);^
In file included from /home/xmu2/bpftrace/src/attached_probe.h:10:0,from /home/xmu2/bpftrace/src/attached_probe.cpp:16:
/usr/include/bcc/libbpf.h:80:8: note: declared herevoid * bpf_attach_uprobe(int progfd, enum bpf_probe_attach_type attach_type,^~~~~~~~~~~~~~~~~
/home/xmu2/bpftrace/src/attached_probe.cpp: In member function ‘void bpftrace::AttachedProbe::attach_tracepoint()’:
/home/xmu2/bpftrace/src/attached_probe.cpp:927:26: error: too few arguments to function ‘void* bpf_attach_tracepoint(int, const char*, const char*, int, int, int, perf_reader_cb, void*)’eventname().c_str());^
In file included from /home/xmu2/bpftrace/src/attached_probe.h:10:0,from /home/xmu2/bpftrace/src/attached_probe.cpp:16:
/usr/include/bcc/libbpf.h:87:8: note: declared herevoid * bpf_attach_tracepoint(int progfd, const char *tp_category,^~~~~~~~~~~~~~~~~~~~~
/home/xmu2/bpftrace/src/attached_probe.cpp: In member function ‘void bpftrace::AttachedProbe::attach_watchpoint(int, const string&)’:
/home/xmu2/bpftrace/src/attached_probe.cpp:1120:23: error: ‘bpf_attach_perf_event_raw’ was not declared in this scopeint perf_event_fd = bpf_attach_perf_event_raw(progfd_, &attr, pid, -1, -1, 0);^~~~~~~~~~~~~~~~~~~~~~~~~
/home/xmu2/bpftrace/src/attached_probe.cpp:1120:23: note: suggested alternative: ‘bpf_attach_perf_event’int perf_event_fd = bpf_attach_perf_event_raw(progfd_, &attr, pid, -1, -1, 0);^~~~~~~~~~~~~~~~~~~~~~~~~bpf_attach_perf_event
src/CMakeFiles/bpftrace.dir/build.make:75: recipe for target 'src/CMakeFiles/bpftrace.dir/attached_probe.cpp.o' failed
make[2]: *** [src/CMakeFiles/bpftrace.dir/attached_probe.cpp.o] Error 1
CMakeFiles/Makefile2:1160: recipe for target 'src/CMakeFiles/bpftrace.dir/all' failed
make[1]: *** [src/CMakeFiles/bpftrace.dir/all] Error 2
Makefile:145: recipe for target 'all' failed
make: *** [all] Error 2

bpftrace安装时报错

CMake Error at /usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):Please run /home/xmu2/bpftrace/build-libs.sh from the build folder first(missing: LIBBCC_LIBRARIES LIBBCC_INCLUDE_DIRS)
Call Stack (most recent call first):/usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)cmake/FindLibBcc.cmake:84 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)CMakeLists.txt:116 (find_package)

按照其他说法安装libbpfcc后仍然报错：

CMake Error at /usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):Please run /home/xmu2/bpftrace/build-libs.sh from the build folder first(missing: LIBBCC_INCLUDE_DIRS)
Call Stack (most recent call first):/usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)cmake/FindLibBcc.cmake:84 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)CMakeLists.txt:116 (find_package)

变成版本llvm-7之后：

/virtual/main.c:138:1: error: expected identifier or '('
段错误

参考：

https://blog.csdn.net/Longyu_wlz/article/details/109900096
https://github.com/bpftools/linux-observability-with-bpf.git

https://blog.csdn.net/qq_34258344/article/details/104400738
https://yaoyao.io/posts/how-to-setup-ebpf-env-on-ubuntu.html

作者回复: libbpf-dev只包含在比较新的发行版中，其他发行版可以从源码安装，具体步骤可以参考 https://github.com/libbpf/libbpf#build。

另外，我们案例的Github中也有源码编译的详细步骤：https://github.com/feiskyer/ebpf-apps/blob/main/bpf-apps/Makefile#L21-L22

---

by the way

按道理这些bug都可以去github开个issue的，不过比较赶时间，也不确定可不可以复现，有谁又遇到了可以去开一个。