一、前言

参考文档：
Building+Lustre+from+Source
Compiling_Lustre

本篇主要介绍Lustre软件源码编译相关过程，供以后参考使用

二、编译说明

OS：CentOS Linux release 7.9.2009 (Core)
Kernel：3.10.0-1160.el7.x86_64
lfs：2.12.9

1、配置yum源

• 添加国内镜像yum源及epel源

yum install wget -y
mv /etc/yum.repos.d/ /etc/yum.repos.d-bak/
mkdir /etc/yum.repos.d/
wget http://mirrors.aliyun.com/repo/Centos-7.repo -P /etc/yum.repos.d/
wget http://mirrors.aliyun.com/repo/epel-7.repo -P /etc/yum.repos.d/

• 添加lustre源

[root@node94 ~]# cat /etc/yum.repos.d/lustre.repo 
[lustre-server]
name=lustre-server
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.12.9/el7.9.2009/server/
gpgcheck=0
enable=1
[root@node94 ~]# yum makecache

2、升级内核

• 安装lustre内核版本

yum install kernel-3.10.0-1160.49.1.el7_lustre kernel-headers-3.10.0-1160.49.1.el7_lustre  -y
yum install kernel-debuginfo-common-x86_64-3.10.0-1160.49.1.el7_lustre kernel-devel-3.10.0-1160.49.1.el7_lustre  -y

注：需要安装对应内核版本kernel-devel和kernel-debuginfo-common软件包，否则编译ldiskfs相关包会失败

Disabling ldiskfs support because complete ext4 source does not exist.If you are building using kernel-devel packages and require ldiskfs
server support then ensure that the matching kernel-debuginfo-common
and kernel-debuginfo-common-<arch> packages are installed.

3、安装依赖

• 下载源码包，安装对应依赖

yum -y groupinstall 'Development Tools'
yum install libyaml libyaml-devel libselinux-devel -y
yum install openssl-devel libselinux-devel -y
yum install libnvpair1 libuutil1 libzfs2 libzfs2-devel libzpool2 -y
yum install spl spl-debuginfo spl-dkms -y
yum install zfs zfs-debuginfo zfs-dkms zfs-test -y

• 解决/usr/bin/ld: cannot find -lzfs/-lnvpair错误

ln -s /usr/lib64/libnvpair.so.1.0.1 /usr/lib64/libnvpair.so
ln -s /usr/lib64/libzfs.so.2.0.0 /usr/lib64/libzfs.so
echo "/usr/lib64/" >> /etc/ld.so.conf
ldconfig

3、源码编译

• 下载源码包

wget https://downloads.whamcloud.com/public/lustre/lustre-2.12.9/el7.9.2009/server/SRPMS/lustre-2.12.9-1.src.rpm
rpm2cpio lustre-2.12.9-1.src.rpm |cpio -div

• 解决zfs编译问题

解决dkms编译问题：Install-Lustre2.12.8-on-CentOS-7.9-2009

修改源码文件lustre-dkms_pre-build.sh，注释掉dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $2; exit 0}' | grep -v ': added$'获取zfs版本命令，修改为dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $1; exit 0}' | cut -f2 -d'/'| cut -f1 -d':'，之后重新打包源码包，重新编译软件

[root@node96 ~]# tar -zxvf lustre-2.12.9.tar.gz
[root@node96 ~]# cat ./lustre-2.12.9/lustre-dkms_pre-build.sh | grep ZFS_VERSION=# ZFS_VERSION=$(dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $2; exit 0}' | grep -v ': added$')ZFS_VERSION=$(dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $1; exit 0}' | cut -f2 -d'/'| cut -f1 -d':')
[root@node96 ~]# tar -zcvf lustre-2.12.9.tar.gz lustre-2.12.9

• 源码编译rpm包

cd lustre-2.12.9
sh ./autogen.sh
./configure 2>&1 | tee configure.log
make -j $(nproc) rpms 2>&1 | tee compile.log

• 编译lustre-zfs-dkms软件包

cp ../lustre-2.12.9.tar.gz /root/rpmbuild/SOURCES/
rpmbuild -bb --with zfs lustre-dkms.spec
cp -ra  /root/rpmbuild/RPMS/noarch/lustre-zfs-dkms-2.12.9-1.el7.noarch.rpm ./

• 编译成功后，会在当前目录下生成对应rpm包

Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/lustre-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/kmod-lustre-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/kmod-lustre-osd-ldiskfs-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/lustre-osd-ldiskfs-mount-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/kmod-lustre-osd-zfs-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/lustre-osd-zfs-mount-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/lustre-resource-agents-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/lustre-tests-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/kmod-lustre-tests-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/lustre-iokit-2.12.9-1.el7.x86_64.rpm
Wrote: /tmp/rpmbuild-lustre-root-bHLH8tPT/RPMS/x86_64/lustre-debuginfo-2.12.9-1.el7.x86_64.rpmWrote: /root/rpmbuild/RPMS/noarch/lustre-zfs-dkms-2.12.9-1.el7.noarch.rpm

• 使用kmod安装lustre zfs文件系统相关软件包（kmod-lustre-osd-zfs），安装失败出现ksym错误，改用dkms方式安装（使用rpm -Uvh安装lustre-osd-zfs-mount和lustre-zfs-dkms软件包）

---> Package lustre-osd-zfs-mount.x86_64 0:2.12.9-1.el7 will be installed
--> Finished Dependency Resolution
Error: Package: kmod-lustre-osd-zfs-2.12.9-1.el7.x86_64 (/kmod-lustre-osd-zfs-2.12.9-1.el7.x86_64)Requires: ksym(sa_bulk_lookup) = 0xbcff886a

三、Q&A

1、/usr/bin/ld: cannot find -lxxx

a、问题说明

编译过程中出现/usr/bin/ld: cannot find -lxxx错误，相关提示信息如下：

/usr/bin/ld: cannot find -lzfs
/usr/bin/ld: cannot find -lnvpair
collect2: error: ld returned 1 exit status

b、原因分析

当出现/usr/bin/ld: cannot find -lxxx错误时，通常是找不到xxx库文件导致（库文件命名一般为libxxx.so），解决思路如下：
通过find / -name *xxx.so*全局搜索，确认是否已安装库文件相关软件

• 如无法搜索到对应so文件，则查找对应库文件由哪个软件提供，安装对应软件即可
• 如可以搜索到对应so文件，但文件名称带有其他后缀信息，则需要更改so文件名称创建一个软链接

#参考示例如下，出现/usr/bin/ld: cannot find -lzfs错误，全局查找存在libzfs.so.2.0.0文件，so后带有其他后缀信息
[root@node94 ~]# find / -name *zfs.so*
/usr/lib64/libzfs.so.2
/usr/lib64/libzfs.so.2.0.0
#创建一个libzfs.so库文件软链接，指向libzfs.so.2.0.0文件
[root@node94 ~]# ln -s /usr/lib64/libzfs.so.2.0.0 /usr/lib64/libzfs.so

• 如可以搜索到对应so文件，且文件名称未带有其他后缀，则可将库文件所在路径添加到/etc/ld.so.conf文件内

echo "/usr/lib64/" >> /etc/ld.so.conf
ldconfig

c、解决措施

• 遍历根目录可查找到对应so文件

[root@node94 ~]# find / -name *zfs.so*
/usr/lib64/libzfs.so.2
/usr/lib64/libzfs.so.2.0.0
[root@node94 ~]# find / -name *nvpair.so*
/usr/lib64/libnvpair.so.1
/usr/lib64/libnvpair.so.1.0.1

• 创建软链接重命名so文件，将so文件路径添加到/etc/ld.so.conf内

ln -s /usr/lib64/libnvpair.so.1.0.1 /usr/lib64/libnvpair.so
ln -s /usr/lib64/libzfs.so.2.0.0 /usr/lib64/libzfs.so
echo "/usr/lib64/" >> /etc/ld.so.conf
ldconfig

2、Deprecated feature: REMAKE_INITRD

a、问题说明

安装完相关软件依赖包之后，执行dmks出现Deprecated feature: REMAKE_INITRD提示信息

[root@node96 lustre-2.12.9]# dkms status
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/spl/0.7.13/source/dkms.conf)
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/zfs/0.7.13/source/dkms.conf)
spl/0.7.13, 3.10.0-1160.49.1.el7_lustre.x86_64, x86_64: installed
zfs/0.7.13, 3.10.0-1160.49.1.el7_lustre.x86_64, x86_64: installed

b、解决措施

• 修改spl和zfs dkms配置文件，注释掉REMAKE_INITRD="no"配置

sed -i "s/REMAKE_INITRD=\"no\"/#REMAKE_INITRD=\"no\"/g" /usr/src/spl-0.7.13/dkms.conf
sed -i "s/REMAKE_INITRD=\"no\"/#REMAKE_INITRD=\"no\"/g" /usr/src/zfs-0.7.13/dkms.conf

• 再次执行dkms status命令，此时没有出现Deprecated feature: REMAKE_INITRD提示信息

[root@node96 lustre-2.12.9]# dkms status
spl/0.7.13, 3.10.0-1160.49.1.el7_lustre.x86_64, x86_64: installed
zfs/0.7.13, 3.10.0-1160.49.1.el7_lustre.x86_64, x86_64: installed

3、源码编译安装lustre-zfs软件包后，缺少osd-zfs内核模块

a、问题说明

使用源码编译后的lustre软件包安装，执行mkfs.lustre --backfstype=zfs格式化zpool成功，但执行mount -t lustre挂载zpool失败，查看dmesg信息出现LustreError: 158-c: Can't load module 'osd-zfs'错误

b、原因分析

官方提供的源码文件中，lustre-dkms_pre-build.sh获取zfs版本方法有误，拿不到zfs版本信息，导致后续的zfs相关内核模块未编译成功

[root@node96 lustre-2.12.9]# cat ./configure | grep zfsverzfsver=$(ls -1 /usr/src/ | grep -m1 zfs | cut -f2 -d'-')zfsdkms="/var/lib/dkms/zfs/${zfsver}"zfssrc1="/usr/src/zfs-${zfsver}/${LINUXRELEASE}"zfssrc2="/usr/src/zfs-${zfsver}"zfsver=$((echo "#include <zfs_config.h>";echo "zfsver=ZFS_META_VERSION-ZFS_META_RELEASE") |grep "^zfsver=" | tr -d \" | cut -d= -f2)zfsver="Not found"{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $zfsver" >&5
$as_echo "$zfsver" >&6; }ZFS_VERSION=${zfsver}ZFS_MAJOR=$(echo $zfsver | sed -re 's/([0-9]+)\.([0-9]+)\.([0-9]+)(\.([0-9]+))?.*/\1/')ZFS_MINOR=$(echo $zfsver | sed -re 's/([0-9]+)\.([0-9]+)\.([0-9]+)(\.([0-9]+))?.*/\2/')ZFS_PATCH=$(echo $zfsver | sed -re 's/([0-9]+)\.([0-9]+)\.([0-9]+)(\.([0-9]+))?.*/\3/')ZFS_FIX=$(echo $zfsver   | sed -re 's/([0-9]+)\.([0-9]+)\.([0-9]+)(\.([0-9]+))?.*/\5/')
[root@node96 lustre-2.12.9]# ls -1 /usr/src/ | grep -m1 zfs | cut -f2 -d'-'
0.7.13
[root@node96 lustre-2.12.9]# cat lustre-dkms_pre-build.sh | grep ZFS_VERSIONZFS_VERSION=$(dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $2; exit 0}' | grep -v ': added$')if [ -z $ZFS_VERSION ] ; then--with-spl=$6/spl-${ZFS_VERSION} \--with-spl-obj=$7/spl/${ZFS_VERSION}/$3/$5 \--with-zfs=$6/zfs-${ZFS_VERSION} \--with-zfs-obj=$7/zfs/${ZFS_VERSION}/$3/$5"
[root@node96 lustre-2.12.9]# dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $2; exit 0}' | grep -v ': added$'[root@node96 lustre-2.12.9]# dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $1; exit 0}' | cut -f2 -d'/'| cut -f1 -d':'
0.7.13

c、解决措施

解决dkms编译问题：Install-Lustre2.12.8-on-CentOS-7.9-2009

修改源码文件lustre-dkms_pre-build.sh，注释掉dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $2; exit 0}' | grep -v ': added$'获取zfs版本命令，修改为dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $1; exit 0}' | cut -f2 -d'/'| cut -f1 -d':'，之后重新打包源码包，重新编译软件

[root@node96 ~]# tar -zxvf lustre-2.12.9.tar.gz
[root@node96 ~]# cat ./lustre-2.12.9/lustre-dkms_pre-build.sh | grep ZFS_VERSION=# ZFS_VERSION=$(dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $2; exit 0}' | grep -v ': added$')ZFS_VERSION=$(dkms status -m zfs -k $3 -a $5 | awk -F', ' '{print $1; exit 0}' | cut -f2 -d'/'| cut -f1 -d':')
[root@node96 ~]# tar -zcvf lustre-2.12.9.tar.gz lustre-2.12.9

4、安装lustre-zfs-dkms软件包，编译过程中出现No openssk-devel headers found警告信息

a、问题说明

安装lustre-zfs-dkms软件包，编译过程中出现以下警告信息

[root@node93 lustre-zfs]# rpm -Uvh *.rpm
Preparing...                          ################################# [100%]
Updating / installing...1:lustre-osd-zfs-mount-2.12.9-1.el7################################# [ 50%]2:lustre-zfs-dkms-2.12.9-1.el7     ################################# [100%]
Loading new lustre-zfs-2.12.9 DKMS files...
Deprecated feature: REMAKE_INITRD (/usr/src/lustre-zfs-2.12.9/dkms.conf)
Building for 3.10.0-1160.49.1.el7_lustre.x86_64
Building initial module for 3.10.0-1160.49.1.el7_lustre.x86_64
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/lustre-zfs/2.12.9/source/dkms.conf)
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/zfs/0.7.13/source/dkms.conf)
configure: WARNING:No openssk-devel headers found, unable to build l_getsepol and SELinux status checkingconfigure: WARNING:No selinux package found, unable to build selinux enabled toolsDone.
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/lustre-zfs/2.12.9/source/dkms.conf)
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/lustre-zfs/2.12.9/source/dkms.conf)

b、解决措施

官方文档相关错误提示判断：lustre-core.m4

• 安装相关依赖包

yum install openssl-devel libselinux-devel -y

• 处理Deprecated feature: REMAKE_INITRD警告信息

sed -i "s/REMAKE_INITRD=\"no\"/#REMAKE_INITRD=\"no\"/g" /usr/src/lustre-zfs-2.12.9/dkms.conf
sed -i "s/REMAKE_INITRD=\"no\"/#REMAKE_INITRD=\"no\"/g" /var/lib/dkms/lustre-zfs/2.12.9/source/dkms.conf
sed -i "s/REMAKE_INITRD=\"no\"/#REMAKE_INITRD=\"no\"/g" /var/lib/dkms/zfs/0.7.13/source/dkms.conf