arm架构部署nexus配置sslcontainerd

背景

合作伙伴私有云的机器架构是arm的，使用registry做为镜像仓库，可以满足基础功能，权限管理等功能无法实现。借鉴现有的架构部署nexus来满足权限管理等需求

思路

翻看dockerhub上没看到有编译好的arm架构的nexus，从github找到源码仓库的dockerfile在arm架构的机器上重新构建arm架构的镜像实现

实施

首先要解决掉机器的网络问题，需要拉取registry.access.redhat.com的镜像，和从download.sonatype.com下载tag包，下载包可以本地下载，改动dockerfile复制进去，

系统信息

root@nexus:/opt/nexus# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
root@nexus:/opt/nexus# uname -a 
Linux nexus 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:10 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
# containerd版本
root@k8s-master01:~# containerd --version 
containerd github.com/containerd/containerd v1.7.20 8fc6bcff51318944179630522a095cc9dbf9f353
# k8s集群版本
root@k8s-master01:~# kubectl version 
Client Version: v1.30.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

构建镜像

git clone https://github.com/sonatype/docker-nexus3.git

略微修改dockerfile的下载源码包，跳过md5校验。

# Download nexus & setup directories
RUN curl -x socks5://xxx:7891 -L ${NEXUS_DOWNLOAD_URL} --output nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz \&& tar -xvf nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz \&& mv nexus-${NEXUS_VERSION} $NEXUS_HOME \&& chown -R nexus:nexus ${SONATYPE_WORK} \&& mv ${SONATYPE_WORK}/nexus3 ${NEXUS_DATA} \&& ln -s ${NEXUS_DATA} ${SONATYPE_WORK}/nexus3

执行构建

docker build --rm=true --tag=sonatype/nexus3-arm 
# 验证
root@nfs:/opt/nexus/docker-nexus3# docker images |grep nexus
sonatype/nexus3-arm                                       latest        fc37defdcdbc   7 weeks ago     913MB
# 导出，拷贝到目标节点上运行
docker save -o nexus3-arm.tar sonatype/nexus3-arm:latest
scp nexus3-arm.tar root@10.17.3.21:/root

启动容器命令

docker run -itd --net=host  --name=nexus3 --user root -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 8084:8084 --privileged=true --restart=always   --ulimit nofile=655350 --ulimit memlock=-1 --memory=7G --memory-swap=-1 --cpuset-cpus='0-3'  -e INSTALL4J_ADD_VM_PARAMS="-Xms2g -Xmx2g -XX:MaxDirectMemorySize=3g" -v /etc/localtime:/etc/localtime         -v /data/nexusdata:/nexus-data sonatype/nexus3-arm:latest

配置ssl

在运行中的nexus容器中执行

NEXUS_DOMAIN=registry.xxx.local     ##更改为你自己的nexus的IP
NEXUS_IP_ADDRESS=192.168.199.12   ##更改为你自己的nexus的IP
# 生成key
# -keystore 文件名
# L 城市 Shanghai Shenzhen都可以 
# ST 城市 Shanghai Shenzhen都可以
# SAN 就是要签发给哪个域名或IP可以使用 
# -validity 签发的证书的有效期
keytool -genkeypair -keystore keystore.jks -alias nexus -keyalg RSA -keysize 2048 -dname "CN=registry.xxx.local, OU=Nexus, O=Nexus, L=ChengShi, ST=ChengShi, C=CN" -ext "SAN=dns:registry.xxx.local,ip:10.17.3.21" -validity 3650
Enter keystore password:  # 输入用于加密keystore.jks文件的密码
Re-enter new password: 
Enter key password for <nexus>(RETURN if same as keystore password):  # 生成cer
[root@nexus ~]# keytool -export -alias nexus -keystore keystore.jks -file keystore.cer -storepass 123456
Certificate stored in file <keystore.cer>Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.jks -deststoretype pkcs12".

从容器中把生成的jks和cer文件拷贝出来

docker cp 46f16d452a65:/opt/sonatype/nexus/etc/ssl/keystore.jks /opt/nexus/nexus-key/
root docker cp 46f16d452a65:/opt/sonatype/nexus/etc/ssl/keystore.cer /opt/nexus/nexus-key/

查看证书的内容

root@nexus:/opt/nexus/nexus-key# openssl x509 -inform der -in keystore.cer -text -noout
Certificate:Data:Version: 3 (0x2)Serial Number: 21382x204 (0x7f72xc)Signature Algorithm: sha256WithRSAEncryptionIssuer: C = CN, ST = xx, L = xx, O = Nexus, OU = Nexus, CN = registry.xx.localValidityNot Before: Sep 11 07:09:57 2024 GMTNot After : Sep  9 07:09:57 2034 GMTSubject: C = CN, ST = xx, L = xx, O = Nexus, OU = Nexus, CN = registry.xx.localSubject Public Key Info:Public Key Algorithm: rsaEncryptionRSA Public-Key: (2048 bit)Modulus:1e:2dExponent: 65537 (0x10001)X509v3 extensions:X509v3 Subject Alternative Name: DNS:registry.xx.local, IP Address:10.17.3.21X509v3 Subject Key Identifier: C5:75:F4:A7:E:62:D1:5F:C9:D7Signature Algorithm: sha256WithRSAEncryption1c:bb:f3:17:47:13:d8:21:21:05:99:cf:ab:5d:43:ae:8e:83:92:99:eb:e8:7d:c7:00:7d:44:fb:68:d1:99:6b:bb:84:79:2a:6d:fc:51:60

重新配置dockerfile

# Download nexus & setup directories
RUN curl -x socks5://xxx:7891 -L ${NEXUS_DOWNLOAD_URL} --output nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz \&& tar -xvf nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz \&& mv nexus-${NEXUS_VERSION} $NEXUS_HOME \&& chown -R nexus:nexus ${SONATYPE_WORK} \&& mv ${SONATYPE_WORK}/nexus3 ${NEXUS_DATA} \&& ln -s ${NEXUS_DATA} ${SONATYPE_WORK}/nexus3# Removing java memory settings from nexus.vmoptions since now we use INSTALL4J_ADD_VM_PARAMS
RUN sed -i '/^-Xms/d;/^-Xmx/d;/^-XX:MaxDirectMemorySize/d' $NEXUS_HOME/bin/nexus.vmoptions
# 新增内容
COPY keystore.cer ${NEXUS_HOME}/etc/ssl/
COPY keystore.jks ${NEXUS_HOME}/etc/ssl/RUN echo "#!/bin/bash" >> ${SONATYPE_DIR}/start-nexus-repository-manager.sh \&& echo "cd /opt/sonatype/nexus" >> ${SONATYPE_DIR}/start-nexus-repository-manager.sh \&& echo "exec ./bin/nexus run" >> ${SONATYPE_DIR}/start-nexus-repository-manager.sh \&& chmod a+x ${SONATYPE_DIR}/start-nexus-repository-manager.sh \&& sed -e '/^nexus-context/ s:$:${NEXUS_CONTEXT}:' -i ${NEXUS_HOME}/etc/nexus-default.properties \&& sed -i '/application-port=8081/i application-port-ssl=443' ${NEXUS_HOME}/etc/nexus-default.properties \ # 监听端口&& sed -i '32c\    <Set name="KeyStorePassword">123456</Set>' ${NEXUS_HOME}/etc/jetty/jetty-https.xml \ # 密钥&& sed -i '33c\    <Set name="KeyManagerPassword">123456</Set>' ${NEXUS_HOME}/etc/jetty/jetty-https.xml \ && sed -i '35c\    <Set name="TrustStorePassword">123456</Set>' ${NEXUS_HOME}/etc/jetty/jetty-https.xml

web界面支持ssl再加两个选项

RUN echo "#!/bin/bash" >> ${SONATYPE_DIR}/start-nexus-repository-manager.sh \&& echo "cd /opt/sonatype/nexus" >> ${SONATYPE_DIR}/start-nexus-repository-manager.sh \&& echo "exec ./bin/nexus run" >> ${SONATYPE_DIR}/start-nexus-repository-manager.sh \&& chmod a+x ${SONATYPE_DIR}/start-nexus-repository-manager.sh \&& sed -e '/^nexus-context/ s:$:${NEXUS_CONTEXT}:' -i ${NEXUS_HOME}/etc/nexus-default.properties \&& sed -i '/application-port=8081/i application-port-ssl=443' ${NEXUS_HOME}/etc/nexus-default.properties \&& sed -i '32c\    <Set name="KeyStorePassword">123456</Set>' ${NEXUS_HOME}/etc/jetty/jetty-https.xml \&& sed -i '33c\    <Set name="KeyManagerPassword">123456</Set>' ${NEXUS_HOME}/etc/jetty/jetty-https.xml \&& sed -i '35c\    <Set name="TrustStorePassword">123456</Set>' ${NEXUS_HOME}/etc/jetty/jetty-https.xml \&& sed -i '/^nexus-args=/ s/$/,${jetty.etc}\/jetty-https.xml/' ${NEXUS_HOME}/etc/nexus-default.properties \&& sed -i '/nexus-args/i ssl.etc=${karaf.data}/etc/ssl' ${NEXUS_HOME}/etc/nexus-default.properties \

构建

docker build --rm=true --tag=sonatype/nexus3-arm:v1.3-ssl .

配置系统信任nexus私有证书

cer转换为pem文件，再把证书导入到节点上，

root@k8s-master01:/usr/local/share/ca-certificates# openssl x509 -inform DER -in /usr/local/share/ca-certificates/keystore.cer -out /usr/local/share/ca-certificates/keystore.pem
# 更新
root@k8s-master01:/usr/local/share/ca-certificates# update-ca-certificates

配置containerd跳过nexus私有证书验证

containerd版本，系统环境信息在上面

containerd_nexus_conf() {mkdir -pv /etc/containerd/certs.d/registry.xx.localcat > /etc/containerd/certs.d/registry.xx.local/hosts.toml <<-EOF
server = "registry.xx.local"
[host."registry.xx.local"]capabilities = ["pull", "resolve", "push"]skip_verify = true
EOFsed -i '/\[plugins\."io\.containerd\.grpc\.v1\.cri"\.registry\]/,/config_path = ""/s|config_path = ""|config_path = "/etc/containerd/certs.d"|g' /etc/containerd/config.tomlsystemctl daemon-reload && systemctl restart containerd;sleep 2containerd_status=$(systemctl is-active containerd)if [ "$containerd_status" == "active" ]; thenecho_green "containerd conf nexus success."else :echo_red "containerd conf nexus failed."exit 3fi
}