云原生Kubernetes: K8S 1.29版本部署Harbor

一、实验

1.环境

2.Linux 部署docker compose

3.证书秘钥配置

4.K8S 1.29版本部署Harbor

5.K8S 1.29版本使用Harbor

二、问题

1.docker 登录harbor失败

一、实验

1.环境

（1）主机

表1 主机

主机	架构	版本	IP	备注
master	K8S master节点	1.29.0	192.168.204.8
node1	K8S node节点	1.29.0	192.168.204.9
node2	K8S node节点	1.29.0	192.168.204.10	已部署Kuboard

（2）master节点查看集群

1）查看node
kubectl get node2)查看node详细信息
kubectl get node -o wide

（3）查看pod

[root@master ~]# kubectl get pod -A

(4) 访问Kuboard

http://192.168.204.10:30080/kuboard/cluster

查看节点

（5）查看docker版本

docker -v

master节点

node1节点

node2节点

2.Linux 部署docker compose

(1) 查阅

https://github.com/docker/compose/releases

最新版为v2.27.0

(2) 下载

https://github.com/docker/compose/releases/download/v2.27.0/docker-compose-linux-x86_64

(3) 查看

[root@master ~]# ls | grep docker

（4）移动并更名为docker-compose

[root@master ~]# mv docker-compose-linux-x86_64 /usr/bin/docker-compose

（5）为二进制文件添加可执行权限

[root@master ~]# chmod +x /usr/bin/docker-compose

（6）查看docker-compse版本

[root@master ~]# docker-compose version

3.证书秘钥配置

（1）确认openssl

[root@master harbor]# rpm -q openssl

(2)创建文件夹用于存放生成的证书秘钥文件

[root@master ~]# mkdir -p /harbor/certs/

(3) 生成私钥

[root@master ~]# openssl genrsa -out ca.key 4096

（4）证书秘钥与IP绑定

openssl req -x509 -new -nodes -sha512 -days 3650 \-subj "/C=CN/ST=Nanjing/L=Nanjing/O=example/OU=Personal/CN=192.168.204.8" \-key ca.key \-out ca.crt

（5）移动证书位置并查看

[root@master ~]# mv ca* /harbor/certs
[root@master ~]# ls /harbor/certs/

4.K8S 1.29版本部署Harbor

(1) 查阅

https://github.com/goharbor/harbor/releases

最新版为v2.10.2

(2) 下载

https://github.com/goharbor/harbor/releases/download/v2.10.2/harbor-offline-installer-v2.10.2.tgz

（3）查看

[root@master ~]# ls | grep harbor

（4）解压

[root@master ~]# tar xf harbor-offline-installer-v2.10.2.tgz

（6）进入目录并查看

[root@master ~]# cd harbor/
[root@master harbor]# ls

（7）修改名称

[root@master harbor]# mv harbor.yml.tmpl harbor.yml

（8）监听端口

8800与8443没有被占用

[root@master harbor]# netstat -antlp | grep 8800
[root@master harbor]# netstat -antlp | grep 8443

（9）修改配置文件

[root@master harbor]# vim harbor.yml

hostname域名修改为要安装harbor的服务器的ip
http端口修改为8800
https端口修改为8443
certificate: 证书路径修改为/harbor/certs/ca.crt
private_key: 秘钥路径修改为/harbor/certs/ca.key
harbor_admin_password: 默认Harbor密码，不修改，就用默认的Harbor12345
data_value: 修改为/data/harbor

修改前：

修改后：

（10）Docker Hub查看镜像

docker pull goharbor/prepare:v2.10.2

(11)拉取镜像

node2

[root@node2 ~]# docker pull goharbor/prepare:v2.10.2

(12) 导出镜像

[root@node2 ~]# docker save -o prepare.tar goharbor/prepare:v2.10.2

(13）复制Docker镜像到master节点

[root@node2 ~]# scp prepare.tar root@master:~

(14）master节点导入Docker镜像

[root@master ~]# docker load -i prepare.tar

（15）执行预备脚本

[root@master harbor]# ./prepare

(16) 执行安装脚本

[root@master harbor]# ./install.sh [Step 0]: checking if docker is installed ...Note: docker version: 24.0.7[Step 1]: checking docker-compose is installed ...Note: Docker Compose version v2.21.0[Step 2]: loading Harbor images ...
a8840e4ae316: Loading layer [==================================================>]  21.63MB/21.63MB
515f0ee642d6: Loading layer [==================================================>]  173.8MB/173.8MB
7e1a15fa7f7c: Loading layer [==================================================>]   25.5MB/25.5MB
4cd39a5aa67d: Loading layer [==================================================>]  18.27MB/18.27MB
3daae6a42996: Loading layer [==================================================>]   5.12kB/5.12kB
e23a6fbc3acf: Loading layer [==================================================>]  6.144kB/6.144kB
87a67eef40a6: Loading layer [==================================================>]  3.072kB/3.072kB
5439373f0cf5: Loading layer [==================================================>]  2.048kB/2.048kB
44c50cb14a57: Loading layer [==================================================>]   2.56kB/2.56kB
e343431b8ac0: Loading layer [==================================================>]   7.68kB/7.68kB
Loaded image: goharbor/harbor-db:v2.10.2
a7437080fdff: Loading layer [==================================================>]  17.16MB/17.16MB
89c54c70338a: Loading layer [==================================================>]  3.584kB/3.584kB
927d190fe457: Loading layer [==================================================>]   2.56kB/2.56kB
dab52f006605: Loading layer [==================================================>]  44.92MB/44.92MB
abc8dcfeeb98: Loading layer [==================================================>]  45.71MB/45.71MB
Loaded image: goharbor/harbor-jobservice:v2.10.2
9343d1011e8e: Loading layer [==================================================>]  21.63MB/21.63MB
d8aa785c45d8: Loading layer [==================================================>]  110.6MB/110.6MB
af30438adf19: Loading layer [==================================================>]  3.072kB/3.072kB
046baf104eef: Loading layer [==================================================>]   59.9kB/59.9kB
b4bf035573f1: Loading layer [==================================================>]  61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.10.2
1a53aa673223: Loading layer [==================================================>]  14.15MB/14.15MB
e3ef818740f3: Loading layer [==================================================>]  4.096kB/4.096kB
f51244ac7c03: Loading layer [==================================================>]  3.072kB/3.072kB
971632584e63: Loading layer [==================================================>]   17.4MB/17.4MB
0cfb29c07835: Loading layer [==================================================>]  18.19MB/18.19MB
Loaded image: goharbor/registry-photon:v2.10.2
9fb8de59ccbd: Loading layer [==================================================>]  130.9MB/130.9MB
b0a99212add7: Loading layer [==================================================>]  3.584kB/3.584kB
c63a00cfdba3: Loading layer [==================================================>]  3.072kB/3.072kB
ea1dbece5b3a: Loading layer [==================================================>]   2.56kB/2.56kB
3c735fb3805e: Loading layer [==================================================>]  3.072kB/3.072kB
1b257c0b8709: Loading layer [==================================================>]  3.584kB/3.584kB
6bb174e7db78: Loading layer [==================================================>]  20.48kB/20.48kB
Loaded image: goharbor/harbor-log:v2.10.2
ea9d3f5fbb0c: Loading layer [==================================================>]  121.2MB/121.2MB
b88bdbd268ea: Loading layer [==================================================>]  6.532MB/6.532MB
a6a0154f827a: Loading layer [==================================================>]  246.8kB/246.8kB
664d5baa31ef: Loading layer [==================================================>]  1.477MB/1.477MB
Loaded image: goharbor/harbor-portal:v2.10.2
fe6dfc816265: Loading layer [==================================================>]  17.16MB/17.16MB
c8d1d3bbdf04: Loading layer [==================================================>]  3.584kB/3.584kB
a95b761bfdbe: Loading layer [==================================================>]   2.56kB/2.56kB
c3c092020a00: Loading layer [==================================================>]  58.96MB/58.96MB
75c8aba5c8aa: Loading layer [==================================================>]  5.632kB/5.632kB
84bdcacb7f37: Loading layer [==================================================>]  123.4kB/123.4kB
c80d23d5cead: Loading layer [==================================================>]  80.38kB/80.38kB
fb9a83422043: Loading layer [==================================================>]  59.95MB/59.95MB
17416c18f4d3: Loading layer [==================================================>]   2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.10.2
6a65c62b44a3: Loading layer [==================================================>]  14.15MB/14.15MB
50d4a017d046: Loading layer [==================================================>]  4.096kB/4.096kB
b00285e03979: Loading layer [==================================================>]   17.4MB/17.4MB
bd20c4f45bc4: Loading layer [==================================================>]  3.072kB/3.072kB
cdef8cae2c49: Loading layer [==================================================>]  32.77MB/32.77MB
8c8e4274f70f: Loading layer [==================================================>]  50.95MB/50.95MB
Loaded image: goharbor/harbor-registryctl:v2.10.2
acbecb34a764: Loading layer [==================================================>]  17.16MB/17.16MB
478163dbc994: Loading layer [==================================================>]  27.56MB/27.56MB
4ec3465523a2: Loading layer [==================================================>]  4.608kB/4.608kB
07fb78bc4c52: Loading layer [==================================================>]  28.35MB/28.35MB
Loaded image: goharbor/harbor-exporter:v2.10.2
113051e862a0: Loading layer [==================================================>]  121.2MB/121.2MB
Loaded image: goharbor/nginx-photon:v2.10.2
2716f555462f: Loading layer [==================================================>]  14.67MB/14.67MB
d4471d1781eb: Loading layer [==================================================>]  4.096kB/4.096kB
2b9cfa166f1f: Loading layer [==================================================>]  3.072kB/3.072kB
4943bbb50494: Loading layer [==================================================>]  209.9MB/209.9MB
1f70ad7459fb: Loading layer [==================================================>]  17.45MB/17.45MB
c6de541f3b98: Loading layer [==================================================>]  228.1MB/228.1MB
Loaded image: goharbor/trivy-adapter-photon:v2.10.2
Loaded image: goharbor/prepare:v2.10.2[Step 3]: preparing environment ...[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/harbor
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dirNote: stopping existing Harbor instance ...[Step 5]: starting Harbor ...
[+] Running 10/10✔ Network harbor_harbor        Created                                                                                                                                      0.1s ✔ Container harbor-log         Started                                                                                                                                      0.0s ✔ Container registry           Started                                                                                                                                      0.1s ✔ Container registryctl        Started                                                                                                                                      0.1s ✔ Container harbor-portal      Started                                                                                                                                      0.1s ✔ Container harbor-db          Started                                                                                                                                      0.1s ✔ Container redis              Started                                                                                                                                      0.1s ✔ Container harbor-core        Started                                                                                                                                      0.0s ✔ Container nginx              Started                                                                                                                                      0.0s ✔ Container harbor-jobservice  Started                                                                                                                                      0.0s 
✔ ----Harbor has been installed and started successfully.----

(17) 查看harbor相关容器

[root@master harbor]# docker ps | grep  goharbor

（18）docker-compose命令显示其管理的harbor容器

[root@master harbor]# docker-compose -f docker-compose.yml ps

（19）访问harbor

https://192.168.204.8:8443/

选高级

继续访问

弹出界面

（20）登录

账号：admin
密码：Harbor12345

成功进入系统

5. K8S 1.29版本使用Harbor

(1) 创建私有项目

(2) 查看

目前项目为空

(3) 创建公开项目

（4）再次查看

目前项目为空

（5）Docker配置Harbor（所有节点）

1）修改
vim /etc/docker/daemon.json
……"insecure-registries": ["https://192.168.204.8:8443"]
……2）重新加载配置
systemctl daemon-reload3）重启服务
systemctl restart dockersystemctl restart cri-docker

修改前：

修改后：

这里以node1节点为例

[root@node1 ~]# systemctl status cri-docker

（6）docker 登录harbor并推送镜像到公开项目

node1节点

[root@node1 ~]# docker login 192.168.204.8:8443 -u admin -p Harbor12345

拉取镜像

打标签

[root@node1 ~]# docker tag alpine:latest 192.168.204.8:8443/devops02/alpine:20240502

推送镜像到公开项目

[root@node1 ~]# docker push 192.168.204.8:8443/devops02/alpine:20240502

查看

删除本地镜像

[root@node1 ~]# docker rmi -f 192.168.204.8:8443/devops02/alpine:20240502

拉取harbor仓库的镜像

[root@node1 ~]# docker pull 192.168.204.8:8443/devops02/alpine:20240502

（7）docker 登录harbor并推送镜像到私有项目

打标签

[root@node1 ~]# docker tag alpine:latest 192.168.204.8:8443/devops01/alpine:20240502

推送镜像到私有项目

[root@node1 ~]# docker push 192.168.204.8:8443/devops01/alpine:20240502

查看

删除本地镜像

[root@node1 ~]# docker rmi -f 192.168.204.8:8443/devops01/alpine:20240502

拉取harbor仓库的镜像

[root@node1 ~]# docker pull 192.168.204.8:8443/devops01/alpine:20240502

(8) 登出账号

[root@node1 ~]# docker logout 192.168.204.8

（9）其他方式的harbor部署

可以参考本人博客：

Docker容器与虚拟化技术：Harbor私有仓库部署与迁移-CSDN博客

云原生Kubernetes：K8S集群使用带凭证的harbor仓库-CSDN博客

二、问题

1.docker 登录harbor失败

（1）报错

Error response from daemon: Get "https://192.168.204.8:8443/v2/": tls: failed to verify certificate: x509: certificate relies on legacy Common Name field, use SANs instead

（2）原因分析

未重启docker。

（3）解决方法

重启docker:

[root@node1 ~]# systemctl restart docker

成功：