目录
Part.01 Kubernets与docker
Part.02 Docker版本
Part.03 Kubernetes原理
Part.04 资源规划
Part.05 基础环境准备
Part.06 Docker安装
Part.07 Harbor搭建
Part.08 K8s环境安装
Part.09 K8s集群构建
Part.10 容器回退
第七章 Harbor搭建
Docker-Compose是用来管理容器的，类似用户容器管家，我们有N多台容器或者应用需要启动的时候，如果手动去操作，是非常耗费时间的，如果有了Docker-Compose只需要一个配置文件就可以帮我们搞定，但是Docker-Compose只能管理当前主机上的Docker，不能去管理其他服务器上的服务。与k8s的区别如下：
- compose是docker推出的（swarm也是，级别同k8s），k8s是CNCF推出的
- compose只能在一台宿主机上编排容器，而k8s可以在很多台机器上编排容器
Docker-Compose由python实现，调用docker服务的API负责实现对docker容器集群的快速编排，即通过一个单独的yaml文件，来定义一组相关的容器来为一个项目服务。因此，harbor也是通过Docker-Compose来实现的。
过程：harbor下有install.sh脚本，里面会调用docker-compose，通过配置文件harbor.yml来实现对harbor的安装。
7.1.安装dockers-compose
docker-compose软件是一个可执行的二进制文件，在harbor01上将二进制文件上传至/usr/local/bin后赋予执行权限。
下载链接：
https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64
cp /opt/harbor/docker-compose-linux-x86_64 /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
查看版本
[root@harbor01 ~]# docker-compose --version
Docker Compose version v2.16.0
7.2.安装harbor
7.2.1.安装
下载harbor安装包，下载页面：
https://github.com/goharbor/harbor/releases/tag/v2.7.2
上传后解压
tar -xvf /opt/harbor/harbor-offline-installer-v2.7.2.tgz -C /opt/harbor/
修改yaml配置文件
cp /opt/harbor/harbor/harbor.yml.tmpl /opt/harbor/harbor/harbor.yml
修改内容如下：
# 修改hostname
hostname: harbor01.k8s.local
# 不使用http协议，注释掉http和port选项
#http:
# port: 80
# 启用https协议
https:port: 443# 证书位置certificate: /opt/harbor/harbor/certs/harbor.crt# 私钥位置private_key: /opt/harbor/harbor/certs/harbor.key
# 页面密码
harbor_admin_password: lnyd@LNsy115
database:# 数据库密码password: root123
# 存储位置
data_volume: /data
创建数据存储目录
mkdir /data
创建证书和私钥对应的路径
mkdir /opt/harbor/harbor/certs
7.2.2.生成自签证书
- 生成证书颁发机构证书
生成CA证书私钥（ca.key）
[root@harbor01 harbor]# cd /opt/harbor/harbor/certs/
[root@harbor01 certs]# openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus
.........++
....................................................................................................................++
e is 65537 (0x10001)
生成CA证书（ca.crt）
调整-subj选项中的值以反映组织信息，如果使用FQDN连接Harbor主机，则必须将其指定为通用名称（CN）属性。
openssl req -x509 -new -nodes -sha512 -days 3650 \-subj "/C=CN/ST=Liaoning/L=Shenyang/O=kubernetes/OU=Personal/CN=harbor01.k8s.local" \-key ca.key \-out ca.crt
- 生成服务器证书
证书通常包含一个.crt文件和一个.key文件
生成私钥（harbor01.k8s.local.key）
[root@harbor01 certs]# openssl genrsa -out harbor01.k8s.local.key 4096
Generating RSA private key, 4096 bit long modulus
........................................................................................................................................++
.........................................................................................................++
e is 65537 (0x10001)
生成证书签名请求（harbor01.k8s.local.csr）
openssl req -sha512 -new \-subj "/C=CN/ST=Liaoning/L=Shenyang/O=kubernetes/OU=Personal/CN=harbor01.k8s.local" \-key harbor01.k8s.local.key \-out harbor01.k8s.local.csr
生成一个x509 v3扩展文件（v3.ext）
无论使用FQDN还是IP地址连接到Harbor主机，都必须创建此文件，以便可以为Harbor主机生成符合主题备用名称（SAN）和x509 v3的证书扩展要求。替换DNS条目以反映域。
cat > v3.ext <<-EOFauthorityKeyIdentifier=keyid,issuerbasicConstraints=CA:FALSEkeyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEnciphermentextendedKeyUsage = serverAuthsubjectAltName = @alt_names[alt_names]DNS.1=harbor01.k8s.localDNS.2=harbor01.k8s.localDNS.3=harbor01.k8s.local
EOF
使用v3.ext文件生成Harbor服务器证书（harbor01.k8s.local.crt）
[root@harbor01 certs]# openssl x509 -req -sha512 -days 3650 \
> -extfile v3.ext \
> -CA ca.crt -CAkey ca.key -CAcreateserial \
> -in harbor01.k8s.local.csr \
> -out harbor01.k8s.local.crt
Signature ok
subject=/C=CN/ST=Liaoning/L=Shenyang/O=kubernetes/OU=Personal/CN=harbor01.k8s.local
Getting CA Private Key
7.2.3.配置daemon.json文件
在master01上配置镜像加速地址以及
{"registry-mirrors": ["https://harbor01.k8s.local"],"exec-opts": ["native.cgroupdriver=systemd"],"bip": "1.1.1.1/24"
}
将daemon.json文件分发至其他节点上
ansible all -m template -a 'src=/etc/docker/daemon.json dest=/etc/docker/'
注：
① docker的cgroup驱动程序默认设置为system，默认情况下Kubernetes cgroup为systemd，因此需要更改Docker cgroup驱动。否则会在后面的kubeadm init时报错；
② Docker从1.3.X之后，与docker registry交互默认使用的是https，http服务则需要增加insecure-registries配置。
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp: lookup localhost on [::1]:53: read udp [::1]:41922->[::1]:53: read: connection refused.
配置完成后，需要重启docker服务
ansible all -m systemd -a 'daemon_reload=yes'
ansible all -m service -a 'name=docker state=restarted'
7.2.4.启动harbor
在/opt/harbor下启动harbor
[root@harbor01 ~]# cd /opt/harbor/harbor
[root@harbor01 harbor]# ./install.sh[Step 0]: checking if docker is installed ...Note: docker version: 23.0.5[Step 1]: checking docker-compose is installed ...Note: Docker Compose version v2.17.3[Step 2]: loading Harbor images ...
17d981d1fd47: Loading layer [==================================================>] 37.78MB/37.78MB
066f24b65b06: Loading layer [==================================================>] 8.91MB/8.91MB
f5c5b2da3f78: Loading layer [==================================================>] 3.584kB/3.584kB
4cd07c2f1254: Loading layer [==================================================>] 2.56kB/2.56kB
90b02d6624a2: Loading layer [==================================================>] 87.15MB/87.15MB
b1c452c676c1: Loading layer [==================================================>] 5.632kB/5.632kB
a07864b2e153: Loading layer [==================================================>] 108kB/108kB
26a29846faca: Loading layer [==================================================>] 44.03kB/44.03kB
15c5d56364b4: Loading layer [==================================================>] 88.09MB/88.09MB
07cc9a12826b: Loading layer [==================================================>] 2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.7.2
d381f65a97a8: Loading layer [==================================================>] 8.91MB/8.91MB
a5ba716047be: Loading layer [==================================================>] 25.63MB/25.63MB
8af720b31993: Loading layer [==================================================>] 4.608kB/4.608kB
cf85d4aafef0: Loading layer [==================================================>] 26.42MB/26.42MB
Loaded image: goharbor/harbor-exporter:v2.7.2
9090e472d914: Loading layer [==================================================>] 6.295MB/6.295MB
95706aae16e4: Loading layer [==================================================>] 4.096kB/4.096kB
1e59d3cfe0b1: Loading layer [==================================================>] 3.072kB/3.072kB
c15f397332af: Loading layer [==================================================>] 190.7MB/190.7MB
625812afd6af: Loading layer [==================================================>] 13.75MB/13.75MB
bc49c81af9a3: Loading layer [==================================================>] 205.2MB/205.2MB
Loaded image: goharbor/trivy-adapter-photon:v2.7.2
d632d8a25428: Loading layer [==================================================>] 91.15MB/91.15MB
cabcd0940bdc: Loading layer [==================================================>] 6.145MB/6.145MB
44ee4d8970ae: Loading layer [==================================================>] 1.249MB/1.249MB
2f6a0dd83f2a: Loading layer [==================================================>] 1.194MB/1.194MB
Loaded image: goharbor/harbor-portal:v2.7.2
1a216f8aa02a: Loading layer [==================================================>] 123.4MB/123.4MB
d089ab0054a9: Loading layer [==================================================>] 24.63MB/24.63MB
8f24b651395d: Loading layer [==================================================>] 5.12kB/5.12kB
f2d321b72ee5: Loading layer [==================================================>] 6.144kB/6.144kB
acee91b49dbe: Loading layer [==================================================>] 3.072kB/3.072kB
73f0a48672cf: Loading layer [==================================================>] 2.048kB/2.048kB
d1137d179e82: Loading layer [==================================================>] 2.56kB/2.56kB
93f0cd1915db: Loading layer [==================================================>] 2.56kB/2.56kB
9c825e10712c: Loading layer [==================================================>] 2.56kB/2.56kB
4cb9928e2724: Loading layer [==================================================>] 9.728kB/9.728kB
Loaded image: goharbor/harbor-db:v2.7.2
bef216058819: Loading layer [==================================================>] 5.767MB/5.767MB
8f27a70b8dba: Loading layer [==================================================>] 4.096kB/4.096kB
6b2d3322e8cd: Loading layer [==================================================>] 17.42MB/17.42MB
4bdfc014a9cd: Loading layer [==================================================>] 3.072kB/3.072kB
dc54a26bde1b: Loading layer [==================================================>] 30.78MB/30.78MB
f22d45960368: Loading layer [==================================================>] 48.99MB/48.99MB
Loaded image: goharbor/harbor-registryctl:v2.7.2
dfef2543aa70: Loading layer [==================================================>] 5.762MB/5.762MB
a68585f608e3: Loading layer [==================================================>] 8.999MB/8.999MB
295d31910dd4: Loading layer [==================================================>] 14.47MB/14.47MB
efd5b1579023: Loading layer [==================================================>] 29.29MB/29.29MB
7dfd2e3fc59e: Loading layer [==================================================>] 22.02kB/22.02kB
faa41d246ac8: Loading layer [==================================================>] 14.47MB/14.47MB
Loaded image: goharbor/notary-signer-photon:v2.7.2
17b21070628b: Loading layer [==================================================>] 5.767MB/5.767MB
65500e78d7c9: Loading layer [==================================================>] 91.76MB/91.76MB
42ee762ff7a8: Loading layer [==================================================>] 3.072kB/3.072kB
26fcbd0bc385: Loading layer [==================================================>] 4.096kB/4.096kB
dce96c29de1b: Loading layer [==================================================>] 92.56MB/92.56MB
Loaded image: goharbor/chartmuseum-photon:v2.7.2
5853ff7207cd: Loading layer [==================================================>] 44.11MB/44.11MB
93590529a39f: Loading layer [==================================================>] 65.93MB/65.93MB
45c0712d114a: Loading layer [==================================================>] 26.14MB/26.14MB
27d6fd7e5535: Loading layer [==================================================>] 65.54kB/65.54kB
b0c1525b1461: Loading layer [==================================================>] 2.56kB/2.56kB
b81d770e8744: Loading layer [==================================================>] 1.536kB/1.536kB
12bbb36d555f: Loading layer [==================================================>] 12.29kB/12.29kB
7a733d55d815: Loading layer [==================================================>] 2.621MB/2.621MB
e4007be64a14: Loading layer [==================================================>] 407kB/407kB
Loaded image: goharbor/prepare:v2.7.2
5bdb50147fe3: Loading layer [==================================================>] 8.909MB/8.909MB
7c7583a1eef8: Loading layer [==================================================>] 3.584kB/3.584kB
f5483be14faa: Loading layer [==================================================>] 2.56kB/2.56kB
9b67b6258fdf: Loading layer [==================================================>] 106.5MB/106.5MB
374df1d91d24: Loading layer [==================================================>] 107.3MB/107.3MB
Loaded image: goharbor/harbor-jobservice:v2.7.2
ec911fc21120: Loading layer [==================================================>] 91.15MB/91.15MB
Loaded image: goharbor/nginx-photon:v2.7.2
631cf08f9ff0: Loading layer [==================================================>] 5.767MB/5.767MB
db4216090ca5: Loading layer [==================================================>] 4.096kB/4.096kB
1f1103a3353e: Loading layer [==================================================>] 3.072kB/3.072kB
5e28d0ce371b: Loading layer [==================================================>] 17.42MB/17.42MB
bbbdbc284648: Loading layer [==================================================>] 18.21MB/18.21MB
Loaded image: goharbor/registry-photon:v2.7.2
3dc8df9174d5: Loading layer [==================================================>] 99.07MB/99.07MB
38e93b103e4f: Loading layer [==================================================>] 3.584kB/3.584kB
74b98ab194ce: Loading layer [==================================================>] 3.072kB/3.072kB
c203b688a2be: Loading layer [==================================================>] 2.56kB/2.56kB
525a15ff6933: Loading layer [==================================================>] 3.072kB/3.072kB
ea4e850eadfa: Loading layer [==================================================>] 3.584kB/3.584kB
5c345ac6af33: Loading layer [==================================================>] 20.48kB/20.48kB
Loaded image: goharbor/harbor-log:v2.7.2
1c464948f4c8: Loading layer [==================================================>] 91.99MB/91.99MB
e23b5317ef75: Loading layer [==================================================>] 3.072kB/3.072kB
ad8e1bb2e672: Loading layer [==================================================>] 59.9kB/59.9kB
2eade6174326: Loading layer [==================================================>] 61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.7.2
dc782aa72031: Loading layer [==================================================>] 5.762MB/5.762MB
aead20724337: Loading layer [==================================================>] 8.999MB/8.999MB
22b6f665e30b: Loading layer [==================================================>] 15.88MB/15.88MB
4ded3a6c4ce0: Loading layer [==================================================>] 29.29MB/29.29MB
258a7b5fb17f: Loading layer [==================================================>] 22.02kB/22.02kB
be68b1b440c0: Loading layer [==================================================>] 15.88MB/15.88MB
Loaded image: goharbor/notary-server-photon:v2.7.2[Step 3]: preparing environment ...[Step 4]: preparing harbor configs ...
prepare base dir is set to /opt/harbor/harbor
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dirNote: stopping existing Harbor instance ...[Step 5]: starting Harbor ...
[+] Running 10/10✔ Network harbor_harbor Created 0.1s✔ Container harbor-log Started 0.7s✔ Container redis Started 1.2s✔ Container harbor-db Started 1.4s✔ Container registry Started 1.5s✔ Container registryctl Started 1.5s✔ Container harbor-portal Started 1.5s✔ Container harbor-core Started 1.8s✔ Container harbor-jobservice Started 2.4s✔ Container nginx Started 2.4s
✔ ----Harbor has been installed and started successfully.----
7.3.向docker主机上添加harbor证书
转换harbor01.k8s.local.crt为harbor01.k8s.local.cert，供Docker使用；Docker守护程序将.crt文件解释为CA证书，并将.cert文件解释为客户端证书
在harbor01上，进行证书转换
cd /opt/harbor/harbor/certs/
openssl x509 -inform PEM -in harbor01.k8s.local.crt -out harbor01.k8s.local.cert
在master01上，直接登录harbor01，会提示证书问题的报错
[root@localhost ansible]# docker login https://harbor01.k8s.local -uadmin
Password:
Error response from daemon: Get "https://harbor01.k8s.local/v2/": x509: certificate signed by unknown authority
将harbor01上的服务器证书、密钥和CA文件复制到/etc/docker/certs.d/harbor01.k8s.local/目录下
ansible all -m file -a 'path=/etc/docker/certs.d/harbor01.k8s.local state=directory'
scp harbor01:/opt/harbor/harbor/certs/harbor01.k8s.local.cert /etc/docker/certs.d/harbor01.k8s.local/
scp harbor01:/opt/harbor/harbor/certs/harbor01.k8s.local.key /etc/docker/certs.d/harbor01.k8s.local/
scp harbor01:/opt/harbor/harbor/certs/ca.crt /etc/docker/certs.d/harbor01.k8s.local/
将harbor的证书复制到master01上，然后分发至所有其他节点上
ansible all -m template -a 'src=/etc/docker/certs.d/harbor01.k8s.local/harbor01.k8s.local.cert dest=/etc/docker/certs.d/harbor01.k8s.local/'
ansible all -m template -a 'src=/etc/docker/certs.d/harbor01.k8s.local/harbor01.k8s.local.key dest=/etc/docker/certs.d/harbor01.k8s.local/'
ansible all -m template -a 'src=/etc/docker/certs.d/harbor01.k8s.local/ca.crt dest=/etc/docker/certs.d/harbor01.k8s.local/'
ansible all -m systemd -a 'daemon_reload=yes'
ansible all -m service -a 'name=docker state=restarted'
重启docker后，需要重新启动harbor
cd /opt/harbor/harbor
./install.sh
登录到私有仓库上，显示“Login Succeeded”表示成功
[root@master01 ansible]# docker login https://harbor01.k8s.local -uadmin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded
7.4.创建项目
访问https://192.168.111.20，用户名admin，密码lnyd@LNsy115
创建项目kubernetes，用于存放kubernetes集群组件的镜像
![[Mac软件]HitPaw Video Converter 功能强大的视频格式转换编辑软件激活版](https://img-blog.csdnimg.cn/direct/f9ef7fcacb2b4e57991388b81056d905.png)
