需求背景
由于项目涉及到较大量的docker镜像，对镜像仓库存储要求较高，需要自建Harbor便于满足其后可能的定制化需求。
部署过程，主要参考了llsydn大佬的博文，并在此基础上补充了：1.、博文中缺省的前置条件；2、在openEuler-22系统下部署Harbor遇到的问题和解决方案。
环境准备
step1. Harbor离线安装包
以 Harbor v2.9.0 为例，在harbor官方网站下载对应版本的离线安装包，下载地址为：harbor-offline-installer-v2.9.0.tgz
由于Harbor 官方仅提供了 x86_64 架构的离线安装包，需要再此基础上手动替换压缩包中的docker 镜像文件为 arm64 架构，也需修改安装脚本文件（install.sh）。
手动打包所需的所有arm64架构的docker镜像
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/harbor-registryctl:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/nginx-photon:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/registry-photon:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/prepare:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/harbor-portal:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/harbor-log:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/harbor-exporter:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/redis-photon:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/trivy-adapter-photon:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/harbor-core:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/harbor-db:v2.9.0
docker pull --platform=linux/arm64 ghcr.io/octohelm/harbor/harbor-jobservice:v2.9.0
docker tag ghcr.io/octohelm/harbor/harbor-registryctl:v2.9.0 goharbor/harbor-registryctl:v2.9.0
docker tag ghcr.io/octohelm/harbor/nginx-photon:v2.9.0 goharbor/nginx-photon:v2.9.0
docker tag ghcr.io/octohelm/harbor/registry-photon:v2.9.0 goharbor/registry-photon:v2.9.0
docker tag ghcr.io/octohelm/harbor/prepare:v2.9.0 goharbor/prepare:v2.9.0
docker tag ghcr.io/octohelm/harbor/harbor-portal:v2.9.0 goharbor/harbor-portal:v2.9.0
docker tag ghcr.io/octohelm/harbor/harbor-log:v2.9.0 goharbor/harbor-log:v2.9.0
docker tag ghcr.io/octohelm/harbor/harbor-exporter:v2.9.0 goharbor/harbor-exporter:v2.9.0
docker tag ghcr.io/octohelm/harbor/redis-photon:v2.9.0 goharbor/redis-photon:v2.9.0
docker tag ghcr.io/octohelm/harbor/trivy-adapter-photon:v2.9.0 goharbor/trivy-adapter-photon:v2.9.0
docker tag ghcr.io/octohelm/harbor/harbor-core:v2.9.0 goharbor/harbor-core:v2.9.0
docker tag ghcr.io/octohelm/harbor/harbor-db:v2.9.0 goharbor/harbor-db:v2.9.0
docker tag ghcr.io/octohelm/harbor/harbor-jobservice:v2.9.0 goharbor/harbor-jobservice:v2.9.0
docker save -o goharbor-harbor-registryctl-v2.9.0.tar goharbor/harbor-registryctl:v2.9.0
docker save -o goharbor-nginx-photon-v2.9.0.tar goharbor/nginx-photon:v2.9.0
docker save -o goharbor-registry-photon-v2.9.0.tar goharbor/registry-photon:v2.9.0
docker save -o goharbor-prepare-v2.9.0.tar goharbor/prepare:v2.9.0
docker save -o goharbor-harbor-portal-v2.9.0.tar goharbor/harbor-portal:v2.9.0
docker save -o goharbor-harbor-log-v2.9.0.tar goharbor/harbor-log:v2.9.0
docker save -o goharbor-harbor-exporter-v2.9.0.tar goharbor/harbor-exporter:v2.9.0
docker save -o goharbor-redis-photon-v2.9.0.tar goharbor/redis-photon:v2.9.0
docker save -o goharbor-trivy-adapter-photon-v2.9.0.tar goharbor/trivy-adapter-photon:v2.9.0
docker save -o goharbor-harbor-core-v2.9.0.tar goharbor/harbor-core:v2.9.0
docker save -o goharbor-harbor-db-v2.9.0.tar goharbor/harbor-db:v2.9.0
docker save -o goharbor-harbor-jobservice-v2.9.0.tar goharbor/harbor-jobservice:v2.9.0
将本地的docker镜像重新打包
tar -czvf harbor.v2.9.0-arm64.tar.gz *.tar
替换掉原离线安装包中的压缩文件
image.png
修改 install.sh
image.png
if [ -f harbor*.tar.gz ]
then
h2 "[Step $item]: loading Harbor images ..."; let item+=1
# docker load -i ./harbor*.tar.gz
rm -rf images && mkdir images
tar -zxvf ./harbor*.tar.gz -C ./images
ls images/*.tar | xargs -n 1 docker load -i
docker images|grep goharbor
fi
echo ""
完成以上步骤后，重新打包
tar -czvf harbor-offline-installer-v2.9.0-arm64.tar.gz harbor
step2. VM环境准备
由于Harbor安装脚本会检测环境是否具有：1. golang；2.docker；3.docker-compose
离线安装 golang
官网下载.tar.gz包
golang官网地址：golang.org/
传到VM后, 解压到 /usr/local
tar -zxf go1.17.2.linux-amd64.tar.gz -C /usr/local
将go添加到环境变量
vim /etc/profile
修改信息如下:
# go 环境变量
export GO111MODULE=on
export GOROOT=/usr/local/go
export GOPATH=/home/gopath
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
应用
source /etc/profile
验证Golang
[root@ecs-k8s-hub ~]# go version
go version go1.18.10 linux/arm64
[root@ecs-k8s-hub ~]#
离线安装 docker
下载官方docker arm64架构离线安装包
Index of linux/centos/8/aarch64/stable/Packages/ (docker.com)
具体流程可参考CentOS7离线部署docker - 掘金 (juejin.cn)
离线安装 docker-compose
从 github 下载docker-compose，并上传到VM
移动下载好的二进制文件并赋予执行权限
mv -f ./docker-compose-linux-x86_64 /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
创建软连接
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
验证Docker-Compose
[root@ecs-k8s-hub ~]# docker-compose -v
Docker Compose version v2.21.0
[root@ecs-k8s-hub ~]#
安装Harbor
解压刚刚打包的压缩包
tar -zxvf harbor-offline-installer-v2.9.0-arm64.tar.gz
修改 harbor.yml 配置文件
可参考官方文档说明修改，文档链接为：Harbor docs | Configure the Harbor YML File (goharbor.io)
执行安装脚本
./install.sh
安装日志如下：
[root@ecs-k8s-hub harbor]# sudo bash install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 19.03.14
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 2.21.0
[Step 2]: loading Harbor images ...
goharbor-harbor-core-v2.9.0.tar
goharbor-harbor-db-v2.9.0.tar
goharbor-harbor-exporter-v2.9.0.tar
goharbor-harbor-jobservice-v2.9.0.tar
goharbor-harbor-log-v2.9.0.tar
goharbor-harbor-portal-v2.9.0.tar
goharbor-harbor-registryctl-v2.9.0.tar
goharbor-nginx-photon-v2.9.0.tar
goharbor-prepare-v2.9.0.tar
goharbor-redis-photon-v2.9.0.tar
goharbor-registry-photon-v2.9.0.tar
goharbor-trivy-adapter-photon-v2.9.0.tar
Loaded image: goharbor/harbor-core:v2.9.0
Loaded image: goharbor/harbor-db:v2.9.0
Loaded image: goharbor/harbor-exporter:v2.9.0
Loaded image: goharbor/harbor-jobservice:v2.9.0
Loaded image: goharbor/harbor-log:v2.9.0
Loaded image: goharbor/harbor-portal:v2.9.0
Loaded image: goharbor/harbor-registryctl:v2.9.0
Loaded image: goharbor/nginx-photon:v2.9.0
Loaded image: goharbor/prepare:v2.9.0
Loaded image: goharbor/redis-photon:v2.9.0
Loaded image: goharbor/registry-photon:v2.9.0
Loaded image: goharbor/trivy-adapter-photon:v2.9.0
goharbor/harbor-portal v2.9.0 29cae4d182e5 6 months ago 192MB
goharbor/harbor-core v2.9.0 2bcbe67b6004 6 months ago 170MB
goharbor/prepare v2.9.0 afc41f59990f 6 months ago 225MB
goharbor/harbor-registryctl v2.9.0 28232e6433cc 6 months ago 149MB
goharbor/harbor-db v2.9.0 a2507b82333a 6 months ago 326MB
goharbor/registry-photon v2.9.0 cd79c62a46c7 6 months ago 89MB
goharbor/harbor-exporter v2.9.0 91ab03541258 6 months ago 111MB
goharbor/harbor-jobservice v2.9.0 e1195a9c6f4d 6 months ago 143MB
goharbor/trivy-adapter-photon v2.9.0 2cd27ed5ed06 6 months ago 447MB
goharbor/nginx-photon v2.9.0 90b63560ebfc 6 months ago 183MB
goharbor/harbor-log v2.9.0 28e054c47632 6 months ago 194MB
goharbor/redis-photon v2.9.0 21d412995e33 6 months ago 202MB
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
Note: stopping existing Harbor instance ...
[Step 5]: starting Harbor ...
[+] Running 10/10
✔ Network harbor_harbor Created 0.3s
✔ Container harbor-log Started 0.2s
✔ Container registryctl Started 0.5s
✔ Container harbor-portal Started 0.6s
✔ Container redis Started 0.5s
✔ Container registry Started 0.5s
✔ Container harbor-db Started 0.5s
✔ Container harbor-core Started 0.2s
✔ Container harbor-jobservice Started 0.2s
✔ Container nginx Started 0.4s
✔ ----Harbor has been installed and started successfully.----
利用docker及docker-compose查看docker容器状态
[root@ecs-k8s-hub harbor]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c83f985b5845 goharbor/nginx-photon:v2.9.0 "nginx -g 'daemon of…" 7 hours ago Up 7 hours (healthy) 0.0.0.0:80->8080/tcp nginx
261577ec5c82 goharbor/harbor-jobservice:v2.9.0 "/harbor/entrypoint.…" 7 hours ago Up 7 hours (healthy) harbor-jobservice
486830b2d36d goharbor/harbor-core:v2.9.0 "/harbor/entrypoint.…" 7 hours ago Up 7 hours (healthy) harbor-core
94ff7bffe35d goharbor/harbor-db:v2.9.0 "/docker-entrypoint.…" 7 hours ago Up 7 hours (healthy) harbor-db
d06e4b53897a goharbor/harbor-portal:v2.9.0 "nginx -g 'daemon of…" 7 hours ago Up 7 hours (healthy) harbor-portal
2957fead290a goharbor/registry-photon:v2.9.0 "/home/harbor/entryp…" 7 hours ago Up 7 hours (healthy) registry
951218e63e06 goharbor/harbor-registryctl:v2.9.0 "/home/harbor/start.…" 7 hours ago Up 7 hours (healthy) registryctl
2a2df4502798 goharbor/redis-photon:v2.9.0 "redis-server /etc/r…" 7 hours ago Up 7 hours (healthy) redis
4ede49cff5d9 goharbor/harbor-log:v2.9.0 "/bin/sh -c /usr/loc…" 7 hours ago Up 7 hours (healthy) 127.0.0.1:1514->10514/tcp harbor-log
[root@ecs-k8s-hub harbor]# docker-compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
harbor-core goharbor/harbor-core:v2.9.0 "/harbor/entrypoint.sh" core 7 hours ago Up 7 hours (healthy)
harbor-db goharbor/harbor-db:v2.9.0 "/docker-entrypoint.sh 13 14" postgresql 7 hours ago Up 7 hours (healthy)
harbor-jobservice goharbor/harbor-jobservice:v2.9.0 "/harbor/entrypoint.sh" jobservice 7 hours ago Up 7 hours (healthy)
harbor-log goharbor/harbor-log:v2.9.0 "/bin/sh -c /usr/local/bin/start.sh" log 7 hours ago Up 7 hours (healthy) 127.0.0.1:1514->10514/tcp
harbor-portal goharbor/harbor-portal:v2.9.0 "nginx -g 'daemon off;'" portal 7 hours ago Up 7 hours (healthy)
nginx goharbor/nginx-photon:v2.9.0 "nginx -g 'daemon off;'" proxy 7 hours ago Up 7 hours (healthy) 0.0.0.0:80->8080/tcp
redis goharbor/redis-photon:v2.9.0 "redis-server /etc/redis.conf" redis 7 hours ago Up 7 hours (healthy)
registry goharbor/registry-photon:v2.9.0 "/home/harbor/entrypoint.sh" registry 7 hours ago Up 7 hours (healthy)
registryctl goharbor/harbor-registryctl:v2.9.0 "/home/harbor/start.sh" registryctl 7 hours ago Up 7 hours (healthy)
查看harbor日志 /var/log/harbor
[root@ecs-k8s-hub harbor]# pwd
/var/log/harbor
[root@ecs-k8s-hub harbor]# ll
total 22M
-rw-r--r-- 1 10000 10000 297 Mar 20 17:39 '#015.log'
-rw-r--r-- 1 10000 10000 387 Mar 20 17:39 Accept.log
-rw-r--r-- 1 10000 10000 7.1M Mar 21 16:48 core.log
-rw-r--r-- 1 10000 10000 474 Mar 20 17:39 Host.log
-rw-r--r-- 1 10000 10000 12M Mar 21 16:48 jobservice.log
-rw-r--r-- 1 10000 10000 926K Mar 21 16:50 portal.log
-rw-r--r-- 1 10000 10000 42K Mar 21 09:43 postgresql.log
-rw-r--r-- 1 10000 10000 807K Mar 21 16:50 proxy.log
-rw-r--r-- 1 10000 10000 84K Mar 21 16:49 redis.log
-rw-r--r-- 1 10000 10000 755K Mar 21 16:50 registryctl.log
-rw-r--r-- 1 10000 10000 866K Mar 21 16:50 registry.log
-rw-r--r-- 1 10000 10000 495 Mar 20 17:39 User-Agent.log
验证Harbor
image.png
其他问题:docker 启动容器报 iptables: No chain/target/match
最后采用重启docker服务后，重建docker容器解决。
Reference
- https://github.com/docker/compose?tab=readme-ov-file
- https://go.dev/
- https://github.com/goharbor/harbor
- https://goharbor.io/docs/2.0.0/install-config/download-installer/
- https://goharbor.io/docs/2.0.0/install-config/configure-https/
- harbor离线安装2.9.0-arm64架构服务制作和升级部署 - 掘金 (juejin.cn)
- v1.5.0 Error response from daemon: configured logging driver does not support reading · Issue #4879 · goharbor/harbor · GitHub
- 安装harbor作为docker镜像仓库 - 掘金 (juejin.cn)
- Centos离线安装Docker、Docker-compose - 掘金 (juejin.cn)
- 解决 docker 启动容器报 iptables: No chain/target/match by that name-CSDN博客
