Tips
- 整个环境是有两台虚拟机作为节点构建k8s集群，第三台虚拟机作为harbor仓库组建；具体的kubernetes集群部署之前已经完成使用kubeadmin配合calico快速搭建kubernetes集群,此篇文章只介绍如何搭建harbor，并在kubernetes中使用。
- Harbor服务器的主机名不要设置成harbor、docker.io这样的主机名，不然可能会在后期push的有问题。
1. 整理部署架构
| 角色 | IP | 服务 | VCPU | 内存 |
|---|---|---|---|---|
| master | 192.168.199.30 | etcd、kube-apiserver、kube-scheduler、kube-controller-manager、kube-proxy、calico-node | 2 | 2 |
| node1 | 192.168.199.31 | kube-proxy、calico-node | 2 | 2 |
| harbor.registry | 192.168.199.32 | docker-compose、harbor | 2 | 4 |
2. 环境准备
(1) 在三台主机上执行以下操作
- 编辑
/etc/hosts文件，将主机映射关系写入：
192.168.199.31 node1
192.168.199.30 master
192.168.199.30 etcd
192.168.199.30 apiserver.k8s
192.168.199.32 harbor.registry
- 关闭防火墙
[root@master yum.repos.d]# systemctl stop firewalld
[root@master yum.repos.d]# systemctl disable firewalld
- 关闭swap
swap，这个当内存不足时，linux会自动使用swap，将部分内存数据存放到磁盘中，这个这样会使性能下降，为了性能考虑推荐关掉
[root@master yum.repos.d]# swapoff -a
- 关闭selinux
vi /etc/selinux/config，关闭SELINUX
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
- 配置yum源，此处配置的是阿里的Centos 7的yum源和kubernetes源
Tips：可以在配置之前将/etc/yum.repos.d/ 下的文件都备份到bak目录下
[root@master ~]# cd /etc/yum.repos.d/ && curl -O http://mirrors.aliyun.com/repo/Centos-7.repo
[root@master yum.repos.d]# vi kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg"
[root@master yum.repos.d]yum clean all
[root@master yum.repos.d]#yum makecache
-
reboot重启操作系统，使hostname、SELINUX配置生效
3. 安装docker和docker-compose
3.1 安装docker
将需要执行的步骤都写入了脚本，直接运行脚本内容即可。 脚本内容如下：
#!/bin/bash
# 先定义一下国内的镜像加速源
export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
# 安装 docker
# 参考文档如下
# https://docs.docker.com/install/linux/docker-ce/centos/
# https://docs.docker.com/install/linux/linux-postinstall/
# 卸载旧版本
yum remove -y docker \
docker-client \
docker-client-latest \
docker-ce-cli \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
# 设置 yum repository
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安装并启动 docker
yum install -y docker-ce-19.03.11 docker-ce-cli-19.03.11 containerd.io-1.2.13
mkdir /etc/docker || true
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors": ["${REGISTRY_MIRROR}"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart Docker
systemctl daemon-reload
systemctl enable docker
systemctl restart docker
docker version
3.2 安装docker-compose
下载软件：docker-compose软件是一个可执行的二进制文件，在要安装Harbor私有镜像仓库的虚拟机中执行以下命令即可从官网中下载docker-compose软件，下载速度很比较慢，如果失败，则需要多试几次。
curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
设置权限：docker-compose可执行文件下载到了/usr/local/bin目录下，但是并不具有可执行的权限，所以要设置可执行的权限。在要安装Harbor私有镜像仓库的虚拟机中执行以下命令即可设置可执行权限。
sudo chmod +x /usr/local/bin/docker-compose
查看版本：给docker-compose可执行文件设置了可执行权限之后，即可执行以下命令以查看安装的docker-compose的版本。
docker-compose --version
4. 安装harbor
个人建议直接到官方下载地址，下载到本地之后再上传到服务器上， 比较快。
此次下载的版本为v1.10.6的版本。Harbor的安装分为离线和在线两种安装方式，此次选择离线的部署方式，文件名为harbor-offline-installer-v1.10.6.tgz。上传到服务器之后，执行以下操作：
- 解压文件夹并进入
[root@harbor ~]# tar -zvxf harbor-offline-installer-v1.10.6.tgz && cd ./harbor
进入文件夹之后，修改配置文件harbor.yml。主要注意以下内容即可：
harbor.yml配置内容
- 执行
[root@harbor harbor]# bash install.sh直接安装即可，运行结束执行docker-compose ps查看部署的容器及结果
[root@harbor harbor]# bash install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 19.03.11
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 1.18.0
[Step 2]: loading Harbor images ...
517216c3ed59: Loading layer [==================================================>] 34.5MB/34.5MB
b9984883b90b: Loading layer [==================================================>] 331.5MB/331.5MB
979e8823fd48: Loading layer [==================================================>] 135.2kB/135.2kB
Loaded image: goharbor/harbor-migrator:v1.10.6
9af36121a2d3: Loading layer [==================================================>] 12.29MB/12.29MB
7469a23fd831: Loading layer [==================================================>] 42.51MB/42.51MB
78b8a6419717: Loading layer [==================================================>] 5.632kB/5.632kB
d82d965b4ef8: Loading layer [==================================================>] 40.45kB/40.45kB
df344cc92b3d: Loading layer [==================================================>] 42.51MB/42.51MB
986058fa362a: Loading layer [==================================================>] 2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v1.10.6
e57060edfa4f: Loading layer [==================================================>] 63.67MB/63.67MB
6b4f0118d2e7: Loading layer [==================================================>] 75.99MB/75.99MB
29a495a70a49: Loading layer [==================================================>] 5.632kB/5.632kB
79b703e7a4e5: Loading layer [==================================================>] 2.56kB/2.56kB
e867025af608: Loading layer [==================================================>] 2.56kB/2.56kB
808ee6ef9f3e: Loading layer [==================================================>] 2.56kB/2.56kB
9dd136dbdbb8: Loading layer [==================================================>] 2.56kB/2.56kB
07a93256363b: Loading layer [==================================================>] 10.75kB/10.75kB
Loaded image: goharbor/harbor-db:v1.10.6
d88d1d4c1e75: Loading layer [==================================================>] 8.515MB/8.515MB
4af0316b2679: Loading layer [==================================================>] 3.584kB/3.584kB
7ffcfabe0b2f: Loading layer [==================================================>] 20.51MB/20.51MB
b2c9e51892ed: Loading layer [==================================================>] 3.072kB/3.072kB
69f04e0492e5: Loading layer [==================================================>] 8.662MB/8.662MB
0befdda31216: Loading layer [==================================================>] 29.99MB/29.99MB
Loaded image: goharbor/harbor-registryctl:v1.10.6
0285f3f1d24f: Loading layer [==================================================>] 10.36MB/10.36MB
Loaded image: goharbor/nginx-photon:v1.10.6
d603a9558e8c: Loading layer [==================================================>] 115.4MB/115.4MB
b23664baab69: Loading layer [==================================================>] 12.15MB/12.15MB
e785164697c1: Loading layer [==================================================>] 3.072kB/3.072kB
ddc72a1554c7: Loading layer [==================================================>] 49.15kB/49.15kB
1bbe40c913b2: Loading layer [==================================================>] 3.584kB/3.584kB
8292a27c053f: Loading layer [==================================================>] 13.03MB/13.03MB
Loaded image: goharbor/clair-photon:v1.10.6
55cfd42af466: Loading layer [==================================================>] 8.515MB/8.515MB
2ef32a6672e4: Loading layer [==================================================>] 9.71MB/9.71MB
516c0b3f7b7d: Loading layer [==================================================>] 9.71MB/9.71MB
Loaded image: goharbor/clair-adapter-photon:v1.10.6
24717191d0eb: Loading layer [==================================================>] 10.36MB/10.36MB
e724890b1092: Loading layer [==================================================>] 7.697MB/7.697MB
a6186df96eea: Loading layer [==================================================>] 223.2kB/223.2kB
9dabcd61b9a2: Loading layer [==================================================>] 195.1kB/195.1kB
70a9eabcf0cf: Loading layer [==================================================>] 15.36kB/15.36kB
6327d3371dbe: Loading layer [==================================================>] 3.584kB/3.584kB
Loaded image: goharbor/harbor-portal:v1.10.6
cbb4f0baa930: Loading layer [==================================================>] 74.79MB/74.79MB
9cff4188aa57: Loading layer [==================================================>] 3.584kB/3.584kB
c2b996a82566: Loading layer [==================================================>] 3.072kB/3.072kB
9829c3787744: Loading layer [==================================================>] 2.56kB/2.56kB
67ddfe98d90a: Loading layer [==================================================>] 3.072kB/3.072kB
ff8b78cb63e7: Loading layer [==================================================>] 3.584kB/3.584kB
854dabac3758: Loading layer [==================================================>] 12.29kB/12.29kB
783d10970f24: Loading layer [==================================================>] 5.632kB/5.632kB
Loaded image: goharbor/harbor-log:v1.10.6
13fd12d41fea: Loading layer [==================================================>] 8.515MB/8.515MB
7f93018d3462: Loading layer [==================================================>] 3.584kB/3.584kB
59f5dbf0affb: Loading layer [==================================================>] 3.072kB/3.072kB
9311a8fa1514: Loading layer [==================================================>] 20.51MB/20.51MB
614eb395ab40: Loading layer [==================================================>] 21.33MB/21.33MB
Loaded image: goharbor/registry-photon:v1.10.6
57010d02c56f: Loading layer [==================================================>] 8.509MB/8.509MB
3742ae96b00e: Loading layer [==================================================>] 6.239MB/6.239MB
216c0af522f9: Loading layer [==================================================>] 14.43MB/14.43MB
09f6f8766ff3: Loading layer [==================================================>] 27.97MB/27.97MB
bffe3cc7a54c: Loading layer [==================================================>] 22.02kB/22.02kB
a60206e2897f: Loading layer [==================================================>] 48.64MB/48.64MB
Loaded image: goharbor/notary-signer-photon:v1.10.6
bbb3a4666fca: Loading layer [==================================================>] 12.29MB/12.29MB
3e7661db06ab: Loading layer [==================================================>] 49.38MB/49.38MB
Loaded image: goharbor/harbor-jobservice:v1.10.6
762f7fd02c27: Loading layer [==================================================>] 98.91MB/98.91MB
738675c5bb2a: Loading layer [==================================================>] 3.072kB/3.072kB
957f6ceb4893: Loading layer [==================================================>] 59.9kB/59.9kB
2ce8a82769c7: Loading layer [==================================================>] 61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v1.10.6
842f97800f3d: Loading layer [==================================================>] 94.53MB/94.53MB
e0d5dfe658d5: Loading layer [==================================================>] 49.92MB/49.92MB
f2d4a8662d68: Loading layer [==================================================>] 2.56kB/2.56kB
5f622d209603: Loading layer [==================================================>] 1.536kB/1.536kB
01094f024495: Loading layer [==================================================>] 157.2kB/157.2kB
f1b9e5269785: Loading layer [==================================================>] 3.017MB/3.017MB
Loaded image: goharbor/prepare:v1.10.6
1f0f4177bbb7: Loading layer [==================================================>] 15.84MB/15.84MB
baf2ca0b445a: Loading layer [==================================================>] 27.97MB/27.97MB
515bf108ac7d: Loading layer [==================================================>] 22.02kB/22.02kB
c2146de85e39: Loading layer [==================================================>] 50.05MB/50.05MB
Loaded image: goharbor/notary-server-photon:v1.10.6
fe546267e7cd: Loading layer [==================================================>] 8.514MB/8.514MB
7de41eeff949: Loading layer [==================================================>] 67.49MB/67.49MB
3204c9160f30: Loading layer [==================================================>] 3.072kB/3.072kB
cb68e6e1c61e: Loading layer [==================================================>] 3.584kB/3.584kB
3b277bfd90f8: Loading layer [==================================================>] 68.32MB/68.32MB
Loaded image: goharbor/chartmuseum-photon:v1.10.6
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /secret/keys/secretkey
Creating harbor-log ... done
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
Creating registry ... done
Creating harbor-core ... done
Creating network "harbor_harbor" with the default driver
Creating nginx ... done
Creating redis ...
Creating registryctl ...
Creating harbor-portal ...
Creating harbor-db ...
Creating registry ...
Creating harbor-core ...
Creating harbor-jobservice ...
Creating nginx ...
✔ ----Harbor has been installed and started successfully.----
[root@harbor harbor]# docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------
harbor-core /harbor/harbor_core Up
harbor-db /docker-entrypoint.sh Up 5432/tcp
harbor-jobservice /harbor/harbor_jobservice ... Up
harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp
harbor-portal nginx -g daemon off; Up 8080/tcp
nginx nginx -g daemon off; Up 0.0.0.0:80->8080/tcp
redis redis-server /etc/redis.conf Up 6379/tcp
registry /home/harbor/entrypoint.sh Up 5000/tcp
registryctl /home/harbor/start.sh Up
- 修改两个节点的daemon.json配置，添加私有镜像仓库地址，并重启docker服务。Tips：所有节点都要配置！
[root@harbor docker]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.cn-hangzhou.aliyuncs.com"],
"insecure-registries": ["harbor.registry"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
[root@harbor docker]# systemctl daemon-reload
[root@harbor docker]# systemctl restart docker
如果在重启docker的时候遇到报错，请仔细检查daemon.json的配置，确认无误之后reload配置文件再次重启docker尝试。、
- 每个节点使用
docker login harbor.registry命令登陆harbor，之后就可以进行pull、tag和push的操作了。
[root@harbor harbor]# docker login harbor.registry
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@harbor harbor]#
Tips：如果在login的时候遇到报错
Error response from daemon: Get harbor.registry/v2/: dial tcp 192.168.199.32:80: connect: connection refused，可能存在以下两种情况：
（1）检查/etc/docker/daemon.json中配置的"insecure-registries"参数是否是正确的harbor主机名或IP地址。
（2）harbor服务异常，切回harbor目录并通过docker-compose ps查看各harbor容器状态，如异常情况执行docker-compose restart即可。注：操作必须要在harbor目录才行。
（3）查看harbar的harbor.yml中的hostname配置的对不对
harbor异常
docker-compose restart
在master节点上的login
至此，所有的harbor搭建阶段已经完成，最后看一下harbor的页面访问效果。
harbor登陆界面
登陆首页
5. 测试
使用之前部署的kubernetes集群的master作为测试节点，向harbor推送镜像，并使用harbor节点pull到本地。
-
在harbor页面新建项目
新建项目 在master节点登陆harbor
[root@master ~]# docker login harbor.registry
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@master ~]#
- 使用
docker image ls查看自身镜像，此次挑选calico/node镜像作为试验。
docker image ls - 使用
docker tag打完镜像标签， 并使用docker push推送到harbor服务器上
[root@master ~]# docker tag calico/node:v3.13.1 harbor.registry/calico/node:v3.13.1
[root@master ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/k8sxio/kube-proxy v1.19.2 d373dd5a8593 2 months ago 118MB
registry.aliyuncs.com/k8sxio/kube-apiserver v1.19.2 607331163122 2 months ago 119MB
registry.aliyuncs.com/k8sxio/kube-controller-manager v1.19.2 8603821e1a7a 2 months ago 111MB
registry.aliyuncs.com/k8sxio/kube-scheduler v1.19.2 2f32d66b884f 2 months ago 45.7MB
registry.aliyuncs.com/k8sxio/etcd 3.4.13-0 0369cf4303ff 2 months ago 253MB
registry.aliyuncs.com/k8sxio/coredns 1.7.0 bfe3a36ebd25 5 months ago 45.2MB
calico/node v3.13.1 2e5029b93d4a 8 months ago 260MB
harbor/calico/node v3.13.1 2e5029b93d4a 8 months ago 260MB
harbor.registry/calico/node v3.13.1 2e5029b93d4a 8 months ago 260MB
calico/pod2daemon-flexvol v3.13.1 e8c600448aae 8 months ago 111MB
calico/cni v3.13.1 6912ec2cfae6 8 months ago 207MB
calico/kube-controllers v3.13.1 3971f13f2c6c 8 months ago 56.6MB
registry.aliyuncs.com/k8sxio/pause 3.2 80d28bedfe5d 9 months ago 683kB
[root@master ~]# docker push harbor.registry/calico/node:v3.13.1
The push refers to repository [harbor.registry/calico/node]
12fd6ed06c05: Pushed
fbde6cdd36b6: Pushed
cd59206a737d: Pushed
2e8e06e347e4: Pushed
0c9d8b0f21a3: Pushed
6861e9e920e0: Pushed
c5e173042e71: Pushed
8c2f5310e5b2: Pushed
783d3f7a3bc6: Pushed
676f013efbd6: Pushed
224fce06dca9: Pushed
27cd2023d60a: Pushed
4b52dfd1f9d9: Pushed
v3.13.1: digest: sha256:cbd5bf2ed8cb93595d358b6f23d3937da1620e8a600c93efbb29f689790b882b size: 3042
-
查看harbor的calico项目下的镜像
web页面的calico项目
Tips:
打标签：docekr tag 镜像名称:标签 你的IP:端口/harbor项目名称/镜像名称:标签 ，如：docker tag calico/node:v3.13.1 harbor.registry/calico/node:v3.13.1
推送：docker push 你的IP:端口/harbor项目名称/镜像名称:标签，如：docker push harbor.registry/calico/node:v3.13.1
删除镜像：docker rmi image_name:tag，如：docker rmi harbor/calico/node:v3.13.1
- 在harbar.registry的节点上作为客户端，pull下刚才上传的镜像
[root@harbor harbor]# docker pull harbor.registry/calico/node:v3.13.1
v3.13.1: Pulling from calico/node
7fe5fcc0340e: Pull complete
086296bbdfc7: Pull complete
352a99f50574: Pull complete
8c35de76d622: Pull complete
90d1c845595c: Pull complete
a0138100ff24: Pull complete
d424c9255a01: Pull complete
9251f9ad4cf0: Pull complete
1895fa439d94: Pull complete
64b522df02c8: Pull complete
058b195a5d5d: Pull complete
996a1138b8e4: Pull complete
d2c562af5e2a: Pull complete
Digest: sha256:cbd5bf2ed8cb93595d358b6f23d3937da1620e8a600c93efbb29f689790b882b
Status: Downloaded newer image for harbor.registry/calico/node:v3.13.1
harbor.registry/calico/node:v3.13.1
[root@harbor harbor]#
