1.梳理k8s 各组件功能

首先有kube-apiserver： K8S的鉴权、准入的入口；runc运行时：CRI(container runtime interface)提供容器运行时接口；  pod是k8s运行容器的最小单元；

1.1、主体服务有，

kube-apiserver K8S的鉴权、准入的入口

kube-scheduler负责资源调度

kube-controller-manager负责接收来自apiserver的请求，控制系统的调度；pod高可用机制；node monitor period节点监视周期5s；node monitor grace节点监视宽限期40s;pod驱逐超时时间5m；

etcd是K8S默认使用的分布式 key-value 数据存储系统，用于保存K8S的所有集群数据，可用于服务发现、共享配置以及一致性保障（如数据库选主、分布式锁等）。

1.2、中间件有，

kube-proxy：管理当前节点上的iptables或者IPVS规则，网络代理

Coredns-DNS： 负责域名的解析;

a.k8s内部的service name(SVC域名解析):nginx -> svc -> pod

b.外网域名(forward)

1.3、功能服务有，

kubectl: 命令行指令管理操作的客户端工具

dashboard: 用户界面看板UI

kubelet:

1.接受apiserver的pod创建并且调用运行时(docker、containerd)

2.对pod进行周期健康状态检查

3.给apiserver反馈pod状态

4.给apiserver反馈node状态

2.基本掌握containerd的安装和使用

1.二进制安装containerd

获取开源软件包

wget https://github.com/containerd/containerd/(加上具体版本号地址)

下载回来后解压缩文件

tar -xvf ./containerd-1.6.20-linux-amd64.tar.gz

cp bin/* /usr/local/bin

2.创建service文件

vim /lib/systemd/system/containerd.service

[Unit]Description=containerd container runtimeDocumentation=https://containerd.ioAfter=network.target local-fs.target

[Service]ExecStartPre=-/usr/sbin/modprobe overlayExecStart=/usr/local/bin/containerd

Type=notifyDelegate=yesKillMode=processRestart=alwaysRestartSec=5

LimitNPROC=infinityLimitCORE=infinityLimitNOFILE=infinityTasksMax=infinityOOMScoreAdjust=-999

[Install]WantedBy=multi-user.target

3.配置config.toml文件

创建一个文件夹

mkdir /etc/containerd/

将源containerd配置默认文件重定向至/etc/coontainerd/config.toml文件

containerd config default >/etc/containerd/config.toml

vim /etc/containerd/config.toml

:set nu#设置行号找到153行，添加镜像源加速配置

[plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint=["https://l21aneue.mirror.aliyuncs.com"]

重启并设置开机自启动

systemctl restart containerd

systemctl enable containerd

`systemctl status containerd`

4.部署runc命令

获取开源软件包

wget https://github.com/opencontainers/runc/releases/(加上具体版本号地址)

下载回来后，添加可执行权限并执行安装/解压缩文件

chmod a+x /usr/local/bin/runc

runc --version

5.测试下载镜像并运行容器

ctr images pull docker.io/library/alpine:latest

ctr run -t --net-host docker.io/library/alpine:latest container1 sh

Ctrl +D 强制退出容器

containerd客户端工具安装

nerdctl-安装和使用

wget https://github.com/containerd/nerdctl/releases/download/v1.5.0/nerdctl-1.5.0-linux-amd64.tar.gz(加上具体版本号地址)

解压缩

tar -xvf nerdctl-1.5.0-linux-amd64.tar.gz

cp nertctl /usr/local/bin

nerdctl --version

再安装cni工具

实现客户端与服务端之间的网络通信

wget https://github.com/containernetworking/plugins/releases /*(加上具体版本号地址)

mkdir /opt/cni/bin/-p

tar -xvf https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz -C /opt/cni/bin

使用客户端命令nerdctl调用containerd创建容器并指定端口

运行nginx

nerdctl run -d -p 88:80 --name=nginx-web1 --restart=always nginx

nerdctl ps

nerdctl exec -it ** bash

再运行Tomcat

nerdctl run -d -p 8080:8080 --name=tomcat-web1 --restart=always tomcat:7.0.88-alpine

nerdctl ps

3.基于kubeadm和containerd部署单master k8s v1.24.x

配置kubernetes镜像源地址

apt-get update && apt-get install -y apt-transport-https

ubuntu更新源报错Certificate verification failed: The certificate is NOT trusted.

方法：把软件源里的https改成http，成功解决.

变更虚拟机的镜像源地址，编辑文件

vim /etc/apt/sources.list # https改成http

阿里云镜像加速，配置参考

https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.3e221b11GAw49R

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main EOF

apt-get update #更新源

下载kubeadm

apt-get install kubelet kubeadm kubectl

apt-get install kubelet=1.24.10-00 kubeadm=1.24.10-00 kubelet=1.24.10-00

使用kubeadm管理kubernetes的初始化安装

kubeadm config images list --kubernetes-version v1.27.4

kubeadm config images list --kubernetes-version v1.24.10

需要下载各组件镜像

编辑一个批量下载各个镜像的脚本

vim images-down.sh #/usr/local/src/#v1.27.4

vim images-down2.sh #/usr/local/src/#1.24.10-00

`#!/bin/bash/

nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.4

nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.27.4

nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.27.4

nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.4

nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9

nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.7-0

nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns/coredns:v1.10.1`

bash images_down2.sh

init初始化K8S安装

备注：提前优化Linux内核参数，需要打开ip_forward以及bridge iptables

vim /etc/sysctl.conf

:set nu

net-bridge.bridge-nf-call-iptables=1

（sysctl -a | grep bridge-nf-call-iptables）#查找出系统参数

net.ipv4.ip_forward=1

加载内核模块

modprobe br_netfilter

然后重启:从配置文件“/etc/sysctl.conf”加载内核参数设置

sysctl -p

初始化安装Kubernetes命令#version=v1.27.4/version=v1.24.10

kubeadm init --apiserver-advertise-address=192.168.235.201 --apiserver-bind-port=6443 --kubernetes-version=v1.24.10 --pod-network-cidr=10.100.0.0/16 --service-cidr=10.200.0.0/16 --service-dns-domain=cluster.local --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --ignore-preflight-errors=swap

提前检查本机交换分区

free -m

又遇到问题：

[preflight] Running pre-flight checks error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists [ERROR Port-10250]: Port 10250 is in use [preflight] If you know what you are doing, you can make a check non-fatal with--ignore-preflight-errors=...To see the stack trace of this error execute with --v=5 or higher

解决方案：

#重置kubeadm

kubeadm reset

仍未解决，再尝试搜索其他方法

由默认安装了v1.27.4最新的版本回退至kubeadm=1.24.10版本

apt-get install kubelet=1.24.10-00 kubeadm=1.24.10-00  kubelet=1.24.10-00

kubeadm config images list --kubernetes-version v1.24.10

vim images_down2.sh

#!/bin/bash/ nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.24.10 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.24.10 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.24.10 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.24.10 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.7 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.6-0 #nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns/coredns:v1.10.1

依然是在master执行完kubeadm  init后一直卡死，然后超时报异常

""""""Waiting for the kubelet to boot up the control plane as static Pods  from directory "/etc/kubernetes/manifests". This can take up to 4m0s

[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:

timed out waiting for the condition

"""

systemctl status kubelet journalctl -xeu kubeletERROR也看不出问题；超过4m0s后，报错。

通过journalctl -f -u containerd看容器引擎的日志，发现：

"

error="failed to do request:  Head "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6": dial tcp 64.233.188.82:443: connect: connection refused" host=registry.k8s.io

"

下载pause这个组件失败？

网上解释：（这里会无视kubelet的配置，如下指定了基础设置image并没用：--pod-infra-container-image=http://registry.aliyuncs.com/google_containers/pause:3.6；kubeadm启动control plane还是会使用http://k8s.gcr.io/pause:3.6）

root@ubuntu20:/usr/local/src# cd /etc/containerd/ vim config.toml sandbox_image = "registry.aliyuncs.com/k8sxio/pause:3.6"systemctl restart containerdkubeadm reset

再一次初始化安装

kubeadm init --

`

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown (id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.235.201:6443 --token v6w2dl.4s5ptclcawb07vbk

--discovery-token-ca-cert-hash sha256:6c0ae50b9fa8f29ad66acb99bf075b704caa112fae57f0d3c898040f6b63ad16

`

解决了。

参考了：

https://zhuanlan.zhihu.com/p/563177876

https://blog.csdn.net/Haskei/article/details/128474534

根据提示安装calico网络通信组件

calico-ipip.yaml

kubectl apply -f calico-ipip.yaml

查看

创建业务容器，检查kubenertes的情况

kubectl create ns myserver

kubectl apply -f nginx.yaml

curl 192.168.235.201:30004

netstat -tanlp | grep 30004

4.部署harbor并实现https(SAN签发证书)

1.安装docker（使用清华镜像源Docker CE软件仓库镜像）

https://mirrors.tuna.tsinghua.edu.cn/help/docker-ce/

2.安装配置docker-compose

chmod a+x docker-compose-linux-x86_64

cp docker-compose-linux-x86_64 /usr/bin/docker-compose

3.下载harbor安装包

tar -xvf harbor-offline-installer-v2.5.3.tgz

cp harbor.yml.tmpl harbor.yaml

cd /usr/local/harbor

vim harbor.yml

修改harbor.yaml文件配置

#根据官方的文档指示进行签发证书

mkdir /apps/harbor/certs

自签名CA机构：按照指令生成本地签发的CA证书

openssl req -x509 -new -nodes -sha512 -days 3650 \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=magedu.com" \ -key ca.key \ -out ca.crt

域名证书申请给客户端使用

openssl genrsa -out magedu.net.key 4096

openssl req -sha512 -new \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=magedu.net" \ -key magedu.net.key \ -out magedu.net.csr

准备签发CA环境

`cat > v3.ext <<-EOF

authorityKeyIdentifier=keyid,issuer

basicConstraints=CA:FALSE

keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment

extendedKeyUsage = serverAuth

subjectAltName = @alt_names

[alt_names]

DNS.1=magedu.com

DNS.2=harbor.magedu.net

DNS.3=harbor.magedu.local

EOF`

#使用自签名CA签发证书

openssl x509 -req -sha512 -days 3650 \ -extfile v3.ext \ -CA ca.crt -CAkey ca.key -CAcreateserial \ -in magedu.net.csr \ -out magedu.net.crt

#生成的证书位置/apps/harbor/certs/

vim /usr/local/harbor/harbor.yml

修改并引用证书

certificate: /apps/harbor/certs/magedu.net.crt private_key: /apps/harbor/certs/magedu.net.key

为防止系统盘数据丢失，创建一块新的磁盘逻辑卷

mkfs.xfs /dev/sdb

在新磁盘创建一块新目录：

mkdir /data/harbor -p

修改harbor.yaml文件参数

vim /usr/local/harbor/harbor.yml

#The default data volume

data_volume: /data/harbor

执行安装脚本

./install.sh --with-trivy --with-chartmuseum

配置本机域名解析与IP地址对应

编辑C:\Windows\System32\drivers\etc\hosts文件

先在对端客户端创建如下目录

`mkdir /etc/docker/certs.d/harbor.magedu.net -p`

将公钥拷贝至对端，同步harbor crt证书

`scp magedu.net.crt 192.168.0.110:/etc/docker/certs.d/harbor.magedu.net`

再其他客户端验证登录harbor

测试push镜像到harbor

5.部署haproxy和keepalived高可用负载均衡

apt update

https更改成http

vim /etc/apt/sources.list

apt install keepalived haproxy

镜像源的问题

apt install keepalived haproxy

寻找到keepalived模版文件

find / -name "keepalived.*"

cp /usr/share/doc/keepalived/samples/keepalived.conf.vrrp /etc/keepalived/keepalived.conf/

LB两台机器修改配置指定负载均衡VIP承载点

vim /etc/keepalived/keepalived.conf

virtual_ipaddress{

192.168.0.188 dev eth0 label eth0:1

192.168.0.189 dev eth0 label eth0:2

}

systemctl restart keepalived.service

systemctl enable keepalived.service

ifconfig#查看虚拟网络是否生成

LB另外一台备份节点同样keepalived配置

scp /etc/keepalived/keepalived.conf 192.168.0.109:/etc/keepalived/keepalived.conf

LB机器再配置haproxy

vim /etc/haproxy/haproxy.cfg

listen harbor-80

listen harbor-80

bind 192.168.0.188:80

mode tcp

server 192.168.0.104 192.168.0.1:80 check inter 3s fall 3 rise 3

listen harbor-443

bind 192.168.0.188:443

mode tcp

server server1 192.168.0.104:443  check inter 3s fall 3 rise 3

systemctl restart haproxy

关于haproxy报错问题查看日志

查看错误日志

haproxy -f /etc/haproxy/haproxy.cfg

systemctl restart keepalived

查看端口状态

ss -tnl

服务域名DNS映射配置：

vim /etc/hosts/