keberneeters的部署安装
1.K8S的基本说明
本文档要实现的内容，以三个服务器完成K8S的部署，master节点可以管理控制node节点，node节点中的容器可以互相访问和调用。
K8S的组件分为kubernetes-master和kubernetes-node两个部分和etcd。
mster节点的组件
- etcd：负责存储通过apiserver的所有数据，并连接flannel网络存储所有node节点数据。
- apiserver：担任K8S集群中交通枢纽的功能，所有命令和信息都是通过apiserver进行传递。
- scheduler：接受apiserver的指令，检索符合的node节点，执行pod调度
- controller：集群管理中心，负责node，pod，namspace，service等资源的故障发现，自愈，扩容缩容。
- flannel：创建一个覆盖网络，让所有容器直接能够互通，且对外访问。
node节点的组件
- kubelet：负责管理调度到该节点的pod，负责整个pod的生命周期。
- proxy： daemon守护进程，通过连接到etcd监控pod的变化，并支持service来完成pod的负载均衡。
2.基础环境的部署
2.1节点规划
172.16.24.41 docker41
172.16.24.42 docker42
172.16.24.43 docker43
2.2基础环境说明
2.2.1关闭Networkmanager（防止network和networkmanager产生冲突）
systemctl stop NetworkManager
systemctl disable NetworkManager
2.2.2 系统信息说明
1.内核版本
[root@docker41 ~]# uname -a
Linux docker41 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
2.3修改主机名
2.3.1修改master主机名
`hostnamectl set-hostame docker41
hostname docker41
`
2.3.2修改node主机名
hostnamectl set-hostname k8s-node1
hostname k8s-node1
hostnamectl set-hostname k8s-node2
hostname k8s-node2
2.4所有节点修改解析文件
2.4.1 viim /etc/hosts
让master节点在连接到node的ip地址时，能够做解析
172.16.23.180 docker41
172.16.23.181 docker42
172.16.23.182 docker43
3.master节点开始部署
3.1master 安装etcd
3.1.1安装etcd软件
`yum install etcd -y
3.1.2 修改etcd配置文件
vim /etc/etcd/etcd.conf
##监听客户端的url，0.0.0.0表示监听本机所有的ip地址
6行：ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0：2379"
##标明ETCD所在的IP和端口
21行：ETCD_ADVERTISE_CLIENT_URLS="http://172.16.23.180:2379"
3.1.3 启动etcd和设置开机自启
systemctl start etcd.service
systemctl enable etcd.service
3.1.4 测试etcd的可用性，设置一个键值对，并且获取到
如能获取到K值，则说明etcd安装成功，至此etcd安装完毕。
etcdctl set testdir/testkey0 0
etcdctl get testdir/testkey0
0`
3.2master安装kubernetes-master
3.2.1 安装master组件
yum install kubernetes-master.x86_64 -y
3.2.2 修改apiserver配置文件
vim /etc/kubernetes/apiserver
##本地服务器apiserver上要监听的地址，0..0.0.0代表所有
8行: KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
##apiserver的端口
11行：KUBE_API_PORT="--port=8080"
##etcd所在的ip地址和端口
17行：KUBE_ETCD_SERVERS="--etcd-servers=http://172.16.23.180:2379"
##这一行需要把ServiceAccount删除，不然后面启动容器时会出现没有认证错误。
23行：KUBE_ADMISSION_CONTROL="--admission
control=NamespaceLifecycle,NamespaceExists,LimitRanger,Security:ContextDeny,ResourceQuota"
3.2.3 修改controller和scheduler配置文件
controller-master，scheduler，proxy共用一个配置文件，所以修改这一个就可以。
vim /etc/kubernetes/config
##为controller和scheduler指明apiserver的IP和端口
22行：KUBE_MASTER="--master=http://172.16.23.180:8080"
3.2.4 重启master上的服务和设置开机自启
systemctl enable kube-apiserver.service
systemctl restart kube-apiserver.service
systemctl enable kube-controller-manager.service
systemctl restart kube-controller-manager.service
systemctl enable kube-scheduler.service
systemctl restart kube-scheduler.service
3.2.5 检查服务是否安装正常
如出现以下状态，则master上安装完毕，服务启动成功。
[root@docker41 ~]# kubectl get componentstatus
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
3.3这里遇到的问题--apiserver组件无法启动
3.3.1 apiserver服务重启失败
[root@docker41 ~]# systemctl restart kube-apiserver.service
Job for kube-apiserver.service failed because the control process exited with error code. See "systemctl status kube-apiserver.service" and "journalctl -xe" for details.
3.3.2 提示启动失败，检查报错
从下面的报错中得知启动频率太快导致，也就是restart，但看不到更多信息。
[root@docker41 ~]# journalctl -xe
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kube-apiserver.service has failed.
--
-- The result is failed.
4月 12 19:15:36 docker41 systemd[1]: Unit kube-apiserver.service entered failed state.
4月 12 19:15:36 docker41 systemd[1]: kube-apiserver.service failed.
4月 12 19:15:36 docker41 systemd[1]: kube-apiserver.service holdoff time over, scheduling restart.
4月 12 19:15:36 docker41 systemd[1]: Stopped Kubernetes API Server.
-- Subject: Unit kube-apiserver.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kube-apiserver.service has finished shutting down.
4月 12 19:15:36 docker41 systemd[1]: start request repeated too quickly for kube-apiserver.service
4月 12 19:15:36 docker41 systemd[1]: Failed to start Kubernetes API Server.
-- Subject: Unit kube-apiserver.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit kube-apiserver.service has failed.
--
-- The result is failed.
4月 12 19:15:36 docker41 systemd[1]: Unit kube-apiserver.service entered failed state.
4月 12 19:15:36 docker41 systemd[1]: kube-apiserver.service failed.
3.3.3接着查看服务状态
从报错中可以看到是由于execstart中某个模块的错误引起的，但得不到更多信息
[root@docker41 ~]# systemctl status kube-apiserver.service
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since 二 2022-04-12 19:28:10 CST; 4min 48s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 20649 ExecStart=/usr/bin/kube-apiserver $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBE_ETCD_SERVERS $KUBE_API_ADDRESS $KUBE_API_PORT $KUBELET_PORT $KUBE_ALLOW_PRIV $KUBE_SERVICE_ADDRESSES $KUBE_ADMISSION_CONTROL $KUBE_API_ARGS (code=exited, status=255)
Main PID: 20649 (code=exited, status=255)
3.3.4 接着查看系统日志
第二行： plugins.go:143] Unknown admission plugin: ServerAccount
从报错中可以得知是由于apiserver的配置文件中有一个模块不能识别而引起的，因此我们需要去进行删除。
cat /var/log/messages
Apr 12 19:28:10 docker41 kube-apiserver: I0412 19:28:10.027128 20649 config.go:562] Will report 172.16.23.180 as public IP address.
Apr 12 19:28:10 docker41 kube-apiserver: F0412 19:28:10.028375 20649 plugins.go:143] Unknown admission plugin: ServerAccount
Apr 12 19:28:10 docker41 systemd: kube-apiserver.service: main process exited, code=exited, status=255/n/a
Apr 12 19:28:10 docker41 systemd: Failed to start Kubernetes API Server.
Apr 12 19:28:10 docker41 systemd: Unit kube-apiserver.service entered failed state.
Apr 12 19:28:10 docker41 systemd: kube-apiserver.service failed.
Apr 12 19:28:10 docker41 systemd: kube-apiserver.service holdoff time over, scheduling restart.
Apr 12 19:28:10 docker41 systemd: Stopped Kubernetes API Server.
Apr 12 19:28:10 docker41 systemd: start request repeated too quickly for kube-apiserver.service
Apr 12 19:28:10 docker41 systemd: Failed to start Kubernetes API Server.
Apr 12 19:28:10 docker41 systemd: Unit kube-apiserver.service entered failed state.
Apr 12 19:28:10 docker41 systemd: kube-apiserver.service failed.
Apr 12 19:30:01 docker41 systemd: Started Session 16 of user root.
3.3.5 删除报错的模块，并重启apiserver
[root@docker41 ~]# vim /etc/kubernetes/apiserver
##删除ServerAccount模块，ServerAccoint用于默认认证，关闭即可。
23 KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
root@docker41 ~]# systemctl restart kube-apiserver.service
3.3.6 检查apiserver状态-正常
[root@docker41 ~]# systemctl status kube-apiserver.service
● kube-apiserver.service - Kubernetes API Server
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
Active: active (running) since 二 2022-04-12 19:43:04 CST; 19s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 20792 (kube-apiserver)
CGroup: /system.slice/kube-apiserver.service
3.3.7 master端配置检查-正常
如master端-出现以下信息，则说明master组件安装完毕，且配置正常，至此master端配置完毕
[root@docker41 ~]# kubectl get componentstatus
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"
4.node节点安装K8S-node（node1，node2）
4.1 node节点安装kubernetes-node
4.1.1 安装组件node组件
yum install kubernetes-node.x86_64 -y
4.1.2 编辑proxy配置文件，让其能够访问到master上的apiserver
vim /etc/kubernetes/config
##apiserver地址与端口
22行：KUBE_MASTER="--master=http://172.16.23.180:8080"
4.1.3 编辑kubelet配置文件
vim /etc/kubernetes/kubelet
##配置kubelet监听的地址，0.0.0.0为监听当前主机所有IP
5行：KUBELET_ADDRESS="--address=0.0.0.0"
监听当前主机10250端口
8行：KUBELET_PORT="--port=10250"
##这里配置node节点当前主机IP地址（我在node1上编辑，所以是172.16.23.181）
11行：KUBELET_HOSTNAME="--hostname-override=172.16.23.181"
##配置apiserver地址和端口
14行：KUBELET_API_SERVER="--api-servers=http://172.16.23.180:8080"
4.1.4 启动kubelet和开机自启
systemctl enable kubelet.service
systemctl start kubelet.service
4.1.5 启动proxy和开机自启
systemctl enable kube-proxy.service
systemctl start kube-proxy.service
4.1.6 在master节点上检查node节点状态
看能否获取到node节点，如master端上能获取到两台主机，出现以下信息则说明配置正常。至此node端配置完毕。
[root@docker41 ~]# kubectl get nodes
NAME STATUS AGE
172.16.23.181 Ready 1m
172.16.23.182 Ready 1m
4.2遇到的第一个问题-kubelet无法启动
4.2.1 启动kubelet出现报错
[root@k8s-node1 ~]# systemctl restart kubelet.service
Failed to restart kubelet.service: Unit is not loaded properly: Invalid argument.
See system logs and 'systemctl status kubelet.service' for details.
4.2.2 查看报错信息
从下面报错信息来看，是由于kubelet无法访问到apiserver所导致。
[root@k8s-node1 ~]# systemctl status kubelet.service
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: inactive (dead) since 二 2022-04-12 20:05:33 CST; 24s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
4月 12 15:06:09 k8s-node1 kubelet[10802]: W0412 15:06:09.845999 10802 reflector.go:319] pkg/kubelet/config/apiserver.go:44: watch of *api.Pod ended with: very short watch
4月 12 15:09:01 k8s-node1 kubelet[10802]: W0412 15:09:01.845071 10802 reflector.go:319] pkg/kubelet/kubelet.go:378: watch of *api.Service ended with: very short watch
4月 12 17:46:32 k8s-node1 kubelet[10802]: E0412 17:46:32.654756 10802 reflector.go:300] pkg/kubelet/kubelet.go:386: Failed to watch *api.Node: Get http://172.1...ute to host
4月 12 20:05:33 k8s-node1 systemd[1]: Stopping Kubernetes Kubelet Server...
4月 12 20:05:33 k8s-node1 systemd[1]: Stopped Kubernetes Kubelet Server.
Hint: Some lines were ellipsized, use -l to show in full.
4.2.3 node1进行访问master验证
访问master时，22端口可以访问，而访问8080和2379端口被拒绝，考虑防火墙设置。
[root@k8s-node1 ~]# curl 172.16.23.181:22
SSH-2.0-OpenSSH_7.4
Protocol mismatch.
curl: (56) Recv failure: Connection reset by peer
[root@k8s-node1 ~]# curl 172.16.23.181:8080
curl: (7) Failed connect to 172.16.23.181:8080; 拒绝连接
[root@k8s-node1 ~]# curl 172.16.23.181:2379
curl: (7) Failed connect to 172.16.23.181:2379; 拒绝连接
4.2.4 设置iptables规则
让master在 172.16.23.0/24网段开放所有端口
[root@docker41 ~]# iptables -I INPUT -s 172.16.23.0/24 -p all -j ACCEPT
[root@docker41 ~]# iptables -I INPUT -s 172.16.23.0/24 -p all -j ACCEPT
4.2.5 重新进行访问验证
访问依然被拒绝，为什么会出现itables设置了规则依然无法访问呢？，研究无果，询问过后发现该系统的防火墙默认为firewall管理，因此进行重新设置。
[root@k8s-node1 ~]# curl 172.16.24.42:8080
curl: (7) Failed connect to 172.16.24.42:8080; 拒绝连接
[root@k8s-node1 ~]# curl 172.16.24.42:2379
curl: (7) Failed connect to 172.16.24.42:2379; 拒绝连接
4.2.6 设置防火墙规则，开放8080和2379
[root@docker41 ~]# firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.23.0/24" port protocol="tcp" port="1024-65535" accept"
success
###让配置生效
[root@k8s-node1 ~]# firewall-cmd --reload
4.2.7 设置完成后可以从node1节点去访问到master上的8080和2379。
[root@k8s-node1 ~]# curl 172.16.24.41:2379
404 page not found
[root@k8s-node1 ~]# curl 172.16.24.41:8080
{
"paths": [
"/api",
"/api/v1",
"/apis",
...
...
4.2.8 这时去master端进行验证
此时master端显示已经可以获取到node节点。
[root@docker41 ~]# kubectl get node
NAME STATUS AGE
172.16.24.42 Ready 1d
172.16.24.43 Ready 1d
4.3 遇到的第二问题-kubelet没有启动，master却可以获取到node。
在上面的问题中，已经可以获取到node节点信息，但我此时去查看node节点上kubelet的状态，却依然是没有启动的。那么为什么master获取到的node节点状态是正常的呢？
这个问题是由于flannel网络没有卸载导致（我重新部署了node节点，却没有卸载flannel网络）。
apiserver可以通过flannel网络访问到container。因此把flannel网络停止，那么master节点获取到的node节点状态就变为了的Notready状态。因此到目前为止master端依然不能获取到node，而且还有一个特别的地方，就是node2可以获取，node1却不能获取，如下所示。那么二者有什么不同呢？
[root@docker41 ~]# kubectl get node
NAME STATUS AGE
172.16.24.42 NotReady 1d
172.16.24.43 Ready 1d
4.3.1 现在来解决node节点的问题-查看所有节点信息
1.node1节点上 proxy的状态-正常
[root@k8s-node1 ~]# systemctl status kube-proxy.service
● kube-proxy.service - Kubernetes Kube-Proxy Server
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 二 2022-04-12 22:55:25 CST; 12min ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 17220 (kube-proxy)
Memory: 32.1M
CGroup: /system.slice/kube-proxy.service
└─17220 /usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://172.16.24.41:8080
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: E0412 22:55:25.939902 17220 server.go:421] Can't get Node "k8s-node1", assuming iptables proxy, err: nodes "k8s-node1" not found
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: I0412 22:55:25.941716 17220 server.go:215] Using iptables Proxier.
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: W0412 22:55:25.942618 17220 server.go:468] Failed to retrieve node info: nodes "k8s-node1" not found
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: W0412 22:55:25.942683 17220 proxier.go:248] invalid nodeIP, initialize kube-proxy with 127.0.0.1 as nodeIP
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: W0412 22:55:25.942689 17220 proxier.go:253] clusterCIDR not specified, unable to distinguish between internal and e...al traffic
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: I0412 22:55:25.942702 17220 server.go:227] Tearing down userspace rules.
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: I0412 22:55:25.957595 17220 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 1048576
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: I0412 22:55:25.957930 17220 conntrack.go:66] Setting conntrack hashsize to 262144
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: I0412 22:55:25.967514 17220 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
4月 12 22:55:25 k8s-node1 kube-proxy[17220]: I0412 22:55:25.967553 17220 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
Hint: Some lines were ellipsized, use -l to show in full.
2.node1节点上kubelet的状态-（不能到达master主机）
[root@k8s-node1 ~]# systemctl status kubelet.service
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since 二 2022-04-12 23:17:58 CST; 4s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 24766 (kubelet)
Memory: 30.7M
CGroup: /system.slice/kubelet.service
├─24766 /usr/bin/kubelet --logtostderr=true --v=0 --api-servers=http://172.16.23.10:8080 --address=0.0.0.0 --port=10250 --hostname-override=172.16.24.42 --allow-...
└─24839 journalctl -k -f
4月 12 23:17:58 k8s-node1 kubelet[24766]: I0412 23:17:58.598177 24766 kubelet_node_status.go:227] Setting node annotation to enable volume controller attach/detach
4月 12 23:17:58 k8s-node1 kubelet[24766]: I0412 23:17:58.599254 24766 kubelet_node_status.go:74] Attempting to register node 172.16.24.42
4月 12 23:17:58 k8s-node1 kubelet[24766]: E0412 23:17:58.601362 24766 eviction_manager.go:204] eviction manager: unexpected err: failed GetNode: node '172.16.2...' not found
4月 12 23:18:00 k8s-node1 kubelet[24766]: E0412 23:18:00.584898 24766 event.go:208] Unable to write event: 'Post http://172.16.23.10:8080/api/v1/namespaces/def...r sleeping)
4月 12 23:18:00 k8s-node1 kubelet[24766]: E0412 23:18:00.584936 24766 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://172.16...ute to host
4月 12 23:18:00 k8s-node1 kubelet[24766]: E0412 23:18:00.584957 24766 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get http:/...ute to host
4月 12 23:18:00 k8s-node1 kubelet[24766]: E0412 23:18:00.584969 24766 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get http://172...ute to host
4月 12 23:18:00 k8s-node1 kubelet[24766]: E0412 23:18:00.584989 24766 kubelet_node_status.go:98] Unable to register node "172.16.24.42" with API server: Post ...ute to host
4月 12 23:18:00 k8s-node1 kubelet[24766]: I0412 23:18:00.785138 24766 kubelet_node_status.go:227] Setting node annotation to enable volume controller attach/detach
4月 12 23:18:00 k8s-node1 kubelet[24766]: I0412 23:18:00.786209 24766 kubelet_node_status.go:74] Attempting to register node 172.16.24.42
Hint: Some lines were ellipsized, use -l to show in full.
3.node2的kubelet节点状态正常
[root@k8s-node2 ~]# systemctl status kubelet.service
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since 三 2022-04-13 11:38:50 CST; 3h 2min ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 6289 (kubelet)
Memory: 43.1M
CGroup: /system.slice/kubelet.service
├─6289 /usr/bin/kubelet --logtostderr=true --v=0 --api-servers=http://172.16.24.41:8080 --address=0.0.0.0 --port=10250 --hostname-override=1...
└─6359 journalctl -k -f
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.498828 6289 factory.go:54] Registering systemd factory
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.498949 6289 factory.go:86] Registering Raw factory
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.499090 6289 manager.go:1106] Started watching for new ooms in manager
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.501145 6289 oomparser.go:185] oomparser using systemd
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.504488 6289 manager.go:288] Starting recovery of all containers
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.540706 6289 manager.go:293] Recovery completed
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.586217 6289 kubelet_node_status.go:227] Setting node annotation to enable volume co...ch/detach
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.587715 6289 kubelet_node_status.go:74] Attempting to register node 172.16.24.43
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.591463 6289 kubelet_node_status.go:113] Node 172.16.24.43 was previously registered
4月 13 11:38:50 k8s-node2 kubelet[6289]: I0413 11:38:50.591480 6289 kubelet_node_status.go:77] Successfully registered node 172.16.24.43
Hint: Some lines were ellipsized, use -l to show in full.
4.master上节点的状态-正常
[root@docker41 ~]# kubectl get componentstatus
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
5.master上获取node节点的状态-只能获取到182也就是node2
182可以正常访问了，说明我的firewall设置是正确的，但为什么181机器一直访问不到，仍然不清楚。
[root@docker41 ~]# kubectl get node
NAME STATUS AGE
172.16.24.42 NotReady 1d
172.16.24.43 Ready 1d
6.查看node2的防火墙规则和node1做对比，发现有几处不同，把不能连接的node1修改为node2.
node1的防火墙状态
[root@k8s-node1 ~]# firewall-cmd --list-all
public
target: default
icmp-block-inversion: no
interfaces:
sources:
services: ssh dhcpv6-client
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
rule family="ipv4" source address="172.16.24.43" port port="2379" protocol="tcp" accept
rule family="ipv4" source address="172.16.24.41" port port="8080" protocol="tcp" accept
rule family="ipv4" source address="172.16.24.41" port port="2379" protocol="tcp" accept
node2的防火墙状态
[root@k8s-node2 ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: ssh dhcpv6-client
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
7.node1改变成node2的规则，让两者一致。
[root@k8s-node1 ~]# firewall-cmd --permanent --remove-rich-rule="rule family="ipv4" source address="172.16.24.41" port protocol="tcp" port="8080" accept"
[root@k8s-node1 ~]# firewall-cmd --permanent --remove-rich-rule="rule family="ipv4" source address="172.16.24.41" port protocol="tcp" port="2379" accept"
[root@k8s-node1 ~]# firewall-cmd --permanent --remove-rich-rule="rule family="ipv4" source address="172.16.24.43" port protocol="tcp" port="2379" accept"
##添加网卡到防火墙规则。
[root@k8s-node1 ~]# firewall-cmd --permanent --zone=public --change-interface=ens192
###让配置生效
[root@k8s-node1 ~]# firewall-cmd --reload
8.现在node1和node2防火墙规则一致了。
[root@k8s-node1 ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: ssh dhcpv6-client
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
9.重启node1服务器，让所有服务开机自启
[root@docker41 ~]# reboot
10.最后检查master节点获取node的情况
到此则说明node端已经配置完毕，node服务正常
[root@docker41 ~]# kubectl get node
NAME STATUS AGE
172.16.24.42 Ready 1d
172.16.24.43 Ready 1d
5.配置flannel网络-实现容器对外服务和互相调用。
5.1所有节点安装flannel网络
5.1.1 所有节点操作
1.安装
yum install -y flannel.x86_64
2.修改flannel网络的配置文件，指明etcd的IP和端口
vim /etc/sysconfig/flanneld
##表明etcd的端口和地址
4 FLANNEL_ETCD_ENDPOINTS="http://172.16.24.41:2379"
##tecd创建一个目录，为flannel网络指明端口范围，这个操作需要在master上的etcd上进行，配置文件中不需要修改。
8 FLANNEL_ETCD_PREFIX="/atomic.io/network"
5.2master节点操作
5.2.1 etcd创建一个目录，且指定flannel网络范围
注意这个网络不能和宿主机的网段一致，不然会导致冲突。
[root@docker41 ~]# etcdctl mk /atomic.io/network/config '{ "Network": "172.19.0.0/16" }'
{ "Network": "172.19.0.0/16" }
5.2.2 安装dockers
因为flannel覆盖网络需要依赖docker，让是master端也能加入flannel网络的必要操作，对于docker不需要进行任何配置，只需要安装
yum install docker -y
5.2.2 重启服务和设置开机自启
systemctl enable flanneld.service
systemctl restart flanneld.service
systemctl start docker
systemctl enable docker
systemctl restart kube-apiserver.service
systemctl restart kube-controller-manager.service
systemctl restart kube-scheduler.service
5.3node节点操作
5.3.1 所有服务重启，以刷新docker0网卡。
systemctl enable flanneld.service
systemctl restart flanneld.service
service docker restart
systemctl restart kubelet.service
systemctl restart kube-proxy.service
5.4所有节点检查网卡
ip a进行检查
如出现docker0，flannel0，且docker，flannel网段在172.19.0.0/16则说明成功
[root@docker41 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 02:51:56:86:d1:ee brd ff:ff:ff:ff:ff:ff
inet 172.16.24.41/24 brd 172.16.23.255 scope global ens192
valid_lft forever preferred_lft forever
inet6 fe80::51:56ff:fe86:d1ee/64 scope link
valid_lft forever preferred_lft forever
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 172.19.35.0/16 scope global flannel0
valid_lft forever preferred_lft forever
inet6 fe80::95b9:4c54:5b33:5878/64 scope link flags 800
valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:41:aa:06:5e brd ff:ff:ff:ff:ff:ff
inet 172.19.35.1/24 scope global docker0
valid_lft forever preferred_lft forever
5.5节点间启动容器进行测试
5.5.1 docker检查可用的镜像
[root@docker41 ~]# docker search centos
INDEX NAME DESCRIPTION STARS OFFICIAL AUTOMATED
docker.io docker.io/centos The official build of CentOS. 7094 [OK]
5.5.2 拉取并启动镜像，查看ip
查看到ip为 172.17.35.2/24
[root@docker41 ~]# docker pull docker.io/centos
[root@docker41 ~]# docker run -it --name="test1" docker.io/centos:latest /bin/bash
[root@9ec701e4bc05 /]# ip a
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UP group default
link/ether 02:42:ac:13:23:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.19.35.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe13:2302/64 scope link
valid_lft forever preferred_lft forever
5.5.3 测试：node主机去ping容器
主机无法访问到容器内，检查防火墙规则，允许flannel网段：172.19.0.0/16网段互相访问。
[root@k8s-node2 ~]# ping 172.19.35.2
PING 172.19.35.2 (172.19.35.2) 56(84) bytes of data.
^C
--- 172.19.35.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
5.5.4 所有节点设置防火墙规则，让flannel网络可以连通。
设置172.19.0.0/16网段可以互相访问。
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.19.0.0/16" port protocol="tcp" port="1024-65535" accept"
firewall-cmd --reload
5.5.5 master查看防火墙规则
[root@docker41 ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: ssh dhcpv6-client
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
rule family="ipv4" source address="172.16.23.0/24" port port="1024-65535" protocol="tcp" accept
rule family="ipv4" source address="172.19.0.0/16" port port="1024-65535" protocol="tcp" accept
5.5.6 node节点查看防火墙规则
[root@k8s-node1 ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens192
sources:
services: ssh dhcpv6-client
ports:
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
rule family="ipv4" source address="172.19.0.0/16" port port="1024-65535" protocol="tcp" accept
5.5.7 再次进行测试
主机不能ping通容器内部，其他的测试-略
[root@docker41 ~]# ping 172
PING 172.75.0 (172.75.0.0) 56(84) bytes of data.
^C
--- 172.75.0 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
5.5.8 测试的结果
- 本机<----->其他主机 可以通信 结论：172.16.23.0/24之间的互相访问没问题
- 本机<----->本机容器 可以通信 结论：flannel0网卡设置没问题
- 本机容器<----->本机容器 可以通信 结论：docker0网卡没问题
- 本机容器----->其他主机 可以通信 结论：172.19.0.0/16网段可以访问172.16.23.0/24网段没问题
- 其他主机----->本机容器 不能通信 结论：flannel网关无法被外网访问
- 其他容器----->本机容器 不能通信 结论：根据上述测试，说明一切问题的都在flannel网卡为什么不能被访问。
5.5.9 问题的解决
k8s安装后默认修改底层iptables规则，为拒绝状态。因此在firewall防火墙上进行设置无法生效。须在iptables上进行配置。
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -F
5.5.10 测试访问结果
容器之间可以互通了
[root@dd274f206524 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
5: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UP group default
link/ether 02:42:ac:13:12:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.19.18.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe13:1202/64 scope link
valid_lft forever preferred_lft forever
[root@dd274f206524 /]# ping 172.19.35.3
PING 172.19.35.3 (172.19.35.3) 56(84) bytes of data.
64 bytes from 172.19.35.3: icmp_seq=27 ttl=60 time=0.378 ms
64 bytes from 172.19.35.3: icmp_seq=28 ttl=60 time=0.377 ms
64 bytes from 172.19.35.3: icmp_seq=29 ttl=60 time=0.403 ms
64 bytes from 172.19.35.3: icmp_seq=30 ttl=60 time=0.382 ms
64 bytes from 172.19.35.3: icmp_seq=31 ttl=60 time=0.368 ms
5.5.11 所有节点把iptables规则保存下来
iptables永久保存
iptables-save >/etc/sysconfig/iptables
iptable规则开机自加载
vim /etc/rc.local
iptables-restore < /etc/sysconfig/iptable
s
5.5.12 关机重启所有服务，再进行测试，以保证服务的稳定。
在两个节点创建各一个容器，并互相访问，然后用主机去访问（要使用pod还需要配置镜像仓库）
[root@docker41 ~]# docker run -it --name="test1" docker.io/centos:latest /bin/bash
[root@9ec701e4bc05 /]# ip a
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UP group default
link/ether 02:42:ac:13:23:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.19.35.2/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe13:2302/64 scope link
valid_lft forever preferred_lft forever
5.5.13 测试容器之间的互访-成功
[root@6bd9ae09f76e /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
7: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1472 qdisc noqueue state UP group default
link/ether 02:42:ac:13:12:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.19.18.3/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe13:1203/64 scope link
valid_lft forever preferred_lft forever
[root@6bd9ae09f76e /]# ping 172.19.18.2
PING 172.19.18.2 (172.19.18.2) 56(84) bytes of data.
64 bytes from 172.19.18.2: icmp_seq=1 ttl=64 time=0.139 ms
64 bytes from 172.19.18.2: icmp_seq=2 ttl=64 time=0.062 ms
64 bytes from 172.19.18.2: icmp_seq=3 ttl=64 time=0.062 ms
64 bytes from 172.19.18.2: icmp_seq=4 ttl=64 time=0.062 ms
64 bytes from 172.19.18.2: icmp_seq=5 ttl=64 time=0.062 ms
至此k8s部署完毕
