k8s安装启动二进制包无安全认证

操作系统

因为要在笔记本上同时开不止一个虚拟机,使用的是CentOS 7最小化安装.。网络使用桥接。.
笔者使用了三台虚拟机,所以以下内容需要在三台虚拟机上都执行一遍。..或在一台上设置好之后再克隆虚拟机.。.IP配置为静态.

环境总览

操作系统 CentOS7
IP配置

主节点 192.168.0.120
worker1 192.168.0.121
worker2 192.168.0.122

k8s版本 1.5.2
etcd版本 3.1.11
docker版本 1.13.1
calico版本 3.13.1

关闭防火墙

查看防火墙状态

systemctl status firewalld

关闭防火墙

systemctl disable firewalld
systemctl stop firewalld

安装后连接网络

vi /etc/sysconfig/network-scripts/ifcfg-ens33

修改 ONBOOT 为 yes

systemctl restart network
ip a

ens33网卡显示IP地址，可正常上网.。

安装必要的软件

yum install vim -y
yum install net-tools -y
yum install telnet -y
yum install wget -y
yum install git -y

网络配置

修改 /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
IPADDR="192.168.0.120" #按照实际的网络环境设置,三台分别指定
NETMASK="255.255.255.0"
NM_CONTROLLED=no
DEFROUTE="yes"
PEERROUTES=yes
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
IPV6_FAILURE_FATAL="no"
NAME="ens33"
UUID="e4b9a998-555f-4d41-8fec-9e7aa09f80b4"
DEVICE="ens33"
ONBOOT="yes"
GATEWAY=192.168.0.1 #按照实际的网络环境设置

修改 /etc/resolv.conf

# Generated by NetworkManager
nameserver 8.8.8.8
nameserver 8.8.4.4

VMWare Fusion 下，网络设置为 Autodetect
重启网络...

systemctl restart network

修改hostname

所有节点的hostname 不能相同，calico向ETCD注册时候会用到。可以使用小写字母，数字，-，和. 修改之后重启一下。。.

hostnamectl set-hostname xxx

安装k8s, docker, etcd

在CentOS下，使用yum 安装..。


yum install kubernetes

yum install docker

yum install etcd

systemctl enable docker

主节点上

etcd

版本 3.3.11 查看etcd的service文件,/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\""
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

找到 EnvironmentFile，编辑它。。。

ETCD_LISTEN_CLIENT_URLS="http://192.168.0.120:2379,http://localhost:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.120:2379"

修改以上两项，默认的localhost只能本机访问

systemctl daemon-reload
systemctl restart etcd
systemctl status etcd

apiserver:

vim /lib/systemd/system/kube-apiserver.service

找到两个环境变量文件,并备份
/etc/kubernetes/config 中
修改 KUBE_ALLOW_PRIV="--allow-privileged=false" 为 KUBE_ALLOW_PRIV="--allow-privileged=true"

/etc/kubernetes/apiserver 中

###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

# The port on the local server to listen on.
# KUBE_API_PORT="--port=8080"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,LimitRanger,DefaultStorageClass,ResourceQuota"

# Add your own!
KUBE_API_ARGS="--kubelet-https=false --enable-swagger-ui=true --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h"

启动apiserver

systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver
journalctl -f -u kube-apiserver

kube-controller-manager

修改 /etc/kubernetes/controller-manager 为

###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 --allocate-node-cidrs=true --service-cluster-ip-range=10.254.0.0/16 --cluster-cidr=172.20.0.0/16 --cluster-name=kubernetes --leader-elect=true --cluster-signing-cert-file= --cluster-signing-key-file= "

启动kube-controller-manager

systemctl enable kube-controller-manager
systemctl start kube-controller-manager & journalctl -f -u kube-controller-manager
systemctl status kube-controller-manager

kube-scheduler

修改/etc/kubernetes/scheduler为

###
###
# kubernetes scheduler config

# default config should be adequate

# Add your own!
KUBE_SCHEDULER_ARGS="--address=127.0.0.1 --master=http://127.0.0.1:8080 --leader-elect=true "

启动scheduler

systemctl enable kube-scheduler
systemctl start kube-scheduler & journalctl -f -u kube-scheduler
systemctl status kube-scheduler

所有节点上

docker

修改/etc/docker/daemon.json为

{
    "cluster-store":"etcd://192.168.0.120:2379"
}

CNI网络插件

CNI插件可供选择的有很多，这里选择Calico
文件 kube-calico.service

 [Unit]
 Description=calico node
 After=docker.service
 Requires=docker.service

 [Service]
 User=root
 PermissionsStartOnly=true
 ExecStart=/usr/bin/docker run --net=host --privileged --name=calico-node \
   -e ETCD_ENDPOINTS=http://192.168.0.120:2379 \
   -e CALICO_LIBNETWORK_ENABLED=true \
   -e CALICO_NETWORKING_BACKEND=bird \
   -e CALICO_DISABLE_FILE_LOGGING=true \
   -e CALICO_IPV4POOL_CIDR=172.20.0.0/16 \
   -e CALICO_IPV4POOL_IPIP=off \
   -e FELIX_DEFAULTENDPOINTTOHOSTACTION=ACCEPT \
   -e FELIX_IPV6SUPPORT=false \
   -e FELIX_LOGSEVERITYSCREEN=info \
   -e FELIX_IPINIPMTU=1440 \
   -e FELIX_HEALTHENABLED=true \
   -e IP=192.168.0.120 \
   -v /var/run/calico:/var/run/calico \
   -v /lib/modules:/lib/modules \
   -v /run/docker/plugins:/run/docker/plugins \
   -v /var/run/docker.sock:/var/run/docker.sock \
   -v /var/log/calico:/var/log/calico \
   -v /var/lib/calico/:/var/lib/calico/ \
   docker.io/calico/node
 ExecStop=/usr/bin/docker rm -f calico-node
 Restart=always
 RestartSec=10

 [Install]
 WantedBy=multi-user.target

cp kube-calico.service /etc/systemd/system
systemctl enable kube-calico
systemctl start kube-calico 
journalctl -f -u kube-calico

安装calicoctl

curl -O -L  https://github.com/projectcalico/calicoctl/releases/download/v3.13.2/calicoctl
chmod +x calicoctl
cp calicoctl /usr/bin

查看calico是否在所有节点上正常运行

calicoctl node status

正常情况下可以看到当前节点外的其他节点

Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 192.168.0.122 | node-to-node mesh | up    | 08:43:35 | Established |
| 192.168.0.121 | node-to-node mesh | up    | 08:44:12 | Established |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

使用kubectl设置集群(任意一个Node上面)

kubectl config set-cluster kubernetes --server=http://192.168.0.120:8080
kubectl config set-context kubernetes --cluster=kubernetes
kubectl config use-context kubernetes
cat ~/.kube/config
kubectl get nodes #No resources found.

在工作节点上

kubelet服务

文件 kubelet.service

[Unit]
 Description=Kubernetes Kubelet
 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
 After=docker.service
 Requires=docker.service

 [Service]
 WorkingDirectory=/var/lib/kubelet
 ExecStart=/usr/bin/kubelet \
   --address=192.168.0.121 \
   --hostname-override=192.168.0.121 \
   --pod-infra-container-image=docker.io/mirrorgooglecontainers/pause-amd64:3.1 \
   --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
   --network-plugin=cni \
   --cni-conf-dir=/etc/cni/net.d \
   --cni-bin-dir=/opt/cni/bin \
   --cluster-dns=10.10.0.1 \
   --cluster-domain=cluster.local. \
   --allow-privileged=true \
   --logtostderr=true \
   --require-kubeconfig \
   --v=2
 ExecStartPost=/sbin/iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 4194 -j ACCEPT
 ExecStartPost=/sbin/iptables -A INPUT -s 172.16.0.0/12 -p tcp --dport 4194 -j ACCEPT
 ExecStartPost=/sbin/iptables -A INPUT -s 192.168.0.0/16 -p tcp --dport 4194 -j ACCEPT
 ExecStartPost=/sbin/iptables -A INPUT -p tcp --dport 4194 -j DROP
 Restart=on-failure
 RestartSec=5

 [Install]
 WantedBy=multi-user.target

kubelet.kubeconfig

 apiVersion: v1
 clusters:
 - cluster:
     insecure-skip-tls-verify: true
     server: http://192.168.0.120:8080
   name: kubernetes
 contexts:
 - context:
     cluster: kubernetes
     user: ""
   name: system:node:kube-master
 current-context: system:node:kube-master
 kind: Config
 preferences: {}
 users: []

calico的配置

10-calico.conf

 {
     "name": "calico-k8s-network",
     "cniVersion": "0.1.0",
     "type": "calico",
     "etcd_endpoints": "http://192.168.0.120:2379",
     "log_level": "info",
     "ipam": {
         "type": "calico-ipam"
     },
     "kubernetes": {
         "k8s_api_root": "http://192.168.0.120:8080"
     }
 }

mkdir -p /var/lib/kubelet
mkdir -p /etc/kubernetes
mkdir -p /etc/cni/net.d
cp kubelet.service /etc/systemd/system
cp kubelet.kubeconfig /etc/kubernetes
cp 10-calico.conf /etc/cni/net.d
systemctl daemon-reload
systemctl enable kubelet 
systemctl start kubelet && journalctl -f -u kubelet

尝试与master节点通讯，在工作节点上

export KUBERNETES_MASTER=http://192.168.0.120:8080
kubectl get pods
# No resources found.

在部署服务之前，还需要下载一下CNI网络插件的可执行文件 /opt/cni/bin这个目录在kubelet.service中已指定

mkdir -p /opt/cni/bin
#calico
curl -L -o /opt/cni/bin/calico https://github.com/projectcalico/cni-plugin/releases/download/v3.8.0/calico-amd64
chmod 755 /opt/cni/bin/calico
curl -L -o /opt/cni/bin/calico-ipam https://github.com/projectcalico/cni-plugin/releases/download/v3.8.0/calico-ipam-amd64
chmod 755 /opt/cni/bin/calico-ipam
#cni-plugins
wget https://github.com/containernetworking/plugins/releases/download/v0.7.5/cni-plugins-amd64-v0.7.5.tgz
tar -xzvf cni-plugins-amd64-v0.7.5.tgz -C /opt/cni/bin 
chmod 755 /opt/cni/bin/*

部署服务试试吧

在主节点上
文件 nginx_deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx
          imagePullPolicy: Always
          name: nginx
          ports:
            - containerPort: 80

kubectl create -f nginx_deployment.yaml && kubectl get po -o wide
#在相应的Node上查看kubelet的日志
journalctl -f -u kubelet
# 查看docker容器是否运行起来
docker ps

其他

calico 默认的 docker 镜像下载不下来，可以用 docker.io/calico/node
同理pause的镜像使用docker.io/mirrorgooglecontainers/pause-amd64:3.1

清除etcd数据

systemctl stop etcd
rm -rf /var/lib/etcd/default.etcd
systemctl start etcd

一定要仔细的看日志，而且新版本的发布，可能会导致配置有些不同

k8s安装启动 二进制包 无安全认证

操作系统

环境总览

关闭防火墙

安装后连接网络

安装必要的软件

网络配置

修改hostname

安装k8s, docker, etcd

主节点上

etcd

apiserver:

kube-controller-manager

kube-scheduler

所有节点上

docker

CNI网络插件

在工作节点上

kubelet服务

calico的配置

部署服务试试吧

其他

清除etcd数据

k8s安装启动二进制包无安全认证