k8s裸机LB支持，ingress服务

metallb简介

官方网站：https://metallb.universe.tf/

MetalLB是使用标准路由协议的裸机Kubernetes集群的软负载均衡器，目前处于测试版本阶段。

私有云裸金属架构的kubernetes集群不支持LoadBalance

Kubernetes没有为裸机群集提供网络负载均衡器（类型为LoadBalancer的服务）的实现，如果你的kubernetes集群没有在公有云的IaaS平台（GCP，AWS，Azure …）上运行，则LoadBalancers将在创建时无限期地保持“挂起”状态，也就是说只有公有云厂商自家的kubernetes支持LoadBalancer。

裸机群集运营商留下了两个较小的工具来将用户流量带入其集群，“NodePort”和“externalIPs”服务。这两种选择都对生产使用产生了重大影响，这使得裸露的金属集群成为Kubernetes生态系统中的二等公民。

部署metallb

metallb 目前有两种部署方式，支持 Helm 和 YAML 两种安装方法


kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml

第一次安装还需要运行以下：


kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

查看：


# kubectl get pod -n metallb-system   

NAME                          READY  STATUS    RESTARTS  AGE

controller-684f5d9b49-x859t  1/1    Running  0          6h3m

speaker-lmrgx                1/1    Running  0          6h3m

speaker-x9jk5                1/1    Running  0          6h3m

创建ConfigMap

下载 config.yaml 模版


wget https://raw.githubusercontent.com/google/metallb/v0.9.3/manifests/example-layer2-config.yaml

这里直接生成


cat >> config.yaml << EOF

apiVersion: v1

kind: ConfigMap

metadata:

  namespace: metallb-system

  name: config

data:

  config: |

    address-pools:

    - name: my-ip-space

      protocol: layer2

      addresses:

      - 192.168.200.130-192.168.200.140

EOF

创建：


# kubectl apply -f config.yaml

查看 configMap 情况


# kubectl describe ConfigMap config -n metallb-system

Name:        config

Namespace:    metallb-system

Labels:      <none>

Annotations: 

Data

====

config:

----

address-pools:

- name: my-ip-space

  protocol: layer2

  addresses:

  - 192.168.200.130-192.168.200.140

Events:  <none>

至此，LoadBalancer的配置工作就完成

创建service测试 LoadBalancer

这里创建一个 nginx 服务，包含deployment和一个LoadBalancer类型的service


# vim nginx-test.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: nginx

spec:

  selector:

    matchLabels:

      app: nginx

  template:

    metadata:

      labels:

        app: nginx

    spec:

      containers:

      - name: nginx

        image: nginx:1

        ports:

        - name: http

          containerPort: 80

---

apiVersion: v1

kind: Service

metadata:

  name: nginx

spec:

  ports:

  - name: http

    port: 80

    protocol: TCP

    targetPort: 80

  selector:

    app: nginx

  type: LoadBalancer

查看service分配的EXTERNAL-IP：


# kubectl get svc

NAME        TYPE          CLUSTER-IP  EXTERNAL-IP      PORT(S)        AGE

kubernetes  ClusterIP      10.0.0.1    <none>            443/TCP        4d5h

nginx        LoadBalancer  10.0.0.86    192.168.200.130  80:32295/TCP  6h1m

集群外的机器访问EXTERNAL-IP 192.168.200.130 成功即可

部署 ingress-nginx 控制器

1、安装

官方文档 https://kubernetes.github.io/ingress-nginx/deploy/

下载


# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml

# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/baremetal/service-nodeport.yaml

# kubctl apply -f mandatory.yaml

创建 NodePort 模式的 service


# kubctl apply -f service-nodeport.yaml

集群有LB的可以创建 LoadBalarcer 模式 service


# cp service-nodeport.yaml service-lb.yaml

# vim service-lb.yaml

spec:

  #type: NodePort

  type: LoadBalancer

...

# kubctl apply -f service-lb.yaml

检查是否安装：


# kubectl get pods -n ingress-nginx

NAME                                        READY  STATUS      RESTARTS  AGE

ingress-nginx-controller-69fb496d7d-t9n6g  1/1    Running    0          18m

2、部署 Ingress

这里创建一个nginx deployment 和 ingress


# vim ingress-test.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: nginx

spec:

  selector:

    matchLabels:

      app: nginx

  template:

    metadata:

      labels:

        app: nginx

    spec:

      containers:

      - name: nginx

        image: nginx:1

        ports:

        - name: http

          containerPort: 80

---

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  name: test-ingress

spec:

  rules:

  - host: k8s.alwooo.cn

    http:

      paths:

      - path: /test

        backend:

          serviceName: nginx

          servicePort: 80

创建：


# kubectl apply -f ingress-test.yaml

Error from server (InternalError): error when creating "ingress-test.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.0.0.171:443: connect: connection timed out

这里发现报错了

错误解决：

查看 node 节点的 kube-proxy


# systemctl status kube-proxy

6月 09 10:17:57 kub-node2 kube-proxy[28434]: E0609 10:17:57.325032  28434 node.go:125] Failed to retrieve node info: nodes "kube-node2" not found

发现其中一台节点 kube-proxy 报错，是配置文件 hostnameOverride 错误导致，修改即可


# vim /opt/kubernetes/cfg/kube-porxy-config.yaml

...

hostnameOverride: kub-node2  # 修改为正确的主机名

...

# systemctl restart kube-proxy

重新部署 ingress 还是报错。

判断是 master 节点无法访问 10.0.0.171:443

查看ingress 服务


# kubectl -n ingress-nginx get svc

NAME                                TYPE        CLUSTER-IP  EXTERNAL-IP  PORT(S)                      AGE

ingress-nginx-controller            NodePort    10.0.0.196  <none>        80:31217/TCP,443:30348/TCP  12s

ingress-nginx-controller-admission  ClusterIP  10.0.0.171  <none>        443/TCP                      12s

svervice 为 NodePort 方式，修改为 LoadBalancer


# vim deploy.yaml

...

apiVersion: v1

kind: Service

metadata:

  labels:

    helm.sh/chart: ingress-nginx-2.0.3

    app.kubernetes.io/name: ingress-nginx

    app.kubernetes.io/instance: ingress-nginx

    app.kubernetes.io/version: 0.32.0

    app.kubernetes.io/managed-by: Helm

    app.kubernetes.io/component: controller

  name: ingress-nginx-controller-admission

  namespace: ingress-nginx

spec:

  type:  LoadBalancer    # 修改此字段

  ports:

    - name: https-webhook

      port: 443

      targetPort: webhook

  selector:

    app.kubernetes.io/name: ingress-nginx

    app.kubernetes.io/instance: ingress-nginx

    app.kubernetes.io/component: controller

---

...

重新部署后，还是一样。

经过多番查阅后发现，是master节点没有部署 kubelet, kube-proxy 导致。必须把master节点当成worker节点。

重新部署kubelet 和 kube-proxy 到master节点后。部署ingerss服务成功


# kubectl apply -f ingress-test.yaml

deployment.apps/nginx created

ingress.extensions/test-ingress created

# kubectl get ingress

NAME          CLASS    HOSTS          ADDRESS          PORTS  AGE

test-ingress  <none>  k8s.alwooo.cn  192.168.200.120  80      41s

如果 service 为 NodePort 模式，则客户端域名指向的地址应为 ADDRESS
如使用是 LoadBalancer 则使用 LB 地址作为 VIP，映射到 ASSRESS


# kubectl get svc -n ingress-nginx

NAME                                TYPE          CLUSTER-IP  EXTERNAL-IP      PORT(S)                      AGE

ingress-nginx-controller            LoadBalancer  10.0.0.207  192.168.200.131  80:30859/TCP,443:30756/TCP  57m

ingress-nginx-controller-admission  ClusterIP      10.0.0.242  <none>            443/TCP                      57m

其他坑 https://www.cnblogs.com/tchua/p/10844013.html

配置 ingress 处理 TLS 传输

创建私钥和证书


# openssl genrsa -out tls.key 2048

# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=GuangDong/L=ZhongShan/O=Anlewo/CN=tomcat.alwooo.cn

创建 k8s Secret


kubectl create secret tls ingress-secret --cert=tls.crt --key=tls.key

添加部署文件


# cat tomcat-ingress-tls.yaml

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  name: ingress-tomcat-tls

  namespace: default

  annotations:

    kubernetes.io/ingress.class: "nginx"

spec:

  tls:

  - hosts:

    - tomcat.alwooo.cn

    secretName: tomcat-ingress-secret

  rules:

  - host: tomcat.alwooo.cn

    http:

      paths:

      - path:

        backend:

          serviceName: tomcat

          servicePort: 8080

部署完后就可以用 https 访问了