前言:
什么是Ingress?
官方的解释是:
Ingress 是对集群中服务的外部访问进行管理的 API 对象,典型的访问方式是 HTTP。
Ingress 可以提供负载均衡、SSL 终结和基于名称的虚拟托管。
-
Ingress简介
Ingress对象,其实就是对“反向代理”的一种抽象,简单的说就是一个全局的负载均衡器,可以通过访问URL定位到后端的Service
有了Ingress这个抽象,K8S就不需要关心Ingress的细节了,实际使用时,只需要选择一个具体的Ingress Controller部署就行了,业界常用的反向代理项目有:Nginx、HAProxy、Envoy、Traefik 都已经成为了K8S专门维护的Ingress Controller
一个Ingress对象的主要内容,就类似Nginx的配置文件描述,对应的转发规则就是ingressRule,
有了Ingress这个对象,用户就可以根据自己的需求选择Ingress Controller,例如,如果应用对代理服务的中断非常敏感,可以使用Treafik这样的Ingress Controller
Ingress工作在七层,Service工作在四层,当想要在Kubernetes里为应用进行TLS配置等HTTPS相关操作时,都必须通过Ingress来进行
ingress-nginx 简单的理解就是你原来需要改 Nginx 配置,然后配置各种域名对应哪个 Service,现在把这个动作抽象出来,变成一个 Ingress 对象,你可以用 yaml 创建,每次不要去改 Nginx 了,直接改 yaml 然后创建/更新就行了;那么问题来了:”Nginx 该怎么处理?”
-
Ingress Controller 这东西就是解决 “Nginx 的处理方式” 的;Ingress Controoler 通过与 Kubernetes API 交互,动态的去感知集群中 Ingress 规则变化,然后读取他,按照他自己模板生成一段 Nginx 配置,再写到 Nginx Pod 里,最后 reload 一下,工作流程如下图:
实际上Ingress也是Kubernetes API的标准资源类型之一,它其实就是一组基于DNS名称(host)或URL路径把请求转发到指定的Service资源的规则。用于将集群外部的请求流量转发到集群内部完成的服务发布。我们需要明白的是,Ingress资源自身不能进行“流量穿透”,仅仅是一组规则的集合,这些集合规则还需要其他功能的辅助,比如监听某套接字,然后根据这些规则的匹配进行路由转发,这些能够为Ingress资源监听套接字并将流量转发的组件就是Ingress Controller
Ingress 两种路由方式
1.虚拟主机
2.URL 路径
Ingree-nginx部署
- ingress-nginx 很多全局配置或修改默认配置都是通过annotations 注释来加载配置文件具体参数详细解释可
参考官方文档:
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/
- 选择NodePort部署方式
- extensions/v1beta1 Ingress资源规范 1.22+版本后会被彻底弃用
apiVersion: extensions/v1betal #资源所属的API群组和版本
Kind: Ingress#资源类型标识符
metadata: #元数据
name <string> #资源名称
annotationsl #资源注解,v1betal使用下面的注解来指定要解析该资源的控制器类型
kubernetes.io/ingress.class: <string> #适配的Ingress控制器类别
namespace <string> #名称空间
spec:
rules <[]Object> #Ingress规则列表;
- host <string> #虚拟主机的FQDN,支持“*"前缀通配,不支持IP,不支持指定端口
http <object>
paths<[]0bject>#虚拟主机PATH定义的列表,由path和backend组成
- path <string> #流量匹配的HTTP PATH,必须以/开头
pathType <string> #匹配机制,支持Exact、_Prefix和ImplementationSpecific
backend <object> #匹配到的流量转发到的目标后端
resource <Object> #引用的同一名称空间下的资源,与下面两个字段互斥
serviceName <string>#引用的Service资源的名称
servicePort <string># Service用于提供服务的端口
tls <[]object> #TLS配置,用于指定上rules中定义的哪些host需要工作HTTPS模式
- hosts <[]string> #使用同一组证书的主机名称列表
secretName <string> #保存于数字证书和私钥信息的secret资源名称
backend <object> #默认backend的定义,可嵌套字段及使用格式跟rules字段中的相同
ingressClassName <string> #ingress类名称,用于指定适配的控制器
- v1 Ingress资源规范
apiVersion: networking.k8s.io/v1 #资源所属的API群组和版本
kind: Ingress #资源类型标识符
metadata: #元数据
name <string> #资源名称
annotations: #资源注解,vlbetal使用下面的注解来指定要解析该资源的控制器类型
kubernetes.io/ingress.class:<string> #适配的Ingress控制器类别
namespace <string> #名称空间
spec:
rules <[]object> #Ingress规则列表
- host <string> #虚拟主机的FQDN,支持“*"前缀通配,不支持IP,不支持指定端口
http <object>
paths <[]object>#虚拟主机PATH定义的列表,由path和backend组成
- path <string> #流量匹配的HTTP PATH,必须以/开头
pathType <string> #支持Exact、Prefix和ImplementationSpecific,必选
backend <Object>#匹配到的流量转发到的目标后端
resource <object> #引用的同一名称空间下的资源,与下面两个字段互斥
service <object> #关联的后端Service对象
name <string> #后端Service的名称
port bject> #后端Service上的端口对象
name <string> #端口名称
number <integer> #端口号
tls <[]object> #TLS配置,用于指定上rules中定义的哪些host需要工作HTTPS模式
- hosts <[]string> #使用同一组证书的主机名称列表
secretName <string> #保存于数字证书和私钥信息的secret资源名称
backend <object> #默认backend的定义,可嵌套字段及使用格式跟rules字段中的相同
ingressClassName <string> #ingress类名称,用于指定适配的控制器
- 添加externalIPs:节点3 IP非必须步骤,方便记忆使用 而不是记忆NodePort端口 NodePort与externalIPs可同时访问
[root@k8s-master Ingress]# vim deploy.yaml
...
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
labels:
helm.sh/chart: ingress-nginx-4.0.1
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/version: 1.0.0
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/component: controller
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
type: NodePort
externalIPs: [192.168.54.173] #添加externalIPs字段 固定访问IP
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
appProtocol: http
- name: https
port: 443
protocol: TCP
targetPort: https
appProtocol: https
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/component: controller
...
- 部署Ingress-nginx
[root@k8s-master Ingress]# kubectl apply -f deploy.yaml
namespace/ingress-nginx unchanged
serviceaccount/ingress-nginx unchanged
configmap/ingress-nginx-controller configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
role.rbac.authorization.k8s.io/ingress-nginx unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
service/ingress-nginx-controller-admission unchanged
service/ingress-nginx-controller configured
deployment.apps/ingress-nginx-controller configured
ingressclass.networking.k8s.io/nginx unchanged
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
serviceaccount/ingress-nginx-admission unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
role.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
job.batch/ingress-nginx-admission-create unchanged
job.batch/ingress-nginx-admission-patch unchanged
[root@k8s-master Ingress]# kubectl get pod
NAME READY STATUS RESTARTS AGE
etcd-operator-646cbffdb6-brbn6 1/1 Running 0 19h
example-etcd-cluster-5fb5d9d6n8 1/1 Running 0 49m
example-etcd-cluster-nc8pdgjrjr 1/1 Running 0 19h
example-etcd-cluster-svgdngq28k 1/1 Running 0 48m
[root@k8s-master Ingress]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.109.24.78 192.168.54.173 80:32493/TCP,443:30239/TCP 120m
ingress-nginx-controller-admission ClusterIP 10.110.72.52 <none> 443/TCP 120m
示例1:创建 Ingress-nginx虚拟主机
- 创建 Deployment 和与之对应的SVC
[root@k8s-master Ingress]# cat deployment-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-demo
namespace: default
spec:
replicas: 4
selector:
matchLabels:
app: demoapp
release: stable
template:
metadata:
labels :
app: demoapp
release: stable
spec:
containers:
- name: demoapp
image: ikubernetes/demoapp:v1.1
ports:
- containerPort: 80
name: http
---
apiVersion: v1
kind: Service
metadata:
name: demoapp-deploy
namespace: default
spec:
selector:
app: demoapp
release: stable
ports:
- name: http
port: 80
targetPort: 80
- 创建 ingress-nginx
[root@k8s-master Ingress]# cat demoapp-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-demo
annotations:
kubernetes.io/ingress.class: "nginx"
namespace: default
spec:
rules:
- host: www.ik8s.io #虚拟主机
http:
paths:
- path: /
pathType: Prefix #前缀匹配
backend:
service:
name: demoapp-deploy
port:
number: 80
[root@k8s-master Ingress]# kubectl apply -f deployment-demo.yaml
[root@k8s-master Ingress]# kubectl apply -f demoapp-ingress.yaml
[root@k8s-master Ingress]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.109.24.78 192.168.54.173 80:32493/TCP,443:30239/TCP 5h50m
ingress-nginx-controller-admission ClusterIP 10.110.72.52 <none> 443/TCP 5h50m
[root@k8s-master Ingress]# kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-demo <none> www.ik8s.io 192.168.4.171 80 3h11m
- 访问测试
[root@bigyong ~]# cat /etc/hosts
# ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
...
192.168.54.173 www.ik8s.io #写hosts
[root@bigyong ~]# curl 192.168.54.173 #直接访问只能到ingress-nginx 没到转发到后端
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
[root@bigyong ~]# curl -H "Host:www.ik8s.io" 192.168.54.173 #访问成功
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-gw6qp, ServerIP: 192.168.113.39
[root@bigyong ~]# curl -H "Host:www.ik8s.io" 192.168.54.173
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-9lnpq, ServerIP: 192.168.12.39!
[root@bigyong ~]# curl -H "Host:www.ik8s.io" 192.168.54.173
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-2zcr5, ServerIP: 192.168.51.61!
[root@bigyong ~]# curl -H "Host:www.ik8s.io" 192.168.54.173
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-9lnpq, ServerIP: 192.168.12.39!
示例2: 创建TLS Ingress HTTPS
[root@k8s-master Ingress]# cat demoapp-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-demo
annotations:
kubernetes.io/ingress.class: "nginx"
namespace: default
spec:
rules:
- host: www.ik8s.io
http:
paths:
- path: /
pathType: Prefix #前缀匹配
backend:
service:
name: demoapp-deploy
port:
number: 80
tls: #添加tls
- hosts:
- www.ik8s.io
secretName: ik8s-tls
[root@k8s-master Ingress]# kubectl apply -f demoapp-ingress.yaml
ingress.networking.k8s.io/ingress-demo configured
[root@k8s-master Ingress]# kubectl describe ingress ingress-demo
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Name: ingress-demo
Namespace: default
Address: 192.168.4.171
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
ik8s-tls terminates www.ik8s.io
Rules:
Host Path Backends
---- ---- --------
www.ik8s.io
/ demoapp-deploy:80 (192.168.113.39:80,192.168.12.39:80)
Annotations: kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 113s (x3 over 6h36m) nginx-ingress-controller Scheduled for sync
- 创建tls自签证书
[root@k8s-master Ingress]# (umask 077); openssl genrsa -out tls.key 2048
Generating RSA private key, 2048 bit long modulus
............................+++
.......................+++
e is 65537 (0x10001)
[root@k8s-master Ingress]# openssl req -new -x509 -key tls.key -out tls.crt -subj "/CN=www.ik8s.io" -days 365
[root@k8s-master Ingress]# ls
demoapp-ingress.yaml deployment-demo.yaml tls.crt
deploy-externalIP.yaml deploy.yaml tls.key
- 创建Secret
[root@k8s-master Ingress]# kubectl create secret tls ik8s-tls --cert=./tls.crt --key=./tls.key
secret/ik8s-tls created
- 访问测试
[root@k8s-master Ingress]# curl -H "Host:www.ik8s.io" 192.168.54.173 #308已经被重定向
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
#通过https访问 提示证书无效不被信任
[root@k8s-master Ingress]# curl -H "Host:www.ik8s.io" https://192.168.54.173
curl: (60) Issuer certificate is invalid.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
- 忽略风险 访问成功
[root@k8s-master Ingress]# curl -k -H "Host:www.ik8s.io" https://192.168.54.173
iKubernetes demoapp v1.1 !! ClientIP: 192.168.113.37, ServerName: deployment-demo-867c7d9d55-9lnpq, ServerIP: 192.168.12.39!
示例3:为dashboard 添加ingress
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/
[root@k8s-master Ingress]# cat ingress-kubernetes-dashboard.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dashboard
annotations:
kubernetes.io/ingress.class: "nginx" #指定控制器
ingress.kubernetes.io/ssl-passthrough: "true" #tcp代理 因为后端dashboard必须通过https访问 这里通过4层转发直接转发到后端Pod
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" #后端直接通过https协议访问
nginx.ingress.kubernetes.io/rewrite-target: /$2 #重写标记 path:/dashboard(/|$)(.*) 这里的第2部分
namespace: kubernetes-dashboard
spec: #这里没有配置host 相当于*号通配所有主机
rules:
- http:
paths:
- path: /dashboard(/|$)(.*)
backend:
serviceName: kubernetes-dashboard
servicePort: 443
[root@k8s-master Ingress]# kubectl apply -f ingress-kubernetes-dashboard.yaml
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
ingress.extensions/dashboard created
[root@k8s-master Ingress]# kubectl describe ingress dashboard -n kubernetes-dashboard
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Name: dashboard
Namespace: kubernetes-dashboard
Address: 192.168.4.171
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
*
/dashboard(/|$)(.*) kubernetes-dashboard:443 (192.168.51.64:8443) #后端pod
Annotations: ingress.kubernetes.io/ssl-passthrough: true
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/rewrite-target: /$2
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 103s (x2 over 2m41s) nginx-ingress-controller Scheduled for sync
- 访问测试
注意:https://192.168.54.173/dashboard/ 一定要带上最后的/ $2为/后面部分 不然无法访问
示例4:longhorn添加ingress-nginx basic认证
参考官方文档
https://longhorn.io/docs/1.2.0/deploy/accessing-the-ui/longhorn-ingress/
- 默认longhorn没有登录认证暴露到公网会存在风险 首先添加basic认证
[root@k8s-master Ingress]# kubectl get pod -n longhorn-system
NAME READY STATUS RESTARTS AGE
....
longhorn-manager-cc8sp 1/1 Running 0 149m
longhorn-manager-fs5tx 1/1 Running 2 149m
longhorn-manager-vwbzn 1/1 Running 1 149m
longhorn-ui-79f8976fbf-c44ct 1/1 Running 1 149m
[root@k8s-master Ingress]# kubectl get svc -n longhorn-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
csi-attacher ClusterIP 10.108.190.205 <none> 12345/TCP 142m
csi-provisioner ClusterIP 10.99.216.181 <none> 12345/TCP 142m
csi-resizer ClusterIP 10.96.161.192 <none> 12345/TCP 141m
csi-snapshotter ClusterIP 10.101.216.72 <none> 12345/TCP 141m
longhorn-backend ClusterIP 10.108.67.31 <none> 9500/TCP 149m
longhorn-frontend ClusterIP 10.107.71.176 <none> 80/TCP 149m
[root@k8s-master Ingress]# USER=user.com; PASSWORD=passwd.com; echo "${USER}:$(openssl passwd -stdin -apr1 <<< ${PASSWORD})" >> auth
[root@k8s-master Ingress]# ls
auth deploy-externalIP.yaml deploy.yaml tls.crt
demoapp-ingress.yaml deployment-demo.yaml ingress-kubernetes-dashboard.yaml tls.key
[root@k8s-master Ingress]# kubectl -n longhorn-system create secret generic basic-auth --from-file=authsecret/basic-auth created
[root@k8s-master Ingress]# cat longhorn-ui-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: longhorn-ingress
namespace: longhorn-system
annotations:
kubernetes.io/ingress.class: "nginx" #指定控制器
# type of authentication
nginx.ingress.kubernetes.io/auth-type: basic #认证类型
# prevent the controller from redirecting (308) to HTTPS
nginx.ingress.kubernetes.io/ssl-redirect: 'false' #http通信
# name of the secret that contains the user/password definitions
nginx.ingress.kubernetes.io/auth-secret: basic-auth #secret名称
# message to display with an appropriate context why the authentication is required
nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required ' #输入提示
# custom max body size for file uploading like backing image uploading
nginx.ingress.kubernetes.io/proxy-body-size: 10000m
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- pathType: Prefix
path: /longhorn(/|$)(.*)
backend:
service:
name: longhorn-frontend
port:
number: 80
[root@k8s-master Ingress]# kubectl apply -f longhorn-ui-ingress.yaml
ingress.networking.k8s.io/longhorn-ingress configured
访问 http://IP/longhorn/ #最后的 "/" 一定要加
-
以下图片IP 重新部署后的服务IP