vm istio治理

服务部署架构

官方文档
https://istio.io/docs/examples/virtual-machines/multi-network/#prerequisites

环境准备

1. k8s集群搭建

注意官方文档只指定的k8s版本有 1.15 1.14 1.13

2. istio搭建

以下步骤在k8s集群上操作：

1）创建一个有meshexpansion-gateways 的Istio部署yaml配置文件

helm template install/kubernetes/helm/istio --name istio --namespace istio-system \
    -f https://github.com/irisdingbj/meshExpansion/blob/master/values-istio-meshexpansion-gateways.yaml \ > $HOME/istio-mesh-expansion-gatways.yaml

2）用生成的文件搭建istio

kubectl create namespace istio-system
#部署时可以加上启动参数打开分布式追踪--set values.tracing.enabled=true
helm template  install/kubernetes/helm/istio-init --name istio-init --namespace istio-system --set values.tracing.enabled=true | kubectl apply -f -
kubectl apply -f $HOME/istio-mesh-expansion-gatways.yaml

3）验证istio是否搭建成功

istioctl verify-install -f $HOME/istio-mesh-expansion-gatways.yaml

4）修改service/istio-ingressgateway
因为我们没有外网ip

a. 配合mesh-gateway使用, 查看meshexpansion-gateway配置如下

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  labels:
    app: gateways
    chart: gateways
    heritage: Tiller
    release: istio
  name: meshexpansion-gateway
  namespace: istio-system
  resourceVersion: "27675"
  selfLink: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/gateways/meshexpansion-gateway
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: tcp-pilot
      number: 15011
      protocol: TCP
  - hosts:
    - '*'
    port:
      name: tcp-citadel
      number: 8060
      protocol: TCP
  - hosts:
    - '*'
    port:
      name: tls-mixer
      number: 15004
      protocol: TLS
    tls:
      mode: AUTO_PASSTHROUGH

b. 把service/istio-ingressgateway的TYPE从LoadBalancer改为NodePort类型, 同时暴露出istio组件相关的端口

  - name: tcp-pilot
    nodePort: 30329
    port: 15011
    protocol: TCP
    targetPort: 15011
  - name: tcp-citadel
    nodePort: 30879
    port: 8060
    protocol: TCP
    targetPort: 8060
  - name: tls-mixer
    nodePort: 31679
    port: 15004
    protocol: TCP
    targetPort: 15004

root@webapp01:# kubectl get svc istio-ingressgateway -n istio-system
NAME                   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                                                                                                     AGE
istio-ingressgateway   NodePort   10.98.149.193   <none>        15020:32252/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31248/TCP,15030:30252/TCP,15031:31682/TCP,15032:32190/TCP,15443:30374/TCP,15011:30329/TCP,8060:30879/TCP,15004:31679/TCP   5d20h

这样就可以通过node的IP+port访问istio相关组件

3.生成虚机连接istio集群需要的配置

1）生成root-cert.pem、key.pem、 cert-chain.pem和cluster.env这几个文件

$ kubectl create ns vm
$ export SERVICE_NAMESPACE="vm"
$ kubectl -n $SERVICE_NAMESPACE get secret istio.default  \
    -o jsonpath='{.data.root-cert\.pem}' | base64 --decode > root-cert.pem
$ kubectl -n $SERVICE_NAMESPACE get secret istio.default  \
    -o jsonpath='{.data.key\.pem}' | base64 --decode > key.pem
$ kubectl -n $SERVICE_NAMESPACE get secret istio.default  \
      -o jsonpath='{.data.cert-chain\.pem}' | base64 --decode > cert-chain.pem
$ echo -e "ISTIO_CP_AUTH=MUTUAL_TLS\nISTIO_SERVICE_CIDR=$ISTIO_SERVICE_CIDR\n" > cluster.env

4. 在虚机上部署istio-sidecar

1）拷贝cluster.env和*.pem文件到虚机上
2）在虚机上安装Envoy sidecar

$ curl -L https://storage.googleapis.com/istio-release/releases/1.4.3/deb/istio-sidecar.deb > istio-sidecar.deb
$ sudo dpkg -i istio-sidecar.deb

3）把istio的网关地址和istio相关组件的域名填上

$ echo "192.144.99.165 istio-citadel istio-pilot istio-pilot.istio-system zipkin.istio-system" | sudo tee -a /etc/hosts

4）安装*.pem和cluster.env

$ sudo mkdir -p /etc/certs
$ sudo cp {root-cert.pem,cert-chain.pem,key.pem} /etc/certs
$ sudo cp cluster.env /var/lib/istio/envoy
$ sudo chown -R istio-proxy /etc/certs /var/lib/istio/envoy

5）修改istio-start.sh
通过测试发现虚机版的istio包有三个bug，需要稍微修改一下istio启动脚本才能正常启动

问题1：
systemctl start 命令启动不了istio，修改istio-start.sh 加上exit $?
解决方法参考于以下issue
https://github.com/istio/istio/issues/19615

function dump {
    iptables-save
    ip6tables-save
    exit $?
}

问题2
istio启动之后，iptables规则创建有点问题，导致重定向不到envoy中
解决方法1：在/var/lib/istio/envoy/sidecar.env文件里配置好iptables用的环境变量
解决方法2：在istio启动之后，手动创建几条规则匹配上
例如：

$ iptables -t nat -A PREROUTING -p tcp --dport 8888 -j ISTIO_INBOUND
$ iptables -t nat -A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN

最终效果见步骤7)中

问题3
istio启动pilot-agent之后，不会指定zipkinAddress,后续分布式追踪获取不到信息
解决方法：
在/usr/local/bin/istio-start.sh脚本中，修改启动pilot-agent代码时候的，加上--zipkinAddress启动参数

exec su -s /bin/bash -c "INSTANCE_IP=${ISTIO_SVC_IP} POD_NAME=${POD_NAME} POD_NAMESPACE=${NS} exec ${ISTIO_BIN_BASE}/pilot-agent proxy ${ISTIO_AGENT_FLAGS_ARRAY[*]} \
    --serviceCluster $SVC \
    --discoveryAddress ${PILOT_ADDRESS} \
    --zipkinAddress zipkin.istio-system:9411 \
    ${CONTROL_PLANE_AUTH_POLICY[*]} \
    2> ${ISTIO_LOG_DIR}/istio.err.log > ${ISTIO_LOG_DIR}/istio.log" ${EXEC_USER}

问题4
要注意istio-pilot和istio-citadel的端口设置与之前istio集群ingressgateway暴露的一致

6）启动istio

$ sudo systemctl start istio-auth-node-agent
$ sudo systemctl start istio
$ sudo systemctl status istio

如果启动不成功可以查看系统日志和istio日志
位置在/var/log/syslog和/var/log/istio/xxx.log

7）检查iptables

$ iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N ISTIO_INBOUND
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-N ISTIO_REDIRECT
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A PREROUTING -p tcp -m tcp --dport 8888 -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 108 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 0 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 108 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 0 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -d 10.254.0.0/16 -j ISTIO_REDIRECT
-A ISTIO_OUTPUT -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001

服务部署流程

集群中部署wkrd webapp

1.部署wkrd

# wrkd
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wrkd
  labels:
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wrkd
      version: v1
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
      labels:
        app: wrkd
        version: v1
    spec:
      containers:
      - name: wrkd
        image: 192.144.99.61/bmi/wrkd:latest
        imagePullPolicy: Always #IfNotPresent
        env:
          - name: WEBAPP_ADDRESS
            value: http://webapp:8889/calculator/bmi
        ports:
        - containerPort: 8890

2）部署webapp

# webapp
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp-v1
  labels:
    version: v1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webapp
      version: v1
  template:
    metadata:
      labels:
        app: webapp
        version: v1
    spec:
      containers:
      - name: webapp
        image: 192.144.99.61/bmi/webapp:latest
        imagePullPolicy: Always #IfNotPresent
        env:
          - name: CALCULATOR_ADDRESS
            value: calculator.vm.svc.cluster.local:8888
        ports:
        - containerPort: 8889

---
apiVersion: v1
kind: Service
metadata:
  name: webapp
  labels:
    app: webapp
spec:
  ports:
  - port: 8889
    name: http
  selector:
    app: webapp

---

3）配置webapp治理规则

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: webapp
spec:
  hosts:
  - webapp
  http:
  - route:
    - destination:
        host: webapp
        subset: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: webapp
spec:
  host: webapp
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

虚机上部署calculator

1.下载代码
2.下载依赖
3.编译calculator
4.设置环境变量
5.运行calculator

配置治理规则

1.创建VirtualService

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: calculator
  namespace: vm
spec:
  hosts:
  - calculator.vm.svc.cluster.local
  http:
  - route:
    - destination:
        host: calculator.vm.svc.cluster.local
        subset: v1
      weight: 80
    - destination:
        host: calculator.vm.svc.cluster.local
        subset: v2
      weight: 20

2.创建DestintationRule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: calculator
  namespace: vm
spec:
  host: calculator.vm.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

3.创建ServiceEntry

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: calculator
  namespace: vm
spec:
  endpoints:
  - address: 192.144.99.163
    labels:
      app: caclulator
      version: v1
    ports:
      grpc: 8888
  - address: 192.144.99.164
    labels:
      app: caclulator
      version: v2
    ports:
      grpc: 8888
  hosts:
  - calculator.vm.svc.cluster.local
  ports:
  - name: grpc
    number: 8888
    protocol: GRPC
  resolution: STATIC

分布式追踪

设置环境变量

JAEGER_REPORTER_LOG_SPANS=true
JAEGER_SERVICE_NAME=calculator
JAEGER_SAMPLER_PARAM=1
JAEGER_PROPAGATION=b3
JAEGER_SAMPLER_TYPE=const
JAEGER_ENDPOINT=http://192.xxx.xx.165:14268/api/traces

暴露jaeger端口
在istio-sidecar启动时，加上zipkin参数

vm istio治理

vm istio治理

服务部署架构

环境准备

1. k8s集群搭建

2. istio搭建

3.生成虚机连接istio集群需要的配置

4. 在虚机上部署istio-sidecar

服务部署流程

集群中部署wkrd webapp

虚机上部署calculator

配置治理规则

分布式追踪

相关阅读更多精彩内容

友情链接更多精彩内容