kubefed实现多集群部署

首先我们需要添加一下kubefed的helm

helm repo add kubefed https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts

然后search一下选择kubefed/kubefed fetch到本地

helm search kubefed
NAME                        CHART VERSION   APP VERSION DESCRIPTION                          
incubator/ack-kubefed       0.0.2                       KubeFed helm chart                   
incubator/ack-kubefed-init  1.0.0           0.1.0       Helm chart to initialize kubefed Envs
kubefed/kubefed             0.6.1                       KubeFed helm chart

这里需要注意crd的apiVersion，如果是apiextensions.k8s.io/v1beta1建议使用chart version 0.4.0，如果是apiextensions.k8s.io/v1，使用最新版本即可

随后我们需要替换一下镜像如果不能翻墙的话

controller:
    annotations: {}
    replicaCount: 2
    repository: quay.mirrors.ustc.edu.cn/kubernetes-multicluster
    image: kubefed
    tag: v0.4.0
    imagePullPolicy: IfNotPresent

然后helm install 指定一下kube-federation-system的namespace即可

kubectl get pod -n kube-federation-system
NAME                                         READY   STATUS    RESTARTS   AGE
kubefed-admission-webhook-76c6847cc6-xj4sf   1/1     Running   0          <invalid>
kubefed-controller-manager-8964d96dd-scqw7   1/1     Running   0          <invalid>
kubefed-controller-manager-8964d96dd-sv5j5   1/1     Running   0          <invalid>

至此kubefed已经部署完毕

之后我们需要安装一下kubefedctl命令

VERSION=0.1.0-rc6
OS=linux
ARCH=amd64
curl -LO https://github.com/kubernetes-sigs/kubefed/releases/download/v${VERSION}/kubefedctl-${VERSION}-${OS}-${ARCH}.tgz
tar -zxvf kubefedctl-*.tgz
chmod u+x kubefedctl
mv kubefedctl /bin/

然后我们需要把其他集群的信息添加到.kube/config文件中

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EVXdPVEF5TXpNeU0xb1hEVE13TURVd056QXlNek15TTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTWc0ClYwYkpVN2h2Q01CNThJWCtJaHpveVBKc0tUeE5zd05CN0g5cnV2QlhESGtHdDU4VDd4RzBQTEdCN1pGSzIzVTcKYTY3cVNvUmJaeXEvMk1zVU5RQ3RFQTB0UHhicm5SZDMzVHlMTWIya0tZcjhvUW5ZUGloNjhMeG5LUG5pQ3E1TgpNQU9MamIybDJZa3hEOWs0N2V6QjBFWkVzZ3lKMWt3ekFQNHYrcTZWK01NVDRtZFpWdlJHMUl2dm1GNkJBRXVICjJhcmV4ai80ckk4MW5Yb0R0TExCZ01PS1ZlWWZYTFJudnJMeGFqYi9xOENtV3AwRXJtcGZMUHBmWnBIb1l1R2IKQytueGNNNjZVcnQ2VWhsOVdRNnQ1RFgxVzBpTWMzVjg1b0lIRzJVclBnRlo2R01nRzRvbWpyMGtseU1Ca3FHNQorMDdaZHprbVk4ZFR3ZCtWQUo4Q0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFBVHFDbmdIUDVrYW9lUGZOMkdEbU1pMHRVV1IKZTV2SEIrUEMrTjBtQVZTK2I5MXoyeDdTYm4zNmVZQ2NZRHI2VkxTRlAwQUVzTklDWXNPejlnemtvdVpqcHZOOQpzbVV1dFE0TlV1eVlsVXBEMWhVYkNpaWgvRnVtT0F2VVdOMlJldm1KaGJYVnM3SjhHeGJNMVlla3pNaGhSQ2NPCjVSOXdiNVh1ekltcG8xcHBERE5PcUhVVWhLWUJuV3NpMTdFUXVPQm5zUDVIVVpEMktDaFI4WVVpQXBZdWlRSEwKZ1l0QnlySUhOQitrZWtCVWowTmVuemcyMXJVMFJqenBpNEI0cjBVeGhFVE5JV2pkUDlvMElwcDBkUG5pYVhuOApUeWUvQVNCZ2xPUXhDb21zb0J5WkxBUGJBZktNdi85anlwT0M3TTc4SE14Mm9QdC9uRHEzcGdnUnF1az0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://kube-vip:16443
  name: kube
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01Ea3hOakF6TXpNd05Wb1hEVE13TURreE5EQXpNek13TlZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHBCCmF0T0E0NWhVQy9TWVkrWkMzR1prVGFCcTJ3d1laT2g3VE1DeFQxekxPQmJoNTl4Unh6S1FDa0hNNzkwbWEvdGsKWWlaWjNuRjg2aFRYZkpuTVJRSEFpSHNwTTBlejFkU1BPbFF1ajlUZm41a1F0MUpZY1dQR1VhNFp0T3owaWdyNwpaalhXLytIaENSRGR6VWtUSkhRWDdjQ2tsb01NbnF3SDAvUFN3bnQzVVdaQ2NMRGtEOWtEblpRQlpaaHJ1a0NkCm1HNlBDSExCMEh0TlFRREQyMkpQbVpxeDMycXdrRlpjbEJWdXI1cjkzZEFndWNPTkUvdjlTQSs0QTdrWmtFVDcKR2Znc2o2NDZ6WHcwU3Z6a2MxNGhnb2FIUmc5UlZPcUIvcUFYMzFTWWcrSW82Q2pJOTdQbmtENXZONHFLd0l0QQp2R2x3ZEJTY0JZemg5U0VhR0tNQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKWDU4eVZhamhjVEkzSmwvenBRNDFiZGtZKzcKWWcxU1NySlJSdXE3RlZjU0l5aVZRa2Y3MzYvMVJ4U2Z2WmdoTzJpVWt4VkpjOFZleFNablB0cVIvVFJmNEJINwpSQUdpVGlhTGIxbEw2cExRejBXUUlGbUtnREVBdU4ybHFiZGxaMkQ4dmhUaC95Zm1vUWxKS050T0FPTTd0ME5lClZkaXNJSUpiQ0F0d2RSYTVUNldvZmdFRXp5MTJKc3F5VitmbGYyRzdDanBqcUl2MXZUdlU5NllGZnJJZlVFbWwKQTNpcjRWeTFaRjFIMmNCUTZPSHVnRFBBaXkvUDJ1RmJvTjVYRlp2L0UrQ2wzVGtDdExrZ2FzaGdMTTF3dTFBago5Z3NzQlpodTgrb1YzNldXVGM4NjJ2cVRxRnlhRkNpNndLaEthc3RTaG9pZEtZNmJlcEM0Z1dyNEgzUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://ksphere-master:6443
  name: ksphere
contexts:
- context:
    cluster: kube
    user: kube-admin
  name: kube
- context:
    cluster: ksphere
    user: ksphere-admin
  name: ksphere
kind: Config
preferences: {}
users:
- name: ksphere-admin
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVycm9sZS1hZ2dyZWdhdGlvbi1jb250cm9sbGVyLXRva2VuLXBwYnc0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXJyb2xlLWFnZ3JlZ2F0aW9uLWNvbnRyb2xsZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzMDdmMzY1MC02ZjEwLTRiNTAtYTc5OS1mOGE0NWVmMGI2NWIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3RlcnJvbGUtYWdncmVnYXRpb24tY29udHJvbGxlciJ9.sVoGWzyodupMj_Jj2ltJBWmUNShd9VR4hwQx981-ZDXOx3B0l7s0G5Tk9L2CeRW4anrjSvIxbZPE5L4qnO-fNkpRkYGCmgE_PGpQxlOtYjnBC6e38spjioPn-OofWndgllnhbIw2DIDiuIUdBb94HTcAhqSekcwSa1Nw0C4-rUkrWJ5cZMbUSVa1L-EpbuJpIGvUlo9CEhJqafvFsIrXKYdXr79IK2AUxDe39M4bTgWnZ_xoNn0pdk-391HrFzIRjfSlbOSnwZZIyPJpvKOYeRO7G8ZbdjlGsjtoOLQcSZe78ayYfnnSPSYIdsFPFa0uCGyh8ThiTTckOe5KbTgLzA
- name: kube-admin
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVycm9sZS1hZ2dyZWdhdGlvbi1jb250cm9sbGVyLXRva2VuLXgycXg1Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXJyb2xlLWFnZ3JlZ2F0aW9uLWNvbnRyb2xsZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI5OGUzZGVkOC05MTlkLTExZWEtOTkxMi05MmJiMTdlNGE3MWYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3RlcnJvbGUtYWdncmVnYXRpb24tY29udHJvbGxlciJ9.l6sUUMf6SA4a_djezN9dcmcHN3iFUaBvWQ_baO4a5B4MxarlZjtsLe3HfQLhBI19O674bO8zHt4mkNMl1X_E7O6aPmkkwlTYMYAxGtkDdq1t80xtRJ2ZZdagwOo_6Pokf5KxSZZaowo05Kz0g3E9Y90zXLfqT26XXdWDttZuMabcYCt5T4cEWjoqeD24bkPBvrbLItAjzSxmc0r9Q-vcYk5h3ZhOHAdonMDTXoP3s7wGaZnPmYTdrU0qyDKcCVy8MDTJBeUl-enzmkr8rc2s0P59sjyMgAV66EPM88NTbJBx-5TepnF3H4xaEi89IyUcbBeqnUHNwvmjOOls87QG0w

这里需要注意的是user token的获取，可以通过一下命令获取到

kubectl describe secret -n kube-system $(kubectl get secret -n kube-system | awk '/clusterrole-aggregation-controller-token/ {print $1}') | awk '/^token:/{print $2}

随后我们就可以用kubefedctl添加我们的集群了

kubefedctl join kube --cluster-context kube --host-cluster-context kube --v=2
kubefedctl join ksphere --cluster-context ksphere --host-cluster-context kube --v=2

查看一下状态为true则表示添加成功

kubectl get kubefedclusters -n kube-federation-system
NAME      AGE   READY
ksphere   15m   True
kube      30m   True

现在我们来测试一下同时部署一个deployment到两个集群中去
首先我们需要创建一个FederatedNamespace

kind: FederatedNamespace
metadata:
  name: test-namespace
  namespace: test-namespace
spec:
  placement:
    clusters:
    - name: kube
    - name: ksphere

这里要注意的是需要提前在两个集群中创建好test-namespace
随后我们只需要创建FederatedDeployment就能在连个集群中同时部署服务了

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: test-deployment
  namespace: test-namespace
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - image: nginx
            name: nginx
  placement:
    clusters:
    - name: kube
    - name: ksphere

最后在两个集群中分别查看是否创建成功

kubectl get pod -n test-namespace
NAME                               READY   STATUS    RESTARTS   AGE
test-deployment-7b4f94bb64-5d6rz   1/1     Running   0          4m36s
test-deployment-7b4f94bb64-bbgr8   1/1     Running   0          4m36s
test-deployment-7b4f94bb64-hj7nr   1/1     Running   0          4m36s

kubectl --context=ksphere-admin@ksphere get pod -n test-namespace
NAME                               READY   STATUS    RESTARTS   AGE
test-deployment-69bddffb77-c9lxd   1/1     Running   0          5m21s
test-deployment-69bddffb77-d8qqw   1/1     Running   0          5m21s
test-deployment-69bddffb77-kv2p6   1/1     Running   0          5m21s