安装kafka
安装zookeeper
安装zookeeper可以采用k8s的StatefulSet方式,非常方便,在安装kafka过程中遇到很多问题,但是zookeeper非常文档
apiVersion: v1
kind: Service
metadata:
name: zk-hs
labels:
app: zk
namespace: kafka
spec:
selector:
app: zk
clusterIP: None
ports:
- name: server
port: 2888
- name: leader-election
port: 3888
---
apiVersion: v1
kind: Service
metadata:
name: zk-cs
labels:
app: zk
namespace: kafka
spec:
selector:
app: zk
type: NodePort
ports:
- name: client
port: 2181
nodePort: 31811
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
namespace: kafka
spec:
selector:
matchLabels:
app: zk
maxUnavailable: 1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: zk
namespace: kafka
spec:
serviceName: "zk-hs"
replicas: 3 # by default is 1
selector:
matchLabels:
app: zk # has to match .spec.template.metadata.labels
updateStrategy:
type: RollingUpdate
podManagementPolicy: Parallel
template:
metadata:
labels:
app: zk # has to match .spec.selector.matchLabels
spec:
containers:
- name: zk
imagePullPolicy: Always
image: leolee32/kubernetes-library:kubernetes-zookeeper1.0-3.4.10
resources:
requests:
memory: "2G"
cpu: "1"
ports:
- containerPort: 2181
name: client
- containerPort: 2888
name: server
- containerPort: 3888
name: leader-election
command:
- sh
- -c
- "start-zookeeper \
--servers=3 \
--data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \
--client_port=2181 \
--election_port=3888 \
--server_port=2888 \
--tick_time=2000 \
--init_limit=10 \
--sync_limit=5 \
--heap=4G \
--max_client_cnxns=60 \
--snap_retain_count=3 \
--purge_interval=12 \
--max_session_timeout=40000 \
--min_session_timeout=4000 \
--log_level=INFO"
readinessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
livenessProbe:
exec:
command:
- sh
- -c
- "zookeeper-ready 2181"
initialDelaySeconds: 10
timeoutSeconds: 5
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper
volumeClaimTemplates:
- metadata:
name: datadir
annotations:
volume.beta.kubernetes.io/storage-class: managed-nfs-storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
安装kafka
采用helm的方式安装kafka最简单,真的不要太方便了,只需要简单的配置即可
借鉴以下文章的方式,镜像采用的是bitnami/kafka
https://artifacthub.io/packages/helm/bitnami/kafka
https://www.shuzhiduo.com/A/n2d9qvb0dD/
下载kafka的chart：bitnami/kafka
下载成功后当前目录有个压缩包：kafka-0.20.8.tgz，解压：tar -zxvf kafka-0.20.8.tgz
进入解压后的kafka目录，编辑values.yaml文件，下面是具体的修改点：
校验安装是否成功
- kubectl exec -ti zk-1 -n kafka -- zkCli.sh -server zk-cs:2181
出现
Welcome to ZooKeeper!
JLine support is enabled
即安装成功
- kubectl exec -ti kafka-0 -n kafka -- /bin/bash
登录到pod里面
执行获取topic : kafka-topics.sh --list --zookeeper zk-cs:2181
kafka-topics.sh --list --zookeeper 10.45.2.13:31811
创建topic : kafka-topics.sh --create --topic testtopic --zookeeper zk-cs:2181 --partitions 1 --replication-factor 1
执行获取topic : kafka-topics.sh --list --zookeeper zk-cs:2181
往topic中生产数据: kafka-console-producer.sh --broker-list nodeport:31092 --topic wisecotest
kafka-2.kafka-cs.kafka.svc.cluster.local:31092
// 对外暴露的地址端口10.233.137.116:9092 32092
往topic中生产数据: kafka-console-producer.sh --broker-list kafka-cs:9092 --topic wisecotest
从topic中获取数据: kafka-console-consumer.sh --bootstrap-server kafka-2.kafka-cs.kafka.svc.cluster.local:31092 --topic wisecotest --from-beginning
安装问题解答
- 出现: 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) didn't match pod anti-affinity rules.
原因是设定了亲和性,导致每个节点上只能安装一个类型,master节点又不允许安装其他的节点,导致失败
- 注意事项,采用stateful set的方式安装kafka在集群内部是可以访问的,但是集群外是不能访问
stateful set 安装kafka后
通过service对外暴露的ip:port能访问到zookeeper,zookeeper返回的是注册的集群内部的地址(这里应该可以修改),导致当前链接不上返回的地址
最后采用了helm的安装方式,等于是对外暴露了三个service,每个service对应pod
然后在zookeeper中看到的注册地址中有个external的存在
