环境信息
- Kubernetes：v1.20.6
- StorageClass：csi-udisk-rssd
- Helm：v3.5.2
- nginx-ingress: 0.47.0
Cortex v1.10.0 依赖
- consul (推荐)
- memcached (可选)
- 如果没有兼容AWS S3协议的对象存储，可以考虑使用minio自建
本安装前准备工作
1. 申请域名证书
可以使用 https://keymanager.org/ 来申请 Let’s Encrypt 提供的免费泛域名证书
2. 创建域名证书 secret
kubectl create namespace cortex
kubectl delete secret tls-cortex-secret -n cortex
kubectl create secret tls tls-cortex-secret \
--cert=onwalk.net.crt --key=onwalk.net.key -n cortex
3. 同步海外源镜像
在国内环境部署应用，经常因为获取国外源站容器镜像超时，导致部署失败，可以提前将容器镜像同步到本地镜像仓库中，以自有镜像仓库harbor.onwalk.net/pts为例，login仓库，执行命令： docker login -u admin -p 'PWxxxxxx' harbor.onwalk.net/pts, 需要同步镜像列表如下：
for IMG in \
quay.io/cortexproject/cortex:v1.10.0 \
docker.io/bitnami/consul:1.10.1-debian-10-r17 \
docker.io/bitnami/memcached:1.6.10-debian-10-r0 \
docker.io/bitnami/minio:2021.6.17-debian-10-r38
do
echo $IMG
docker pull $IMG
docker tag $IMG harbor.onwalk.net/pts/$IMG
docker push harbor.onwalk.net/pts/$IMG
done
关于docker pull tag push 操作可以参考：
- https://docs.docker.com/engine/reference/commandline/pull/
- https://docs.docker.com/engine/reference/commandline/tag/
- https://docs.docker.com/engine/reference/commandline/push/
4. 创建 imagePullSecrets
创建容器集群访问仓库地址 harbor.onwalk.net/pts，拉取镜像需要的 secret
kubectl create namespace cortex
kubectl create secret docker-registry registry-harbor-secret -n cortex \
--docker-server=harbor.onwalk.net/pts \
--docker-username='admin' \
--docker-password='PWxxxxxx'
5. 添加 Helm仓库
这里选用cortex官方和bitnami提供的chart仓库
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add cortex https://cortexproject.github.io/cortex-helm-chart
helm repo update
安装依赖组建
1. 安装 consul
cat > consul-values.yaml << EOF
global:
imageRegistry: "harbor.onwalk.net/pts/docker.io"
imagePullSecrets:
- registry-harbor-secret
storageClass: "csi-udisk-rssd"
clusterDomain: admin.local
EOF
helm upgrade --install consul bitnami/consul -n cortex -f consul-values.yaml
关键配置参数说明: clusterDomain 这里要和所在K8S集群的配置一致
安装后如需要重启：kubectl rollout restart -n cortex statefulset.apps/consul
consul的服务端口：consul-headless:8500
2. 安装 memcached
cat > memcached-values.yaml << EOF
global:
nameOverride: consul
imageRegistry: "harbor.onwalk.net/pts/docker.io"
imagePullSecrets:
- registry-harbor-secret
storageClass: "csi-udisk-rssd"
clusterDomain: admin.local
EOF
helm upgrade --install memcached bitnami/memcached -n cortex \
-f memcached-values.yaml
关键配置参数说明: clusterDomain 这里要和所在K8S集群的配置一致
memcached的服务端口：memcached:11211
3. 安装 minio
生成访问对象存储的Key
accessKey=`cat /dev/random | head -c20 | base64`
secretKey=`cat /dev/random | head -c50 | base64`
记录下 accessKey 和 secretKey 部署minio 和访问 minio 提供的对象存储都会需要
cat > minio-values.yaml << EOF
global:
imageRegistry: "harbor.onwalk.net/pts/docker.io"
imagePullSecrets:
- registry-harbor-secret
storageClass: "csi-udisk-rssd"
minio:
accessKey: $accessKey
secretKey: $secretKey
clusterDomain: admin.local
mode: distributed
ingress:
enabled: true
certManager: false
hostname: cortex-minio.onwalk.net
extraTls:
- hosts:
- cortex-minio.onwalk.net
secretName: tls-cortex-secret
EOF
helm delete minio -n cortex
helm upgrade --install minio bitnami/minio -n cortex -f minio-values.yaml
关键配置参数说明: clusterDomain 这里要和所在K8S集群的配置一致
minio的服务端口：minio:9000
minio server 上创建 bucket 操作参考
docker run -it --entrypoint=/bin/sh minio/mc
mc alias set minio http://minio:9000 minio accesskeyxxxx secretkeyxxx
mc mb minio/cortex-tsdb
mc mb minio/cortex-ruler
mc mb minio/cortex-alertmanager
完成 Cortex 安装
cat > cortex-values.yaml << EOF
image:
repository: harbor.onwalk.net/pts/quay.io/cortexproject/cortex
tag: v1.10.0
pullSecrets:
- myRegistrKeySecretName
clusterDomain: admin.local
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
hosts:
- host: cortex-gateway.onwalk.net
paths:
- /
tls:
- secretName: tls-cortex-secret
hosts:
- cortex-gateway.onwalk.net
nginx:
enabled: true
replicas: 2
http_listen_port: 80
config:
dnsResolver: kube-dns.kube-system.svc.admin.local
ingester:
replicas: 3
persistentVolume:
enabled: true
accessModes:
- ReadWriteOnce
size: 10Gi
storageClass: "csi-udisk-rssd"
compactor:
enabled: true
replicas: 1
persistentVolume:
enabled: true
accessModes:
- ReadWriteOnce
size: 10Gi
storageClass: "csi-udisk-rssd"
store_gateway:
replicas: 1
persistentVolume:
enabled: true
accessModes:
- ReadWriteOnce
size: 10Gi
storageClass: "csi-udisk-rssd"
config:
auth_enabled: false
distributor:
shard_by_all_labels: true
pool:
health_check_ingesters: true
instance_limits:
max_ingestion_rate: 0
max_inflight_push_requests: 0
ingester_client:
grpc_client_config:
# Configure the client to allow messages up to 100MB.
max_recv_msg_size: 104857600
max_send_msg_size: 104857600
grpc_compression: gzip
ingester:
lifecycler:
# We want to start immediately.
join_after: 0
final_sleep: 0s
num_tokens: 512
ring:
kvstore:
store: consul
consul:
host: consul-headless:8500
replication_factor: 1
instance_limits:
max_ingestion_rate: 0
max_tenants: 0
max_series: 0
max_inflight_push_requests: 0
querier:
query_ingesters_within: 3h
# Used when the blocks sharding is disabled.
store_gateway_addresses: store-gateway-1:9008,store-gateway-2:9009
blocks_storage:
backend: s3
tsdb:
dir: /data/cortex-tsdb-ingester
ship_interval: 1m
block_ranges_period: [ 2h ]
retention_period: 3h
max_exemplars: 50000
bucket_store:
sync_dir: /data/cortex-tsdb-querier
consistency_delay: 5s
index_cache:
backend: memcached
memcached:
addresses: memcached:11211
chunks_cache:
backend: memcached
memcached:
addresses: memcached:11211
metadata_cache:
backend: memcached
memcached:
addresses: memcached:11211
s3:
endpoint: minio:9000
bucket_name: cortex-tsdb
access_key_id: $accessKey
secret_access_key: $secretKey
insecure: true
ruler:
enable_api: true
enable_sharding: true
ring:
heartbeat_period: 5s
heartbeat_timeout: 15s
kvstore:
store: consul
consul:
host: consul-headless:8500
alertmanager_url: http://alertmanager-1:8031/alertmanager,http://alertmanager-2:8032/alertmanager,http://alertmanager-3:8033/alertmanager
enable_alertmanager_v2: false
ruler_storage:
backend: s3
s3:
bucket_name: cortex-ruler
endpoint: minio:9000
access_key_id: $accessKey
secret_access_key: $secretKey
insecure: true
alertmanager:
enable_api: true
sharding_enabled: true
sharding_ring:
replication_factor: 3
heartbeat_period: 5s
heartbeat_timeout: 15s
kvstore:
store: consul
consul:
host: consul-headless:8500
alertmanager_storage:
backend: s3
s3:
bucket_name: cortex-alertmanager
endpoint: minio:9000
access_key_id: $accessKey
secret_access_key: $secretKey
insecure: true
storage:
engine: blocks
compactor:
compaction_interval: 30s
data_dir: /data/cortex-compactor
consistency_delay: 1m
sharding_enabled: true
cleanup_interval: 1m
tenant_cleanup_delay: 1m
sharding_ring:
kvstore:
store: consul
consul:
host: consul-headless:8500
store_gateway:
sharding_enabled: true
sharding_ring:
replication_factor: 1
heartbeat_period: 5s
heartbeat_timeout: 15s
kvstore:
store: consul
consul:
host: consul-headless:8500
frontend:
query_stats_enabled: true
frontend_worker:
frontend_address: "query-frontend:9007"
match_max_concurrent: true
# scheduler_address: "query-scheduler:9012"
query_range:
split_queries_by_interval: 24h
limits:
# Limit max query time range to 31d
max_query_length: 744h
EOF
helm upgrade --install cortex cortex/cortex -n cortex -f cortex-values.yaml
Cortex部署完毕后，
- Grafana 的数据源接入点：https://cortex-gateway.onwalk.net/api/prom
- Prometheus的远端存储 : https://cortex-gateway.onwalk.net/api/prom/push
压测工具
Prombench
