【prometheus-operator】k8s监控redis

1、准备exporter

https://github.com/oliver006/redis_exporter

oliver006-redis_exporter-amd64.tar

# 安装镜像

docker load -i oliver006-redis_exporter-amd64.tar

# 上传镜像
docker tag oliver006/redis_exporter ip/monitor/redis_exporter:latest
docker push ip/monitor/redis_exporter:latest

2、部署exporter

kind: Deployment
apiVersion: apps/v1
metadata:name: redis-exporternamespace: devopslabels:app: redis-exporterannotations:deployment.kubernetes.io/revision: '1'kubesphere.io/creator: adminkubesphere.io/description: 监控使用
spec:replicas: 1selector:matchLabels:app: redis-exportertemplate:metadata:labels:app: redis-exporterannotations:kubesphere.io/containerSecrets: '{"container-zsbagq":"harbor"}'spec:containers:- name: container-zsbagqimage: 'ip:8888/monitor/redis_exporter:latest'args:- '-redis.addr'- 'redis-hip-redis-cluster-headless.hip-mid:6379'- '-redis.password'- '111111'ports:- name: tcp-9121containerPort: 9121protocol: TCPimagePullPolicy: IfNotPresentrestartPolicy: AlwaysimagePullSecrets:- name: harbor

3、创建serviceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:labels:app: redis-exportername: redis-exporternamespace: devops
spec:endpoints:- interval: 30sport: tcp-9121params:target:- redis-hip-redis-cluster-headless.hip-mid:6379relabelings:- sourceLabels: [__param_target]targetLabel: instanceselector:matchLabels:app: redis-exporternamespaceSelector:matchNames:- devops

创建成功，访问prometheus targets界面，显示如图

4、告警规则

获取规则：

kubectl get prometheusrules prometheus-k8s-rules -n {namespace} -o yaml > prometheus-rule.yaml

  - name: redis_altersrules:- alert: redis_exporter_statusannotations:description: job-{{$labels.job}}, instance-{{ $labels.instance }} 监控exporter下线，请检查！summary: 监控exporter下线预警expr: up{namespace="monitor",instance=~".*:9122"} == 0for: 3mlabels:severity: emergency- alert: redis_statusannotations:description: job-{{$labels.job}}, instance-{{ $labels.instance }} 节点下线，请检查！summary: 节点下线预警expr: redis_up{namespace="monitor"} == 0for: 3mlabels:severity: emergency- alert: redis_memory_usageannotations:description: job-{{$labels.job}}, instance-{{ $labels.instance }} 内存超过90%，请检查！summary: 内存预警expr: redis_memory_used_bytes{namespace="monitor"} / redis_memory_max_bytes{namespace="monitor"}* 100 > 90for: 15slabels:severity: emergency- alert: redis_rejected_connectionannotations:description: job-{{$labels.job}}, instance-{{ $labels.instance }} 一些连接被拒绝，请检查！summary: 连接被拒绝expr: increase(redis_rejected_connections_total{namespace="monitor"}[1m]) >0for: 1mlabels:severity: emergency- alert: redis_max_clientannotations:description: job-{{$labels.job}}, instance-{{ $labels.instance }} 连接总数达到总量的90%，请检查！summary: 连接数过多expr: redis_connected_clients{namespace="monitor"} > redis_config_maxclients{namespace="monitor"}* 0.90for: 15slabels:severity: emergency