[k8s]metricbeat的kubernetes模块kube-metric模块

正确姿势启动metricbeat

metricbeat.modules:
- module: systemmetricsets:- cpu- filesystem- memory- network- processenabled: trueperiod: 10sprocesses: ['.*']cpu_ticks: falseoutput.elasticsearch:hosts: ["http://192.168.x.x:9200"]setup.template.name: "metricbeat"
setup.template.fields: "fields.yml"
setup.template.overwrite: true
setup.template.settings:index.number_of_shards: 1index.number_of_replicas: 1setup.kibana.host: "192.168.x.x:5601"
setup.dashboards.enabled: true

./metricbeat -e -c metricbeat.yml -d "publish"

metricbeat的Kubernetes插件

metricbeat.modules:
# Node metrics, from kubelet:
- module: kubernetesmetricsets:- node- system- pod- container- volumeperiod: 10shosts: ["192.168.x.132:10255","192.168.x.133:10255"]# State metrics from kube-state-metrics service:  #这部分需要kube-state-metrics支撑(Nodes/deploy个数),因此集群里需要安装kube-state-metrics组件.默认情况下这里没有的话图中上半部分没有数据
- module: kubernetesenabled: false  ## 这里一定要是truemetricsets:- state_node- state_deployment- state_replicaset- state_pod- state_containerperiod: 10shosts: ["192.168.x.x:8080"]# Kubernetes events
- module: kubernetesenabled: falsemetricsets:- eventoutput.elasticsearch:hosts: ["http://192.168.x.x:9200"]

既然需要kube-state-metrics了,那就研究下吧.

监控思路

• vm思路:遵从大道至简的原则
  

• 容器监控思路:
  pod的:
  top cpu
  top mem

• 别人家的监控-只能流口水了
  

kube-state-metrics提供监控k8s集群的metric

kube-state-metrics本质上是提供了k8s集群的metric
部署参考:
https://github.com/kubernetes/kube-state-metrics

克隆代码: create这几个即可

访问:

报错解决:
kubectl -n kube-system logs -f kube-state-metrics-215312065-1vzg2 -c kube-state-metrics
时候发现一些api错误

解决: 给apiserver启动添加这个参数: 它github上有说

--runtime-config=batch/v2alpha1=true

即目前完整的apiserver参数如下

 kube-apiserver     --service-cluster-ip-range=10.254.0.0/16     --etcd-servers=http://127.0.0.1:2379     --insecure-bind-address=0.0.0.0     --admission-control=ServiceAccount     --service-account-key-file=/root/ssl/ca.key --client-ca-file=/root/ssl/ca.crt    --tls-cert-file=/root/ssl/server.crt --tls-private-key-file=/root/ssl/server.key --allow-privileged=true --runtime-config=batch/v2alpha1=true --v=2

k8s插件监控集群效果图

vm监控效果图

贴上kube-metric的yaml

参考: https://github.com/kubernetes/kube-state-metrics

修改点:

• deploy的api版本

• svc为nodeport模式

• 镜像拉取

docker pull lanny/gcr.io_google_containers_kube-state-metrics_v1.1.0
dokcer pull lanny/gcr.io_google_containers_addon-resizer_1.0
docker tag lanny/gcr.io_google_containers_kube-state-metrics_v1.1.0 gcr.io/google_containers/kube-state-metrics:v1.1.0 
docker tag lanny/gcr.io_google_containers_addon-resizer_1.0 gcr.io/google_containers/addon-resizer:1.0 

$ cat kube-state-metrics-deployment.yaml 
apiVersion: apps/v1beta1 # for versions before 1.8.0 use apps/v1beta1  ## 这里默认是v1beat1,我用的是1.7.8,因此apiVersion: apps/v1beta2是不适用的.
kind: Deployment
metadata:name: kube-state-metricsnamespace: kube-system
spec:selector:matchLabels:k8s-app: kube-state-metricsreplicas: 1template:metadata:labels:k8s-app: kube-state-metricsspec:serviceAccountName: kube-state-metricscontainers:- name: kube-state-metricsimage: quay.io/coreos/kube-state-metrics:v1.1.0ports:- name: http-metricscontainerPort: 8080readinessProbe:httpGet:path: /healthzport: 8080initialDelaySeconds: 5timeoutSeconds: 5- name: addon-resizerimage: gcr.io/google_containers/addon-resizer:1.0resources:limits:cpu: 100mmemory: 30Mirequests:cpu: 100mmemory: 30Mienv:- name: MY_POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: MY_POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespacecommand:- /pod_nanny- --container=kube-state-metrics- --cpu=100m- --extra-cpu=1m- --memory=100Mi- --extra-memory=2Mi- --threshold=5- --deployment=kube-state-metrics

$ cat kube-state-metrics-service.yaml 
apiVersion: v1
kind: Service
metadata:name: kube-state-metricsnamespace: kube-systemlabels:k8s-app: kube-state-metricsannotations:prometheus.io/scrape: 'true'
spec:type: NodePort ## 这里需要改下,外界通过nodeport方式访问metric apiports:- name: http-metricsport: 8080targetPort: http-metricsprotocol: TCPselector:k8s-app: kube-state-metrics

$ cat kube-state-metrics-service-account.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:name: kube-state-metricsnamespace: kube-system