基于cobra开发的k8s命令行管理工具k8s-manager
- 如果觉得好用，麻烦给个Star!
- 通用配置
- 1 node 分析所有node的资源情况
- 2 analysis 分析Node节点上的资源使用构成
- 3 image 获取指定namespace的所有镜像地址
- 4 resource 获取指定namespace的所有limit 与 Requests大小
- 5 top 获取指定namespace的资源使用情况
- 常用的使用套路
- 1 k8s节点异常卡顿，容器频繁重启
- 2 优化limit requests
如果觉得好用，麻烦给个Star!
项目介绍：Github地址 ，代码已开源
- https://github.com/qinlang258/cobra-k8s-manager/tree/main
该命令有以下几个功能：analysis，image，node，resource，top
使用 方法：下载项目githun空间下,使用k8s-manager即可，无需进行额外配置。
longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.Usage:k8s-manager [flags]k8s-manager [command]Available Commands:analysis 分析某一节点的资源使用情况completion Generate the autocompletion script for the specified shellhelp Help about any commandimage 获取镜像信息node 获取节点的资源信息resource 获取pod资源的相关 Limit与Resource信息top 获取容器的实际使用资源开销Flags:--analysis string 请输入想分析的Node名字-h, --help help for k8s-manager--kubeconfig string 请输入 kubeconfig的文件路径 (default "/root/.kube/config")--name string 请输入资源的name信息-n, --namespace string 请输入 namespace空间，如果不填写则输出所有空间下的镜像 (default "all")--node string 请输入想要查询的Node名字--workload string 请输入 workload的种类，如果不填写输出所有类型的镜像 (default "all")Use "k8s-manager [command] --help" for more information about a command.
通用配置
- 所有命令均可附带 --kubeconfig指定配置文件
./k8s-manager --kubeconfig <指定使用的k8s配置文件>
1 node 分析所有node的资源情况
示例代码
1 获取所有节点的资源信息
./k8s-manager node
示例代码
root@k8s:/usr/local/cobra-k8s-manager# k8s-manager node
+--------+----------------+--------------------+-------------+---------------------------+-------------+-----------+-------------------------+--------------+------------+--------------------------+
| 节点名 | 节点IP | OS镜像 | KUBELET版本 | CONTAINER RUNTIME VERSION | 已使用的CPU | CPU总大小 | CPU使用占服务器的百分比 | 已使用的内存 | 内存总大小 | 内存使用占服务器的百分比 |
+--------+----------------+--------------------+-------------+---------------------------+-------------+-----------+-------------------------+--------------+------------+--------------------------+
| k8s | 192.168.44.134 | Ubuntu 22.04.4 LTS | v1.26.7 | containerd://1.6.8 | 363.00m | 8000m | 4.54% | 6585.25Mi | 7901.89Mi | 83.34% |
+--------+----------------+--------------------+-------------+---------------------------+-------------+-----------+-------------------------+--------------+------------+--------------------------+
2 analysis 分析Node节点上的资源使用构成
1 分析指定节点上的所有容器的资源开销
./k8s-manager analysis --node <节点名>
示例代码
root@k8s:/usr/local/cobra-k8s-manager# k8s-manager analysis --node k8s
E1222 05:55:41.502661 75816 analysis.go:46] context.BackgroundError fetching metrics for pod ingress-nginx-admission-create-r24ff: pods "ingress-nginx-admission-create-r24ff" not found
E1222 05:55:41.504124 75816 analysis.go:46] context.BackgroundError fetching metrics for pod ingress-nginx-admission-patch-w5gnn: pods "ingress-nginx-admission-patch-w5gnn" not found
+--------+---------------+--------------------------------------------------+---------------------------------+-----------------+-------------------------+------------------+--------------------------+
| 节点名 | NAMESPACE | POD NAME | 容器名 | 当前已使用的CPU | CPU使用占服务器的百分比 | 当前已使用的内存 | 内存使用占服务器的百分比 |
+--------+---------------+--------------------------------------------------+---------------------------------+-----------------+-------------------------+------------------+--------------------------+
| k8s | kube-system | kube-apiserver-k8s | kube-apiserver | 35.00m | 0.44% | 492.68m | 6.24% |
| k8s | monitoring | prometheus-k8s-0 | prometheus | 10.00m | 0.12% | 383.23m | 4.85% |
| k8s | monitoring | prometheus-k8s-1 | prometheus | 17.00m | 0.21% | 342.59m | 4.34% |
| k8s | ingress-nginx | ingress-nginx-controller-f87d69b54-t8kd8 | controller | 1.00m | 0.01% | 166.57m | 2.11% |
| k8s | monitoring | grafana-9bb74449d-8m8xl | grafana | 4.00m | 0.05% | 134.36m | 1.70% |
| k8s | kube-system | calico-node-7zbfk | calico-node | 24.00m | 0.30% | 102.68m | 1.30% |
| k8s | kube-system | etcd-k8s | etcd | 18.00m | 0.22% | 86.12m | 1.09% |
| k8s | kube-system | kube-controller-manager-k8s | kube-controller-manager | 11.00m | 0.14% | 85.29m | 1.08% |
| k8s | monitoring | prometheus-adapter-854d95bc45-pvfh7 | prometheus-adapter | 3.00m | 0.04% | 58.35m | 0.74% |
| k8s | monitoring | prometheus-operator-57cf88fbcb-wks8t | prometheus-operator | 1.00m | 0.01% | 45.33m | 0.57% |
| k8s | kube-system | coredns-5bbd96d687-x9qmp | coredns | 2.00m | 0.03% | 43.17m | 0.55% |
| k8s | kube-system | metrics-server-7d5c696976-wlms5 | metrics-server | 3.00m | 0.04% | 38.56m | 0.49% |
| k8s | kube-system | kube-scheduler-k8s | kube-scheduler | 2.00m | 0.03% | 35.71m | 0.45% |
| k8s | monitoring | node-exporter-6cqz8 | kube-rbac-proxy | 1.00m | 0.01% | 34.83m | 0.44% |
| k8s | monitoring | kube-state-metrics-79996cfcc5-5286s | kube-state-metrics | 1.00m | 0.01% | 33.56m | 0.42% |
| k8s | kube-system | calico-kube-controllers-57b57c56f-d5p6n | calico-kube-controllers | 1.00m | 0.01% | 31.89m | 0.40% |
| k8s | monitoring | prometheus-adapter-854d95bc45-tz822 | prometheus-adapter | 3.00m | 0.04% | 30.73m | 0.39% |
| k8s | kube-system | kube-proxy-nb9tq | kube-proxy | 1.00m | 0.01% | 29.99m | 0.38% |
| k8s | monitoring | alertmanager-main-1 | alertmanager | 2.00m | 0.03% | 29.04m | 0.37% |
| k8s | kube-system | coredns-5bbd96d687-thl59 | coredns | 2.00m | 0.03% | 29.00m | 0.37% |
| k8s | monitoring | alertmanager-main-0 | alertmanager | 2.00m | 0.03% | 27.72m | 0.35% |
| k8s | monitoring | alertmanager-main-2 | alertmanager | 2.00m | 0.03% | 27.71m | 0.35% |
| k8s | monitoring | alertmanager-main-0 | config-reloader | 0.00m | 0.00% | 22.56m | 0.29% |
| k8s | monitoring | blackbox-exporter-59dddb7bb6-8lp69 | blackbox-exporter | 1.00m | 0.01% | 21.18m | 0.27% |
| k8s | monitoring | prometheus-k8s-1 | config-reloader | 1.00m | 0.01% | 20.79m | 0.26% |
| k8s | monitoring | alertmanager-main-1 | config-reloader | 1.00m | 0.01% | 20.40m | 0.26% |
| k8s | monitoring | prometheus-k8s-0 | config-reloader | 0.00m | 0.00% | 20.16m | 0.26% |
| k8s | monitoring | alertmanager-main-2 | config-reloader | 1.00m | 0.01% | 18.77m | 0.24% |
| k8s | nfs | nfs-subdir-external-provisioner-65664b8954-qrs2q | nfs-subdir-external-provisioner | 1.00m | 0.01% | 17.30m | 0.22% |
| k8s | monitoring | node-exporter-6cqz8 | node-exporter | 13.00m | 0.16% | 15.90m | 0.20% |
| k8s | monitoring | kube-state-metrics-79996cfcc5-5286s | kube-rbac-proxy-main | 1.00m | 0.01% | 9.80m | 0.12% |
| k8s | monitoring | prometheus-operator-57cf88fbcb-wks8t | kube-rbac-proxy | 1.00m | 0.01% | 9.67m | 0.12% |
| k8s | monitoring | blackbox-exporter-59dddb7bb6-8lp69 | kube-rbac-proxy | 1.00m | 0.01% | 9.62m | 0.12% |
| k8s | monitoring | kube-state-metrics-79996cfcc5-5286s | kube-rbac-proxy-self | 1.00m | 0.01% | 9.15m | 0.12% |
| k8s | monitoring | blackbox-exporter-59dddb7bb6-8lp69 | module-configmap-reloader | 0.00m | 0.00% | 4.00m | 0.05% |
+--------+---------------+--------------------------------------------------+---------------------------------+-----------------+-------------------------+------------------+--------------------------+
3 image 获取指定namespace的所有镜像地址
示例代码
1 获取所有namespace的镜像地址
./k8s-manager image
2 获取指定namespace的镜像地址
./k8s-manager image -n <namespace>
示例代码
root@k8s:/usr/local/cobra-k8s-manager# k8s-manager image
+---------------+--------------+---------------------------------+---------------------------------+---------------------------------------------------------------------------------+
| NAMESPACE | 资源类型 | 资源名 | 容器名 | 镜像地址 |
+---------------+--------------+---------------------------------+---------------------------------+---------------------------------------------------------------------------------+
| ingress-nginx | deployment | ingress-nginx-controller | controller | registry.cn-zhangjiakou.aliyuncs.com/jcrose-k8s/ingress-nginx-controller:v1.7.0 |
| kube-system | deployment | calico-kube-controllers | calico-kube-controllers | docker.io/calico/kube-controllers:v3.25.0 |
| kube-system | deployment | coredns | coredns | registry.aliyuncs.com/google_containers/coredns:v1.9.3 |
| kube-system | deployment | metrics-server | metrics-server | k8s.dockerproxy.net/metrics-server/metrics-server:v0.7.2 |
| kube-system | daemonsets | calico-node | calico-node | docker.io/calico/node:v3.25.0 |
| kube-system | daemonsets | kube-proxy | kube-proxy | registry.aliyuncs.com/google_containers/kube-proxy:v1.26.7 |
| monitoring | deployment | blackbox-exporter | blackbox-exporter | quay.io/prometheus/blackbox-exporter:v0.24.0 |
| monitoring | deployment | blackbox-exporter | module-configmap-reloader | jimmidyson/configmap-reload:v0.5.0 |
| monitoring | deployment | blackbox-exporter | kube-rbac-proxy | quay.io/brancz/kube-rbac-proxy:v0.14.2 |
| monitoring | deployment | grafana | grafana | grafana/grafana:9.5.3 |
| monitoring | deployment | kube-state-metrics | kube-state-metrics | bitnami/kube-state-metrics:2.9.2 |
| monitoring | deployment | kube-state-metrics | kube-rbac-proxy-main | quay.io/brancz/kube-rbac-proxy:v0.14.2 |
| monitoring | deployment | kube-state-metrics | kube-rbac-proxy-self | quay.io/brancz/kube-rbac-proxy:v0.14.2 |
| monitoring | deployment | prometheus-adapter | prometheus-adapter | xuxiaoweicomcn/prometheus-adapter:v0.11.1 |
| monitoring | deployment | prometheus-operator | prometheus-operator | quay.io/prometheus-operator/prometheus-operator:v0.67.1 |
| monitoring | deployment | prometheus-operator | kube-rbac-proxy | quay.io/brancz/kube-rbac-proxy:v0.14.2 |
| monitoring | statefulsets | alertmanager-main | alertmanager | quay.io/prometheus/alertmanager:v0.26.0 |
| monitoring | statefulsets | alertmanager-main | config-reloader | quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1 |
| monitoring | statefulsets | prometheus-k8s | prometheus | quay.io/prometheus/prometheus:v2.46.0 |
| monitoring | statefulsets | prometheus-k8s | config-reloader | quay.io/prometheus-operator/prometheus-config-reloader:v0.67.1 |
| monitoring | daemonsets | node-exporter | node-exporter | quay.io/prometheus/node-exporter:v1.6.1 |
| monitoring | daemonsets | node-exporter | kube-rbac-proxy | quay.io/brancz/kube-rbac-proxy:v0.14.2 |
| nfs | deployment | nfs-subdir-external-provisioner | nfs-subdir-external-provisioner | dyrnq/nfs-subdir-external-provisioner:v4.0.2 |
+---------------+--------------+---------------------------------+---------------------------------+---------------------------------------------------------------------------------+
4 resource 获取指定namespace的所有limit 与 Requests大小
新增 Java XMX XMS的展示，比较直观的展示 CPU内存的Limit与Requests，Prometheus根据7天查询的内存CPU使用（已经根据转换得到与kubectl top po一直的数据格式），Java_opts的XMX,XMS，可以作为一个参考依据来调整。
示例代码
1 获取所有namespace的limit 与 Requests大小
./k8s-manager resource
2 获取指定namespace的limit 与 Requests大小
./k8s-manager resource -n <namespace>3 在prometheus查询最近七天的内存CPU使用情况
./k8s-manager resource prometheus -u <prometheus访问地址>
示例代码：获取requests与Limit,并且查询prometheus的实际开销
root@k8s:/usr/local/cobra-k8s-manager# k8s-manager resource prometheus -u http://192.168.44.134:20248/ -n monitoring
+------------+--------------------------------------+---------------------------+---------+---------+--------------------+----------+----------+---------------------+
| NAMESPACE | POD NAME | 容器名 | CPU限制 | CPU所需 | 最近7天已使用的CPU | 内存限制 | 内存所需 | 最近7天已使用的内存 |
+------------+--------------------------------------+---------------------------+---------+---------+--------------------+----------+----------+---------------------+
| monitoring | alertmanager-main-0 | alertmanager | 100m | 4m | 7.76m | 100Mi | 100Mi | 23.12MI |
| monitoring | alertmanager-main-0 | config-reloader | 10m | 10m | 1.13m | 50Mi | 50Mi | 17.61MI |
| monitoring | alertmanager-main-1 | alertmanager | 100m | 4m | 6.68m | 100Mi | 100Mi | 23.20MI |
| monitoring | alertmanager-main-1 | config-reloader | 10m | 10m | 0.56m | 50Mi | 50Mi | 17.49MI |
| monitoring | alertmanager-main-2 | alertmanager | 100m | 4m | 8.00m | 100Mi | 100Mi | 23.27MI |
| monitoring | alertmanager-main-2 | config-reloader | 10m | 10m | 0.58m | 50Mi | 50Mi | 17.42MI |
| monitoring | blackbox-exporter-59dddb7bb6-8lp69 | blackbox-exporter | 20m | 10m | 3.24m | 40Mi | 40Mi | 14.40MI |
| monitoring | blackbox-exporter-59dddb7bb6-8lp69 | module-configmap-reloader | 20m | 10m | 0.00m | 40Mi | 40Mi | 1.24MI |
| monitoring | blackbox-exporter-59dddb7bb6-8lp69 | kube-rbac-proxy | 20m | 10m | 0.54m | 40Mi | 40Mi | 8.84MI |
| monitoring | grafana-9bb74449d-8m8xl | grafana | 200m | 100m | 13.44m | 200Mi | 200Mi | 78.36MI |
| monitoring | kube-state-metrics-79996cfcc5-5286s | kube-state-metrics | 100m | 10m | 2.65m | 250Mi | 250Mi | 18.55MI |
| monitoring | kube-state-metrics-79996cfcc5-5286s | kube-rbac-proxy-main | 40m | 20m | 0.98m | 40Mi | 40Mi | 9.24MI |
| monitoring | kube-state-metrics-79996cfcc5-5286s | kube-rbac-proxy-self | 20m | 10m | 0.52m | 40Mi | 40Mi | 8.93MI |
| monitoring | node-exporter-6cqz8 | node-exporter | 250m | 102m | 42.76m | 180Mi | 180Mi | 9.23MI |
| monitoring | node-exporter-6cqz8 | kube-rbac-proxy | 20m | 10m | 2.64m | 40Mi | 40Mi | 11.93MI |
| monitoring | prometheus-adapter-854d95bc45-pvfh7 | prometheus-adapter | 250m | 102m | 10.88m | 180Mi | 180Mi | 31.21MI |
| monitoring | prometheus-adapter-854d95bc45-tz822 | prometheus-adapter | 250m | 102m | 9.31m | 180Mi | 180Mi | 31.08MI |
| monitoring | prometheus-k8s-0 | prometheus | 0 | 0 | 59.38m | 0 | 0 | 321.25MI |
| monitoring | prometheus-k8s-0 | config-reloader | 10m | 10m | 0.35m | 50Mi | 50Mi | 18.61MI |
| monitoring | prometheus-k8s-1 | prometheus | 0 | 0 | 53.57m | 0 | 0 | 321.04MI |
| monitoring | prometheus-k8s-1 | config-reloader | 10m | 10m | 0.67m | 50Mi | 50Mi | 18.47MI |
| monitoring | prometheus-operator-57cf88fbcb-wks8t | prometheus-operator | 200m | 100m | 2.03m | 200Mi | 200Mi | 27.48MI |
| monitoring | prometheus-operator-57cf88fbcb-wks8t | kube-rbac-proxy | 20m | 10m | 0.71m | 40Mi | 40Mi | 9.12MI |
+------------+--------------------------------------+---------------------------+---------+---------+--------------------+----------+----------+---------------------+
5 top 获取指定namespace的资源使用情况
这个命令是查看以namespace为单位的，不能top node节点，如果需要看 node信息，使用 k8s-manager node 或者 k8s-manager analysis --node
1 获取所有namespace的资源开销
./k8s-manager top
2 获取指定namespace的资源开销
./k8s-manager top -n <namespace>
示例代码
root@k8s:/usr/local/cobra-k8s-manager# k8s-manager top -n monitoring
+------------+-------------+--------------------------------+--------------------------------------+-------------+--------------+
| NAMESPACE | 资源类型 | 资源名 | POD NAME | 已使用的CPU | 已使用的内存 |
+------------+-------------+--------------------------------+--------------------------------------+-------------+--------------+
| monitoring | StatefulSet | alertmanager-main | alertmanager-main-0 | 2.00m | 28.20m |
| monitoring | StatefulSet | alertmanager-main | alertmanager-main-1 | 1.00m | 19.89m |
| monitoring | StatefulSet | alertmanager-main | alertmanager-main-2 | 2.00m | 27.96m |
| monitoring | ReplicaSet | blackbox-exporter-59dddb7bb6 | blackbox-exporter-59dddb7bb6-8lp69 | 0.00m | 4.00m |
| monitoring | ReplicaSet | grafana-9bb74449d | grafana-9bb74449d-8m8xl | 5.00m | 134.92m |
| monitoring | ReplicaSet | kube-state-metrics-79996cfcc5 | kube-state-metrics-79996cfcc5-5286s | 1.00m | 9.84m |
| monitoring | DaemonSet | node-exporter | node-exporter-6cqz8 | 9.00m | 16.10m |
| monitoring | ReplicaSet | prometheus-adapter-854d95bc45 | prometheus-adapter-854d95bc45-pvfh7 | 3.00m | 57.54m |
| monitoring | ReplicaSet | prometheus-adapter-854d95bc45 | prometheus-adapter-854d95bc45-tz822 | 3.00m | 32.05m |
| monitoring | StatefulSet | prometheus-k8s | prometheus-k8s-0 | 12.00m | 397.64m |
| monitoring | StatefulSet | prometheus-k8s | prometheus-k8s-1 | 10.00m | 375.96m |
| monitoring | ReplicaSet | prometheus-operator-57cf88fbcb | prometheus-operator-57cf88fbcb-wks8t | 1.00m | 43.32m |
+------------+-------------+--------------------------------+--------------------------------------+-------------+--------------+
常用的使用套路
1 k8s节点异常卡顿，容器频繁重启
k8s-manager analysis --node k8s #查看实际node上的开销情况，数据是由metrics-server提供的，使用的资源在服务器的占比
2 优化limit requests
支持 namespace与 node的筛选
root@k8s:/usr/local/cobra-k8s-manager# go run main.go resource prometheus -u http://192.168.44.134:20248/ --node k8s -n nfs
+--------+-----------+--------------------------------------------------+---------------------------------+---------+---------+--------------------+----------+----------+---------------------+
| 节点名 | NAMESPACE | POD NAME | 容器名 | CPU限制 | CPU所需 | 最近7天已使用的CPU | 内存限制 | 内存所需 | 最近7天已使用的内存 |
+--------+-----------+--------------------------------------------------+---------------------------------+---------+---------+--------------------+----------+----------+---------------------+
| k8s | nfs | nfs-subdir-external-provisioner-65664b8954-qrs2q | nfs-subdir-external-provisioner | 0 | 0 | 4.10m | 0 | 0 | 9.60Mi |
+--------+-----------+--------------------------------------------------+---------------------------------+---------+---------+--------------------+----------+----------+---------------------+
---详细版)
