七点半的菜市场

large picture of the cover

二、Prometheus使用ServiceMonitor监控ETCD和Mariadb

📦 Kubernetes版本 1.18，使用容器部署 Prometheus-operator版本为 0.6

🏆 Prometheus监控Etcd和Mariadb

❓ 云原生应用

为什么拿这两个做监控呢？之前说过云原生应用和非云原生应用，而 ETCD就数据云原生应用而 Mariadb反之

Etcd会暴露一个 /metrics接口，而 mariadb这种非云原生应用就需要通过 exporter来进行数据收集，且有 exporter来暴露 /metrics

Kubernetes 本身的宿主机肯定不是，它是一个系统，肯定也没有暴露 /metrics,所以就需要使用到 exporter了，在安装好 kube-prometheus之后就已经通过 daemnoset的方式安装好 node-exporter了

[root@k8s-master mariadb]# kubectl  get pod -n monitoring  | grep node-exporter
node-exporter-wkdmk                    2/2     Running   2          45h
node-exporter-xgrf2                    2/2     Running   2          45h

Promethues就可以通过 node-exporter暴露的 9100端口来进行采集

curl 127.0.0.1:9100/metrics | tail -n 1
promhttp_metric_handler_requests_total{code="503"} 0

如果你是通过 Kube-Prometheus安装的，就需要通过一个 ServiceMonitor来自动发现监控目标并动态的生成配置，这个是一个 CRD由 operator自动创建，只要创建了 ServiceMonitor，Pormetheus operator就会自动的去解析并生成对应的配置

⭐️ 监控 ETCD

使用 Prometheus监控 kubernetes集群的 etcd

1️⃣ 首先查看 prometheus和 etcd服务是正常运行

##Prometheus
[root@k8s-master etcd]# kubectl get pod -n monitoring 
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   2          26h
alertmanager-main-1                    2/2     Running   2          26h
alertmanager-main-2                    2/2     Running   2          26h
kube-state-metrics-69d4c7c69d-wdqpj    3/3     Running   3          26h
node-exporter-wkdmk                    2/2     Running   2          26h
node-exporter-xgrf2                    2/2     Running   2          26h
prometheus-adapter-66b855f564-j69k6    1/1     Running   1          26h
prometheus-k8s-0                       3/3     Running   1          24m
prometheus-k8s-1                       3/3     Running   1          24m
prometheus-operator-75c98bcfd7-lglzj   2/2     Running   2          26h

##Etcd
[root@k8s-master etcd]# kubectl get pod -n kube-system | grep etcd
etcd-k8s-master                      1/1     Running   16         22d

2️⃣ Kubernetes的 etcd是直接使用了 hostIP，由于它是云原生应用所以它会提供一个 metrics接口，使用如下命令访问

注意 ip地址记得更换，如果修改过 etcd端口的也需要更换

[root@k8s-master etcd]# curl -s --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key https://10.96.38.117:2379/metrics -k  | tail -n 1
promhttp_metric_handler_requests_total{code="503"} 0

3️⃣ 由于 ETCD是使用的 hostIP,ServiceMonitor需要对接 Service，先给 etcd创建一个 svc代理，yaml文件如下

apiVersion: v1
kind: Service
metadata:
  name: etcd-svc-prom
  labels:
    app: etcd
  namespace: kube-system ###请注意你的etcd的名称空间
spec:
  selector:
    component: etcd
  ports:
  - name: https-metrics
    port: 2379
    protocol: TCP
  type: ClusterIP

创建之后查看

[root@k8s-master etcd]# kubectl get svc -n kube-system  -l app=etcd
NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
etcd-svc-prom   ClusterIP   10.96.38.117   <none>        2379/TCP   39m

4️⃣ 由于 etcd默认是 tls，必须通过 https访问所以需要添加证书

这里请注意证书地址，不同的搭建k8s集群的证书生成地址可能不一致

[root@k8s-master etcd]# kubectl create secret generic etcd-ssl --from-file=/etc/kubernetes/pki/etcd/ca.crt --from-file=/etc/kubernetes/pki/etcd/server.key --from-file=/etc/kubernetes/pki/etcd/server.crt  -n monitoring

创建完成后需要挂载至 Prometheus容器之中

##直接使用kebectl edit 命令
[root@k8s-master etcd]# kubectl edit prometheus k8s -n monitoring
....
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  secrets: ##添加
  - etcd-ssl ##添加
....

保存退出之后查看普罗米修斯是否正常运行

[root@k8s-master etcd]# kubectl  get pod -n monitoring 
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   2          26h
alertmanager-main-1                    2/2     Running   2          26h
alertmanager-main-2                    2/2     Running   2          26h
kube-state-metrics-69d4c7c69d-wdqpj    3/3     Running   3          26h
node-exporter-wkdmk                    2/2     Running   2          26h
node-exporter-xgrf2                    2/2     Running   2          26h
prometheus-adapter-66b855f564-j69k6    1/1     Running   1          26h
prometheus-k8s-0                       3/3     Running   1          37m
prometheus-k8s-1                       3/3     Running   1          37m
prometheus-operator-75c98bcfd7-lglzj   2/2     Running   2          26h

注意挂载位置为如下

[root@k8s-master etcd]# kubectl  exec -n monitoring prometheus-k8s-0 -c prometheus -- ls /etc/prometheus/secrets/etcd-ssl/
ca.crt
server.crt
server.key

6️⃣ 创建 ServiceMonitor对接 etcd-svc，如下 yaml文件

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: etcd-svc-monitor
  namespace: monitoring
  labels:
    app: etcd
spec:
  jobLabel: k8s-etcd
  endpoints:
  - interval: 10s
    port: https-metrics
    scheme: https
    tlsConfig:
      caFile: /etc/prometheus/secrets/etcd-ssl/ca.crt
      certFile: /etc/prometheus/secrets/etcd-ssl/server.crt
      keyFile: /etc/prometheus/secrets/etcd-ssl/server.key
      insecureSkipVerify: true
  selector:
    matchLabels:
      app: etcd ##注意需要和service的标签一致
  namespaceSelector:
    matchNames:
    - kube-system

解释：

  • honorLabels: 如果目标标签和服务器标签冲突，是否保留目标标签
  • interval: 监控数据抓取的时间间隔
  • prot: 暴露的 metrics端口
  • selector： 监控的目标 Service标签
  • namespaceSelector: 被监控的 Service所在的名称空间

5️⃣ 创建完成之后进入 Prometheus Dashboard查看

Status ---> Service Discovery

image-20241021224842188

进入 Grafana导入 3070模板

image-20241021224951238

⭐️ 监控 Mariadb

1️⃣ 首先需要你先安装一个 Mariadb，可以先查看前几条博客 Kubernets搭建 Mariadb主从

[root@k8s-master ~]# kubectl  get pod -n mall-app  -l app=mariadb
NAME        READY   STATUS    RESTARTS   AGE
mariadb-0   2/2     Running   10         5d20h
mariadb-1   2/2     Running   10         5d20h

进入主节点容器，输入如下命令，注意数据库的用户和密码填写你自定义的

[root@k8s-master mariadb]# kubectl  exec -it -n mall-app mariadb-0 -c mariadb -- mysql -u xxx -p xxxx
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 601
Server version: 10.6.4-MariaDB-1:10.6.4+maria~focal-log mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> CREATE USER 'exporter'@'%' IDENTIFIED BY 'exporter123' WITH MAX_USER_CONNECTIONS 3;
MariaDB [(none)]> GRANT PROCESS, REPLICATION CLIENT, SELECT,SUPER ON *.* TO 'exporter'@'%';

2️⃣ 安装 mysql-export使用如下yaml文档

使用的镜像为https://github.com/prometheus/mysqld_exporter/tree/v0.14.0，详细操作可以查看文档

配置数据连接语法文档在https://github.com/go-sql-driver/mysql#dsn-data-source-name,其中 mariadb.mall-app.svc.cluster.local为你的 maraidb主节点的内部域名

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql-export er
  namespace: monitoring #注意
spec:
  selector:
    matchLabels:
      monitor: mysql
  replicas: 1
  template:
    metadata:
      name: mysql-exporter
      labels:
        monitor: mysql
    spec:
      containers:
      - name: mysql-exporter
        image: prom/mysqld-exporter:v0.14.0 ##注意0.15之后貌似env不是DATA_SOURCE_NAME了
        env:
        - name: DATA_SOURCE_NAME ##配置数据库的连接，注意语法
          value: "exporter:exporter123@(mariadb.mall-app.svc.cluster.local:3306)/"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 9104 ##注意端口
---
apiVersion: v1
kind: Service
metadata:
  name: mysql-exporter
  namespace: monitoring ##注意
  labels:
    monitor: mysql ## 注意label
spec:
  selector:
    monitor: mysql
  ports:
  - name: api
    port: 9104
    protocol: TCP
  type: ClusterIP

创建之后查看是否运行成功

[root@k8s-master mariadb]# kubectl  get pod -n monitoring  -l monitor=mysql
NAME                              READY   STATUS    RESTARTS   AGE
mysql-exporter-6b4bbdb6d4-wslt6   1/1     Running   0          4h11m

3️⃣ 创建 ServiceMonitor 绑定 exporter的 svc

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mysql-exporter
  namespace: monitoring ##注意
  labels:
    monitor: mysql
spec:
  jobLabel: k8s-app
  endpoints:
  - port: api
    interval: 30s
    scheme: http
  selector:
    matchLabels:
      monitor: mysql ##注意label和svc的label一致
  namespaceSelector:
    matchNames:
    - monitoring

创建成功后查看

[root@k8s-master mariadb]# kubectl get servicemonitors.monitoring.coreos.com -n monitoring  -l monitor=mysql
NAME             AGE
mysql-exporter   4h12m

4️⃣ 接下来进入 Prometheus的 WebUI即可查看

image-20241022180607501

进入 grafana，同样选择 dashboard导入 13106

image-20241022180817614

site logo

Hi,Friend

© 2024 七点半的菜市场