服务器多节点docker部署Grafana、Prometheus、Node-Exporter、alertmanager、prometheus-alert并对接飞书通知

本文最后更新于 2026年7月25日 上午

要在多台服务器上部署 Grafana、Prometheus 和 Node-Exporter，并且其中一台服务器专门用于 Grafana 和 Prometheus 的部署

1. 准备工作

  • 服务器信息：
    • Server 1：用于部署 Grafana 和 Prometheus、alertmanager、prometheus-alert
    • Server 2-n：用于部署 Node-Exporter。
  • Docker：确保所有服务器上已安装 Docker 和 Docker Compose。

https://getdocker.quickso.cn/

2. 在 Server 1 上部署 Grafana 和 Prometheus等

2.1 创建 Docker Compose 文件

在 Server 1 上创建一个 docker-compose.yml 文件，内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    restart: always
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus:/etc/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.enable-lifecycle'

  grafana:
    image: grafana/grafana
    container_name: grafana
    restart: always
    ports:
      - "3001:3000"

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    restart: always
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml

  blackbox_exporter:
    image: prom/blackbox-exporter
    container_name: blackbox_exporter
    restart: always
    ports:
      - "9115:9115"
    volumes:
      - ./blackbox_exporter/blackbox.yml:/etc/blackbox_exporter/config.yml
    command:
      - '--config.file=/etc/blackbox_exporter/config.yml'

2.2 创建 配置文件

在 Server 1 上创建一个 prometheus.yml 文件，内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - /etc/prometheus/alerts.yml
  
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['prometheus:9090']      
  - job_name: "node_exporter"
    static_configs:
      - targets: ["IP:9100"]
        labels:
          instance: '服务器0'
      - targets: ["IP:9100"]
        labels:
          instance: 服务器1'   
  - job_name: 'blackbox_http_2xx'
    metrics_path: /probe
    params:
      module: [http_2xx]  #配置get请求检测
    static_configs:
      - targets:
        - https://www.baidu.com
        - https://www.google.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115  #blackbox地址和端口号
        
  - job_name: 'blackbox_tcp_connect' # 检测某些端口是否在线
    scrape_interval: 30s
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
        - IP:22
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115 # blackbox-exporter 服务所在的机器和端口

alerts.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
groups:
- name: node_alerts
  rules:

  # ==========================
  # 节点状态
  # ==========================
  - alert: NodeDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} 已离线"
      description: |
        节点 {{ $labels.instance }} 已离线超过 1 分钟。

  # ==========================
  # CPU
  # ==========================
  - alert: NodeCPUHigh
    expr: |
      100 - (
        avg by(instance)(
          rate(node_cpu_seconds_total{mode="idle"}[5m])
        ) * 100
      ) > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} CPU 使用率过高"
      description: |
        CPU 使用率超过 80%
        当前值：{{ printf "%.2f" $value }}%

  - alert: NodeCPUCritical
    expr: |
      100 - (
        avg by(instance)(
          rate(node_cpu_seconds_total{mode="idle"}[5m])
        ) * 100
      ) > 90
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} CPU 使用率严重过高"
      description: |
        CPU 使用率超过 90%
        当前值：{{ printf "%.2f" $value }}%

  # ==========================
  # Memory
  # ==========================
  - alert: NodeMemoryHigh
    expr: |
      (
        node_memory_MemTotal_bytes
        - node_memory_MemAvailable_bytes
      )
      /
      node_memory_MemTotal_bytes
      *100 >90
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} 内存使用率过高"
      description: |
        内存使用率超过 90%
        当前值：{{ printf "%.2f" $value }}%

  - alert: NodeMemoryCritical
    expr: |
      (
        node_memory_MemTotal_bytes
        - node_memory_MemAvailable_bytes
      )
      /
      node_memory_MemTotal_bytes
      *100 >90
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} 内存使用率严重过高"
      description: |
        内存使用率超过 90%
        当前值：{{ printf "%.2f" $value }}%

  # ==========================
  # Disk
  # ==========================
  - alert: NodeDiskSpaceHigh
    expr: |
      (
        1 -
        (
          node_filesystem_avail_bytes{
            fstype!~"tmpfs|overlay",
            mountpoint!="/boot"
          }
          /
          node_filesystem_size_bytes{
            fstype!~"tmpfs|overlay",
            mountpoint!="/boot"
          }
        )
      ) *100 >80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} 磁盘空间不足"
      description: |
        {{ $labels.mountpoint }} 使用率超过80%
        当前值：{{ printf "%.2f" $value }}%

  - alert: NodeDiskSpaceCritical
    expr: |
      (
        1 -
        (
          node_filesystem_avail_bytes{
            fstype!~"tmpfs|overlay",
            mountpoint!="/boot"
          }
          /
          node_filesystem_size_bytes{
            fstype!~"tmpfs|overlay",
            mountpoint!="/boot"
          }
        )
      ) *100 >90
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} 磁盘空间严重不足"
      description: |
        {{ $labels.mountpoint }} 使用率超过90%
        当前值：{{ printf "%.2f" $value }}%

  # ==========================
  # inode
  # ==========================
  - alert: NodeInodeLow
    expr: |
      (
        node_filesystem_files_free
        /
        node_filesystem_files
      ) *100 <10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} inode 不足"
      description: |
        {{ $labels.mountpoint }} inode 剩余不足10%

  # ==========================
  # Load
  # ==========================
  - alert: NodeLoadHigh
    expr: |
      node_load15
      /
      count by(instance)(
        node_cpu_seconds_total{mode="system"}
      ) >2
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} Load Average 过高"
      description: |
        Load15 超过 CPU 核数两倍
        当前值：{{ printf "%.2f" $value }}

  # ==========================
  # OOM
  # ==========================
  - alert: NodeOOMKill
    expr: increase(node_vmstat_oom_kill[5m]) >0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} 发生 OOM"
      description: |
        最近5分钟发生 OOM Killer

- name: blackbox_alerts
  rules:

  # ==========================
  # Probe Failed
  # ==========================
  - alert: BlackboxProbeFailed
    expr: probe_success == 0
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} 无法访问"
      description: |
        Blackbox Probe 检测失败

  # ==========================
  # Response Slow
  # ==========================
  - alert: BlackboxSlowResponse
    expr: avg_over_time(probe_duration_seconds[5m]) > 3
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} 响应缓慢"
      description: |
        平均响应时间超过3秒
        当前值：{{ printf "%.2f" $value }} 秒

  # ==========================
  # HTTP Status
  # ==========================
  - alert: BlackboxHttpStatusError
    expr: probe_http_status_code >=400
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} HTTP 状态异常"
      description: |
        HTTP Status={{ $value }}

  # ==========================
  # SSL
  # ==========================
  - alert: BlackboxSSLExpire30Days
    expr: probe_ssl_earliest_cert_expiry-time()<86400*30
    for: 30m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.instance }} SSL证书30天内过期"
      description: |
        SSL证书将在30天内过期

  - alert: BlackboxSSLExpire7Days
    expr: probe_ssl_earliest_cert_expiry-time()<86400*7
    for: 30m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} SSL证书7天内过期"
      description: |
        SSL证书将在7天内过期

  - alert: BlackboxSSLExpired
    expr: probe_ssl_earliest_cert_expiry-time()<=0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "{{ $labels.instance }} SSL证书已过期"
      description: |
        SSL证书已过期

alertmanager.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
global:
 resolve_timeout: 1m
route:
receiver: feishu-webhook
group_by:
    - alertname
group_wait: 5s
group_interval: 1m
repeat_interval: 30m
receivers:
- name: feishu-webhook
    webhook_configs:
    - url: http://IP:5000/webhook
        send_resolved: true

2.3 启动服务

在 Server 1 上运行以下命令启动：

1
docker-compose up -d

2.4.Prometheus Webhook for Feishu (飞书)

该项目提供了一个 Webhook 服务，用于接收来自 Prometheus Alertmanager 的告警通知，并将其作为交互式卡片转发到飞书。

https://github.com/muzihuaner/prometheus-webhook-feishu

3.在 Server 2-n 上部署 Node-Exporter，并开放端口

Docker方式

1
2
3
4
5
docker run -d \
  --name node-exporter \
  -p 9100:9100 \
  --restart always \
  prom/node-exporter

二进制（Debian）

1
apt install prometheus-node-exporter

4. 配置 Grafana

4.1 访问 Grafana

在浏览器中访问 http://<Server1-IP>:3000，使用默认用户名 admin 和密码 admin 登录。

4.2 添加 Prometheus 数据源

在 Grafana 中，点击左侧菜单的 Configuration -> Data Sources。

  1. 点击 Add data source，选择 Prometheus。
  2. 在 URL 字段中输入 http://prometheus:9090，然后点击 Save & Test。

4.3 导入仪表盘

在 Grafana 中，点击左侧菜单的 + -> Import。

  1. 在 Grafana.com Dashboard 字段中输入 ID，然后点击 Load。
  2. 选择 Prometheus 数据源，然后点击 Import。
常用仪表盘
  1. Caddy host 24146 https://grafana.com/grafana/dashboards/24146-caddy-hosts/
  2. minecraft 14756 https://grafana.com/grafana/dashboards/14756-unifiedmetrics-0-3-x-prometheus/
  3. node-exporter 16098 https://grafana.com/grafana/dashboards/16098-node-exporter-dashboard-20240520-job/
  4. Proxmox 10347 https://grafana.com/grafana/dashboards/10347-proxmox-via-prometheus/
  5. ipmi-exporter
  6. frp
  7. Prometheus 2.0 Stats
  8. Websites Overview

4. 验证部署

  • Prometheus：访问 http://<Server1-IP>:9090/targets，确保所有 Node-Exporter 目标（包括 Server 1）的状态为 UP。
  • Grafana：访问 http://<Server1-IP>:3000，查看仪表盘是否显示所有服务器的监控数据。
  • Alertmanager: 访问 http://<Server1-IP>:9093，告警管理组件，负责处理、分组、抑制和发送告警通知。
教程
#Prometheus #Grafana #Node-Exporter
服务器多节点docker部署Grafana、Prometheus、Node-Exporter、alertmanager、prometheus-alert并对接飞书通知
https://blog.quickso.cn/2025/03/14/服务器多节点Docker部署Grafana、Prometheus、Node-Exporter、alertmanager、prometheus-alert并对接飞书通知/
作者
木子欢儿
发布于
2025年3月14日
许可协议