Prometheus+Grafana监控

部署根目录/home/ops_app_user/middleware/prometheus一、部署概述1.1 适用场景单业务服务器/小规模集群的主机系统、Docker容器、MySQL数据库、Redis缓存一体化监控满足生产环境持久化、权限管控、告警预留、可维护性要求。1.2 组件清单固定稳定版本组件版本作用端口Prometheusv2.45.0 LTS时序数据库、指标抓取、告警引擎9090Grafana10.2.0可视化仪表盘、告警通知3000node_exporterv1.6.1Linux主机CPU/内存/磁盘/网络指标采集9100cAdvisorv0.47.0Docker容器资源指标采集8081mysqld_exporterv0.15.1MySQL数据库性能指标采集9104redis_exporterv1.55.0Redis缓存指标采集9121二、目录规划与权限配置2.1 完整目录结构/home/ops_app_user/middleware/prometheus ├── docker-compose.yml # 统一编排文件 ├── prometheus │ ├── prometheus.yml # Prometheus主配置 │ ├── rules/ # 告警规则目录 │ │ └── host_rules.yml │ └── data/ # 时序数据持久化 └── grafana └── data/ # Grafana面板/账号/数据源持久化2.2 创建目录与权限设置生产环境禁止777权限按容器内运行用户UID分配对应权限# 创建完整目录树mkdir-p/home/ops_app_user/middleware/prometheus/{prometheus/{data,rules},grafana/data}# Prometheus 容器内用户为 nobody(UID 65534)chown-R65534:65534 /home/ops_app_user/middleware/prometheus/prometheus/datachmod755/home/ops_app_user/middleware/prometheus/prometheus# Grafana 容器内用户为 grafana(UID 472)chown-R472:472 /home/ops_app_user/middleware/prometheus/grafana/datachmod755/home/ops_app_user/middleware/prometheus/grafana三、核心配置文件3.1 Prometheus 主配置文件路径/home/ops_app_user/middleware/prometheus/prometheus/prometheus.yml生产级特性30s抓取周期、超时控制、告警规则预留、热加载支持所有目标地址请替换为服务器实际IP。global:scrape_interval:30sevaluation_interval:30sscrape_timeout:10s# 告警规则文件rule_files:-/etc/prometheus/rules/*.yml# Alertmanager 接入部署告警组件后启用# alerting:# alertmanagers:# - static_configs:# - targets: [alertmanager:9093]scrape_configs:-job_name:prometheusstatic_configs:-targets:[prometheus:9090]labels:instance:prometheus-server-job_name:node-exporterstatic_configs:-targets:[node-exporter:9100]labels:instance:prod-server-01-targets:[node-exporter:9100]labels:instance:prod-server-02-job_name:cadvisorstatic_configs:-targets:[cadvisor:8080]labels:instance:prod-server-01-job_name:mysqld-exporterstatic_configs:-targets:[mysqld-exporter:9104]labels:instance:prod-mysql-01-job_name:redis-exporterstatic_configs:-targets:[redis-exporter:9121]labels:instance:prod-redis-013.2 基础告警规则文件路径/home/ops_app_user/middleware/prometheus/prometheus/rules/host_rules.yml覆盖主机离线、CPU/磁盘/内存使用率超标等核心告警groups:-name:host-base-alertrules:-alert:HostDownexpr:up{jobnode-exporter} 0for:1mlabels:severity:criticalannotations:summary:主机 {{ $labels.instance }} 离线description:服务器超过1分钟无法采集指标请立即排查-alert:HighCpuUsageexpr:100-(avg by(instance) (irate(node_cpu_seconds_total{modeidle}[5m])) * 100)80for:5mlabels:severity:warningannotations:summary:主机 {{ $labels.instance }} CPU使用率过高description:当前使用率 {{ $value | printf \%.2f\ }}%持续5分钟-alert:HighDiskUsageexpr:100-(node_filesystem_avail_bytes{fstype~ext4|xfs,mountpoint!~/var/lib/docker/.*}/ node_filesystem_size_bytes * 100)85for:5mlabels:severity:warningannotations:summary:主机 {{ $labels.instance }} 磁盘使用率过高description:分区 {{ $labels.mountpoint }} 使用率 {{ $value | printf \%.2f\ }}%四、Docker Compose 编排文件文件路径/home/ops_app_user/middleware/prometheus/docker-compose.yml生产级特性固定镜像版本、自定义网络隔离、资源上限限制、开机自启、数据持久化、安全参数。# version 字段在 Compose V2 仅为兼容提示不影响功能可保留或删除version:3.8networks:monitor:driver:bridgeservices:# 主机系统指标采集node-exporter:image:prom/node-exporter:v1.6.1container_name:node-exporterrestart:alwaysports:-9100:9100volumes:-/proc:/host/proc:ro-/sys:/host/sys:ro-/:/rootfs:rocommand:---path.procfs/host/proc---path.sysfs/host/sys---path.rootfs/rootfs---collector.filesystem.ignored-mount-points^/(sys|proc|dev|host|etc|rootfs/var/lib/docker)($$|/)networks:-monitordeploy:resources:limits:cpus:0.2memory:128M# Docker容器指标采集已替换为国内可访问镜像cadvisor:image:google/cadvisor:latestcontainer_name:cadvisorrestart:alwaysprivileged:trueports:-8081:8080volumes:-/:/rootfs:ro-/var/run:/var/run:rw-/sys:/sys:ro-/var/lib/docker/:/var/lib/docker:ro-/dev/disk/:/dev/disk:rodevices:-/dev/kmsg:/dev/kmsgnetworks:-monitordeploy:resources:limits:cpus:0.3memory:256M# MySQL指标采集mysqld-exporter:image:prom/mysqld-exporter:v0.15.1container_name:mysqld-exporterrestart:alwaysports:-9104:9104environment:# ⚠️ 原密码含 会破坏 DSN 格式user:password(host:port) 中 是分隔符# 已将密码中的 转义为 %40若实际密码不含第二个 请自行修正DATA_SOURCE_NAME:root:xxx(xxx:3020)/networks:-monitordeploy:resources:limits:cpus:0.2memory:128M# Redis指标采集redis-exporter:image:oliver006/redis_exporter:v1.55.0container_name:redis-exporterrestart:alwaysports:-9121:9121command:---redis.addrredis://xxx:6379---redis.passwordxxxnetworks:-monitordeploy:resources:limits:cpus:0.2memory:128M# Prometheus 服务端prometheus:image:prom/prometheus:v2.45.0container_name:prometheusrestart:alwaysports:-9090:9090volumes:-./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro-./prometheus/rules:/etc/prometheus/rules:ro-./prometheus/data:/prometheuscommand:---config.file/etc/prometheus/prometheus.yml---storage.tsdb.path/prometheus---storage.tsdb.retention.time30d---storage.tsdb.retention.size20GB---web.enable-lifecycle---web.enable-admin-api---log.levelinfouser:65534:65534networks:-monitordepends_on:-node-exporter-cadvisordeploy:resources:limits:cpus:2memory:4G# Grafana 可视化grafana:image:grafana/grafana:10.2.0container_name:grafanarestart:alwaysports:-3000:3000volumes:-./grafana/data:/var/lib/grafanaenvironment:GF_SECURITY_ADMIN_PASSWORD:Admin2024GF_USERS_ALLOW_SIGN_UP:falseGF_AUTH_ANONYMOUS_ENABLED:falseuser:472:472networks:-monitordepends_on:-prometheusdeploy:resources:limits:cpus:1memory:1G五、部署启动流程5.1 前置准备替换prometheus.yml中所有targets的IP为服务器实际地址修改mysqld-exporter的MySQL账号密码建议创建最小权限监控账号CREATE USER monitor% IDENTIFIED BY Monitor123; GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO monitor%; FLUSH PRIVILEGES;修改redis-exporter的Redis连接地址与密码无密码删除对应参数修改Grafana初始管理员密码生产环境必须使用强密码5.2 启动服务cd /home/ops_app_user/middleware/prometheus # 一键启动全部组件 docker-compose up -d # 查看运行状态 docker-compose ps5.3 配置热加载修改配置文件后无需重启容器执行以下命令生效curl -X POST [http://localhost:9090/-/reload](http://localhost:9090/-/reload)六、验证与Grafana接入6.1 组件可用性验证Prometheus访问http://服务器IP:9090进入Status - Targets所有Job状态均为UP采集器分别访问对应端口/metrics路径可正常输出指标文本6.2 Grafana 接入数据源访问http://服务器IP:3000使用admin账号登录左侧菜单Connections - Data sources - Add data source选择PrometheusPrometheus server URL填写http://prometheus:9090同自定义网络可直接解析点击Save test提示绿色成功即为连通6.3 推荐生产仪表盘直接导入Grafana Labs官方高星模板主机监控ID 8919Node Exporter Full容器监控ID 14282cAdvisor ExporterMySQL监控ID 7362MySQL OverviewRedis监控ID 763Redis Dashboard操作路径Dashboards - New - Import输入ID后选择Prometheus数据源即可。七、生产级运维规范7.1 安全加固端口管控9090、9100、8081等采集端口禁止公网暴露仅开放内网运维网段接口鉴权Prometheus前置Nginx反向代理并开启Basic Auth禁止裸访问权限最小化数据库监控账号仅授予只读权限禁止全库权限7.2 数据与备份每周全量备份/home/ops_app_user/middleware/prometheus下的prometheus/data与grafana/data目录保留3份历史副本监控数据目录磁盘使用率达到70%提前扩容或调整数据保留周期定期使用promtool检查配置与TSDB健康状态7.3 告警体系落地补充文档章节风格对齐原文可直接粘贴进文档前置在现有 compose 清单追加 alertmanager 服务、prometheus.yml 开启 alerting 配置、新增 alertmanager 配置文件目录和整套监控栈共用monitor网桥。7.3.1 告警架构说明告警链路Prometheus匹配告警规则 → Alertmanager分组、降噪、去重、抑制、路由分发 → 通知渠道钉钉/企业微信Prometheus 只负责判断告警是否触发告警收敛、静默、重复推送间隔、多渠道分发全部交给 Alertmanager避免短时间大量同类型指标抖动产生告警风暴。分级策略severity: critical核心故障主机失联、数据库采集中断、磁盘满工作群强提醒按需对接电话/短信告警severity: warning性能预警CPU偏高、负载上涨普通工作群通知不做电话触达。核心能力告警分组、重复通知间隔、告警抑制、告警静默、告警恢复通知。7.3.2 新增目录规划在根目录/home/ops_app_user/middleware/prometheus下新增 alertmanager 配置目录mkdir -p /home/ops_app_user/middleware/prometheus/alertmanager目录结构追加后/home/ops_app_user/middleware/prometheus ├── alertmanager │ └── alertmanager.yml # Alertmanager路由、渠道配置 ├── docker-compose.yml ├── prometheus │ ├── prometheus.yml │ ├── rules/ │ └── data/ └── grafana └── data/7.3.3 docker-compose.yml 追加 Alertmanager 服务片段# 告警分发组件 Alertmanager alertmanager: image: prom/alertmanager:v0.26.0 container_name: alertmanager restart: always ports: - 9093:9093 volumes: - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro command: - --config.file/etc/alertmanager/alertmanager.yml - --web.external-urlhttp://【服务器内网IP】:9093 networks: - monitor deploy: resources: limits: cpus: 0.2 memory: 128M同步修改prometheus/prometheus.yml取消 alerting 注释alerting: alertmanagers: - static_configs: - targets: [alertmanager:9093]7.3.4 alertmanager.yml 模板钉钉机器人生产收敛版路径/home/ops_app_user/middleware/prometheus/alertmanager/alertmanager.ymlglobal: resolve_timeout: 5m route: group_by: [alertname, instance] group_wait: 10s group_interval: 10s repeat_interval: 5m # 同一个告警多久重复发一次 receiver: dingtalk receivers: - name: dingtalk webhook_configs: - url: https://oapi.dingtalk.com/robot/send?access_tokenxxx send_resolved: true # 告警恢复时也发送通知 inhibit_rules: - source_match: severity: critical target_match: severity: warning equal: [alertname, instance]如需企业微信替换 webhook 地址即可格式保持不变支持多 receiver、多路由实现不同告警推送到不同群。7.3.5 告警规则扩展追加到 prometheus/rules/host_rules.yml可选原有主机规则基础上补充采集器离线告警和原有规则合并- alert: ExporterDown expr: up{job~mysqld-exporter|redis-exporter|cadvisor} 0 for: 1m labels: severity: critical annotations: summary: 采集器 {{ $labels.job }} 离线 description: 实例 {{ $labels.instance }} 的指标采集中断请检查 exporter 进程、账号、网络7.3.6 部署、校验与验证命令# 重新加载编排 cd /home/ops_app_user/middleware/prometheus docker-compose up -d alertmanager # 校验 alertmanager 配置语法 docker compose exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml # 校验 prometheus 配置并热加载 curl -X POST [http://localhost:9090/-/reload](http://localhost:9090/-/reload)访问地址Alertmanager 管理页面http://【服务器内网IP】:9093可手动查看活跃告警、静默告警。7.3.7 运维管控规范告警静默变更维护窗口停机升级、扩容前在 Alertmanager WebUI 创建静默规则匹配instance/alertname设置静默起止时间避免变更期间大量无效告警静默到期自动解除禁止长期静默。告警路由拆分后续业务量大后新增 route 匹配不同 job、severity实现数据库告警发给DBA群主机告警发给运维群。告警链路巡检每周抽查告警推送链路可临时调低告警阈值触发一次测试告警验证通知通道可用性测试完成恢复阈值。权限管控9093 端口同样只对内网运维网段开放禁止公网暴露Alertmanager 无内置鉴权高安全环境前置 Nginx Basic Auth。告警风暴治理优先依靠inhibit_rules抑制次要告警避免编写粒度过细、极易抖动的短周期告警for时间合理拉长规避瞬时抖动误报。7.3.8 故障排障思路PrometheusAlerts页面告警状态为firing钉钉收不到消息优先校验机器人 token、IP白名单、关键词/加签配置告警已经恢复但钉钉不发恢复通知确认send_resolved: true同故障疯狂刷屏调大repeat_interval检查是否重复定义多套 receiver 路由。