VLLM部署模型的日志实时监控系统

在我的公司环境中基于华为昇腾算力通过VLLM Ascend部署了多个大模型包括了Deepseek V4 Flash, GLM 5.2等。在日常运营中我们需要了解用户在调用大模型时是否遇到问题并及时解决但是很多大模型都是采用PD分离方式部署的而且通常都是多个实例进行负载均衡。当遇到问题需要翻查日志时需要去到多个服务器上进行查找十分不便。因此我计划构建一个基于Grafana Loki Fluent-bit的集中日志监控系统可以统一采集各个服务器的日志信息并在Grafana里面进行查询。VLLM部署模型开启日志输出我这边都是通过容器方式运行VLLM需要把VLLM的日志输出到前台保存到宿主机的容器日志目录中。VLLM的日志默认是采用NewLineFormatter格式化它的作用是在多行消息的换行符后添加日志前缀让每一行看起来都“对齐”了。但需要特别注意只有第一行包含完整的时间戳和日志级别后续行虽然也有前缀但时间戳等信息是重复或空缺的。例如一个错误日志可能长这样[10-10 15:20:30] [vllm/engine/llm_engine.py:521] ERROR: CUDA out of memory. Tried to allocate 2048MiB (GPU 0; 23.7GiB total capacity; 20.1GiB already allocated)这种方式虽然方便阅读但是不方便程序进行处理如果逐行采集这些行会被当作独立事件导致告警和分析时无法关联到同一个错误。一个好的处理方式是直接让VLLM输出JSON格式的日志通过环境变VLLM_LOGGING_CONFIG_PATH指定自定义的 JSON 日志配置。在服务器上创建一个配置文件例如vllm-logging.json。以下配置使用了python-json-logger库来格式化输出在VLLM的官方Docker镜像已经包含了这个库了{ version: 1, disable_existing_loggers: false, formatters: { json: { class: pythonjsonlogger.jsonlogger.JsonFormatter, format: %(asctime)s %(levelname)s %(name)s %(message)s } }, handlers: { console: { class: logging.StreamHandler, formatter: json, level: INFO, stream: ext://sys.stdout } }, loggers: { vllm: { handlers: [console], level: INFO, propagate: false } } }然后就可以通过VLLM来启动模型了如以下示例docker run -d \ --gpus all \ --name qwen \ -p 8000:8000 \ -v /path/Qwen3-0.6B:/model \ -v /path/vllm-logging.json:/tmp/vllm-logging.json \ -e VLLM_LOGGING_CONFIG_PATH/tmp/vllm-logging.json \ -e VLLM_LOGGING_LEVELINFO \ vllm/vllm-openai:latest \ /model \ --served-model-name qwen3启动成功之后通过docker logs -f qwen可以看到日志已经是按照json格式输出了如下所示(APIServer pid1) {asctime: 2026-10-03 09:30:36,948, levelname: ERROR, name: vllm.entrypoints.openai.chat_completion.serving, message: Error with model errorErrorInfo(messageThe model /qwen3 does not exist., typeNotFoundError, parammodel, code404)} (APIServer pid1) {asctime: 2026-10-03 09:31:02,088, levelname: INFO, name: vllm.v1.metrics.loggers, message: Engine 000: Avg prompt throughput: 1.5 tokens/s, Avg generation throughput: 12.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%} (APIServer pid1) {asctime: 2026-10-03 09:31:12,089, levelname: INFO, name: vllm.v1.metrics.loggers, message: Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%}设置Fluent-Bit采集日志信息在运行VLLM的服务器中运行Fluent-Bit容器采集VLLM容器的日志信息。这里我采用的是ghcr.m.daocloud.io/openinsight-proj/fluent-bit:4.0.1这个社区版镜像因为官方镜像的jemalloc内存分配器对系统的内存页大小由严格要求必须是4096。但是在昇腾服务器的麒麟Linux里面默认页大小是65536会出现不匹配的问题。配置fluent-bit.conf[SERVICE] Flush 1 Log_Level info Parsers_File /fluent-bit/etc/parsers.conf [INPUT] Name tail Path /var/lib/docker/containers/*/*-json.log Parser docker Tag docker.* Refresh_Interval 5 Rotate_Wait 30 #Docker_Mode true Path_Key filepath #Read_from_Head true #Ignore_Older 0 DB /fluent-bit/db/flb_tail.db #Docker_Mode true #Docker_Mode_Flush 5 # 只采集运行中的容器 #Interval_Sec 1 [FILTER] Name lua Match docker.* script /fluent-bit/etc/enrich_record.lua call enrich_record [FILTER] Name rewrite_tag Match docker.* Rule $container_name ^(qwen.*)$ vllm.$1 false Rule $container_name ^(apiserver.*)$ apiserver.$1 false Rule $container_name .* other.$1 false [FILTER] Name parser Match vllm.* Key_Name log Parser vllm_prefix Reserve_Data On [FILTER] Name parser Match vllm.* Key_Name payload Parser vllm_payload Reserve_Data On [FILTER] Name modify Match vllm.* Add hostname ${HOSTNAME} Remove filepath [OUTPUT] Name loki Match vllm.* Host 10.135.248.240 Port 3100 Labels jobvllm, instance${HOST_IP}, modelQwen3.6-35B, role$role, index$index, level$levelname #Label_keys $levelname Line_format json [FILTER] Name parser Match apiserver.* Key_Name log Parser apiserver_json Reserve_Data On [FILTER] Name modify Match apiserver.* Add hostname ${HOSTNAME} Remove filepath [OUTPUT] Name loki Match apiserver.* Host 10.135.248.240 Port 3100 Labels jobapiserver, instance${HOST_IP}, level$levelname Line_format json在配置文件里面INPUT后面跟着的第一个lua FILTER是用一个Lua程序对宿主机的容器日志文件按照容器名来过滤并且提取日志中感兴趣的信息。程序如下:function enrich_record(tag, timestamp, record) local log record[log] if not log then return 2, timestamp, record end -- 清洗 ANSI 转义序列 local cleaned string.gsub(log, \27%[[%d;]*m, ) cleaned string.gsub(cleaned, \27%[[%d;]*[A-Za-z], ) cleaned string.gsub(cleaned, \27, ) record[log] cleaned -- 提取 levelname local level string.match(cleaned, levelname%s*:%s*([^])) if not level then level string.match(cleaned, level%s*:%s*([^])) end if level then record[levelname] level end -- 从 filepath 提取容器 ID local path record[filepath] if not path then return 2, timestamp, record end local container_id string.match(path, containers/([^/])/) if not container_id then return 2, timestamp, record end -- 读取 config.v2.json 获取容器名 local config_path /var/lib/docker/containers/ .. container_id .. /config.v2.json local f io.open(config_path, r) if not f then return 2, timestamp, record end local content f:read(*all) f:close() local name string.match(content, Name%s*:%s*/([^])) if not name then return 2, timestamp, record end record[container_name] name -- 解析 role 和 index local role, index string.match(name, _(prefill)_(%d)$) if not role then role, index string.match(name, _(decode)_(%d)$) end if role then record[role] role record[index] index end return 2, timestamp, record end在rewrite_tag FILTER里面定义了几个RULE根据不同的容器名来打上不同的tag。配置文件里面提到的Parser在以下parsers.conf文件中定义[PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name vllm_prefix Format regex Regex ^\((?process[^)])\)\s(?payload\{.*) [PARSER] Name vllm_payload Format json [PARSER] Name apiserver_json Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L%z Time_Keep On最后运行容器即可docker run -d \ --name fluent-bit \ --user root \ -v /var/lib/docker/containers:/var/lib/docker/containers:ro \ -v /var/run/docker.sock:/var/run/docker.sock:ro \ -v /opt/fluent-bit/db:/fluent-bit/db \ -v /opt/fluent-bit/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro \ -v /opt/fluent-bit/parsers.conf:/fluent-bit/etc/parsers.conf:ro \ -v /opt/fluent-bit/enrich_record.lua:/fluent-bit/etc/enrich_record.lua:ro \ -e HOSTNAME$(hostname) \ ghcr.m.daocloud.io/openinsight-proj/fluent-bit:4.0.1设置LOKI接收所有Fluent-Bit容器推送的日志docker run -d --name loki \ --restart unless-stopped \ -p 3100:3100 \ -v /data/loki/local-config.yaml:/etc/loki/local-config.yaml:ro \ -v /data/loki/data:/loki \ --config.file/etc/loki/local-config.yaml \ grafana/loki:latestGrafana查询显示Loki日志最后在Grafana里面配置添加数据源输入Loki服务器的IP和端口即可。然后在探索里面就可以对日志进行查询了。如下图