发布于 2024-07-08 14:06:14

Grafana + Loki + promtail 日志搭建

概念与ELK相比

grafana :kibana
loki : elasticsearch
promtail : logstash

准备

docker镜像拉不下来问题： https://www.maocaoying.com/topic/1195
版本： loki promtail 都是3.0.0 ; grafana选的算是比较新的 10.4.5
非k8s环境。 (k8s反而简单)

1. Grafana

docker run --name grafana -d --user=0 -v /root/qin/mydata/grafana:/var/lib/grafana -p 3000:3000 grafana/grafana:10.4.5
默认账号密码 admin/admin

2. Loki

docker run --name loki -d --user=0 -v /root/qin/mydata/loki/:/mnt/config -p 3100:3100 grafana/loki:3.0.0 -config.file=/mnt/config/loki-local-config.yaml

[root@devtest-service loki]# cat loki-local-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 0

common:
  instance_addr: 0.0.0.0
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# # analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
# #
# # Statistics help us better understand how Loki is used, and they show us performance
# # levels for most users. This helps us prioritize features and documentation.
# # For more information on what's sent, look at
# # https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# # Refer to the buildReport method to see what goes into a report.
# #
# # If you would like to disable reporting, uncomment the following lines:
analytics:
  reporting_enabled: false
#

3. promtail

docker run --name promtail -d -v /root/qin/mydata/promtail:/mnt/config -v /var/run/docker.sock:/var/run/docker.sock --link loki grafana/promtail:3.0.0 -config.file=/mnt/config/promtail-docker-config.yaml

主要就是 -v /var/run/docker.sock:/var/run/docker.sock 通过socket联通所有的docker日志。
否则需要配置/var/logs//.log
每个要监控的机器，装一个promtail .地址填目标loki的地址。这里的--link是本机的loki hostname

[root@devtest-service promtail]# cat promtail-docker-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /mnt/config/positions.yaml # 这是内部变量。配置-v .转化为宿主机的

clients:
  - url: http://loki:3100/loki/api/v1/push # 上边--link loki用的

scrape_configs:
  - job_name: dockerlogs
    docker_sd_configs:
      - host: "unix:///var/run/docker.sock"
        refresh_interval: "1s"
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container_name

这里 [__meta_docker_container_name] 解析docker container的名字，作为标签
https://grafana.com/docs/loki/latest/send-data/promtail/configuration/

docker logs tips

容器运行时的是docker，正常情况下我们可以在节点的docker路径中找到容器的stdout的日志，

默认为/var/lib/docker/containers/{containerId}/{containerId}-json.log。

在Kubernetes 1.14版本之前，kubelet会在/var/log/pods/<podUID>/<containerName>/<num>.log建立一个软链接到stdout文件中。
在Kubernetes 1.14版本之后，改成了/var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/<num>.log的形式。

4. docker-compose还没测试，一开始各种跑不起来，分开部署的。

#docker-compose -f loki-logs.yml up -d

services:
  loki:
    image: grafana/loki
    container_name: loki
    restart: always
    ports:
      - "3100:3100"
    volumes:
      - /root/qin/mydata/loki:/mnt/config
    command: -config.file=/mnt/config/loki-local-config.yaml
    user: root
  promtail:
    image: grafana/promtail
    container_name: promtail
    restart: always
    volumes:
      - /root/qin/mydata/promtail:/mnt/config
      -v /var/run/docker.sock:/var/run/docker.sock
    command:
      -config.file="/mnt/config/promtail-docker-config.yaml"
    user: root
  grafana:
    image: grafana/grafana:master
    restart: always
    container_name: grafana
    volumes:
      - /root/qin/mydata/grafana:/var/lib/grafana
    environment:
    - TZ=Asia/Shanghai
    - LANG=zh_CN.UTF-8
    ports:
    - 3000:3000
    user: root

5. prometheus 时序库搭建

充当监控数据

6. prometheus 监控系统采集

每个node节点（物理机）搭建node_exporter。每台机器的内存、硬盘、网络io等导入prometheus . 可以共用grafana看
springboot 集成prometheus。配合actuator 数据自动拉取到prometheus 。JVM、自建监控（线城市,非xxljob定时任务）等
grafana 有针对各种业务开源的dashboard 直接导入

技术 #logs

浏览 (604)