Skip to content

Observability — Setup

Loki (logs), Prometheus (metrics), and Grafana (dashboards) run as Docker containers on LXC 108. Grafana Alloy (unified agent) collects both logs and metrics from Docker hosts, Alpine LXCs, and the observability stack itself, pushing logs to Loki and exposing metrics for Prometheus to scrape.

  • Grafana Alloy — https://github.com/grafana/alloy · https://grafana.com/docs/alloy/
  • Grafana — https://github.com/grafana/grafana · https://grafana.com
  • Loki — https://github.com/grafana/loki · https://grafana.com/oss/loki
  • Prometheus — https://github.com/prometheus/prometheus · https://prometheus.io

Infrastructure

Component Host Internal Role
Loki Docker (LXC 108) 192.168.1.108:3100 Log aggregation
Prometheus Docker (LXC 108) 192.168.1.108:9090 Metrics storage
Grafana Docker (LXC 108) 192.168.1.108:3001 → https://grafana.eva-00.network Dashboards + alerting
Alloy Docker (LXC 108) loki-stack sidecar Loki-stack container logs
Alloy Native (LXC 100) forgejo Forgejo application logs
Alloy Native (LXC 101) runner Forgejo Runner logs
Alloy Native (LXC 105) caddy Caddy journal logs
Alloy Native (LXC 107) ollama Ollama journal logs
Alloy Native (LXC 109) minecraft Minecraft + Playit journal logs
Alloy Docker (LXC 110) gluetun Gluetun container logs
Alloy Docker (LXC 111) seedbox Seedbox container logs
Alloy Docker (LXC 113) mediabot Mediabot container logs
Alloy Docker (LXC 114) jellyfin Jellyfin container logs
Alloy Docker (LXC 115) netbird NetBird container logs
Alloy Docker (LXC 116) all-might All-Might container logs
Alloy Docker (LXC 117) karakeep Karakeep container logs
Alloy Docker (LXC 118) tools code-server, thelounge, qbitwebui logs
Alloy Docker (LXC 119) infra-apps gatus, ntfy, glance, oauth2-proxy logs
Alloy Docker (LXC 120) automation n8n container logs
Alloy Docker (LXC 121) matrix Synapse container logs
Alloy Docker (LXC 122) ai Open WebUI container logs
Alloy Docker (LXC 123) auth PocketID container logs
Python cron Native (LXC 100) forgejo Forgejo action log decompression

Architecture

LXC 100 (forgejo, Alpine)    ── Alloy binary → file tail → Loki
LXC 101 (runner, Alpine)     ── Alloy binary → file tail → Loki
LXC 105 (caddy, Debian)      ── Alloy binary → journal  → Loki
LXC 107 (ollama, Debian)     ── Alloy binary → journal  → Loki
LXC 109 (minecraft, Debian)  ── Alloy binary → journal  → Loki
LXC 110 (gluetun, Debian)    ── Alloy Docker → container logs → Loki
LXC 111 (seedbox, Debian)    ── Alloy Docker → container logs → Loki
LXC 113 (mediabot, Debian)   ── Alloy Docker → container logs → Loki
LXC 114 (jellyfin, Debian)   ── Alloy Docker → container logs → Loki
LXC 115 (netbird, Debian)    ── Alloy Docker → container logs → Loki
LXC 116 (all-might, Debian)  ── Alloy Docker → container logs → Loki
LXC 117 (karakeep, Debian)   ── Alloy Docker → container logs → Loki
LXC 118 (tools, Debian)      ── Alloy Docker → container logs → Loki
LXC 119 (infra-apps, Debian) ── Alloy Docker → container logs → Loki
LXC 120 (automation, Debian) ── Alloy Docker → container logs → Loki
LXC 121 (matrix, Debian)     ── Alloy Docker → container logs → Loki
LXC 122 (ai, Debian)         ── Alloy Docker → container logs → Loki
LXC 123 (auth, Debian)       ── Alloy Docker → container logs → Loki

LXC 108 (observability-host)
  Alloy sidecar ──→ loki-stack container logs ──→ Loki :3100
                                                 ↓
                            ┌────────────────────┴─────────────────┐
                            ↓                                       ↓
                        Grafana :3001 ←── Prometheus :9090 ←── Alloy :12345

IaC

Alloy on Docker-based LXCs (110, 111, 113-123) is deployed as part of each service's own playbook using a shared services/alloy/ config template. The alloy_job_name variable is set per-playbook to label logs by LXC.

Artifact Path
Loki-stack playbook ansible/playbooks/loki-stack.yml → targets observability_hosts
Loki-stack workflow .forgejo/workflows/loki-stack.yml
Alloy playbook (native agents) ansible/playbooks/observability-agents.yml → targets Alpine/native host groups
Alloy workflow (native agents) .forgejo/workflows/observability-agents.yml
Alloy Docker template services/alloy/config.alloy.j2 (shared by all Docker LXCs)
Alloy Docker compose services/alloy/docker-compose.yml (shared by all Docker LXCs)
Native Alloy configs services/alloy-*/config.alloy (per-LXC for native installs)
Prometheus config services/loki-stack/prometheus-config.yml (scrape targets)

Secrets

secret/grafana         → pocketid_client_id, pocketid_client_secret
secret/homelab-sso     → grafana_admin_password

Access

  • URL: https://grafana.eva-00.network
  • Auth: PocketID native OIDC
  • Internal: http://192.168.1.108:3001

Adding a new Prometheus scrape target

Edit services/loki-stack/prometheus-config.yml, add the target under scrape_configs, push to main. Prometheus will scrape at the next config reload.

Collecting logs from a new Docker LXC

Docker-based LXCs get Alloy deployed as part of their service playbook. The Alloy block at the end of each playbook:

  1. Creates /opt/alloy/ and /opt/alloy/data/ directories
  2. Copies services/alloy/docker-compose.yml
  3. Templates services/alloy/config.alloy.j2 using the playbook's alloy_job_name variable
  4. Deploys Alloy via docker_compose_v2
  5. Restarts Alloy if the config changed

No separate workflow needed — Alloy deploys with the service.

Collecting logs from a new native (non-Docker) LXC

  1. Create services/alloy-<service>/config.alloy with loki.source.file for the log path
  2. Create services/alloy-<service>/alloy.initd (copy template from alloy-runner or alloy-forgejo)
  3. Add a play to observability-agents.yml for the new host group
  4. Update the workflow path triggers to include services/alloy-<service>/**