Skip to content

ADR-005 — Backup Infrastructure Migration: Databasement + Backrest + PBS

Date: 2026-04-09 Status: Accepted

Context

The homelab backup infrastructure on cajita-elite currently uses:

  • PBS — whole-LXC block-level snapshots (critical nightly, non-critical weekly)
  • Backrest (restic) — application-level backups via custom shell hook scripts that SSH into LXCs, run dump commands, and snapshot the output

The Backrest hook scripts (dump-postgres.sh, dump-mariadb.sh, dump-sqlite.sh, snapshot-vault.sh, collect-configs.sh, notify.sh) are hand-written shell that:

  • SSH into each LXC and run pg_dumpall, mariadb-dump, docker cp + sqlite3 .backup
  • Require sqlite3 installed on target LXC hosts
  • Need special handling per service (Forgejo uses forgejo dump, PocketID has DB on host, others need docker cp)
  • Are fragile to debug (no structured logging, opaque failures)
  • Don't follow IaC principles — they're imperative scripts, not declarative config

This conflicts with the homelab's core principle: everything is IaC.

Decision

Replace the custom Backrest hook scripts with Databasement for all database backups. Keep Backrest for file/config backups and Vault raft snapshots. Keep PBS for whole-LXC disaster recovery.

New architecture (three layers)

Layer Tool What it covers
Database dumps Databasement PostgreSQL, MariaDB, SQLite across all LXCs
File/config backups Backrest (restic) Databasement dump output, Vault raft snapshots, app configs
Whole-LXC snapshots PBS Full rootfs of all 22 LXCs (disaster recovery)
Offline/media 18TB external unohana, filedump, PBS export (manual rsync)

Databasement details

  • What: Self-hosted database backup manager with web UI (David-Crty/databasement)
  • Deployment: Single Docker container on cajita-elite (davidcrty/databasement:1, port 2226)
  • Databases supported: MySQL, PostgreSQL, MariaDB, MongoDB, SQLite, Redis/Valkey
  • Connection method: SSH tunnels from cajita-elite into LXCs (reuses existing SSH key infrastructure)
  • Backup methods: pg_dump for PostgreSQL, mariadb-dump for MariaDB, file copy via SFTP for SQLite
  • Storage: Local filesystem (/data volume on cajita-elite)
  • Retention: GFS policy (7 daily, 4 weekly, 12 monthly)
  • Notifications: Ntfy webhook for success/failure
  • Auth: PocketID (native OIDC, no oauth2-proxy needed), holo as admin via OAUTH_DEFAULT_ROLE=admin
  • API: REST API + MCP server for automation/AI integration
  • License: MIT

Database inventory

Database LXC Engine Databasement connection
Paperless 124 PostgreSQL SSH tunnel → port 5432 (Docker)
MediaManager 112 PostgreSQL SSH tunnel → port 5432 (Docker)
Grimmory + RomM 116 MariaDB SSH tunnel → port 3306 (Docker)
Forgejo 100 SQLite SFTP → /var/lib/forgejo/db/forgejo.db
Synapse 121 SQLite SFTP → bind-mounted host path
PocketID 123 SQLite SFTP → host filesystem path
N8N 120 SQLite SFTP → bind-mounted host path
Jellyfin 114 SQLite SFTP → bind-mounted host path
Open-WebUI 122 SQLite SFTP → bind-mounted host path
Shoko 116 SQLite SFTP → bind-mounted host path
Karakeep 117 SQLite SFTP → bind-mounted host path

Future-ready: Immich (PostgreSQL), Tube Archivist (Elasticsearch).

Backrest reconfiguration

Plan Schedule What
databasement-dumps 2:30am daily Restic snapshot of Databasement's /data directory
vault-snapshot 2:15am daily Vault raft snapshot via HTTP API (single curl, kept as inline hook)
app-configs 2:20am daily SSH collection of Caddy, Forgejo, Proxmox configs

Removed: postgres-dumps, mariadb-dumps, sqlite-dumps plans and all 6 hook scripts.

PBS (unchanged)

Schedule What
Nightly 4am Critical LXCs: 100, 105, 106, 108, 119, 120, 123, 124
Weekly Monday 4am Non-critical LXCs: 101, 102, 104, 107, 109, 110, 112, 113, 114, 115, 116, 117, 118, 121, 122
Saturday 3am Garbage collection
Daily 5am Prune (7 daily, 4 weekly, 2 monthly)

Daily timeline

2:00am  Databasement: all DB dumps (PG, MariaDB, SQLite)
2:15am  Backrest: vault-snapshot (API call)
2:20am  Backrest: app-configs (SSH collection)
2:30am  Backrest: databasement-dumps (restic snapshot of dump output)
3:00am  PBS: garbage collection (Saturdays)
4:00am  PBS: critical LXC snapshots (nightly)
4:00am  PBS: non-critical LXC snapshots (Mondays)
5:00am  PBS: prune
6:00am  Backrest: repo prune (Sundays)
7:00am  Backrest: repo integrity check (1st of month)

Trade-offs

Accepted

  • SQLite backup safety: Databasement uses cp (file copy via SFTP) for SQLite, not sqlite3 .backup. For homelab traffic levels, corruption risk during a copy is negligible. Mitigated by: most apps use WAL mode, PBS provides a separate full-LXC snapshot, and backups run at 2am when traffic is near zero.
  • SQLite DBs must be on host-accessible paths: Docker containers with SQLite must bind-mount their DB files to host paths so Databasement can reach them via SFTP. This aligns with the existing CLAUDE.md rule ("Docker services use bind mounts to /opt/<app>/{config,database,assets}").
  • Vault remains a custom hook: No backup tool natively supports Vault raft snapshots. The single curl call is kept as a Backrest inline pre-hook — 3 lines, not a fragile script.
  • Three tools instead of one: No single tool covers DB dumps + file backups + LXC snapshots. Each tool does one thing well with minimal overlap.
  • Meilisearch data on urahara: Karakeep's Meilisearch uses a 2TB sparse data.mdb file (~1MB real data). This caused PBS backups to take 40+ minutes despite only containing negligible real data. Moved to urahara bind mount so PBS skips it. Meilisearch data survives restarts and PBS restores (bind mount re-attaches). Only requires manual reindex via Karakeep admin UI if the urahara disk is lost. Both disks (heavy-pool SSD, urahara SSD) have comparable I/O — no performance impact. See: Meilisearch storage docs.

Rejected alternatives

  • Borgmatic: Native YAML DB dump declarations, but requires switching from restic to Borg. No web UI. Uses sqlite3 .dump (SQL text) instead of .backup (binary safe copy). Gains don't justify the migration cost.
  • Databasus: Similar to Databasement but no SQLite support, requires PostgreSQL + Valkey for its own backend (heavier deployment). Databasement uses SQLite internally.
  • Custom unified tool: Building a single tool covering all backup types was considered but the development effort is unjustified when three mature tools cover the same ground.
  • Switching all DBs to one engine: Converting all services to PostgreSQL/MongoDB would eliminate engine diversity but requires upstream support per application and significant migration effort.

Implementation plan

Phase 0: Undo current hook infrastructure

  1. SSH to cajita-elite, stop Backrest DB dump plans, delete hook scripts and dump directories
  2. Remove services/backrest/hooks/ from chizuru-v2 repo
  3. Simplify backrest.yml playbook (remove hook deployment tasks)
  4. Clean up debug artifacts on runner (LXC 101)

Phase 1: Deploy Databasement

  1. Create services/databasement/docker-compose.yml
  2. Create ansible/playbooks/databasement.yml (deploy Docker Compose on cajita-elite)
  3. Create .forgejo/workflows/databasement.yml
  4. Add Caddy route for databasement.eva-00.network (native OIDC, no oauth2-proxy)
  5. Add Databasement to Glance dashboard
  6. Verify all SQLite DBs are bind-mounted to host paths; fix any that aren't
  7. Configure database servers and backup schedules via Databasement web UI
  8. Set up Ntfy notifications

Phase 2: Reconfigure Backrest

  1. Remove postgres-dumps, mariadb-dumps, sqlite-dumps plans from Backrest UI
  2. Add databasement-dumps plan (snapshot Databasement's /data output)
  3. Keep vault-snapshot as inline pre-hook (no external script)
  4. Keep app-configs plan unchanged
  5. Update Backrest playbook to reflect new plan structure

Phase 3: QA validation

  1. Create test LXCs: one PostgreSQL, one MariaDB, one SQLite service
  2. Populate with test data
  3. Test Databasement backup + restore for each DB type
  4. Test Backrest file restore from databasement-dumps snapshot
  5. Test PBS whole-LXC restore
  6. Test combined disaster recovery scenario (LXC destroyed → PBS restore → verify DB data intact)

Phase 4: Documentation

  1. Update setup.md with Databasement deployment details
  2. Update whats-backed-up.md with new tool assignments
  3. Update runbook.md with Databasement restore procedures
  4. Clean up references to removed hook scripts

References