Backup — Setup

Backup infrastructure runs on cajita-elite (standalone PC, not a Proxmox LXC). Three tools: PBS for whole-LXC snapshots, Databasement for database backups, Backrest (restic) for file/config/vault backups.

Links

PBS: https://pbs.proxmox.com/docs/
Backrest: https://github.com/garethgeorge/backrest
Databasement: https://github.com/David-Crty/databasement
Restic: https://restic.net

Infrastructure

Component	Host	IP	Port	URL
PBS	cajita-elite	192.168.1.196	8007	https://pbs.eva-00.network
Backrest	cajita-elite	192.168.1.196	9898	https://backrest.eva-00.network
Databasement	cajita-elite	192.168.1.196	2226	https://databasement.eva-00.network

cajita-elite hardware

Component	Details
CPU	Intel i5-8500T (6 cores)
RAM	8GB
OS	PBS 4.0 on Debian 13
nvme0n1	931GB — PBS OS + Backrest repos + Databasement data (`/opt/backrest/`, `/opt/databasement/`)
nvme1n1	931GB — ZFS, PBS datastore `pbs-local`

IaC

Artifact	Path
PBS playbook	`ansible/playbooks/pbs.yml`
PBS workflow	`.forgejo/workflows/pbs.yml`
Backrest playbook	`ansible/playbooks/backrest.yml`
Backrest workflow	`.forgejo/workflows/backrest.yml`
Backrest config	`services/backrest/config.json`
Databasement playbook	`ansible/playbooks/databasement.yml`
Databasement workflow	`.forgejo/workflows/databasement.yml`
Databasement compose	`services/databasement/docker-compose.yml`
Inventory entry	`ansible/inventory/hosts.yml` (group: `backup`)

Architecture

                 ┌───────────────────────────────────────────┐
                 │           cajita-elite (PBS host)         │
                 │                                           │
  4am ────────►  │  nvme1n1: PBS datastore (pbs-local)       │
  LXC snapshots  │  - Critical LXCs: nightly                 │
  from chizuru   │  - Non-critical LXCs: weekly Mon          │
                 │  - Retention: 7d / 4w / 2m                │
                 │  - GC: Saturday 3am                       │
                 │  - Prune: daily 5am                       │
                 │                                           │
  2am ────────►  │  nvme0n1: Databasement (Docker)           │
  DB dumps via   │  - PostgreSQL dumps (pg_dump)             │
  SSH tunnels    │  - MariaDB dumps (mariadb-dump)           │
                 │  - SQLite backups (file copy via SFTP)    │
                 │  - Retention: GFS (7d / 4w / 12m)        │
                 │                                           │
  2:15am ─────►  │  nvme0n1: Backrest repos                  │
  Vault + configs│  - Vault raft snapshots (API)             │
  + Databasement │  - App config collection (SSH)            │
  dump snapshot  │  - Databasement dump snapshot             │
                 │  - Retention: 7d / 4w / 2m                │
                 │                                           │
                 │  Alloy (Docker)                           │
                 │  - Logs → Loki (LXC 108:3100)            │
                 │  - Metrics → Prometheus (LXC 108:9090)   │
                 │  - Ntfy notifications → backups topic     │
                 └───────────────────────────────────────────┘

                    │                              │
                    │ on-demand (plug-in)           │ continuous
                    ▼                              ▼
          ┌──────────────────────┐    ┌──────────────────────────┐
          │  senku (T7 Shield)   │    │  Grafana (LXC 108)       │
          │  1.8TB USB SSD, ext4 │    │  Backups dashboard:      │
          │  - PBS mirror        │    │  - PBS snapshot table     │
          │  - Backrest mirror   │    │  - Databasement table     │
          │  - Databasement copy │    │  - Datastore usage gauge  │
          │  UUID: c45c6f1e-...  │    │  - Backup event feed      │
          │  Mount: /mnt/senku   │    │  Embedded in Glance tab   │
          └──────────────────────┘    └──────────────────────────┘

Tool responsibilities

Layer	Tool	What it covers
Database dumps	Databasement	PostgreSQL, MariaDB, SQLite across all LXCs
File/config backups	Backrest (restic)	Databasement dump output, Vault raft snapshots, app configs
Whole-LXC snapshots	PBS	Full rootfs of all 22 LXCs (disaster recovery)
On-demand offsite	senku (USB SSD)	rsync mirror of PBS + Backrest + Databasement

senku (USB backup drive)

Samsung T7 Shield 1.8TB USB SSD. On-demand backup-of-backups — plug it in, sync runs automatically via udev trigger.

How it works

Plug in senku → udev detects UUID → triggers mnt-senku.mount + backup-sync-senku.service
Ntfy "started" notification sent immediately
rsync mirrors PBS datastore, Backrest repo, Databasement data to /mnt/senku/
Ntfy "complete" notification sent on finish (or "partial"/"failed" on error)
Unplug when done — data persists on the SSD

Manual trigger: systemctl start backup-sync-senku.service

What's synced

Source	Path on cajita-elite	Path on senku
PBS datastore	`/mnt/datastore/pbs-local/`	`/mnt/senku/pbs-local/`
Backrest	`/opt/backrest/`	`/mnt/senku/backrest/`
Databasement	`/opt/databasement/data/`	`/mnt/senku/databasement/`

IaC

Artifact	Path
Playbook	`ansible/playbooks/backup-sync.yml`
Workflow	`.forgejo/workflows/backup-sync.yml`

Drive details

Property	Value
Model	Samsung PSSD T7 Shield
Serial	S6NLNS0W603300R
Size	1.8 TB
Filesystem	ext4, label `senku`
UUID	`c45c6f1e-92b7-4441-a55e-575c014980d6`
Mount point	`/mnt/senku`

Databasement

Deployment

Single Docker container on cajita-elite. Configured via web UI at https://databasement.eva-00.network.

# services/databasement/docker-compose.yml
services:
  databasement:
    image: davidcrty/databasement:1
    container_name: databasement
    ports:
      - "2226:2226"
    environment:
      - DB_CONNECTION=sqlite
      - DB_DATABASE=/data/database.sqlite
      - ENABLE_QUEUE_WORKER=true
      - APP_URL=https://databasement.eva-00.network
      - APP_TIMEZONE=America/New_York
    volumes:
      - /opt/databasement/data:/data
    restart: unless-stopped

Database connections

Databasement connects to databases via SSH tunnels from cajita-elite into LXCs. It reuses the same SSH key infrastructure as Backrest (ed25519 key at /root/.ssh/id_ed25519, distributed to all LXCs by the backrest playbook).

Database	LXC	Engine	Install Type	Connection	Host path for SQLite
Paperless	124	PostgreSQL	Docker	SSH tunnel → Docker port 5432	—
MediaManager	113	PostgreSQL	Docker (mediabot stack)	SSH tunnel → Docker port 5432 (bound on 127.0.0.1)	—
Grimmory + RomM	116	MariaDB	Docker	SSH tunnel → Docker port 3306	—
Forgejo	100	SQLite	Native (Alpine pkg)	SFTP file copy	`/var/lib/forgejo/db/forgejo.db`
PocketID	123	SQLite	Docker	SFTP file copy	`/opt/pocketid/data/pocket-id.db`
N8N	120	SQLite	Docker	SFTP file copy	`/opt/n8n/data/database.sqlite`
Jellyfin	114	SQLite	Docker	SFTP file copy	`/opt/jellyfin/config/data/jellyfin.db`
Open-WebUI	122	SQLite	Docker	SFTP file copy	`/opt/open-webui/data/webui.db`
Shoko	116	SQLite	Docker	SFTP file copy	`/opt/shoko/config/Shoko.CLI/SQLite/JMMServer.db3`
Karakeep	117	SQLite	Docker	SFTP file copy	`/opt/karakeep/data/db.db`, `queue.db`

Note: Install type (Docker vs native) doesn't affect Databasement's SSH tunnel approach. For PostgreSQL/MariaDB, the tunnel connects to 127.0.0.1:<port> regardless of whether the DB runs in Docker or natively. For SQLite, the DB file must be on a host-accessible path (bind mount for Docker, direct path for native).

SQLite backup note

Databasement uses file copy (via SFTP) for SQLite backups, not sqlite3 .backup. For homelab traffic at 2am, corruption risk is negligible. All apps use WAL mode, and PBS provides a separate full-LXC snapshot as a safety net.

Prerequisite: All SQLite databases must be on bind-mounted host paths (not Docker named volumes) so Databasement can access them via SFTP. Jellyfin and Karakeep were migrated from named volumes to bind mounts as part of this setup.

Meilisearch (Karakeep)

Karakeep's Meilisearch search index uses a sparse data.mdb file (2TB apparent, <1MB real data). This file was causing PBS backups of LXC 117 to take 40+ minutes.

Solution: Meilisearch data is stored on urahara at /mnt/pve/urahara/karakeep/meilisearch, bind-mounted into LXC 117 at /opt/karakeep/meilisearch. PBS automatically excludes bind mounts, so backups are fast.

Behavior:

Scenario	Result
Restart / reboot	Meilisearch finds data via bind mount. No reindex.
PBS restore of LXC 117	Re-attach bind mount from urahara. Data is still there. No reindex.
urahara disk failure	Index lost. Manual reindex via Karakeep Admin > Background Jobs > "Reindex All Bookmarks".

Performance: Both heavy-pool (Samsung 860 EVO) and urahara (Crucial MX500) are SATA SSDs. Meilisearch uses LMDB (memory-mapped), so reads mostly hit page cache. No measurable performance difference.

Backrest

Plans

Plan	Schedule	What
vault-snapshot	2:15am daily	Vault raft snapshot via HTTP API (inline hook)
app-configs	2:20am daily	SSH collection of Caddy, Forgejo, Proxmox configs
databasement-dumps	2:30am daily	Restic snapshot of Databasement's `/opt/databasement/data`

Vault raft snapshot

The vault-snapshot plan uses an inline pre-hook (no external script) that hits the Vault API directly from cajita-elite over LAN (http://192.168.1.106:8200/v1/sys/storage/raft/snapshot). Token stored at /opt/backrest/.vault-token, deployed by the backrest playbook. The token's policy (claude) includes sys/storage/raft/snapshot read access.

App config collection

The app-configs plan collects via SCP:

Source	LXC/Host	What's collected
Caddy	105	`/opt/caddy/Caddyfile`
Forgejo	100	`/etc/forgejo/app.ini`
Proxmox	chizuru	`/etc/pve/storage.cfg`, all LXC configs

PBS (whole LXC rootfs snapshots)

Critical — nightly 4am

LXC	Service	Why critical
100	Forgejo	Source code, CI config
105	Caddy	Reverse proxy, TLS certs
106	Vault	All secrets
108	Observability	Grafana dashboards, Prometheus data
119	Infra-apps	Gatus, Ntfy, Glance, OAuth2 proxies
120	Automation (n8n)	Workflows
123	Auth (PocketID)	SSO config
124	Paperless	Documents

Non-critical — weekly Monday 4am

LXC	Service
101	Forgejo runner
102	FileDump
104	Homebridge
107	Ollama
109	Minecraft
110	dlbox
113	Mediabot
114	Jellyfin
115	NetBird
116	all-might
117	Karakeep
118	Tools
122	AI (Open WebUI)

Not backed up by PBS: Bind-mounted media (unohana, seedbox, filedump, urahara assets) — these live on host disks and are not part of ZFS rootfs snapshots.

Not backed up (by design)

Data	Location	Reason
unohana (media)	sdc 4TB `/mnt/all-might`	Too large for PBS; backed up to 18TB external
seedbox downloads	sdb 2TB `/mnt/seedbox`	Re-downloadable; optionally backed up to 18TB
filedump	sda 12TB `/mnt/filedump`	Backed up to 18TB external
urahara assets	sdd 2TB `/mnt/pve/urahara`	Karakeep, Paperless media, Ollama models

Observability

cajita-elite runs Grafana Alloy (Docker) to ship logs and metrics to the Grafana stack on LXC 108.

What's collected

Source	Type	How
Databasement container	Logs	Docker log driver via Alloy
PBS, Backrest, senku sync	Logs	systemd journal via Alloy
cajita-elite host	Metrics	Node exporter (CPU, disk, memory, network)

All data lands in Loki/Prometheus with job="cajita-elite".

Grafana Backups Dashboard

Provisioned automatically at https://grafana.eva-00.network/d/backups-overview/backups. Panels:

Panel	Datasource	What it shows
PBS - LXC Snapshots	Infinity → PBS API	Per-LXC snapshot count, latest backup time, size
Databasement - Database Backups	Infinity → Databasement API	Per-service backup status, engine, size
PBS Datastore Usage	Infinity → PBS API	Gauge showing pbs-local disk usage
PBS Datastore Details	Infinity → PBS API	Used/Total/Available breakdown
Backup Events	Loki	Live log feed from cajita-elite (backup start/complete/fail)

The Infinity datasource plugin queries PBS and Databasement REST APIs directly. Auth tokens are provisioned in grafana-datasources.yml.

Glance Backups Tab

The Glance dashboard has a "Backups" tab that embeds the Grafana dashboard via iframe (&kiosk mode) plus links to PBS, Backrest, and Databasement UIs.

IaC

Artifact	Path
Alloy config (cajita-elite)	`services/alloy/config-cajita-elite.alloy`
Alloy compose (cajita-elite)	`services/alloy/docker-compose-cajita-elite.yml`
Grafana dashboard JSON	`services/observability/dashboards/backups.json`
Dashboard provisioning	`services/observability/grafana-dashboard-provisioning.yml`
Datasource provisioning	`services/observability/grafana-datasources.yml`

Notifications

All backup jobs send start and completion notifications to the backups ntfy topic at https://ntfy.eva-00.network/backups.

What sends notifications

Source	Start	Complete	Fail
senku sync	Yes	Yes	Yes (partial or full)
Backrest: vault-snapshot	Yes	Yes	Yes
Backrest: app-configs	Yes	Yes	Yes
Backrest: databasement-dumps	Yes	Yes	Yes
PBS (Proxmox backup jobs)	No (built-in)	No	Email on failure
Databasement	No (no hook system)	No	No

Receiving notifications

Install the ntfy app on your phone:

iOS: ntfy on App Store
Android: ntfy on Google Play or F-Droid

Setup:

Open the ntfy app
Tap the server icon (top-left) → Add server
Enter https://ntfy.eva-00.network as the server URL
Go back, tap + to subscribe to a topic
Enter backups as the topic name
You'll now receive push notifications for all backup events

You can also subscribe to other topics (e.g., service-specific alerts) the same way.

Auth

Service	Auth method	Caddy route
Backrest	oauth2-proxy on LXC 119 (port 8590)	backrest.eva-00.network
PBS	oauth2-proxy on LXC 119 (port 8591)	pbs.eva-00.network
Databasement	Native OIDC (PocketID direct)	databasement.eva-00.network → cajita-elite:2226

Databasement uses native OIDC support — no oauth2-proxy needed. First user auto-created as admin via OAUTH_DEFAULT_ROLE=admin. OIDC client_id/secret deployed via .env from Vault.

Secrets

Vault path	Keys
`secret/backrest`	restic_password, backrest_password
`secret/cajita-elite`	root_password
`secret/external-oauth2-proxies`	backrest_client_id/secret/cookie, pbs_client_id/secret/cookie, databasement_client_id/secret/cookie

SSH access

cajita-elite root SSH uses password auth. Backrest playbook generates ed25519 key and distributes pubkey to all LXCs — this key is also used by Databasement for SSH tunnel connections.