Skip to content

Backup — Runbook

Check backup status

Databasement

  1. Open https://databasement.eva-00.network (PocketID login)
  2. Dashboard shows all database servers, backup jobs, last run times, and snapshot history
  3. Ntfy notifications go to the backups topic on success/failure

Backrest

  1. Open https://backrest.eva-00.network (PocketID login)
  2. Dashboard shows all plans, last run times, and snapshot history
  3. Plans: vault-snapshot, app-configs, databasement-dumps

PBS

  1. Open https://pbs.eva-00.network (PocketID login)
  2. Dashboard shows last backup time, datastore usage, and task history
  3. Or via CLI on cajita-elite:
ssh [email protected]
proxmox-backup-client list --repository localhost:pbs-local

Trigger a manual backup

Databasement — single database

Open Databasement UI > Backups > click "Run Now" on the desired backup job.

Backrest — single plan

Open Backrest UI > Plans > click "Backup Now" on the desired plan.

PBS — single LXC

From chizuru (192.168.1.125):

vzdump <VMID> --storage cajita-elite --mode snapshot --compress zstd

Restore from Databasement

Restore a database

  1. Open Databasement UI > Backups > select the backup job
  2. Find the snapshot to restore from
  3. Click "Restore" and select the target database server
  4. Databasement handles the import automatically

Restore manually from Databasement files

If Databasement UI is unavailable, backups are also captured by Backrest:

ssh [email protected]
export RESTIC_REPOSITORY=/opt/backrest/repos/homelab
export RESTIC_PASSWORD="<from Vault secret/backrest>"

# List databasement-dumps snapshots
restic snapshots --tag databasement-dumps

# Restore to temp directory
restic restore <SNAPSHOT_ID> --target /tmp/restore

# Databasement stores dumps in /opt/databasement/data/
# Import manually:
# PostgreSQL: psql -f /tmp/restore/opt/databasement/data/<dump_file>.sql
# MariaDB: mariadb < /tmp/restore/opt/databasement/data/<dump_file>.sql
# SQLite: cp /tmp/restore/opt/databasement/data/<db_file>.db /path/to/target

Restore from Backrest (restic)

List snapshots

ssh [email protected]
export RESTIC_REPOSITORY=/opt/backrest/repos/homelab
export RESTIC_PASSWORD="<from Vault secret/backrest>"
restic snapshots

Restore Vault from raft snapshot

# Get snapshot from restic
restic restore <SNAPSHOT_ID> --target /tmp/restore

# On Vault LXC (106)
VAULT_ADDR=http://127.0.0.1:8200 vault operator raft snapshot restore \
  /tmp/restore/opt/backrest/dumps/vault/vault_YYYYMMDD.snap

Restore app configs

restic restore <SNAPSHOT_ID> --target /tmp/restore
# Files in /tmp/restore/opt/backrest/dumps/configs/{caddy,forgejo,proxmox}/
# Copy to target locations as needed

Restore from PBS

Restore entire LXC

From Proxmox UI: 1. Datacenter > Storage > cajita-elite > Content 2. Find the backup, click Restore 3. Choose target storage pool and LXC ID

From CLI:

# On chizuru
pct restore <NEW_VMID> <BACKUP_FILE> --storage <POOL> --force

Restore single file from PBS

# On cajita-elite, mount a backup
proxmox-backup-client mount <SNAPSHOT> /mnt/restore --repository localhost:pbs-local
# Browse and copy files
cp /mnt/restore/path/to/file /destination
# Unmount when done
umount /mnt/restore

18TB external drive workflow

When the external drive is plugged in:

  1. Connect to cajita-elite or chizuru (wherever the drive is attached)
  2. Mount the drive: mount /dev/sdX1 /mnt/external
  3. Sync unohana: rsync -avP /mnt/all-might/ /mnt/external/unohana/
  4. Optionally sync PBS datastore: rsync -avP /mnt/datastore/pbs-local/ /mnt/external/pbs/
  5. Unmount: umount /mnt/external

Troubleshooting

Databasement issues

  • SSH tunnel connection failed: Verify cajita-elite can SSH to the target LXC: ssh root@<IP> true
  • Database dump failed: Check if the Docker container is running: ssh root@<IP> docker ps
  • SQLite file not found: Verify the file path exists on the LXC host (must be bind-mounted, not in a Docker named volume)
  • Backup not running: Check Databasement logs: ssh [email protected] docker logs databasement

Backrest issues

  • Vault snapshot 403: Token expired or missing — re-run backrest playbook to redeploy token, verify /opt/backrest/.vault-token exists on cajita-elite
  • Vault sealed: Unseal Vault before vault-snapshot can run
  • Config collection SSH failure: First connection to a new LXC may fail — run ssh -o StrictHostKeyChecking=accept-new root@<IP> true from cajita-elite
  • Restic repo locked: restic unlock (usually means a previous backup was interrupted)

PBS backup failure

Check task log in PBS UI or:

ssh [email protected]
proxmox-backup-manager task list --limit 10
proxmox-backup-manager task log <UPID>

Common issues: - LXC locked (another operation in progress): wait or pct unlock <VMID> on chizuru - Datastore full: check usage, run GC manually: proxmox-backup-manager garbage-collection start pbs-local

Ntfy notifications not arriving

Test manually from cajita-elite:

curl -X POST "https://ntfy.eva-00.network/backups" -H "Title: Test" -d "Backup test notification"

If that fails, check DNS resolution on cajita-elite (/etc/resolv.conf must have public DNS).