Skip to content

Backup QA Test Plan

Validate that all three backup layers (Databasement, Backrest, PBS) can back up and restore data correctly before relying on them in production.

Test environment

Create temporary test LXCs on chizuru to avoid touching production data. Each test LXC runs one DB engine with sample data.

Test LXCs to create

LXC Purpose Template Storage pool IP
200 PostgreSQL test Debian 12 apps-pool 192.168.1.200
201 MariaDB test Debian 12 apps-pool 192.168.1.201
202 SQLite test Debian 12 apps-pool 192.168.1.202

Each LXC needs: - Docker installed - SSH access from cajita-elite (add backrest pubkey to authorized_keys) - Test database container running with sample data

Sample data setup

PostgreSQL (LXC 200): 

docker run -d --name test-pg -p 5432:5432 \
  -e POSTGRES_PASSWORD=testpass \
  -v /opt/test-pg/data:/var/lib/postgresql/data \
  postgres:16

# Insert test data
docker exec test-pg psql -U postgres -c "
  CREATE DATABASE testdb;
  \c testdb;
  CREATE TABLE items (id SERIAL PRIMARY KEY, name TEXT, created_at TIMESTAMP DEFAULT NOW());
  INSERT INTO items (name) VALUES ('item1'), ('item2'), ('item3');
"

MariaDB (LXC 201): 

docker run -d --name test-mariadb -p 3306:3306 \
  -e MARIADB_ROOT_PASSWORD=testpass \
  -v /opt/test-mariadb/data:/var/lib/mysql \
  mariadb:11

# Insert test data
docker exec test-mariadb mariadb -u root -ptestpass -e "
  CREATE DATABASE testdb;
  USE testdb;
  CREATE TABLE items (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(255), created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP);
  INSERT INTO items (name) VALUES ('item1'), ('item2'), ('item3');
"

SQLite (LXC 202): 

apt-get install -y sqlite3
mkdir -p /opt/test-sqlite
sqlite3 /opt/test-sqlite/test.db "
  CREATE TABLE items (id INTEGER PRIMARY KEY, name TEXT, created_at TEXT DEFAULT CURRENT_TIMESTAMP);
  INSERT INTO items (name) VALUES ('item1'), ('item2'), ('item3');
"

Test 1: Databasement — backup and restore each DB type

1a. PostgreSQL backup + restore

  1. Register LXC 200 as a PostgreSQL server in Databasement (SSH tunnel, port 5432, user: postgres, password: testpass)
  2. Create a backup job and run it manually
  3. Verify backup appears in Databasement UI with correct size
  4. Delete test data: docker exec test-pg psql -U postgres -d testdb -c "DELETE FROM items;"
  5. Restore from Databasement UI
  6. Verify data: docker exec test-pg psql -U postgres -d testdb -c "SELECT * FROM items;"
  7. Pass criteria: All 3 rows restored with correct values

1b. MariaDB backup + restore

  1. Register LXC 201 as a MariaDB server in Databasement
  2. Create a backup job and run it manually
  3. Delete test data
  4. Restore from Databasement UI
  5. Verify data restored
  6. Pass criteria: All 3 rows restored

1c. SQLite backup + restore

  1. Register LXC 202 as an SQLite server in Databasement (SFTP, path: /opt/test-sqlite/test.db)
  2. Create a backup job and run it manually
  3. Delete test data: sqlite3 /opt/test-sqlite/test.db "DELETE FROM items;"
  4. Restore from Databasement UI (file copy back)
  5. Verify: sqlite3 /opt/test-sqlite/test.db "SELECT * FROM items;"
  6. Pass criteria: All 3 rows restored

Test 2: Backrest — restic snapshot of Databasement output

  1. After Test 1 backups complete, trigger Backrest's databasement-dumps plan manually
  2. Verify snapshot appears in Backrest UI
  3. Simulate Databasement data loss: rename /opt/databasement/data on cajita-elite
  4. Restore from Backrest: restic restore <SNAPSHOT_ID> --target /tmp/restore
  5. Verify dump files exist in /tmp/restore/opt/databasement/data/
  6. Pass criteria: Dump files present and non-empty

Test 3: Backrest — Vault raft snapshot

  1. Trigger vault-snapshot plan manually from Backrest UI
  2. Verify snapshot appears and .snap file is in /opt/backrest/dumps/vault/
  3. Verify file size is reasonable (should be > 1MB for a populated Vault)
  4. Pass criteria: Snapshot file created, restic snapshot recorded

Test 4: PBS — whole LXC backup + restore

4a. Backup test LXC

  1. Trigger a manual PBS backup of LXC 200: vzdump 200 --storage cajita-elite --mode snapshot --compress zstd
  2. Verify backup appears in PBS UI under cajita-elite datastore

4b. Restore to new LXC

  1. Restore LXC 200's backup to a new VMID (203): pct restore 203 <BACKUP_FILE> --storage apps-pool
  2. Start LXC 203 and verify Docker container + data is intact
  3. Pass criteria: PostgreSQL container starts, all 3 rows present

4c. Full disaster recovery simulation

  1. Destroy LXC 200: pct destroy 200
  2. Restore from PBS to same VMID: pct restore 200 <BACKUP_FILE> --storage apps-pool
  3. Start LXC 200, verify everything works
  4. Pass criteria: LXC fully functional after restore, same IP, Docker running, data intact

Test 5: Combined — DB dump + LXC restore

Verify that Databasement dumps survive a PBS restore (they should, since dumps live on cajita-elite, not the LXC).

  1. Run Databasement backup of LXC 200
  2. Insert new data into LXC 200 that is NOT yet backed up by Databasement
  3. Destroy and PBS-restore LXC 200 (restores to pre-new-data state)
  4. Use Databasement to restore the latest dump (which also doesn't have the new data)
  5. Verify: data matches the Databasement backup point, not the new data
  6. Pass criteria: Confirms that Databasement and PBS are independent backup chains

Test 6: Notifications

  1. Run each Databasement backup job — verify Ntfy notification on backups topic
  2. Run each Backrest plan — verify Ntfy notification
  3. Intentionally break a connection (wrong password) and trigger — verify failure notification
  4. Pass criteria: Success and failure notifications arrive on Ntfy

Cleanup

After all tests pass:

# On chizuru
pct stop 200 && pct destroy 200
pct stop 201 && pct destroy 201
pct stop 202 && pct destroy 202
pct stop 203 && pct destroy 203 2>/dev/null  # if created during Test 4b

# In Databasement UI: remove test database servers and backup jobs
# In Backrest: no cleanup needed (test snapshots will be pruned by retention)

Checklist summary

  • [ ] Test 1a: PostgreSQL backup + restore via Databasement
  • [ ] Test 1b: MariaDB backup + restore via Databasement
  • [ ] Test 1c: SQLite backup + restore via Databasement
  • [ ] Test 2: Backrest restic snapshot of Databasement output
  • [ ] Test 3: Vault raft snapshot via Backrest
  • [ ] Test 4a: PBS backup of test LXC
  • [ ] Test 4b: PBS restore to new LXC
  • [ ] Test 4c: PBS full disaster recovery simulation
  • [ ] Test 5: Combined DB dump + LXC restore independence
  • [ ] Test 6: Ntfy notifications (success + failure)
  • [ ] Cleanup: destroy test LXCs and remove test configs