Skip to content

Forgejo — Runbook

Runner

Check runner status

ssh -i ~/.ssh/homelab_claude [email protected] "rc-service forgejo-runner status"

Restart runner

ssh -i ~/.ssh/homelab_claude [email protected] "rc-service forgejo-runner restart"

Authentication

Sign in with PocketID

  1. Go to https://git.eva-00.network
  2. Click Sign in with pocketid on the login page

Re-configure the OIDC auth source

Trigger the Configure Forgejo workflow from Forgejo Actions (workflow_dispatch). It will: - Fetch secret/forgejo from Vault (written by the pocketid-setup workflow) - Add or update the pocketid auth source in Forgejo - Restart Forgejo

Rotate OIDC client secret

  1. Run pocketid-setup workflow — generates new client secret and writes to Vault
  2. Run Configure Forgejo workflow — picks up new credentials from Vault

Logs

Log Contents Location Loki label Format Example query
Forgejo application Git operations, web requests, auth events LXC 100 /var/log/forgejo/gitea.log job="forgejo" Plain text {job="forgejo"} \|= "error"
Forgejo Runner daemon Task pickup, start/stop events, connection status LXC 101 /var/log/forgejo-runner.log job="forgejo-runner" Plain text {job="forgejo-runner"} \|= "error"
Workflow step output Full stdout/stderr from each workflow step LXC 100 .log.zst files job="forgejo-actions" Plain text {job="forgejo-actions", task_id="820"}

Notes: - Forgejo Actions logs have a ~2 minute delay (cron-based push from compressed .log.zst files) - Use task_id to query a specific workflow run. Get task IDs from the Forgejo Actions UI or API: GET /api/v1/repos/holo/homelab/actions/tasks?limit=10 - Additional labels for actions: owner, repo, task_id - SSH fallback for action logs: ssh [email protected] "pct exec 100 -- zstdcat /var/lib/forgejo/data/actions_log/<owner>/<repo>/<hash>/<task_id>.log.zst"

Cancelling workflow runs

Forgejo does not support cancelling runs via API. Use the database directly.

Cancel all queued/running runs except a specific one

# From chizuru — set KEEP_ID to the run you want to keep
KEEP_ID=1245
pct exec 100 -- sqlite3 /var/lib/forgejo/db/forgejo.db \
  "UPDATE action_run SET status=4, stopped=strftime('%s','now') WHERE status IN (2,3) AND id != $KEEP_ID;
   UPDATE action_run_job SET status=4 WHERE run_id IN (SELECT id FROM action_run WHERE status=4 AND id >= 1217) AND status IN (2,3,6);"

Cancel all queued/running runs

pct exec 100 -- sqlite3 /var/lib/forgejo/db/forgejo.db \
  "UPDATE action_run SET status=4, stopped=strftime('%s','now') WHERE status IN (2,3);
   UPDATE action_run_job SET status=4 WHERE run_id IN (SELECT id FROM action_run WHERE status=4) AND status IN (2,3,6);"

Flush internal queues

pct exec 100 -- su -s /bin/sh forgejo -c "forgejo manager flush-queues"

Status codes: 1=success, 2=running, 3=waiting, 4=cancelled, 5=failure, 6=job running

After DB changes: restart Forgejo (pct exec 100 -- rc-service forgejo restart) for the UI to reflect the updates.

Troubleshooting

Workflow stuck / runner not picking up jobs

  1. Check runner status: rc-service forgejo-runner status
  2. Check runner logs for connection errors
  3. Verify runner config URL matches current Forgejo URL: cat /root/.runner
  4. Restart runner: rc-service forgejo-runner restart

Runner registered but jobs fail immediately

  • Check the runner SSH key is still authorised on chizuru: ssh -i /root/.ssh/id_ed25519 [email protected] "echo ok"
  • If not, re-add the runner's public key to /root/.ssh/authorized_keys on chizuru

SSO login creates a new account instead of signing into existing

  • Ensure Forgejo admin email exactly matches the PocketID profile email ([email protected])
  • If accounts are split: merge them via Forgejo admin panel, or update the email on one side to match