Karakeep — Runbook
Quick Reference
| Item | Value |
|---|---|
| LXC | 117 @ 192.168.1.217 |
| URL | https://karakeep.eva-00.network |
| Health | curl http://192.168.1.217:3000 (200 or 302) |
| Vault | secret/data/karakeep |
| Deploy | Forgejo Actions → Deploy Karakeep |
Check Service Status
ssh [email protected] docker compose -f /opt/homelab/karakeep/docker-compose.yml ps
Restart Services
ssh [email protected] docker compose -f /opt/homelab/karakeep/docker-compose.yml restart
View Logs
Via Loki (preferred)
{container_name="karakeep-karakeep-1"}
{container_name=~"karakeep.*"} |= "error"
Via SSH (fallback)
ssh [email protected] docker compose -f /opt/homelab/karakeep/docker-compose.yml logs -f --tail 100
Fresh Redeploy
Trigger via Forgejo Actions with force_clean=true, or manually:
ssh [email protected]
cd /opt/homelab/karakeep
docker compose down
docker volume rm karakeep_data karakeep_meilisearch
Then re-run the workflow — it will re-create the admin user and API key.
Memory / Performance
LXC resource allocation
| Resource | Value | Notes |
|---|---|---|
| RAM | 8192 MB | Meilisearch RSS grows with bookmark count |
| Swap | 1024 MB | Safety net; normal operation should not swap |
| Cores | 2 |
Meilisearch RAM guidance
Meilisearch uses LMDB (memory-mapped files). Virtual memory is always huge (normal); what matters is RSS.
| Collection size | Steady-state RAM | Indexing peak |
|---|---|---|
| ~1,000 bookmarks | ~300–500 MB | ~1–2 GB |
| ~5,000 bookmarks | ~500 MB–1.5 GB | ~3–5 GB |
| ~10,000 bookmarks | ~1–3 GB | ~5–10 GB |
| ~20,000+ bookmarks | ~2–5 GB | ~8–15 GB |
MEILI_EXPERIMENTAL_REDUCE_INDEXING_MEMORY_USAGE=true is set in the compose file to cap indexing spikes. If you grow past ~10k bookmarks and see slowness, increase LXC RAM via Proxmox MCP:
# Bump to 12 GB if needed
pct set 117 --memory 12288
Disk usage
Full-page archives average 2–5 MB each. At 10k bookmarks expect 20–50 GB. The rootfs is currently 8 GB — expand before hitting that scale:
pct resize 117 rootfs +20G
OOM Kill Detection
How OOM kills appear (and why Docker lies)
When a container exceeds the LXC cgroup memory limit, the Linux kernel OOM killer terminates it — not Docker. Docker then restarts the container via restart: unless-stopped and reports ExitCode: 0 and OOMKilled: false. This is misleading.
Docker inspect is unreliable for cgroup-level OOM kills. The only reliable sources are:
| Method | Command | Notes |
|---|---|---|
| Grafana alert | Fires automatically | Requires Alloy on chizuru (see below) |
| Prometheus query | increase(node_vmstat_oom_kill{job="chizuru"}[1h]) |
Shows OOM kill count |
| Loki query | {job="chizuru-syslog"} \|= "oom-kill" |
Shows kernel log lines |
| dmesg fallback | ssh [email protected] "dmesg \| grep -i oom" |
Direct host check |
Grafana alert
An alert rule "OOM Kill Detected on chizuru" is provisioned in Grafana under the Homelab Alerts folder. It fires immediately (no delay) when node_vmstat_oom_kill increments. The alert annotation points to both the Prometheus metric and the Loki query to identify which process was killed.
This alert requires Alloy running on chizuru (ansible/playbooks/alloy-chizuru.yml).
How to identify which process was killed
# On the Grafana Explore page, run:
{job="chizuru-syslog"} |= "oom-kill"
# Or via SSH on chizuru:
ssh [email protected] "dmesg | grep -E 'oom-kill|Out of memory|Killed process'"
The kernel log includes: process name, PID, RSS at time of kill, and the cgroup path (e.g., /lxc/117/...) identifying which LXC it was in.
Troubleshooting
Container restarts with ExitCode 0 (suspected OOM)
Do NOT trust docker inspect for OOM. Check the host kernel instead:
ssh [email protected] "dmesg | tail -50 | grep -i oom"
A line like Memory cgroup out of memory: Killed process ... (meilisearch) confirms an OOM kill. Fix: increase LXC RAM (see Memory section above).
OIDC login redirects to wrong URL or loops
Check that NEXTAUTH_URL in /opt/homelab/karakeep/.env exactly matches the URL you're accessing. Must be https://karakeep.eva-00.network with no trailing slash.
AI tagging not working
- Check Ollama is reachable:
curl http://192.168.1.107:11434/api/tags - Check Karakeep logs for inference errors:
{container_name="karakeep-karakeep-1"} |= "inference" - Verify models are pulled in Ollama:
llama3.1andllava
Search not returning results
Meilisearch may need time to index. Check its logs:
{container_name="karakeep-meilisearch-1"} |= "error"
API key not working
Verify the key exists in the database:
ssh [email protected] docker exec karakeep-karakeep-1 sh -c \
"sqlite3 /data/db.db 'SELECT name, keyId FROM apiKey;'"