matt/echo6-docs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

auto: docs sync 2026-07-12T00:00:23+00:00

Browse source 
Files changed: ", c.get(k))\nPY\n\\\"\n\"" engine/config.yaml engine/lib/lint.py engine/lib/vocab_gen.py engine/lint-report.md engine/sweep.sh vault/.obsidian/workspace.json vault/archive/projects/meshai-native-fire-severity-audit-cc-handoff.md vault/archive/projects/vaultwarden-plan.md vault/docs/matrix/matrix_host.md vault/docs/matrix/synapse.md vault/docs/services/services.md vault/docs/software/authentik.md vault/docs/software/caddy.md vault/docs/software/dns.md vault/docs/software/recon.md vault/docs/software/searxng.md vault/glossary.md vault/notes/echo6-landing-page-data-export.md vault/projects/matrix-synapse-deployment.md vault/projects/meshai.md vault/projects/meshtastic-headscale-runbook.md vault/projects/mmud-project.md vault/runbooks/add-peertube-channel.md vault/runbooks/authentik-access-groups.md vault/runbooks/authentik-create-invitation.md vault/runbooks/authentik-oidc-application.md vault/runbooks/authentik-upgrade.md vault/runbooks/expose-service-contabo.md vault/runbooks/lxc-service-migration.md vault/runbooks/mailcow-create-mailbox.md vault/runbooks/meshtastic-sidecar-node.md vault/runbooks/meshtasticd-sim-nodes-runbook.md vault/runbooks/proxmox-create-ubuntu-vm.md vault/runbooks/recon-operations.md vault/runbooks/recon-service-integration.md vault/runbooks/syncthing-add-node.md
This commit is contained in:
echo6-autocommit 2026-07-12 00:00:23 +00:00
commit cce29c595d
37 changed files with 336 additions and 262 deletions
Download patch file Download diff file Expand all files Collapse all files

270
vault/archive/projects/meshai-native-fire-severity-audit-cc-handoff.md Normal file
View file

@ -0,0 +1,270 @@
---
title: "MeshAI Native Fire-Severity Audit — CC Handoff (archived)"
type: project
tags:
- mesh
status: archived
updated: 2026-07-11
---
> **ARCHIVED / SUPERSEDED — fix deployed.** This was a one-time Claude Code handoff for the native fire-severity regression (audit 2026-07-08). The fix has been deployed; this runbook is retained for history only. Its host/access details are STALE — meshai runs as utility CT 108 (`pct exec 108` from root@utility; zvx has no sudo), NOT at 192.168.1.144 with sudo. Do not execute.
# MeshAI Audit Handoff — Native Fire Severity Regression
**For:** Claude Code (CC), operating from cortex against the MeshAI LXC (192.168.1.144)
**Repo:** zvx-echo6/meshai (deployed via docker compose on the LXC)
**Date of audit:** 2026-07-08, against HEAD `aef9877` (#95)
**Scope:** One minimal backend fix in `work/meshai/env/store.py`, plus a mandatory DB pre-seed. Nothing else. No drive-by refactors.
---
## 0. Standing operational rules (non-negotiable)
1. **SSH prereq gate.** Before ANY work: verify SSH from cortex to the MeshAI LXC (192.168.1.144) using the configured method in `~/.ssh/config`. If auth fails, STOP and report. Never assume SSH works.
2. **Never use `docker compose run`.** Ever. Use `docker exec` against the running container, or host tooling.
3. **Verbatim diffs only.** Apply the diff in §10 exactly. If the anchor lines don't match, STOP and report — do not improvise.
4. **Plain-English explanation before every commit.** Matt's understanding is the gate. Present the summary, wait for his confirmation, then commit.
5. **Every status report back to Matt ends with `sudo docker ps`.**
6. **Rebuild/deploy is ONLY:** `sudo docker compose build meshai && sudo docker compose up -d`.
7. **Mesh-spam stop gate** — see §1. Binding.
---
## 1. ⚠️ MESH-SPAM STOP GATE — BINDING
Deploying the fix in this document **will release a burst of fire broadcasts** unless the pre-seed in §9 is completed and verified first.
Mechanics: every fire currently blocked by this bug retries its "New" broadcast **on every poll** and dies silently at the dispatcher severity floor. Because delivery never succeeds, the decider's commit never runs, so its `fires` row keeps `last_broadcast_at = NULL`. The first poll after the fixed code starts will pass **every one of those fires at once** — and there is no pacer on the native path (`FirePacer` is wired to the Central consumer only, `main.py:128–136`; its own docstring says it serves the Central reconnect/drain path).
**Rule:** Phase −1 (§7) and Phase 0 (§8) are read-only and may run immediately. **Do not execute anything in §9 onward until Matt replies verbatim: `I know and I accept`.** When you reach that point, report the burst size from Phase 0 step D and ask for the ack explicitly. Even with the pre-seed done, note to Matt that legitimate post-fix fire traffic will begin (that's the point), and multiple genuinely-new fires arriving in one poll batch will emit together.
---
## 2. Symptom
Since the central→native swap (Jul 4–6), the mesh carries only WZDx, NWS weather alerts, satpass, and band conditions. Fire alerts — which should be active in southern Idaho in July — are silent. (Avalanche silence is seasonal and expected; it is NOT part of this problem.)
## 3. Root cause — plain English
The central→native swap dropped the decider's severity promotion, and native fire events now die at the dispatcher's `min_severity` floor — silently, with no log line and no drop counter.
Evidence chain (all verified by reading the code at HEAD):
1. **Central path (old behavior):** `central/wfigs_handler.py:238, 348` stamped `data["_severity_override"] = "priority"` on every fire New/Update. `central/consumer.py:649–654` promoted it onto the Event: `severity=sev_override or map_severity(...)`. Every fire reached the dispatcher as **priority**.
2. **Native adapter severity:** `env/fires.py` (~lines 139–148) assigns severity by anchor proximity — `"priority"` only within 25 km of a region anchor, **`"routine"` otherwise**. `to_event` (line 333+) passes it through unchanged.
3. **The decider still stamps the override:** `notifications/gating/fire.py::_broadcast_patch` sets `"_severity_override": "priority"` on every New and Update GateResult. The intent is unchanged from Central.
4. **The promotion is lost:** `env/store.py::_emit_event` (line 678+) builds the Event from `adapter.to_event()` FIRST, then merges the decider patch into **`event.data` only** (`event.data.update(gate.data_patch)`, ~line 731). Nothing anywhere in the native delivery path promotes `_severity_override` onto `event.severity` — verified by grep across dispatcher, composer, and events modules: zero consumers.
5. **The floor:** `notifications/pipeline/dispatcher.py` Section 2 (~line 558): `if event_rank < SEVERITY_RANK[tog.min_severity]: return` — a bare return. **No log, no counter** (the four persisted counters cover cold-start/stale/cooldown/dedup, not the severity floor). The matrix path (Section 1.5) has an equivalent per-cell floor with an "authoritative no-send" return.
6. **The default floor is "priority":** `config.py:641` (`NotificationToggle.min_severity: str = "priority"`) and again at the factory default, `config.py:681`.
Net: routine-severity fire → silent drop, every poll, forever. Fire is forced through the decider regardless of cutover state (`NATIVE_ALWAYS_DECIDE`, `notifications/cutover.py:44`), so every native fire New/Update is affected.
**Timeline:** WFIGS→decider migration `8bc9b14` (Jul 4, 22:51), received-delta gate `0c76a80` (Jul 5), native WFIGS routed through decider `8d61b16` (Jul 6, 15:35). Fire silence began at the native flip — matches the operator's report exactly.
## 4. Why the surviving feeds survive
They handle severity in-adapter, so they never depended on the lost promotion:
- `env/satpass.py:250` — reads `severity=data.get("_severity_override", "routine")` itself. **This is the correct pattern the fix replicates at the store level.**
- `env/nws.py:39` — derives severity in-adapter via `_map_nws_severity` from CAP severity; warnings clear the floor.
- `env/wzdx.py:365` — full closures are `"priority"`.
- Band conditions — scheduled broadcaster (`BandConditionsScheduler`) that calls the dispatcher directly.
Of the six deciders that emit `_severity_override` (fire, nws, quake, swpc, avalanche, firms), only **firms** (`env/firms.py:456`) and **satpass** consume it in-adapter.
## 5. Ruled out during the audit
- Avalanche: out of season; expected silence. Not a bug.
- Received-delta gate (`_delta_emit`, commit `0c76a80`): fires deliberately bypass it (`store.py::_ingest`, `nifc` branch) — the decider is the fire gate. Not the drop point.
- FirePacer black hole: pacer is Central-consumer-only; native fires go straight to the bus. Not a factor.
- Toggle mapping: `wildfire_declared`/`wildfire_incident` both map to the `fire` toggle (`notifications/categories.py:~380–390`). Not a factor.
- Coverage filter: fail-open for non-weather categories; fires carry centroids. Not the primary drop.
- ToggleFilter/inhibitor/grouper: all log their drops; fire family is presumed enabled (verify in Phase 0).
## 6. Secondary findings — explicitly OUT OF SCOPE for this change (backlog)
1. **Restart-window ignition swallowing.** `_fires_seeded` is per-process (`store.py::_ingest_fires`); every restart's first poll silent-seeds any fire that first appeared while the container was down — it is never announced as New. Deliberate anti-backlog design, but during dense deploy periods in fire season it eats real ignitions. Needs a design decision (e.g., persisted seen-baseline), not a hotfix.
2. **Zero-acre New fires deferred.** `env/fires.py::to_event` returns None when `not acres`; fresh WFIGS ignitions often report 0 acres initially. Self-heals when acreage posts (row exists with NULL `last_broadcast_at` → decider fires "New"), but delays first announcements.
3. **Same bug class armed for quake/swpc/avalanche/nws.** Their deciders emit `_severity_override` with no native consumer; currently masked behind `MESHAI_CUTOVER_CATEGORIES` shadow gating. The §10 fix resolves this at the shared choke point for all cut-over categories too.
4. **No native fire pacer.** Multiple legit new fires in one poll batch will emit back-to-back. Consider extending FirePacer to the native emit path later.
---
## 7. Phase −1 — SSH prereq gate (mandatory first step)
From cortex:
```bash
ssh <configured-alias-or-user>@192.168.1.144 'echo SSH_OK && hostname'
```
If this fails for any reason: STOP. Report to Matt. Do not proceed.
Then locate the deployment on the LXC (do not guess paths):
```bash
sudo docker ps --format '{{.Names}}\t{{.Image}}' | grep -i meshai
find /opt /root /home -maxdepth 3 -name docker-compose.yml 2>/dev/null | xargs grep -l meshai 2>/dev/null
```
Record the compose project directory and the container name (assumed `meshai` below — substitute if different). Confirm `git status` in the repo checkout is clean before any edit; if dirty, STOP and report.
## 8. Phase 0 — read-only diagnostics (safe to run now)
Run all of these and report results to Matt before anything else. Determine the DB path first:
```bash
sudo docker exec meshai ls /app/data
```
**A. Are fire events reaching the bus and dying downstream?** (Diagnosis predicts: repeatedly nonzero.)
```bash
sudo docker logs meshai --since 24h 2>&1 | grep -c "Emitted nifc"
sudo docker logs meshai --since 24h 2>&1 | grep "Emitted nifc" | tail -5
```
**B. Live fire family config — the floor:**
```bash
sudo docker exec meshai grep -B2 -A8 -i "fire" /app/config/config.yaml | grep -i -B1 -A4 "min_severity\|enabled\|region"
```
**C. Cutover state (determines blast radius of the bug class):**
```bash
sudo docker exec meshai printenv MESHAI_CUTOVER_CATEGORIES
```
**D. Burst size if fixed today** — this number goes in the stop-gate report to Matt:
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> \
"SELECT COUNT(*) FROM fires WHERE last_broadcast_at IS NULL;"
sudo docker exec meshai sqlite3 /app/data/<DBFILE> \
"SELECT irwin_id, incident_name, current_acres, current_contained_pct, last_broadcast_at
FROM fires ORDER BY last_event_at DESC LIMIT 20;"
```
**E. Dispatcher drop counters** (severity-floor drops are NOT counted — expected to look quiet; that's part of the finding):
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> "SELECT * FROM dispatcher_state;"
```
**Decision matrix:**
- A nonzero AND B shows fire `min_severity: priority` (or matrix fire cells with priority floors) → **diagnosis confirmed.** Proceed to the stop gate.
- A is ZERO → fires aren't reaching the bus at all; the diagnosis does NOT hold as-is. STOP. Report findings; do not apply the fix. (Check adapter enablement, NIFC fetch errors: `sudo docker logs meshai --since 24h 2>&1 | grep -i "nifc\|wfigs" | tail -20`.)
- A nonzero but B shows fire floor already `routine` → drop is elsewhere (matrix cells or region scope). STOP. Report B plus the matrix config; do not apply the fix blind.
End the Phase 0 report with `sudo docker ps`.
---
## ⛔ EVERYTHING BELOW REQUIRES MATT'S VERBATIM ACK: `I know and I accept`
Report the Phase 0 results including the §8-D burst count, restate the risk (§1), and wait.
## 9. Phase 1 — pre-seed the fires table (mandatory BEFORE the fixed code ever starts)
Run against the **still-running old container** (old code keeps suppressing at the floor, so this changes nothing live; after pre-seed the decider simply returns cooldown for those rows):
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> "
UPDATE fires
SET last_broadcast_at = strftime('%s','now'),
first_broadcast_at = COALESCE(first_broadcast_at, strftime('%s','now')),
last_broadcast_acres = current_acres,
last_broadcast_contained = current_contained_pct
WHERE last_broadcast_at IS NULL;"
```
Verify — MUST return 0 before proceeding:
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> \
"SELECT COUNT(*) FROM fires WHERE last_broadcast_at IS NULL;"
```
If `sqlite3` is not present in the image: stop the container, run the same SQL with host `sqlite3` against the bind-mounted data directory (check the volume mapping in docker-compose.yml), verify count 0, and only then continue. Do NOT start the new image before the verify shows 0.
Consequence to state to Matt: currently-active fires will not get a retroactive "New" announcement; they will broadcast on their next real growth/containment change. Only genuinely new post-deploy ignitions announce as New.
## 10. Phase 2 — the fix (verbatim diff)
Target: `work/meshai/env/store.py`, inside `_emit_event` (function starts line 678). Anchor by grepping `event.data.update(gate.data_patch)` — it occurs once. Verify both anchor lines exist verbatim, then apply exactly:
```diff
--- a/work/meshai/env/store.py
+++ b/work/meshai/env/store.py
@@ def _emit_event(self, adapter, raw_evt: dict):
# Apply data_patch into event.data
event.data.update(gate.data_patch)
+ # Promote decider overrides onto the Event itself, mirroring
+ # the Central path (central/consumer.py:649-653). The gating
+ # deciders stamp _severity_override (fire: "priority" on every
+ # New/Update) and, for fire New/tombstone, a category
+ # override. Merging them into event.data alone leaves
+ # event.severity at the adapter's value ("routine" for fires
+ # >=25 km from an anchor), which silently fails the
+ # toggle/matrix min_severity floors (default "priority",
+ # config.py:641). Native and Central must share identical
+ # broadcast decisions.
+ _sev_override = gate.data_patch.get("_severity_override")
+ if _sev_override:
+ event.severity = _sev_override
+ _cat_override = gate.data_patch.get("category")
+ if _cat_override:
+ event.category = _cat_override
if gate.commit is not None:
event.data["_on_broadcast_committed"] = gate.commit
```
Notes for review:
- `Event` is a plain mutable `@dataclass` (`notifications/events.py:38`); attribute assignment is valid.
- Category promotion is intentional and safe: `wildfire_declared` maps to the `fire` toggle (`categories.py:~387`); formatter branch selection already falls back to `data["category"]`; cooldown keys become category-scoped, matching Central behavior (New vs Update cooldowns separate).
- This is the shared choke point: any category later added to `MESHAI_CUTOVER_CATEGORIES` (quake, swpc, avalanche, nws) gets the correct promotion automatically.
**Tests:** if a pytest environment exists on the LXC checkout, run the targeted subsets and compare against the known 10-failure baseline (per commit `0c76a80`, 1697 passed):
```bash
python3 -m pytest work/tests -k "fire or store or dispatch or gating" -q
```
If no local pytest env exists, note that in the report and proceed — the change is 16 lines with fully-traced semantics. Do NOT use `docker compose run` to run tests.
## 11. Phase 3 — commit and deploy
1. Present Matt a plain-English summary of the change (what/why in 3–5 sentences, drawn from §3). Wait for his confirmation. His understanding is the gate.
2. Commit with a message in the repo's convention, e.g.:
`fix(native): promote decider _severity_override/category onto Event — restores Central-parity fire severity (silent min_severity floor drop)`
3. Deploy exactly:
```bash
sudo docker compose build meshai && sudo docker compose up -d
```
## 12. Phase 4 — post-deploy verification
Watch the first 2–3 NIFC poll cycles (read the poll interval from the wfigs/nifc adapter config during Phase 0):
```bash
sudo docker logs -f meshai 2>&1 | grep -Ei "Emitted nifc|matrix: dispatched|dispatched|SUPPRESSED|cold-start"
```
Expected: NO burst at startup (pre-seed verified 0); cold-start seed lines for any brand-new first-sight fires; subsequent genuine New/growth events showing `matrix: dispatched` or toggle-path delivery lines. Confirm on-air on the mesh when the first legit event lands.
Then confirm the counters are stable and report:
```bash
sudo docker exec meshai sqlite3 /app/data/<DBFILE> "SELECT * FROM dispatcher_state;"
sudo docker ps
```
Every report back to Matt ends with `sudo docker ps`.
## 13. Backlog (do NOT do now — list for future sessions)
1. Persist the fire seen-baseline across restarts to close the restart-window ignition gap (§6.1).
2. Decide policy for zero-acre first sightings (§6.2).
3. Extend pacing to the native emit path for multi-fire poll batches (§6.4).
4. Before cutting over quake/swpc/avalanche/nws via `MESHAI_CUTOVER_CATEGORIES`, re-verify override promotion end-to-end (covered by this fix, but confirm with shadow logs).
5. Add a drop counter + log line for the dispatcher severity-floor return (it is currently the only fully silent drop in the pipeline).

219
vault/archive/projects/vaultwarden-plan.md Normal file
View file

@ -0,0 +1,219 @@
---
title: Vaultwarden → edge2 LXC — Migration Pilot (+ reusable LXC-migration runbook) — v2
type: note
tags:
- proxmox
aliases: []
related:
- [[lxc-service-migration]]
- [[expose-service-edge2]]
- [[edge2-access-reference]]
- [[caddy]]
- [[ip-allocation]]
status: archived
updated: 2026-07-11
---
> **ARCHIVED / COMPLETED 2026-06-16.** Vaultwarden was migrated to edge2 CT 102 (100.64.0.33) on 2026-06-16 — see [[services]]. This plan is retained for history only; its future-tense provisioning steps and `root@100.64.0.1` (dead Contabo) access are STALE. Do not re-execute.
# Vaultwarden → edge2 LXC — Migration Pilot (+ reusable LXC-migration runbook) — v2
## Changes from v1 (what was corrected and why)
- **CT ID 101 → 102.** CT 101 is the **live, running `wordpress` container** on edge2 (`pct list`: 100=pdm, 101=wordpress, both running, /etc/pve/lxc/101.conf exists). v1's `pct create 101` would error on ID-in-use, and a careless rollback `pct destroy 101` would **destroy WordPress**. New ID = **102** (`pvesh get /cluster/nextid` → 102; confirmed free, no 102.conf, no qemu config). IP `10.10.10.20/24` is still free, so only the CTID changes.
- **edge2 access path pinned.** `root@100.64.0.26` over the tailnet is **REFUSED**. All edge2 `pct`/`pvesm`/`pveam` commands run as `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`) then **`sudo`**. (Contabo/headscale commands stay `root@100.64.0.1` — different host, unchanged.)
- **Debian 12 template must be downloaded first.** Only `debian-13-standard_13.1-2` is cached locally; Debian 12 is in the catalog but **not present**. Added explicit `pveam download local debian-12-standard_12.12-1_amd64.tar.zst` step (volid `local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst`).
- **Right-sized RAM/disk.** v1 copied pdm's `memory 2048 / rootfs local:16`. Vaultwarden is a single Rust binary (idle RSS ~30–60 MB) over ~5.3 MB of SQLite. Downsized to **`memory 1024 / swap 512 / rootfs local:8`** — generous (8 GB rootfs holds Debian 12 + Docker + image + data with ~5 GB free).
- **`--ostype debian` added** to `pct create` (CT 100 has it; v1 omitted it).
- **Live-WAL data-consistency handling made explicit & zero-loss.** Source DB is in WAL mode with a **live 2.4 MB `db.sqlite3-wal`** — a bare `cp` of `db.sqlite3` loses ~2.4 MB of changes. Verified that `sqlite3 .backup` produces **one self-contained file with NO `-wal`/`-shm` side files**. For this single-user instance we **stop the source container before `.backup`** (quiescent DB, zero post-snapshot writes), then keep the source intact as rollback baseline.
- **Compose port edit corrected.** Source binds `127.0.0.1:8086:80` (container listens on **80**). The edit is `127.0.0.1:8086:80` → **`VW_TS_IP:8086:80`** (keep the container-side `:80`). v1's `…→ VW_TS_IP:8086` dropped the target port and would break the mapping.
- **Cutover line corrected — NO scheme.** v1 said `reverse_proxy http://VW_TS_IP:8086`. Vaultwarden's backend is **plain HTTP**; bare `host:port` already defaults to http. Adding `http://` is non-idiomatic and risks a validate failure. The minimal edit is **one token**: `127.0.0.1:8086` → `VW_TS_IP:8086` (the `proxmox.echo6.co` block uses a scheme only because *that* backend is HTTPS).
- **G9 wording softened (accurate failure mode).** `rsa_key.pem` signs JWTs (session/device/push tokens); it is **not** the vault-data encryption key. Missing it = mass session invalidation (forced re-login on every client) + push breakage, **not** undecryptable data. Carrying it is still required; the "generating new RSA key" log gate stays.
- **G10 blast radius noted.** `systemctl restart caddy` momentarily drops **all** Contabo-Caddy-fronted sites (auth/forge/mail/vault…) for ~1–2 s, not just vault. Acceptable, but stated. Also: `systemctl reload caddy` / `caddy reload` **fail** under `admin off` (ExecReload uses `caddy reload --force`) — restart is hard-required.
- **Stale-archive guard added.** `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md` is STALE (shows nonexistent `:3012` port, `/notifications/hub` route, dnsmasq entry, `/oidc-signin` URI). Phase 4 copies **only from the live `/opt/vaultwarden` host**, never the archive. Phase 9 retires the archive doc.
- **Composed cutover path treated as unproven → hard gate.** Contabo-Caddy → edge2-LXC tailnet path is new (precedent proxies to the *data* node, not an edge2 LXC). Phase 3/6 keep a **hard HTTP `curl …/alive` pre-cutover gate** from Contabo.
---
## Context
We're consolidating both Contabo VPSes onto Proxmox. edge2 is up and proven (PVE 8 + PDM). The next phase is evacuating [[services]] off the **main Contabo** so it can be rebuilt as `edge1` — moving each service into an LXC on edge2, with the live Contabo as rollback, mail handled last on its original IP.
**Vaultwarden is the pilot** because it's the lowest-risk service: a single container, ~5.3 MB of SQLite data, no external DB, and `SSO_ONLY=false` so local master-password login works even if [[authentik]] is down. This run **doubles as the reusable per-service LXC-migration template** — phases are tagged `[G]` generic vs `[S]` service-specific.
The cutover uses a **proven, low-risk pattern**: public [[dns]] never changes; we stand up the service on edge2, then re-point *one token* of Contabo's [[caddy]] backend to the new instance over the tailnet (exactly how `proxmox.echo6.co` already routes to `100.64.0.6:8006`). Rollback = revert that one token + restart source (~1–2 s).
**edge2 access (pinned):** `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`), then `sudo` for every `pct`/`pvesm`/`pveam` command. `root@100.64.0.26` is refused — do not use it.
## Approach & decisions (defaults baked in)
- **Zero data loss for this single-user vault.** We **stop the source container immediately before** the SQLite hot `.backup`, so the snapshot is taken from a quiescent DB and no post-snapshot writes can occur. The source container + its `data/` stay **intact and become the rollback baseline** (do not delete). WAL mtime shows ~0 write activity (only WS keepalives), so this is the simplest, safest path. (If true zero-downtime were ever required: accept a brief no-write window, or do a WAL-aware delta re-snapshot at cutover.)
- **CT 102**, hostname `vaultwarden`, **static `10.10.10.20/24` gw `10.10.10.1` ns `1.1.1.1`** (edge2 `vmbr0` has no DHCP), storage `local` (dir, ~368 GB free; no local-lvm exists), `--ostype debian`, unprivileged, `nesting=1,keyctl=1`, `--onboot 1`, **right-sized `memory 1024 / swap 512 / rootfs local:8`**.
- **Bind Vaultwarden to the LXC's tailnet IP:8086** (tight — only Contabo Caddy reaches it over the tailnet), not `0.0.0.0`.
- **Source NOT deleted in the pilot** — stopped at Phase 5, kept as a cold backup + final tarball before any eventual removal.
- **Local-login assurance:** `SSO_ONLY=false` preserved (carried in the untouched `.env`); immediately after cutover, test **email + master-password (local) login** in the browser — instant rollback if it fails.
## Phases
### Phase 0 — Recon & baseline `[G]`
- On Contabo (`ssh root@100.64.0.1`): capture the **verbatim** `vault.echo6.co` Caddy block (rollback baseline — see Phase 7) and grep the whole Caddyfile for any `vault`/`notifications`/`3012` sub-route (none exists — confirm; resolves **G1**).
- Re-read live `/opt/vaultwarden/docker-compose.yml` + `.env` + `ls -la data/` **from the running host** (never the stale `.ref/archive` doc).
- On edge2 (`ssh admin@184.174.35.153 'sudo …'`):
- Confirm **CT 102** and **10.10.10.20** are free: `sudo pct list` (expect 100=pdm@.10, 101=wordpress@.11), `sudo pvesh get /cluster/nextid` (→ 102), `sudo pct status 102` (→ does not exist). **Do not touch 101 — it is live WordPress.**
- Download the Debian 12 template (NOT cached yet): `sudo pveam download local debian-12-standard_12.12-1_amd64.tar.zst`, then `sudo pvesm list local | grep debian-12`.
- Confirm `local` storage free space: `sudo pvesm status`.
- **Rollback:** n/a (read-only except the template download).
### Phase 0a — Validate a LOCAL user exists + record to credentials `[S]` ← user-requested GATE
- Take a read-only hot snapshot of the live DB and query `users`: confirm **≥1 account has a non-empty `password_hash`** (= master password set → local email+password login works, independent of SSO). [[recon]] already confirms: **one account `matt@echo6.co`, password_hash non-empty, email verified, enabled → GATE PASS.** Record email(s), `login_count`, verified status.
- **Record to `/home/zvx/projects/.ref/credentials`** under a `# Vaultwarden (vault.echo6.co)` section: the account email(s), `SSO_ONLY=false` (local login enabled), and a note that the **master password is Matt's own secret — not stored anywhere in plaintext** (Vaultwarden keeps only the PBKDF2/Argon2 hash). The `ADMIN_TOKEN` (from `.env`) goes here as admin-panel break-glass.
- **GATE:** if NO account has a master password set, **STOP and surface it** — the local-login safety net wouldn't exist. Also confirm with Matt he knows the master password for at least one listed account (we can't recover it, only verify the account exists).
### Phase 1 — Provision the LXC `[G]`
- Create the container (right-sized, ID 102, ostype debian):
```
sudo pct create 102 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
--hostname vaultwarden --cores 2 --memory 1024 --swap 512 --rootfs local:8 \
--net0 name=eth0,bridge=vmbr0,ip=10.10.10.20/24,gw=10.10.10.1 \
--nameserver 1.1.1.1 --ostype debian \
--features nesting=1,keyctl=1 --unprivileged 1 --onboot 1
```
- Append the TUN device lines to `/etc/pve/lxc/102.conf` (verbatim, matching CT 100):
```
printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/102.conf
```
(Resulting lines: `lxc.cgroup2.devices.allow: c 10:200 rwm` and `lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file`.)
- `sudo pct start 102`; verify Debian 12, internet via NAT masquerade, DNS resolution (same health checks as the pdm CT).
- **Rollback:** `sudo pct stop 102 && sudo pct destroy 102`. (Production untouched. Never `destroy 101`.)
### Phase 2 — Docker baseline `[G]`
- Inside CT 102 (`sudo pct exec 102 -- …`): install Docker + compose plugin (`curl -fsSL get.docker.com | sh`), `systemctl enable --now docker`. (`nesting=1`+`keyctl=1` from Phase 1 make Docker-in-LXC work.)
- **Rollback:** destroy CT 102.
### Phase 3 — Tailscale onboard `[G]`
- Create preauth key on Contabo (user 1 = `echo6`, the infra user):
```
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```
- In CT 102 (`sudo pct exec 102 -- …`): install tailscale; add the **DNS-bootstrap drop-in** so `vpn.echo6.co` resolves at boot before MagicDNS is up (CT uses ns `1.1.1.1`):
```
mkdir -p /etc/systemd/system/tailscaled.service.d
cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<'EOF'
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
systemctl daemon-reload && systemctl restart tailscaled
```
Then bring it up:
```
tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=vaultwarden
```
- **Capture `VW_TS_IP`:** `tailscale ip -4`. (Do NOT hardcode — allocation has gaps, e.g. `.31` free, highest is `.32`; the next IP is assigned dynamically at registration.) Confirm the node appears in `ssh root@100.64.0.1 'docker exec headscale headscale nodes list'` (no existing `vaultwarden` node — clean onboard).
- **HARD pre-cutover reachability gate (run again after Phase 6):** the composed Contabo-Caddy → edge2-LXC tailnet path is new, so prove it over HTTP (not ICMP — edge2↔Contabo may not ICMP-ping although both online):
```
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://VW_TS_IP:8086/alive"
```
Will return `000` until Phase 6 (nothing listening yet) — that only proves the tailnet route; the **200** gate is enforced in Phase 6.
- **Note (G10/systemd):** the bundled `caddy.service` `ExecReload` runs `caddy reload --force`, which **errors under `admin off`**. Do not run `systemctl reload caddy` anywhere in this runbook — **restart** is required (see Phase 7).
- **Rollback:** destroy CT 102 + `ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'`.
### Phase 4 — Stage compose + .env `[S]`
- Copy `/opt/vaultwarden/docker-compose.yml` + `.env` **byte-for-byte FROM the live running Contabo host** into the CT (e.g. `/opt/vaultwarden/`). **Never copy from `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md`** — it is stale (shows a nonexistent `:3012` port + `/notifications/hub` route).
- `.env` stays **untouched** → `SSO_ONLY=false`, `DOMAIN=https://vault.echo6.co`, `ADMIN_TOKEN`, `SIGNUPS_ALLOWED=false`, `WEBSOCKET_ENABLED=true`, `SSO_ENABLED=true`, `SSO_CLIENT_ID/SECRET`, `SSO_AUTHORITY=https://auth.echo6.co/application/o/vaultwarden/`, `SSO_PKCE=true` all preserved.
- **The ONE compose edit (G3):** change the single `ports` line `127.0.0.1:8086:80` → **`VW_TS_IP:8086:80`** (keep the container-side `:80`; the container listens on 80, 8086 is only the host-published port). Everything else stays byte-for-byte: `image: vaultwarden/server:latest`, `volume ./data:/data`, `env_file .env`, `restart: unless-stopped`, `TZ`. Confirm there is **exactly one** ports line and **NO `3012` line**.
- **Rollback:** destroy CT 102.
### Phase 5 — Migrate data (zero-loss snapshot) `[S]`
- **Stop the source** so the DB is quiescent (zero post-snapshot writes): on Contabo `cd /opt/vaultwarden && docker compose stop`. (Keep the container + `data/` intact as rollback baseline — do **not** `down`/delete.)
- **Hot consistent snapshot** on the source host (`/usr/bin/sqlite3` v3.45.1 confirmed present):
```
sqlite3 /opt/vaultwarden/data/db.sqlite3 ".backup /tmp/vw-snap.sqlite3"
sqlite3 /tmp/vw-snap.sqlite3 "PRAGMA integrity_check" # must == ok BEFORE transfer
```
`.backup` uses the online backup API and writes **one self-contained file with NO `-wal`/`-shm` side files** (verified). The output's `journal_mode=wal` readout is just a persisted header setting — it is fully self-contained; do not be alarmed.
- **File manifest:**
- **MUST travel:** `vw-snap.sqlite3` → target `data/db.sqlite3`; **`rsa_key.pem`** byte-for-byte (JWT signing key — G9).
- **OPTIONAL:** `icon_cache/` (cosmetic favicon cache, auto-rebuilds).
- **MUST NOT travel:** `db.sqlite3-wal`, `db.sqlite3-shm`, the bare live `db.sqlite3`, `tmp/`. (Copying a live `-wal` alongside a separately-snapshotted db risks recovery corruption.)
- Verify in CT 102: `sqlite3 data/db.sqlite3 "PRAGMA integrity_check"` == ok; `rsa_key.pem` present (header `-----BEGIN RSA PRIVATE KEY-----`, ~1679 B).
- **Note:** when Vaultwarden opens the migrated db it will create **fresh** `db.sqlite3-wal`/`-shm` on the target automatically — expected and correct; do not pre-create or copy them.
- **Rollback:** destroy CT 102; restart source: `cd /opt/vaultwarden && docker compose up -d`.
### Phase 6 — Start + health gate `[S]`
- `cd /opt/vaultwarden && docker compose up -d` in CT 102.
- **Health gates (all must pass before cutover):**
1. **Pre-start integrity:** `sqlite3 data/db.sqlite3 "PRAGMA integrity_check"` == ok (done in Phase 5).
2. **RSA key loaded:** container logs **must NOT** contain `generating new RSA key` (would mean `rsa_key.pem` didn't travel/load → mass session invalidation), and should show normal startup.
3. **`/alive` == 200** over the tailnet from Contabo (the HARD composed-path gate from Phase 3):
```
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://VW_TS_IP:8086/alive"
```
4. **Row-count gate:** `sqlite3 data/db.sqlite3 "SELECT count(*) FROM users"` == **1** (matches source `matt@echo6.co`).
- **Rollback:** destroy CT 102; restart source (`docker compose up -d` on Contabo).
### Phase 7 — Cutover `[G]` ← the primary, instant-rollback step
- On Contabo, capture a verbatim on-box baseline first:
```
cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-prevault
```
- **The ONE edit:** in `/etc/caddy/Caddyfile`, replace **only the upstream token** on the `vault.echo6.co` backend line — `127.0.0.1:8086` → `VW_TS_IP:8086` (**no scheme** — bare host:port defaults to http; the proxmox block uses a scheme only because that backend is HTTPS). Keep `header_up X-Real-IP {remote_host}` and the braces untouched (G5). Resulting block must read exactly:
```
vault.echo6.co {
reverse_proxy VW_TS_IP:8086 {
header_up X-Real-IP {remote_host}
}
}
```
- Validate (works under `admin off`), then **restart** (G10 — `caddy reload` and `systemctl reload caddy` both FAIL under `admin off`):
```
caddy validate --config /etc/caddy/Caddyfile
systemctl restart caddy
```
- **Blast-radius note (G10):** `systemctl restart caddy` briefly drops **all** Contabo-Caddy-fronted sites (auth/forge/mail/vault…) for ~1–2 s, not just vault. Acceptable; validate-before-restart prevents restarting into a broken config.
- **Rollback (primary, ~1–2 s, public DNS never moved):** either
```
cp /etc/caddy/Caddyfile.bak-prevault /etc/caddy/Caddyfile && systemctl restart caddy
```
OR revert the line to exactly:
```
vault.echo6.co {
reverse_proxy 127.0.0.1:8086 {
header_up X-Real-IP {remote_host}
}
}
```
then `caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy`, and ensure the source container is running (`cd /opt/vaultwarden && docker compose up -d`).
### Phase 8 — End-to-end verification `[G/S]`
- `vault.echo6.co` resolves unchanged → `5.189.158.149`; `curl -I https://vault.echo6.co` 200, valid cert.
- **Local login:** log in at `https://vault.echo6.co` with **email + master password** (non-SSO) → vault items decrypt + list (proves local-login safety net + data integrity). ← user-requested assurance.
- **SSO login:** the "Log in with SSO" path round-trips through Authentik and lands back **logged-in** on the **unchanged** `https://vault.echo6.co`. Authentik needs **no** change (DNS + `DOMAIN` unchanged → registered redirect URI stays valid) (G6). Verify only that the round-trip completes logged-in — do **not** assert a specific redirect path (the archive's `/oidc-signin` vs `/callback` is stale/inconsistent and irrelevant since Authentik is untouched).
- **Websocket sync:** browser-extension live sync works (101 Upgrade through Caddy) (G7). No `/notifications/hub` or `:3012` route exists or is needed — WS is in-process on `:8086`, so the single upstream carries the Upgrade.
- **Reboot survival:** `sudo pct reboot 102` → CT comes back, Docker + container auto-start (`--onboot 1` + `restart: unless-stopped`), tailscale reconnects (DNS-bootstrap drop-in), `/alive` 200 (G8).
- **Rollback:** Phase 7 revert.
### Phase 9 — Decommission (deferred) `[G]`
- Source stays **installed + stopped** as a cold rollback for a soak period. Final `tar` of `/opt/vaultwarden` before any eventual deletion.
- **Retire the stale archive doc** `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md` (it misdescribes ports/routes/URI).
- Update `.ref` (services.md, environment.md, ip-allocation.md → CT 102 @ 10.10.10.20, the new headscale node). Promote this file to `.ref/runbooks/lxc-service-migration.md`.
## Gotchas
- **G1** Stale `caddy.md`/archive mentions a `:3012`/`notifications/hub` WS route — live Caddy has **none** (in-process WS on `:8086`). Confirmed in Phase 0.
- **G2** edge2 `vmbr0` has **no DHCP** and no `local-lvm` → static IP `10.10.10.20/24` + `local` (dir) storage. (Overrides [[ct-runbook]] defaults.)
- **G3** Source binds loopback-only (`127.0.0.1:8086:80`); LXC must bind the tailnet IP (`VW_TS_IP:8086:80`) so Caddy reaches it. Keep the container-side `:80`.
- **G4** SQLite is in WAL mode with a **live 2.4 MB WAL** → use hot `.backup` (single self-contained file, no `-wal`/`-shm`); never copy a live `db.sqlite3`+WAL naively. Here we also **stop the source first** for zero-loss.
- **G5** Preserve `header_up X-Real-IP {remote_host}` in the new Caddy backend line (edit only the upstream token).
- **G6** SSO redirect derives from the unchanged `DOMAIN`/DNS → **do not touch Authentik**. Verify the round-trip lands logged-in; don't assert a specific redirect path.
- **G7** Caddy passes websocket upgrades by default; no WS-specific directive needed (in-process WS on `:8086`).
- **G8** Reboot survival needs all three: `--onboot 1`, `restart: unless-stopped`, and the DNS-bootstrap drop-in (so `vpn.echo6.co` resolves at boot).
- **G9** `rsa_key.pem` is the **JWT signing key** — must travel byte-for-byte. **Accurate failure mode:** if missing, Vaultwarden generates a new one and **all existing device/session tokens are invalidated (forced re-login everywhere) + push breaks** — vault DATA still decrypts on master-password login (data is client-side encrypted). The "generating new RSA key" log gate (Phase 6) catches this.
- **G10** Contabo Caddy admin API is **off** → `caddy validate` then **`systemctl restart caddy`**. Both `caddy reload` and `systemctl reload caddy` FAIL (`ExecReload` uses `caddy reload --force`). Restart briefly drops all Contabo-fronted sites (~1–2 s).
- **N (new):** the composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised → keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use curl/http, not ICMP).
## Generalize for future services
Template = phases **0→1→2→3→ (4,5,6 swap per service) →7→8→9**. Generic, identical for every Contabo-Caddy-fronted service: free-CTID pick (`pvesh get /cluster/nextid`) + LXC provision (right-sized per service), Docker, Tailscale onboard + DNS-bootstrap drop-in, the one-token Caddy cutover + its verbatim rollback. Service-specific only: the compose/`.env` (Phase 4 — copy from the **live host**, never archives), what "data" is + its consistency method (Phase 5 — stateless services collapse this; SQLite uses stop-then-`.backup`; Postgres-backed services dump/restore instead of file-copy), and the login/functional checks (Phases 6/8). Always pin edge2 access to `admin@184.174.35.153` + `sudo`, and always download the right template first.
## What I'll need to do (permissions)
- SSH to **Contabo** as `root@100.64.0.1` (Caddy edit, headscale preauth/node, source sqlite3 `.backup`, source `docker compose stop/up`).
- SSH to **edge2** as **`admin@184.174.35.153`** (key `~/.ssh/contabo2_ed25519`, alias `edge2`) then **`sudo`** for all `pct`/`pvesm`/`pveam` commands. (NOT `root@100.64.0.26` — refused.)
- `pveam download` one Debian 12 template; `pct`/`docker`/`tailscale` inside CT 102; create one Headscale preauth key + node; edit **one token** of Contabo's Caddyfile; copy ~5 MB of data.
- **No public DNS changes, no Authentik changes, no dnsmasq changes; source left intact (stopped) as rollback.** Never touch CT 101 (live WordPress).