echo6-docs/plans/vaultwarden-plan.md

# Vaultwarden → edge2 LXC — Migration Pilot (+ reusable LXC-migration runbook) — v2

## Changes from v1 (what was corrected and why)

- **CT ID 101 → 102.** CT 101 is the **live, running `wordpress` container** on edge2 (`pct list`: 100=pdm, 101=wordpress, both running, /etc/pve/lxc/101.conf exists). v1's `pct create 101` would error on ID-in-use, and a careless rollback `pct destroy 101` would **destroy WordPress**. New ID = **102** (`pvesh get /cluster/nextid` → 102; confirmed free, no 102.conf, no qemu config). IP `10.10.10.20/24` is still free, so only the CTID changes.
- **edge2 access path pinned.** `root@100.64.0.26` over the tailnet is **REFUSED**. All edge2 `pct`/`pvesm`/`pveam` commands run as `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`) then **`sudo`**. (Contabo/headscale commands stay `root@100.64.0.1` — different host, unchanged.)
- **Debian 12 template must be downloaded first.** Only `debian-13-standard_13.1-2` is cached locally; Debian 12 is in the catalog but **not present**. Added explicit `pveam download local debian-12-standard_12.12-1_amd64.tar.zst` step (volid `local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst`).
- **Right-sized RAM/disk.** v1 copied pdm's `memory 2048 / rootfs local:16`. Vaultwarden is a single Rust binary (idle RSS ~30–60 MB) over ~5.3 MB of SQLite. Downsized to **`memory 1024 / swap 512 / rootfs local:8`** — generous (8 GB rootfs holds Debian 12 + Docker + image + data with ~5 GB free).
- **`--ostype debian` added** to `pct create` (CT 100 has it; v1 omitted it).
- **Live-WAL data-consistency handling made explicit & zero-loss.** Source DB is in WAL mode with a **live 2.4 MB `db.sqlite3-wal`** — a bare `cp` of `db.sqlite3` loses ~2.4 MB of changes. Verified that `sqlite3 .backup` produces **one self-contained file with NO `-wal`/`-shm` side files**. For this single-user instance we **stop the source container before `.backup`** (quiescent DB, zero post-snapshot writes), then keep the source intact as rollback baseline.
- **Compose port edit corrected.** Source binds `127.0.0.1:8086:80` (container listens on **80**). The edit is `127.0.0.1:8086:80` → **`VW_TS_IP:8086:80`** (keep the container-side `:80`). v1's `…→ VW_TS_IP:8086` dropped the target port and would break the mapping.
- **Cutover line corrected — NO scheme.** v1 said `reverse_proxy http://VW_TS_IP:8086`. Vaultwarden's backend is **plain HTTP**; bare `host:port` already defaults to http. Adding `http://` is non-idiomatic and risks a validate failure. The minimal edit is **one token**: `127.0.0.1:8086` → `VW_TS_IP:8086` (the `proxmox.echo6.co` block uses a scheme only because *that* backend is HTTPS).
- **G9 wording softened (accurate failure mode).** `rsa_key.pem` signs JWTs (session/device/push tokens); it is **not** the vault-data encryption key. Missing it = mass session invalidation (forced re-login on every client) + push breakage, **not** undecryptable data. Carrying it is still required; the "generating new RSA key" log gate stays.
- **G10 blast radius noted.** `systemctl restart caddy` momentarily drops **all** Contabo-Caddy-fronted sites (auth/forge/mail/vault…) for ~1–2 s, not just vault. Acceptable, but stated. Also: `systemctl reload caddy` / `caddy reload` **fail** under `admin off` (ExecReload uses `caddy reload --force`) — restart is hard-required.
- **Stale-archive guard added.** `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md` is STALE (shows nonexistent `:3012` port, `/notifications/hub` route, dnsmasq entry, `/oidc-signin` URI). Phase 4 copies **only from the live `/opt/vaultwarden` host**, never the archive. Phase 9 retires the archive doc.
- **Composed cutover path treated as unproven → hard gate.** Contabo-Caddy → edge2-LXC tailnet path is new (precedent proxies to the *data* node, not an edge2 LXC). Phase 3/6 keep a **hard HTTP `curl …/alive` pre-cutover gate** from Contabo.

---

## Context

We're consolidating both Contabo VPSes onto Proxmox. edge2 is up and proven (PVE 8 + PDM). The next phase is evacuating services off the **main Contabo** so it can be rebuilt as `edge1` — moving each service into an LXC on edge2, with the live Contabo as rollback, mail handled last on its original IP.

**Vaultwarden is the pilot** because it's the lowest-risk service: a single container, ~5.3 MB of SQLite data, no external DB, and `SSO_ONLY=false` so local master-password login works even if Authentik is down. This run **doubles as the reusable per-service LXC-migration template** — phases are tagged `[G]` generic vs `[S]` service-specific.

The cutover uses a **proven, low-risk pattern**: public DNS never changes; we stand up the service on edge2, then re-point *one token* of Contabo's Caddy backend to the new instance over the tailnet (exactly how `proxmox.echo6.co` already routes to `100.64.0.6:8006`). Rollback = revert that one token + restart source (~1–2 s).

**edge2 access (pinned):** `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`), then `sudo` for every `pct`/`pvesm`/`pveam` command. `root@100.64.0.26` is refused — do not use it.

## Approach & decisions (defaults baked in)

- **Zero data loss for this single-user vault.** We **stop the source container immediately before** the SQLite hot `.backup`, so the snapshot is taken from a quiescent DB and no post-snapshot writes can occur. The source container + its `data/` stay **intact and become the rollback baseline** (do not delete). WAL mtime shows ~0 write activity (only WS keepalives), so this is the simplest, safest path. (If true zero-downtime were ever required: accept a brief no-write window, or do a WAL-aware delta re-snapshot at cutover.)
- **CT 102**, hostname `vaultwarden`, **static `10.10.10.20/24` gw `10.10.10.1` ns `1.1.1.1`** (edge2 `vmbr0` has no DHCP), storage `local` (dir, ~368 GB free; no local-lvm exists), `--ostype debian`, unprivileged, `nesting=1,keyctl=1`, `--onboot 1`, **right-sized `memory 1024 / swap 512 / rootfs local:8`**.
- **Bind Vaultwarden to the LXC's tailnet IP:8086** (tight — only Contabo Caddy reaches it over the tailnet), not `0.0.0.0`.
- **Source NOT deleted in the pilot** — stopped at Phase 5, kept as a cold backup + final tarball before any eventual removal.
- **Local-login assurance:** `SSO_ONLY=false` preserved (carried in the untouched `.env`); immediately after cutover, test **email + master-password (local) login** in the browser — instant rollback if it fails.

## Phases

### Phase 0 — Recon & baseline `[G]`
- On Contabo (`ssh root@100.64.0.1`): capture the **verbatim** `vault.echo6.co` Caddy block (rollback baseline — see Phase 7) and grep the whole Caddyfile for any `vault`/`notifications`/`3012` sub-route (none exists — confirm; resolves **G1**).
- Re-read live `/opt/vaultwarden/docker-compose.yml` + `.env` + `ls -la data/` **from the running host** (never the stale `.ref/archive` doc).
- On edge2 (`ssh admin@184.174.35.153 'sudo …'`):
  - Confirm **CT 102** and **10.10.10.20** are free: `sudo pct list` (expect 100=pdm@.10, 101=wordpress@.11), `sudo pvesh get /cluster/nextid` (→ 102), `sudo pct status 102` (→ does not exist). **Do not touch 101 — it is live WordPress.**
  - Download the Debian 12 template (NOT cached yet): `sudo pveam download local debian-12-standard_12.12-1_amd64.tar.zst`, then `sudo pvesm list local | grep debian-12`.
  - Confirm `local` storage free space: `sudo pvesm status`.
- **Rollback:** n/a (read-only except the template download).

### Phase 0a — Validate a LOCAL user exists + record to credentials `[S]` ← user-requested GATE
- Take a read-only hot snapshot of the live DB and query `users`: confirm **≥1 account has a non-empty `password_hash`** (= master password set → local email+password login works, independent of SSO). Recon already confirms: **one account `matt@echo6.co`, password_hash non-empty, email verified, enabled → GATE PASS.** Record email(s), `login_count`, verified status.
- **Record to `/home/zvx/projects/.ref/credentials`** under a `# Vaultwarden (vault.echo6.co)` section: the account email(s), `SSO_ONLY=false` (local login enabled), and a note that the **master password is Matt's own secret — not stored anywhere in plaintext** (Vaultwarden keeps only the PBKDF2/Argon2 hash). The `ADMIN_TOKEN` (from `.env`) goes here as admin-panel break-glass.
- **GATE:** if NO account has a master password set, **STOP and surface it** — the local-login safety net wouldn't exist. Also confirm with Matt he knows the master password for at least one listed account (we can't recover it, only verify the account exists).

### Phase 1 — Provision the LXC `[G]`
- Create the container (right-sized, ID 102, ostype debian):
  ```
  sudo pct create 102 local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst \
    --hostname vaultwarden --cores 2 --memory 1024 --swap 512 --rootfs local:8 \
    --net0 name=eth0,bridge=vmbr0,ip=10.10.10.20/24,gw=10.10.10.1 \
    --nameserver 1.1.1.1 --ostype debian \
    --features nesting=1,keyctl=1 --unprivileged 1 --onboot 1
  ```
- Append the TUN device lines to `/etc/pve/lxc/102.conf` (verbatim, matching CT 100):
  ```
  printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/102.conf
  ```
  (Resulting lines: `lxc.cgroup2.devices.allow: c 10:200 rwm` and `lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file`.)
- `sudo pct start 102`; verify Debian 12, internet via NAT masquerade, DNS resolution (same health checks as the pdm CT).
- **Rollback:** `sudo pct stop 102 && sudo pct destroy 102`. (Production untouched. Never `destroy 101`.)

### Phase 2 — Docker baseline `[G]`
- Inside CT 102 (`sudo pct exec 102 -- …`): install Docker + compose plugin (`curl -fsSL get.docker.com | sh`), `systemctl enable --now docker`. (`nesting=1`+`keyctl=1` from Phase 1 make Docker-in-LXC work.)
- **Rollback:** destroy CT 102.

### Phase 3 — Tailscale onboard `[G]`
- Create preauth key on Contabo (user 1 = `echo6`, the infra user):
  ```
  ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
  ```
- In CT 102 (`sudo pct exec 102 -- …`): install tailscale; add the **DNS-bootstrap drop-in** so `vpn.echo6.co` resolves at boot before MagicDNS is up (CT uses ns `1.1.1.1`):
  ```
  mkdir -p /etc/systemd/system/tailscaled.service.d
  cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<'EOF'
  [Service]
  Environment=TS_DEBUG_RESOLV_CONF=1
  EOF
  systemctl daemon-reload && systemctl restart tailscaled
  ```
  Then bring it up:
  ```
  tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=vaultwarden
  ```
- **Capture `VW_TS_IP`:** `tailscale ip -4`. (Do NOT hardcode — allocation has gaps, e.g. `.31` free, highest is `.32`; the next IP is assigned dynamically at registration.) Confirm the node appears in `ssh root@100.64.0.1 'docker exec headscale headscale nodes list'` (no existing `vaultwarden` node — clean onboard).
- **HARD pre-cutover reachability gate (run again after Phase 6):** the composed Contabo-Caddy → edge2-LXC tailnet path is new, so prove it over HTTP (not ICMP — edge2↔Contabo may not ICMP-ping although both online):
  ```
  ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://VW_TS_IP:8086/alive"
  ```
  Will return `000` until Phase 6 (nothing listening yet) — that only proves the tailnet route; the **200** gate is enforced in Phase 6.
- **Note (G10/systemd):** the bundled `caddy.service` `ExecReload` runs `caddy reload --force`, which **errors under `admin off`**. Do not run `systemctl reload caddy` anywhere in this runbook — **restart** is required (see Phase 7).
- **Rollback:** destroy CT 102 + `ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'`.

### Phase 4 — Stage compose + .env `[S]`
- Copy `/opt/vaultwarden/docker-compose.yml` + `.env` **byte-for-byte FROM the live running Contabo host** into the CT (e.g. `/opt/vaultwarden/`). **Never copy from `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md`** — it is stale (shows a nonexistent `:3012` port + `/notifications/hub` route).
- `.env` stays **untouched** → `SSO_ONLY=false`, `DOMAIN=https://vault.echo6.co`, `ADMIN_TOKEN`, `SIGNUPS_ALLOWED=false`, `WEBSOCKET_ENABLED=true`, `SSO_ENABLED=true`, `SSO_CLIENT_ID/SECRET`, `SSO_AUTHORITY=https://auth.echo6.co/application/o/vaultwarden/`, `SSO_PKCE=true` all preserved.
- **The ONE compose edit (G3):** change the single `ports` line `127.0.0.1:8086:80` → **`VW_TS_IP:8086:80`** (keep the container-side `:80`; the container listens on 80, 8086 is only the host-published port). Everything else stays byte-for-byte: `image: vaultwarden/server:latest`, `volume ./data:/data`, `env_file .env`, `restart: unless-stopped`, `TZ`. Confirm there is **exactly one** ports line and **NO `3012` line**.
- **Rollback:** destroy CT 102.

### Phase 5 — Migrate data (zero-loss snapshot) `[S]`
- **Stop the source** so the DB is quiescent (zero post-snapshot writes): on Contabo `cd /opt/vaultwarden && docker compose stop`. (Keep the container + `data/` intact as rollback baseline — do **not** `down`/delete.)
- **Hot consistent snapshot** on the source host (`/usr/bin/sqlite3` v3.45.1 confirmed present):
  ```
  sqlite3 /opt/vaultwarden/data/db.sqlite3 ".backup /tmp/vw-snap.sqlite3"
  sqlite3 /tmp/vw-snap.sqlite3 "PRAGMA integrity_check"   # must == ok BEFORE transfer
  ```
  `.backup` uses the online backup API and writes **one self-contained file with NO `-wal`/`-shm` side files** (verified). The output's `journal_mode=wal` readout is just a persisted header setting — it is fully self-contained; do not be alarmed.
- **File manifest:**
  - **MUST travel:** `vw-snap.sqlite3` → target `data/db.sqlite3`; **`rsa_key.pem`** byte-for-byte (JWT signing key — G9).
  - **OPTIONAL:** `icon_cache/` (cosmetic favicon cache, auto-rebuilds).
  - **MUST NOT travel:** `db.sqlite3-wal`, `db.sqlite3-shm`, the bare live `db.sqlite3`, `tmp/`. (Copying a live `-wal` alongside a separately-snapshotted db risks recovery corruption.)
- Verify in CT 102: `sqlite3 data/db.sqlite3 "PRAGMA integrity_check"` == ok; `rsa_key.pem` present (header `-----BEGIN RSA PRIVATE KEY-----`, ~1679 B).
- **Note:** when Vaultwarden opens the migrated db it will create **fresh** `db.sqlite3-wal`/`-shm` on the target automatically — expected and correct; do not pre-create or copy them.
- **Rollback:** destroy CT 102; restart source: `cd /opt/vaultwarden && docker compose up -d`.

### Phase 6 — Start + health gate `[S]`
- `cd /opt/vaultwarden && docker compose up -d` in CT 102.
- **Health gates (all must pass before cutover):**
  1. **Pre-start integrity:** `sqlite3 data/db.sqlite3 "PRAGMA integrity_check"` == ok (done in Phase 5).
  2. **RSA key loaded:** container logs **must NOT** contain `generating new RSA key` (would mean `rsa_key.pem` didn't travel/load → mass session invalidation), and should show normal startup.
  3. **`/alive` == 200** over the tailnet from Contabo (the HARD composed-path gate from Phase 3):
     ```
     ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://VW_TS_IP:8086/alive"
     ```
  4. **Row-count gate:** `sqlite3 data/db.sqlite3 "SELECT count(*) FROM users"` == **1** (matches source `matt@echo6.co`).
- **Rollback:** destroy CT 102; restart source (`docker compose up -d` on Contabo).

### Phase 7 — Cutover `[G]` ← the primary, instant-rollback step
- On Contabo, capture a verbatim on-box baseline first:
  ```
  cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-prevault
  ```
- **The ONE edit:** in `/etc/caddy/Caddyfile`, replace **only the upstream token** on the `vault.echo6.co` backend line — `127.0.0.1:8086` → `VW_TS_IP:8086` (**no scheme** — bare host:port defaults to http; the proxmox block uses a scheme only because that backend is HTTPS). Keep `header_up X-Real-IP {remote_host}` and the braces untouched (G5). Resulting block must read exactly:
  ```
  vault.echo6.co {
      reverse_proxy VW_TS_IP:8086 {
          header_up X-Real-IP {remote_host}
      }
  }
  ```
- Validate (works under `admin off`), then **restart** (G10 — `caddy reload` and `systemctl reload caddy` both FAIL under `admin off`):
  ```
  caddy validate --config /etc/caddy/Caddyfile
  systemctl restart caddy
  ```
- **Blast-radius note (G10):** `systemctl restart caddy` briefly drops **all** Contabo-Caddy-fronted sites (auth/forge/mail/vault…) for ~1–2 s, not just vault. Acceptable; validate-before-restart prevents restarting into a broken config.
- **Rollback (primary, ~1–2 s, public DNS never moved):** either
  ```
  cp /etc/caddy/Caddyfile.bak-prevault /etc/caddy/Caddyfile && systemctl restart caddy
  ```
  OR revert the line to exactly:
  ```
  vault.echo6.co {
      reverse_proxy 127.0.0.1:8086 {
          header_up X-Real-IP {remote_host}
      }
  }
  ```
  then `caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy`, and ensure the source container is running (`cd /opt/vaultwarden && docker compose up -d`).

### Phase 8 — End-to-end verification `[G/S]`
- `vault.echo6.co` resolves unchanged → `5.189.158.149`; `curl -I https://vault.echo6.co` 200, valid cert.
- **Local login:** log in at `https://vault.echo6.co` with **email + master password** (non-SSO) → vault items decrypt + list (proves local-login safety net + data integrity). ← user-requested assurance.
- **SSO login:** the "Log in with SSO" path round-trips through Authentik and lands back **logged-in** on the **unchanged** `https://vault.echo6.co`. Authentik needs **no** change (DNS + `DOMAIN` unchanged → registered redirect URI stays valid) (G6). Verify only that the round-trip completes logged-in — do **not** assert a specific redirect path (the archive's `/oidc-signin` vs `/callback` is stale/inconsistent and irrelevant since Authentik is untouched).
- **Websocket sync:** browser-extension live sync works (101 Upgrade through Caddy) (G7). No `/notifications/hub` or `:3012` route exists or is needed — WS is in-process on `:8086`, so the single upstream carries the Upgrade.
- **Reboot survival:** `sudo pct reboot 102` → CT comes back, Docker + container auto-start (`--onboot 1` + `restart: unless-stopped`), tailscale reconnects (DNS-bootstrap drop-in), `/alive` 200 (G8).
- **Rollback:** Phase 7 revert.

### Phase 9 — Decommission (deferred) `[G]`
- Source stays **installed + stopped** as a cold rollback for a soak period. Final `tar` of `/opt/vaultwarden` before any eventual deletion.
- **Retire the stale archive doc** `/home/zvx/projects/.ref/archive/projects/vaultwarden-deployment.md` (it misdescribes ports/routes/URI).
- Update `.ref` (services.md, environment.md, ip-allocation.md → CT 102 @ 10.10.10.20, the new headscale node). Promote this file to `.ref/runbooks/lxc-service-migration.md`.

## Gotchas
- **G1** Stale `caddy.md`/archive mentions a `:3012`/`notifications/hub` WS route — live Caddy has **none** (in-process WS on `:8086`). Confirmed in Phase 0.
- **G2** edge2 `vmbr0` has **no DHCP** and no `local-lvm` → static IP `10.10.10.20/24` + `local` (dir) storage. (Overrides ct-runbook defaults.)
- **G3** Source binds loopback-only (`127.0.0.1:8086:80`); LXC must bind the tailnet IP (`VW_TS_IP:8086:80`) so Caddy reaches it. Keep the container-side `:80`.
- **G4** SQLite is in WAL mode with a **live 2.4 MB WAL** → use hot `.backup` (single self-contained file, no `-wal`/`-shm`); never copy a live `db.sqlite3`+WAL naively. Here we also **stop the source first** for zero-loss.
- **G5** Preserve `header_up X-Real-IP {remote_host}` in the new Caddy backend line (edit only the upstream token).
- **G6** SSO redirect derives from the unchanged `DOMAIN`/DNS → **do not touch Authentik**. Verify the round-trip lands logged-in; don't assert a specific redirect path.
- **G7** Caddy passes websocket upgrades by default; no WS-specific directive needed (in-process WS on `:8086`).
- **G8** Reboot survival needs all three: `--onboot 1`, `restart: unless-stopped`, and the DNS-bootstrap drop-in (so `vpn.echo6.co` resolves at boot).
- **G9** `rsa_key.pem` is the **JWT signing key** — must travel byte-for-byte. **Accurate failure mode:** if missing, Vaultwarden generates a new one and **all existing device/session tokens are invalidated (forced re-login everywhere) + push breaks** — vault DATA still decrypts on master-password login (data is client-side encrypted). The "generating new RSA key" log gate (Phase 6) catches this.
- **G10** Contabo Caddy admin API is **off** → `caddy validate` then **`systemctl restart caddy`**. Both `caddy reload` and `systemctl reload caddy` FAIL (`ExecReload` uses `caddy reload --force`). Restart briefly drops all Contabo-fronted sites (~1–2 s).
- **N (new):** the composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised → keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use curl/http, not ICMP).

## Generalize for future services
Template = phases **0→1→2→3→ (4,5,6 swap per service) →7→8→9**. Generic, identical for every Contabo-Caddy-fronted service: free-CTID pick (`pvesh get /cluster/nextid`) + LXC provision (right-sized per service), Docker, Tailscale onboard + DNS-bootstrap drop-in, the one-token Caddy cutover + its verbatim rollback. Service-specific only: the compose/`.env` (Phase 4 — copy from the **live host**, never archives), what "data" is + its consistency method (Phase 5 — stateless services collapse this; SQLite uses stop-then-`.backup`; Postgres-backed services dump/restore instead of file-copy), and the login/functional checks (Phases 6/8). Always pin edge2 access to `admin@184.174.35.153` + `sudo`, and always download the right template first.

## What I'll need to do (permissions)
- SSH to **Contabo** as `root@100.64.0.1` (Caddy edit, headscale preauth/node, source sqlite3 `.backup`, source `docker compose stop/up`).
- SSH to **edge2** as **`admin@184.174.35.153`** (key `~/.ssh/contabo2_ed25519`, alias `edge2`) then **`sudo`** for all `pct`/`pvesm`/`pveam` commands. (NOT `root@100.64.0.26` — refused.)
- `pveam download` one Debian 12 template; `pct`/`docker`/`tailscale` inside CT 102; create one Headscale preauth key + node; edit **one token** of Contabo's Caddyfile; copy ~5 MB of data.
- **No public DNS changes, no Authentik changes, no dnsmasq changes; source left intact (stopped) as rollback.** Never touch CT 101 (live WordPress).