echo6-docs/runbooks/lxc-service-migration.md

# LXC Service Migration — Contabo → edge2

> Proven pilot: **Vaultwarden → edge2 CT 102** (2026-06-16). This runbook generalizes that pattern into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.

---

## Overview

Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo Caddy frontend unchanged (public DNS never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.

**Architecture after migration:**

```
Internet → 5.189.158.149 (Contabo Caddy) → 100.64.0.XX:PORT (edge2 LXC, via tailnet)
```

**edge2 access (always):** `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`), then `sudo` for every `pct`/`pvesm`/`pveam` command. `root@100.64.0.26` is refused — do not use it.

---

## Phases

### Phase 0 — Recon & baseline `[G]`

- On Contabo (`ssh root@100.64.0.1`): capture the **verbatim** `<service>.echo6.co` Caddy block (this is your rollback baseline). Grep the whole Caddyfile for any sub-routes related to the service.
- Read the live compose file + `.env` + data directory listing **from the running host**. Never use `.ref/archive` docs — they may be stale.
- On edge2 (`ssh edge2`):
  - Confirm the next free CT ID: `sudo pvesh get /cluster/nextid`
  - Confirm chosen IP is free: check `sudo pct list` and the `10.10.10.x` allocation table in `ip-allocation.md`
  - Check for needed template: `sudo pvesm list local | grep <os>`. If absent: `sudo pveam download local <template-volid>`
  - Confirm storage free space: `sudo pvesm status`
- **Rollback:** n/a (read-only except optional template download).

**Service-specific (example: Vaultwarden):** greppeed for `:3012`/`notifications/hub` sub-routes — none existed. Confirmed CT 102 and 10.10.10.20 free.

---

### Phase 0a — Pre-migration gate `[S]`

Confirm any service-specific preconditions before provisioning (e.g., local login works, data is sane, credentials are recorded).

**Service-specific (example: Vaultwarden):** Verified ≥1 account has a non-empty `password_hash` (local login works without SSO). Recorded `ADMIN_TOKEN` to credentials file.

---

### Phase 1 — Provision the LXC `[G]`

Pick right-sized resources for the service (not a copy of another CT's sizing).

```bash
ssh edge2 "sudo pct create <CTID> local:vztmpl/<template>.tar.zst \
  --hostname <service> --cores <N> --memory <MB> --swap <MB> --rootfs local:<GB> \
  --net0 name=eth0,bridge=vmbr0,ip=10.10.10.<X>/24,gw=10.10.10.1 \
  --nameserver 1.1.1.1 --ostype <debian|ubuntu> \
  --features nesting=1,keyctl=1 --unprivileged 1 --onboot 1"
```

Append TUN device lines to the CT config (required for Tailscale):

```bash
ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/<CTID>.conf"
ssh edge2 "sudo pct start <CTID>"
```

Verify: internet access via NAT, DNS resolution.

**Rollback:** `ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'`

**Service-specific (example: Vaultwarden):** CT 102, hostname `vaultwarden`, 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs, IP 10.10.10.20/24, Debian 12.

---

### Phase 2 — Docker baseline `[G]`

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c 'curl -fsSL https://get.docker.com | sh && systemctl enable --now docker'"
```

(`nesting=1` + `keyctl=1` from Phase 1 make Docker-in-LXC work.)

**Rollback:** destroy CT.

---

### Phase 3 — Tailscale onboard `[G]`

Create a preauth key on the Headscale host:

```bash
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
```

Install Tailscale + DNS-bootstrap drop-in inside the CT:

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c '
  curl -fsSL https://tailscale.com/install.sh | sh
  mkdir -p /etc/systemd/system/tailscaled.service.d
  cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<\"EOF\"
[Service]
Environment=TS_DEBUG_RESOLV_CONF=1
EOF
  systemctl daemon-reload && systemctl restart tailscaled
  tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<service>
'"
```

Capture the tailnet IP: `sudo pct exec <CTID> -- tailscale ip -4` → call it `SVC_TS_IP`.

**Hard pre-cutover reachability gate** (run now, should return `000`; must return `200` after Phase 6):

```bash
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
```

**Rollback:** destroy CT + `ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'`

**Service-specific (example: Vaultwarden):** tailnet IP became 100.64.0.33, Headscale node id 45. Health path: `/alive`.

---

### Phase 4 — Stage compose + config `[G]`

Copy compose file + config **byte-for-byte from the live running host** into the CT. Never copy from `.ref/archive` docs.

```bash
# Copy from Contabo to edge2 CT (via intermediate)
scp root@100.64.0.1:/opt/<service>/docker-compose.yml /tmp/
ssh edge2 "sudo pct exec <CTID> -- mkdir -p /opt/<service>"
# push to CT via edge2:
scp /tmp/docker-compose.yml edge2:/tmp/
ssh edge2 "sudo pct push <CTID> /tmp/docker-compose.yml /opt/<service>/docker-compose.yml"
# Repeat for .env / other config files
```

**The ONE compose edit `[G3]`:** change the `ports` bind address from `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>` (keep the container-side port unchanged). This binds the service to the tailnet IP so Caddy can reach it — but nothing else can.

Confirm there are no unexpected sub-routes or extra ports in the compose file.

**Rollback:** destroy CT.

**Service-specific (example: Vaultwarden):** `.env` carried untouched (SSO_ONLY=false, DOMAIN, ADMIN_TOKEN, SSO_* preserved). Changed `127.0.0.1:8086:80` → `100.64.0.33:8086:80`. Confirmed no `:3012` line.

---

### Phase 5 — Migrate data `[S]`

This phase is entirely service-specific. Common patterns:

| Data type | Method |
|-----------|--------|
| Stateless (no persistent data) | Skip this phase |
| SQLite (WAL mode) | Stop source → `sqlite3 .backup` → verify integrity → transfer |
| PostgreSQL | `pg_dump` → transfer → `psql` restore |
| File data | `rsync` or `tar` |

**Zero-loss for single-user SQLite:**
```bash
# Stop source (keeps container + data intact as rollback baseline)
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'

# Snapshot + verify
ssh root@100.64.0.1 'sqlite3 /opt/<service>/data/db.sqlite3 ".backup /tmp/svc-snap.sqlite3" && sqlite3 /tmp/svc-snap.sqlite3 "PRAGMA integrity_check"'

# Transfer to CT
scp root@100.64.0.1:/tmp/svc-snap.sqlite3 /tmp/
ssh edge2 "sudo pct push <CTID> /tmp/svc-snap.sqlite3 /opt/<service>/data/db.sqlite3"
```

Do NOT copy live `-wal`/`-shm` files alongside the snapshot.

**Rollback:** destroy CT + `ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'`

**Service-specific (example: Vaultwarden):** Also transferred `rsa_key.pem` byte-for-byte (JWT signing key — see G9). Did NOT transfer `db.sqlite3-wal`, `db.sqlite3-shm`, `tmp/`.

---

### Phase 6 — Start + health gate `[G/S]`

```bash
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d'"
```

All must pass before cutover:

1. **Data integrity check** (if applicable, e.g. `sqlite3 db.sqlite3 "PRAGMA integrity_check"` == `ok`)
2. **Service-specific startup log checks** (no error markers)
3. **`/alive` == 200 over tailnet from Contabo** (the HARD composed-path gate):
   ```bash
   ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
   ```
   Must return `200` — do not proceed to Phase 7 if it doesn't.

**Rollback:** destroy CT + restart source on Contabo.

**Service-specific (example: Vaultwarden):** Also checked that container logs did NOT contain `generating new RSA key` (confirms `rsa_key.pem` loaded — G9). Checked `SELECT count(*) FROM users` == 1.

---

### Phase 7 — Cutover `[G]`

On Contabo, take a backup first:

```bash
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-pre<service>'
```

**The ONE edit:** replace only the upstream token in the `<service>.echo6.co` block — `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>`. No scheme prefix (bare `host:port` defaults to HTTP). Preserve all other directives (e.g. `header_up X-Real-IP {remote_host}`).

Validate, then restart (G10 — `caddy reload` fails under `admin off`):

```bash
ssh root@100.64.0.1 'caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy'
```

**Blast radius (G10):** `systemctl restart caddy` briefly drops all Contabo-Caddy-fronted sites for ~1–2 s. Validate before restart prevents broken-config restarts.

**Rollback (~1–2 s, public DNS never moved):**

```bash
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile.bak-pre<service> /etc/caddy/Caddyfile && systemctl restart caddy'
# Then if source container was stopped:
ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'
```

**Service-specific (example: Vaultwarden):** Final block:
```caddyfile
vault.echo6.co {
    reverse_proxy 100.64.0.33:8086 {
        header_up X-Real-IP {remote_host}
    }
}
```

---

### Phase 8 — End-to-end verification `[G/S]`

- `<service>.echo6.co` DNS still resolves to `5.189.158.149` (unchanged)
- `curl -I https://<service>.echo6.co` → 200, valid cert
- Functional login + core workflow test
- **Reboot survival:** `ssh edge2 'sudo pct reboot <CTID>'` → CT comes back, Docker + container auto-start (`--onboot 1` + `restart: unless-stopped`), Tailscale reconnects (DNS-bootstrap drop-in), `/alive` 200 `[G8]`
- **Rollback:** Phase 7 revert.

**Service-specific (example: Vaultwarden):** Tested local (email + master-password) login AND SSO login. Verified websocket sync via browser extension. Verified reboot survival.

---

### Phase 9 — Decommission (deferred) `[G]`

- Source stays **installed + stopped** as cold rollback during soak period
- Before eventual deletion: `tar -czf <service>-source-backup-$(date +%Y%m%d).tar.gz /opt/<service>`
- Update `.ref` docs (services.md, environment.md, ip-allocation.md)
- Retire or banner any stale archive docs
- Promote migration plan to this runbook (done)

---

## Gotchas

| ID | Issue |
|----|-------|
| G1 | Stale docs may show sub-routes (e.g. `:3012`/`notifications/hub`) that don't exist in the live Caddyfile. Always read from the live host in Phase 0. |
| G2 | edge2 `vmbr0` has **no DHCP** and no `local-lvm` → must use static IP + `local` (dir) storage. |
| G3 | Source binds `127.0.0.1:<PORT>:<CONTAINER_PORT>`; LXC must bind `<SVC_TS_IP>:<PORT>:<CONTAINER_PORT>`. Keep the container-side port unchanged. |
| G4 | SQLite in WAL mode has a live `.wal` file — never copy `db.sqlite3` + `-wal` naively. Use `sqlite3 .backup` (produces one self-contained file). Stop the source first for zero-loss. |
| G5 | Preserve all existing Caddy directives on the backend block (e.g. `header_up X-Real-IP {remote_host}`) — edit only the upstream token. |
| G6 | SSO redirect URIs derive from the unchanged `DOMAIN`/DNS — **do not touch Authentik**. Verify the round-trip lands logged-in; don't assert a specific redirect path. |
| G7 | Caddy passes websocket upgrades by default; no WS-specific directive needed when WS is in-process on the same port. |
| G8 | Reboot survival requires all three: `--onboot 1`, `restart: unless-stopped`, and the Tailscale DNS-bootstrap drop-in. |
| G9 | JWT signing keys (e.g. `rsa_key.pem` for Vaultwarden) **must travel byte-for-byte**. If missing, existing sessions are invalidated (forced re-login everywhere) but data remains intact. Check startup logs for "generating new key" messages. |
| G10 | Contabo Caddy runs with `admin off` → `caddy reload` and `systemctl reload caddy` FAIL. Always use `caddy validate` then `systemctl restart caddy`. Restart drops all Contabo-fronted sites for ~1–2 s. |
| N | The composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised for each new service. Keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use `curl`, not ICMP). |

---

## Template Summary

**Generic phases (identical for every service):** 0 (recon) → 1 (LXC provision) → 2 (Docker) → 3 (Tailscale + DNS-bootstrap + reachability pre-gate) → 7 (one-token Caddy cutover + backup + validate + restart) → 8 (end-to-end + reboot survival) → 9 (deferred decommission).

**Service-specific phases:** 0a (pre-migration gate), 4 (compose/config — copy from live host), 5 (data migration method depends on storage type), 6 (health gates — service-specific checks before cutover).

**Access pins:**
- Contabo: `ssh root@100.64.0.1`
- edge2: `ssh admin@184.174.35.153` (alias `edge2`) + `sudo` for pct/pvesm/pveam
- `root@100.64.0.26` is refused on edge2

---

*Last updated: 2026-06-17*