auto: docs sync 2026-06-17T12:00:04+00:00
Files changed: .obsidian/workspace.json INDEX.md archive/projects/vaultwarden-deployment.md docs/hardware/environment.md docs/hardware/ip-allocation.md docs/services/services.md runbooks/lxc-service-migration.md
This commit is contained in:
parent
e9cb7777fe
commit
91a08df645
7 changed files with 342 additions and 17 deletions
24
.obsidian/workspace.json
vendored
24
.obsidian/workspace.json
vendored
|
|
@ -189,12 +189,22 @@
|
|||
},
|
||||
"active": "ea4cc678c44e8b67",
|
||||
"lastOpenFiles": [
|
||||
"credentials.tmp.3861660.701c90c30e54",
|
||||
"INDEX.md.tmp.40509.2a05e7708182",
|
||||
"runbooks/lxc-service-migration.md",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.38316817de79",
|
||||
"archive/projects/vaultwarden-deployment.md.tmp.40509.88bb66425f5e",
|
||||
"docs/hardware/ip-allocation.md.tmp.40509.b03329ad68b8",
|
||||
"docs/hardware/ip-allocation.md.tmp.40509.008cf455ddde",
|
||||
"docs/hardware/ip-allocation.md.tmp.40509.8b2ec85a60b8",
|
||||
"docs/hardware/environment.md.tmp.40509.ad3a0bc1d9bf",
|
||||
"docs/hardware/environment.md.tmp.40509.e90afdbac998",
|
||||
"docs/hardware/environment.md.tmp.40509.727e35e81611",
|
||||
"docs/services/services.md.tmp.40509.e97391cd330d",
|
||||
"docs/services/services.md.tmp.40509.3adb6587d024",
|
||||
"runbooks/edge2-access-reference.md",
|
||||
"runbooks/expose-service-edge2.md",
|
||||
"plans/vaultwarden-plan.md",
|
||||
"plans/vaultwarden-migration.md",
|
||||
"plans",
|
||||
"rules/radio.md",
|
||||
"rules/tak.md",
|
||||
"archive/matrix/PHASE6_DECISION.md",
|
||||
|
|
@ -211,20 +221,10 @@
|
|||
"rules/watchtower.md",
|
||||
"rules/argus.md",
|
||||
"rules/aurora.md",
|
||||
"Untitled.base",
|
||||
"docs/matrix/synapse_homeserver.yaml.sanitized",
|
||||
"Untitled 1.base",
|
||||
"projects/argus.md",
|
||||
"archive/matrix/PLAN.md",
|
||||
"archive/matrix/archivist_discovery.md",
|
||||
"archive/matrix/appservices.md",
|
||||
"archive/matrix/archive_receiver_discovery.md",
|
||||
"notes",
|
||||
"archive/matrix",
|
||||
"docs/matrix",
|
||||
"archive/projects/mmud/mmud-prompts/mmud-prompts",
|
||||
"archive/projects/mmud/mmud-prompts.tar.gz",
|
||||
"archive/projects/mmud/mmud-prompts",
|
||||
"assets/echo6yellow_logo_422x422_square.png",
|
||||
"assets/echo6yellow_logo_422x81.png",
|
||||
"assets/echo6_logo.png",
|
||||
|
|
|
|||
1
INDEX.md
1
INDEX.md
|
|
@ -48,6 +48,7 @@ The map of this vault — start here. `docs/` is current reference, `runbooks/`
|
|||
|
||||
- **Authentik:** [[authentik-oidc-application]] · [[authentik-access-groups]] · [[authentik-create-invitation]] · [[authentik-upgrade]]
|
||||
- **Exposing a service:** [[expose-service-contabo]] · [[expose-service-home]]
|
||||
- **Service migration:** [[lxc-service-migration]] — move a Contabo-Caddy-fronted service to edge2 LXC (Vaultwarden pilot 2026-06-16)
|
||||
- **Proxmox / hosts:** [[ct-runbook]] · [[proxmox-create-ubuntu-vm]] · [[proxmox-onboard-node]] · [[pi-nas-omv-runbook]] · [[headscale-onboard-node]]
|
||||
- **RECON:** [[recon-operations]] · [[recon-service-integration]]
|
||||
- **Mesh / Meshtastic:** [[meshtastic-sidecar-node]] · [[meshtasticd-sim-nodes-runbook]] · [[idahomesh-bridge-setup]] · [[idahomesh-vpn-device-setup]] · [[meshmonitor-password-reset]]
|
||||
|
|
|
|||
|
|
@ -1,7 +1,19 @@
|
|||
> [!warning] STALE / SUPERSEDED — DO NOT USE AS REFERENCE
|
||||
>
|
||||
> **This document describes the original Contabo deployment (2026-02-05) and is no longer accurate.**
|
||||
>
|
||||
> - The service was **migrated to edge2 CT 102** (10.10.10.20 / Tailscale 100.64.0.33) on **2026-06-16**.
|
||||
> - The `:3012` websocket port, `/notifications/hub` Caddy route, dnsmasq split-DNS entry, and `/oidc-signin` redirect URI described below **do not exist** in the live deployment.
|
||||
> - **Current reference:** `docs/services/services.md` (Vaultwarden entry) and `runbooks/lxc-service-migration.md`.
|
||||
> - **Rollback info:** Contabo source is stopped-but-intact; `/etc/caddy/Caddyfile.bak-prevault` exists on Contabo for instant rollback.
|
||||
|
||||
---
|
||||
|
||||
# Vaultwarden Deployment
|
||||
|
||||
**Deployed:** 2026-02-05
|
||||
**Location:** Contabo VPS (5.189.158.149 / 100.64.0.1)
|
||||
**Superseded:** 2026-06-16 (migrated to edge2 CT 102 — see services.md)
|
||||
**Location (historical):** Contabo VPS (5.189.158.149 / 100.64.0.1)
|
||||
**URL:** https://vault.echo6.co
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -87,7 +87,7 @@ Five nodes running Proxmox VE:
|
|||
| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, Authentik, Headscale, Forge, Matrix |
|
||||
| edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB |
|
||||
|
||||
*Last updated: 2026-06-16 — Added edge2 CTs (pdm CT 100, wordpress CT 101), Caddy reverse proxy on edge2 host*
|
||||
*Last updated: 2026-06-17 — Added edge2 CT 102 (vaultwarden), Headscale node 45 (100.64.0.33); previously added edge2 CTs pdm CT 100 / wordpress CT 101*
|
||||
|
||||
## LXC Containers
|
||||
|
||||
|
|
@ -106,6 +106,7 @@ Five nodes running Proxmox VE:
|
|||
| peertube | media (CT 110) | 192.168.1.170 | 100.64.0.23 | PeerTube video streaming |
|
||||
| pdm | edge2 (CT 100) | 10.10.10.10 | 100.64.0.28 | Proxmox Datacenter Manager |
|
||||
| wordpress | edge2 (CT 101) | 10.10.10.11 | 100.64.0.31 | WordPress for intermountainmesh.com |
|
||||
| vaultwarden | edge2 (CT 102) | 10.10.10.20 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
|
||||
|
||||
## IP Allocation Scheme
|
||||
|
||||
|
|
@ -153,6 +154,7 @@ Current registered nodes (26 total):
|
|||
| bluefin | 100.64.0.30 | Desktop |
|
||||
| wordpress | 100.64.0.31 | LXC |
|
||||
| meshai | 100.64.0.32 | LXC |
|
||||
| vaultwarden | 100.64.0.33 | LXC (edge2 CT 102) |
|
||||
|
||||
## IdahoMesh Headscale Node List
|
||||
|
||||
|
|
|
|||
|
|
@ -47,6 +47,16 @@
|
|||
| .182 | immich (CT 120) | cloud | Immich photos |
|
||||
| .183 | nextcloud (CT 121) | cloud | Nextcloud AIO |
|
||||
|
||||
### edge2 LXC Containers (10.10.10.x, vmbr0)
|
||||
|
||||
edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate 10.10.10.0/24 subnet on vmbr0 (gw 10.10.10.1). No DHCP — all IPs are static. Storage: `local` (dir, no local-lvm).
|
||||
|
||||
| IP | Container | CTID | Tailscale | Purpose |
|
||||
|----|-----------|------|-----------|---------|
|
||||
| 10.10.10.10 | pdm | CT 100 | 100.64.0.28 | Proxmox Datacenter Manager |
|
||||
| 10.10.10.11 | wordpress | CT 101 | 100.64.0.31 | WordPress for intermountainmesh.com |
|
||||
| 10.10.10.20 | vaultwarden | CT 102 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
|
||||
|
||||
### VMs (.150-.199)
|
||||
| IP | VM | Host | Purpose |
|
||||
|----|-----|------|---------|
|
||||
|
|
@ -95,6 +105,7 @@
|
|||
| 100.64.0.22 | cloud | 192.168.1.242 |
|
||||
| 100.64.0.26 | edge2 | 184.174.35.153 (external) |
|
||||
| 100.64.0.28 | pdm (CT 100 on edge2) | 10.10.10.10 (vmbr0) |
|
||||
| 100.64.0.33 | vaultwarden (CT 102 on edge2) — node id 45 | 10.10.10.20 (vmbr0) |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -107,4 +118,4 @@
|
|||
|
||||
---
|
||||
|
||||
*Last updated: 2026-06-16*
|
||||
*Last updated: 2026-06-17 — Added edge2 CT 102 (vaultwarden) at 10.10.10.20 / 100.64.0.33 (Headscale node 45)*
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@
|
|||
| Headscale | Contabo | 5.189.158.149 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) |
|
||||
| Headplane | Contabo | 127.0.0.1:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) |
|
||||
| Mailcow | Contabo | 5.189.158.149 | https://mail.echo6.co | Email server |
|
||||
| Vaultwarden | Contabo | 127.0.0.1:8086 | https://vault.echo6.co | Password manager (SSO enabled) |
|
||||
| Vaultwarden | edge2 (CT 102) | 100.64.0.33:8086 | https://vault.echo6.co | Password manager (SSO enabled) — fronted by Contabo Caddy (reverse_proxy 100.64.0.33:8086) |
|
||||
| Syncthing | Contabo | 100.64.0.1:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
|
||||
| Syncthing | cortex | 100.64.0.14:22000 | Internal (Tailscale) | File sync — ~/.claude/, ~/projects/ |
|
||||
| Proxmox VE | data node | 192.168.1.240:8006 | https://proxmox.echo6.co | Cluster web UI (via Caddy+Tailscale) |
|
||||
|
|
@ -254,12 +254,21 @@
|
|||
- Static MAC: A7:A1:30:79:BB:BB
|
||||
- Tailscale registered on IdahoMesh Headscale (vpn.idahomesh.com) under malice user
|
||||
|
||||
### edge2 - CT 102 (10.10.10.20 / Tailscale: 100.64.0.33, node 45 `vaultwarden`)
|
||||
- Vaultwarden password manager (port 8086, https://vault.echo6.co, Docker)
|
||||
- Headscale node id 45, name `vaultwarden`, user `echo6`
|
||||
- Compose path: `/opt/vaultwarden/docker-compose.yml`; data: `./data/`
|
||||
- Binds to tailnet IP `100.64.0.33:8086:80`; Contabo Caddy proxies here over tailnet
|
||||
- SSO via Authentik (`SSO_ONLY=false` — local email+master-password login also works)
|
||||
- Source (Contabo `/opt/vaultwarden`) STOPPED but intact as cold rollback; rollback = restore `/etc/caddy/Caddyfile.bak-prevault` + `systemctl restart caddy` + `docker compose up -d` on Contabo
|
||||
- **Migrated from Contabo to edge2 CT 102 on 2026-06-16**
|
||||
|
||||
### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1)
|
||||
- Authentik (SSO, Echo6 branded — custom CSS, dark theme, logo, favicon, flow titles)
|
||||
- Forge (Git)
|
||||
- Headscale (mesh VPN)
|
||||
- Mailcow (email)
|
||||
- Vaultwarden (passwords)
|
||||
- Vaultwarden — **migrated to edge2 CT 102 on 2026-06-16** (Caddy now proxies to 100.64.0.33:8086)
|
||||
- Syncthing (syncs with cortex)
|
||||
- WATCHTOWER (ops dashboard, port 8099, https://wt.echo6.co)
|
||||
- Matrix Synapse homeserver (port 8008, https://matrix.echo6.co, Docker, SSO via Authentik)
|
||||
|
|
|
|||
290
runbooks/lxc-service-migration.md
Normal file
290
runbooks/lxc-service-migration.md
Normal file
|
|
@ -0,0 +1,290 @@
|
|||
# LXC Service Migration — Contabo → edge2
|
||||
|
||||
> Proven pilot: **Vaultwarden → edge2 CT 102** (2026-06-16). This runbook generalizes that pattern into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo Caddy frontend unchanged (public DNS never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.
|
||||
|
||||
**Architecture after migration:**
|
||||
|
||||
```
|
||||
Internet → 5.189.158.149 (Contabo Caddy) → 100.64.0.XX:PORT (edge2 LXC, via tailnet)
|
||||
```
|
||||
|
||||
**edge2 access (always):** `ssh admin@184.174.35.153` (alias `edge2`, key `~/.ssh/contabo2_ed25519`), then `sudo` for every `pct`/`pvesm`/`pveam` command. `root@100.64.0.26` is refused — do not use it.
|
||||
|
||||
---
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 0 — Recon & baseline `[G]`
|
||||
|
||||
- On Contabo (`ssh root@100.64.0.1`): capture the **verbatim** `<service>.echo6.co` Caddy block (this is your rollback baseline). Grep the whole Caddyfile for any sub-routes related to the service.
|
||||
- Read the live compose file + `.env` + data directory listing **from the running host**. Never use `.ref/archive` docs — they may be stale.
|
||||
- On edge2 (`ssh edge2`):
|
||||
- Confirm the next free CT ID: `sudo pvesh get /cluster/nextid`
|
||||
- Confirm chosen IP is free: check `sudo pct list` and the `10.10.10.x` allocation table in `ip-allocation.md`
|
||||
- Check for needed template: `sudo pvesm list local | grep <os>`. If absent: `sudo pveam download local <template-volid>`
|
||||
- Confirm storage free space: `sudo pvesm status`
|
||||
- **Rollback:** n/a (read-only except optional template download).
|
||||
|
||||
**Service-specific (example: Vaultwarden):** greppeed for `:3012`/`notifications/hub` sub-routes — none existed. Confirmed CT 102 and 10.10.10.20 free.
|
||||
|
||||
---
|
||||
|
||||
### Phase 0a — Pre-migration gate `[S]`
|
||||
|
||||
Confirm any service-specific preconditions before provisioning (e.g., local login works, data is sane, credentials are recorded).
|
||||
|
||||
**Service-specific (example: Vaultwarden):** Verified ≥1 account has a non-empty `password_hash` (local login works without SSO). Recorded `ADMIN_TOKEN` to credentials file.
|
||||
|
||||
---
|
||||
|
||||
### Phase 1 — Provision the LXC `[G]`
|
||||
|
||||
Pick right-sized resources for the service (not a copy of another CT's sizing).
|
||||
|
||||
```bash
|
||||
ssh edge2 "sudo pct create <CTID> local:vztmpl/<template>.tar.zst \
|
||||
--hostname <service> --cores <N> --memory <MB> --swap <MB> --rootfs local:<GB> \
|
||||
--net0 name=eth0,bridge=vmbr0,ip=10.10.10.<X>/24,gw=10.10.10.1 \
|
||||
--nameserver 1.1.1.1 --ostype <debian|ubuntu> \
|
||||
--features nesting=1,keyctl=1 --unprivileged 1 --onboot 1"
|
||||
```
|
||||
|
||||
Append TUN device lines to the CT config (required for Tailscale):
|
||||
|
||||
```bash
|
||||
ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file\n' | sudo tee -a /etc/pve/lxc/<CTID>.conf"
|
||||
ssh edge2 "sudo pct start <CTID>"
|
||||
```
|
||||
|
||||
Verify: internet access via NAT, DNS resolution.
|
||||
|
||||
**Rollback:** `ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'`
|
||||
|
||||
**Service-specific (example: Vaultwarden):** CT 102, hostname `vaultwarden`, 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs, IP 10.10.10.20/24, Debian 12.
|
||||
|
||||
---
|
||||
|
||||
### Phase 2 — Docker baseline `[G]`
|
||||
|
||||
```bash
|
||||
ssh edge2 "sudo pct exec <CTID> -- bash -c 'curl -fsSL https://get.docker.com | sh && systemctl enable --now docker'"
|
||||
```
|
||||
|
||||
(`nesting=1` + `keyctl=1` from Phase 1 make Docker-in-LXC work.)
|
||||
|
||||
**Rollback:** destroy CT.
|
||||
|
||||
---
|
||||
|
||||
### Phase 3 — Tailscale onboard `[G]`
|
||||
|
||||
Create a preauth key on the Headscale host:
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
|
||||
```
|
||||
|
||||
Install Tailscale + DNS-bootstrap drop-in inside the CT:
|
||||
|
||||
```bash
|
||||
ssh edge2 "sudo pct exec <CTID> -- bash -c '
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
mkdir -p /etc/systemd/system/tailscaled.service.d
|
||||
cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<\"EOF\"
|
||||
[Service]
|
||||
Environment=TS_DEBUG_RESOLV_CONF=1
|
||||
EOF
|
||||
systemctl daemon-reload && systemctl restart tailscaled
|
||||
tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<service>
|
||||
'"
|
||||
```
|
||||
|
||||
Capture the tailnet IP: `sudo pct exec <CTID> -- tailscale ip -4` → call it `SVC_TS_IP`.
|
||||
|
||||
**Hard pre-cutover reachability gate** (run now, should return `000`; must return `200` after Phase 6):
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
|
||||
```
|
||||
|
||||
**Rollback:** destroy CT + `ssh root@100.64.0.1 'docker exec headscale headscale nodes delete -i <id>'`
|
||||
|
||||
**Service-specific (example: Vaultwarden):** tailnet IP became 100.64.0.33, Headscale node id 45. Health path: `/alive`.
|
||||
|
||||
---
|
||||
|
||||
### Phase 4 — Stage compose + config `[G]`
|
||||
|
||||
Copy compose file + config **byte-for-byte from the live running host** into the CT. Never copy from `.ref/archive` docs.
|
||||
|
||||
```bash
|
||||
# Copy from Contabo to edge2 CT (via intermediate)
|
||||
scp root@100.64.0.1:/opt/<service>/docker-compose.yml /tmp/
|
||||
ssh edge2 "sudo pct exec <CTID> -- mkdir -p /opt/<service>"
|
||||
# push to CT via edge2:
|
||||
scp /tmp/docker-compose.yml edge2:/tmp/
|
||||
ssh edge2 "sudo pct push <CTID> /tmp/docker-compose.yml /opt/<service>/docker-compose.yml"
|
||||
# Repeat for .env / other config files
|
||||
```
|
||||
|
||||
**The ONE compose edit `[G3]`:** change the `ports` bind address from `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>` (keep the container-side port unchanged). This binds the service to the tailnet IP so Caddy can reach it — but nothing else can.
|
||||
|
||||
Confirm there are no unexpected sub-routes or extra ports in the compose file.
|
||||
|
||||
**Rollback:** destroy CT.
|
||||
|
||||
**Service-specific (example: Vaultwarden):** `.env` carried untouched (SSO_ONLY=false, DOMAIN, ADMIN_TOKEN, SSO_* preserved). Changed `127.0.0.1:8086:80` → `100.64.0.33:8086:80`. Confirmed no `:3012` line.
|
||||
|
||||
---
|
||||
|
||||
### Phase 5 — Migrate data `[S]`
|
||||
|
||||
This phase is entirely service-specific. Common patterns:
|
||||
|
||||
| Data type | Method |
|
||||
|-----------|--------|
|
||||
| Stateless (no persistent data) | Skip this phase |
|
||||
| SQLite (WAL mode) | Stop source → `sqlite3 .backup` → verify integrity → transfer |
|
||||
| PostgreSQL | `pg_dump` → transfer → `psql` restore |
|
||||
| File data | `rsync` or `tar` |
|
||||
|
||||
**Zero-loss for single-user SQLite:**
|
||||
```bash
|
||||
# Stop source (keeps container + data intact as rollback baseline)
|
||||
ssh root@100.64.0.1 'cd /opt/<service> && docker compose stop'
|
||||
|
||||
# Snapshot + verify
|
||||
ssh root@100.64.0.1 'sqlite3 /opt/<service>/data/db.sqlite3 ".backup /tmp/svc-snap.sqlite3" && sqlite3 /tmp/svc-snap.sqlite3 "PRAGMA integrity_check"'
|
||||
|
||||
# Transfer to CT
|
||||
scp root@100.64.0.1:/tmp/svc-snap.sqlite3 /tmp/
|
||||
ssh edge2 "sudo pct push <CTID> /tmp/svc-snap.sqlite3 /opt/<service>/data/db.sqlite3"
|
||||
```
|
||||
|
||||
Do NOT copy live `-wal`/`-shm` files alongside the snapshot.
|
||||
|
||||
**Rollback:** destroy CT + `ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'`
|
||||
|
||||
**Service-specific (example: Vaultwarden):** Also transferred `rsa_key.pem` byte-for-byte (JWT signing key — see G9). Did NOT transfer `db.sqlite3-wal`, `db.sqlite3-shm`, `tmp/`.
|
||||
|
||||
---
|
||||
|
||||
### Phase 6 — Start + health gate `[G/S]`
|
||||
|
||||
```bash
|
||||
ssh edge2 "sudo pct exec <CTID> -- bash -c 'cd /opt/<service> && docker compose up -d'"
|
||||
```
|
||||
|
||||
All must pass before cutover:
|
||||
|
||||
1. **Data integrity check** (if applicable, e.g. `sqlite3 db.sqlite3 "PRAGMA integrity_check"` == `ok`)
|
||||
2. **Service-specific startup log checks** (no error markers)
|
||||
3. **`/alive` == 200 over tailnet from Contabo** (the HARD composed-path gate):
|
||||
```bash
|
||||
ssh root@100.64.0.1 "curl -sS -o /dev/null -w '%{http_code}' http://<SVC_TS_IP>:<PORT>/<healthcheck-path>"
|
||||
```
|
||||
Must return `200` — do not proceed to Phase 7 if it doesn't.
|
||||
|
||||
**Rollback:** destroy CT + restart source on Contabo.
|
||||
|
||||
**Service-specific (example: Vaultwarden):** Also checked that container logs did NOT contain `generating new RSA key` (confirms `rsa_key.pem` loaded — G9). Checked `SELECT count(*) FROM users` == 1.
|
||||
|
||||
---
|
||||
|
||||
### Phase 7 — Cutover `[G]`
|
||||
|
||||
On Contabo, take a backup first:
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak-pre<service>'
|
||||
```
|
||||
|
||||
**The ONE edit:** replace only the upstream token in the `<service>.echo6.co` block — `127.0.0.1:<PORT>` → `<SVC_TS_IP>:<PORT>`. No scheme prefix (bare `host:port` defaults to HTTP). Preserve all other directives (e.g. `header_up X-Real-IP {remote_host}`).
|
||||
|
||||
Validate, then restart (G10 — `caddy reload` fails under `admin off`):
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 'caddy validate --config /etc/caddy/Caddyfile && systemctl restart caddy'
|
||||
```
|
||||
|
||||
**Blast radius (G10):** `systemctl restart caddy` briefly drops all Contabo-Caddy-fronted sites for ~1–2 s. Validate before restart prevents broken-config restarts.
|
||||
|
||||
**Rollback (~1–2 s, public DNS never moved):**
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 'cp /etc/caddy/Caddyfile.bak-pre<service> /etc/caddy/Caddyfile && systemctl restart caddy'
|
||||
# Then if source container was stopped:
|
||||
ssh root@100.64.0.1 'cd /opt/<service> && docker compose up -d'
|
||||
```
|
||||
|
||||
**Service-specific (example: Vaultwarden):** Final block:
|
||||
```caddyfile
|
||||
vault.echo6.co {
|
||||
reverse_proxy 100.64.0.33:8086 {
|
||||
header_up X-Real-IP {remote_host}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 8 — End-to-end verification `[G/S]`
|
||||
|
||||
- `<service>.echo6.co` DNS still resolves to `5.189.158.149` (unchanged)
|
||||
- `curl -I https://<service>.echo6.co` → 200, valid cert
|
||||
- Functional login + core workflow test
|
||||
- **Reboot survival:** `ssh edge2 'sudo pct reboot <CTID>'` → CT comes back, Docker + container auto-start (`--onboot 1` + `restart: unless-stopped`), Tailscale reconnects (DNS-bootstrap drop-in), `/alive` 200 `[G8]`
|
||||
- **Rollback:** Phase 7 revert.
|
||||
|
||||
**Service-specific (example: Vaultwarden):** Tested local (email + master-password) login AND SSO login. Verified websocket sync via browser extension. Verified reboot survival.
|
||||
|
||||
---
|
||||
|
||||
### Phase 9 — Decommission (deferred) `[G]`
|
||||
|
||||
- Source stays **installed + stopped** as cold rollback during soak period
|
||||
- Before eventual deletion: `tar -czf <service>-source-backup-$(date +%Y%m%d).tar.gz /opt/<service>`
|
||||
- Update `.ref` docs (services.md, environment.md, ip-allocation.md)
|
||||
- Retire or banner any stale archive docs
|
||||
- Promote migration plan to this runbook (done)
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
| ID | Issue |
|
||||
|----|-------|
|
||||
| G1 | Stale docs may show sub-routes (e.g. `:3012`/`notifications/hub`) that don't exist in the live Caddyfile. Always read from the live host in Phase 0. |
|
||||
| G2 | edge2 `vmbr0` has **no DHCP** and no `local-lvm` → must use static IP + `local` (dir) storage. |
|
||||
| G3 | Source binds `127.0.0.1:<PORT>:<CONTAINER_PORT>`; LXC must bind `<SVC_TS_IP>:<PORT>:<CONTAINER_PORT>`. Keep the container-side port unchanged. |
|
||||
| G4 | SQLite in WAL mode has a live `.wal` file — never copy `db.sqlite3` + `-wal` naively. Use `sqlite3 .backup` (produces one self-contained file). Stop the source first for zero-loss. |
|
||||
| G5 | Preserve all existing Caddy directives on the backend block (e.g. `header_up X-Real-IP {remote_host}`) — edit only the upstream token. |
|
||||
| G6 | SSO redirect URIs derive from the unchanged `DOMAIN`/DNS — **do not touch Authentik**. Verify the round-trip lands logged-in; don't assert a specific redirect path. |
|
||||
| G7 | Caddy passes websocket upgrades by default; no WS-specific directive needed when WS is in-process on the same port. |
|
||||
| G8 | Reboot survival requires all three: `--onboot 1`, `restart: unless-stopped`, and the Tailscale DNS-bootstrap drop-in. |
|
||||
| G9 | JWT signing keys (e.g. `rsa_key.pem` for Vaultwarden) **must travel byte-for-byte**. If missing, existing sessions are invalidated (forced re-login everywhere) but data remains intact. Check startup logs for "generating new key" messages. |
|
||||
| G10 | Contabo Caddy runs with `admin off` → `caddy reload` and `systemctl reload caddy` FAIL. Always use `caddy validate` then `systemctl restart caddy`. Restart drops all Contabo-fronted sites for ~1–2 s. |
|
||||
| N | The composed **Contabo-Caddy → edge2-LXC tailnet** path is unexercised for each new service. Keep the Phase 6 HTTP `/alive` 200 gate as a HARD pre-cutover requirement (use `curl`, not ICMP). |
|
||||
|
||||
---
|
||||
|
||||
## Template Summary
|
||||
|
||||
**Generic phases (identical for every service):** 0 (recon) → 1 (LXC provision) → 2 (Docker) → 3 (Tailscale + DNS-bootstrap + reachability pre-gate) → 7 (one-token Caddy cutover + backup + validate + restart) → 8 (end-to-end + reboot survival) → 9 (deferred decommission).
|
||||
|
||||
**Service-specific phases:** 0a (pre-migration gate), 4 (compose/config — copy from live host), 5 (data migration method depends on storage type), 6 (health gates — service-specific checks before cutover).
|
||||
|
||||
**Access pins:**
|
||||
- Contabo: `ssh root@100.64.0.1`
|
||||
- edge2: `ssh admin@184.174.35.153` (alias `edge2`) + `sudo` for pct/pvesm/pveam
|
||||
- `root@100.64.0.26` is refused on edge2
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-06-17*
|
||||
Loading…
Add table
Add a link
Reference in a new issue