docs: migrate Authentik (SSO keystone) to edge2 CT 105

- Authentik -> edge2 CT 105 (Postgres pg_dump/restore; SECRET_KEY carried verbatim; zero-downtime until ~2s cutover)
- Multi-block Caddy cutover: auth.echo6.co + notes.echo6.co outpost/forward_auth -> 100.64.0.36:9000
- runbook: add reboot tailscale-before-docker gotcha; clarify dnsmasq must NOT be repointed (points at Caddy host)
- source left stopped + intact on Contabo as cold rollback

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matt Johnson 2026-06-18 05:49:07 +00:00
commit 44f0257376
140 changed files with 4013 additions and 24 deletions

View file

@ -0,0 +1,100 @@
# Headscale / Tailscale — Onboard a New Node
Standard procedure to bring any new system (Proxmox host, bare-metal, VM, or LXC) onto the Echo6 tailnet.
## Overview
- **Control server:** Headscale on Contabo — `ssh root@100.64.0.1`, Docker container `headscale`.
- **Login server:** `https://vpn.echo6.co` · **Admin UI (Headplane):** `https://vpn.echo6.co/admin`
- **User/namespace:** all Echo6 nodes join `echo6` (user ID 1).
- **MagicDNS base domain:** `echo6.mesh` (e.g. `ping data.echo6.mesh`).
- **Naming:** lowercase, hyphens OK, no dots/underscores. Matches `--hostname` and the cortex SSH alias.
## 1. Generate a preauth key (on Contabo)
```bash
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user echo6 --reusable --expiration 1h'
# If the name form errors, use the user ID: --user 1
```
## 2. Install Tailscale on the node
```bash
curl -fsSL https://tailscale.com/install.sh | sh
systemctl enable --now tailscaled
```
## 3. DNS-bootstrap drop-in (REQUIRED on LXCs, recommended everywhere)
Without this, tailscaled can't resolve `vpn.echo6.co` on a cold boot. Makes onboarding survive a reboot.
```bash
mkdir -p /etc/systemd/system/tailscaled.service.d
cat > /etc/systemd/system/tailscaled.service.d/dns-bootstrap.conf <<'EOF'
[Service]
ExecStartPre=/bin/sh -c "grep -q nameserver /etc/resolv.conf || echo nameserver 1.1.1.1 > /etc/resolv.conf"
EOF
systemctl daemon-reload
```
## 4. LXC containers only — TUN device prerequisite
On the Proxmox host, before starting the container (needs `nesting=1,keyctl=1` features set):
```bash
pct stop <CTID>
cat >> /etc/pve/lxc/<CTID>.conf <<'EOF'
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file
EOF
pct start <CTID>
```
(Privileged variant: create with `--unprivileged 0`, use `lxc.mount.entry: /dev/net dev/net none bind,create=dir`.)
## 5. Register with Headscale
```bash
tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=<NAME>
```
- Standard nodes pass **no** `--advertise-routes` / `--accept-routes` / `--advertise-tags`.
- Add `--ssh` to enable Tailscale SSH.
- **Always include `--login-server=https://vpn.echo6.co`** — without it the node registers against Tailscale's public servers instead of Echo6 Headscale.
- Subnet-router exception: the mesh-bridge advertises `100.100.0.0/16 --accept-routes`, then enable on Headscale (`headscale routes list` / `routes enable -r <ID>`).
## 6. Verify
```bash
tailscale status
tailscale ip -4
ssh root@100.64.0.1 'docker exec headscale headscale nodes list' # confirm node + assigned 100.64.0.x
```
Headscale assigns the IP **sequentially at registration** — don't pre-assign; read it back here.
## 7. SSH config aliases on cortex (`~/.ssh/config`)
```
Host <alias>
HostName <local-or-public-IP>
User root
Host ts-<alias>
HostName <tailscale-IP>
User root
```
Most nodes use `User root`. Per-node exceptions exist — e.g. **edge2** uses `User admin` + `IdentityFile ~/.ssh/contabo2_ed25519` (cloud image, no root login).
## 8. Update inventory
- `docs/hardware/environment.md` — Key Servers table **and** Headscale Node List (bump count + "updated" line).
- `docs/hardware/ip-allocation.md` — Tailscale IPs section.
- `CLAUDE.md` cluster cheat-sheet — Proxmox hosts only.
- `docs/services/services.md` — once services are deployed on the node.
## Worked example — edge2 (2026-06-16)
Proxmox host on Contabo. Preauth key → `tailscale up --login-server=https://vpn.echo6.co --auth-key=<KEY> --hostname=edge2` → assigned **100.64.0.26** (Headscale node 42). SSH aliases `edge2` / `ts-edge2` use `User admin` + `~/.ssh/contabo2_ed25519`.
## Gotchas
- `environment.md`'s Headscale Node List is the authoritative IP source — `ip-allocation.md`'s table has gone stale before.
- A node deleted from Headscale does not get its old IP auto-reused; confirm the real next IP with `headscale nodes list`.
- The old `ct-runbook` `tailscale up --authkey ... --ssh` form omits `--login-server` — don't copy it blindly.
## Checklist
```
[ ] Preauth key generated on Contabo (user echo6)
[ ] Tailscale installed + tailscaled enabled
[ ] DNS-bootstrap drop-in in place (LXC: required)
[ ] (LXC) TUN device added to CT config
[ ] tailscale up with --login-server=https://vpn.echo6.co --hostname=<name>
[ ] Node appears in headscale nodes list; IP captured
[ ] SSH aliases (<alias> + ts-<alias>) added on cortex and tested
[ ] Inventory updated (environment.md, ip-allocation.md, CLAUDE.md)
```