auto: docs sync 2026-06-21T12:00:06+00:00

Files changed: engine/changelog.md engine/lint-report.md vault/.obsidian/workspace.json vault/projects/fleet-patch-audit.md vault/projects/meshtastic-headscale-runbook.md

vault/projects/meshtastic-headscale-runbook.md

@@ -10,7 +10,7 @@ related:
   - [[meshtastic-sidecar-node]]
   - [[headscale-onboard-node]]
   - [[caddy]]
-updated: 2026-06-18
+updated: 2026-06-21
 ---
 # IdahoMesh Tailnet Runbook
 
@@ -572,4 +572,96 @@ ping 100.64.0.14  # cortex — should timeout/unreachable
 
 ---
 
-*Last updated: 2026-02-11*
+## Headscale 0.28 → 0.29 Upgrade Guide
+
+**Rehearsed on CT 106 (meshtastic-hs) 2026-06-21. Result: success. Use this as the playbook for CT 107 (mesh-bridge / Echo6 Headscale).**
+
+### Breaking changes that require action before starting
+
+#### 1. `randomize_client_port` removed from config.yaml (HARD BLOCKER)
+
+Headscale 0.29 **refuses to start** if `randomize_client_port` exists in `config.yaml`. Remove the line:
+
+```bash
+sed -i '/^randomize_client_port:/d' /etc/headscale/config.yaml
+```
+
+If the feature is needed, move it to the ACL policy file (`/etc/headscale/acl.json`) as a top-level key:
+```json
+{ "randomizeClientPort": true }
+```
+
+Default is `false`; simply removing the key is equivalent to `randomize_client_port: false`.
+
+#### 2. `ephemeral_node_inactivity_timeout` removed from top-level config (deprecation → removed)
+
+The old top-level key is gone in 0.29. Replace with the new nested path:
+
+```yaml
+# OLD (remove this):
+ephemeral_node_inactivity_timeout: 30m
+
+# NEW (replace with):
+node:
+  ephemeral:
+    inactivity_timeout: 30m
+```
+
+0.29 accepts the old key with a warning and uses the correct value, but the warning appears on every startup. Fix it to keep logs clean.
+
+#### 3. Strict version upgrade path enforced
+
+0.29 blocks skipping minor versions. If upgrading from 0.27 or earlier, you MUST go 0.27 → 0.28 → 0.29 one step at a time. We were on 0.28, so this is not a blocker for this fleet.
+
+#### 4. ACL wildcard `*` behavior changed (review your policy)
+
+In 0.29, bare `*` in ACL `src`/`dst` resolves to the CGNAT+ULA range (tailnet nodes only) instead of all IPs. If you had `"dst": ["*:*"]` meaning "any internet IP," you must switch to `autogroup:danger-all` as a source or explicit CIDRs. **The IdahoMesh ACL does not use bare `*` — no action needed for this tailnet.**
+
+#### 5. Minimum Tailscale client version raised to v1.80.0
+
+Any tailscale client older than v1.80.0 will be rejected by 0.29. Verify all registered nodes are running a sufficiently recent tailscale before upgrading.
+
+### Other notable 0.29 changes (no action required, FYI)
+
+- GivenName collision suffix changed from random hash (`laptop-abc12xyz`) to monotonic numeric (`laptop`, `laptop-1`). MagicDNS names for nodes with old random suffixes will change on upgrade. The raw Hostname column is unchanged.
+- `oidc.expiry` removed; use `node.expiry` for all registration methods.
+- `headscale nodes register` deprecated in favour of `headscale auth register --auth-id`.
+- `--namespace` flag removed from nodes commands (was already replaced by `--user`).
+- New: `trusted_proxies` config option for True-Client-IP headers (previously honoured from any client).
+- DB migration is automatic on first start — no manual migration command needed.
+
+### Rollback procedure (if upgrade fails)
+
+1. Stop the service: `systemctl stop headscale`
+2. Reinstall the 0.28.0 binary: `cp /usr/local/bin/headscale.bak-v0.28.0-pre029 /usr/local/bin/headscale`
+3. Restore config: `cp /etc/headscale/config.yaml.bak-pre029-20260621 /etc/headscale/config.yaml`
+4. Restore DB: `cp /root/headscale-db-pre029-20260621.sqlite /var/lib/headscale/db.sqlite`
+5. Start: `systemctl start headscale`
+
+> **Note:** 0.29 blocks downgrading once the DB migration has run. The DB backup from step 4 predates the migration, so this rollback is clean.
+
+### CT 106 (meshtastic-hs) upgrade record
+
+- **Date:** 2026-06-21
+- **From:** v0.28.0 → **To:** v0.29.1
+- **Deployment:** native systemd binary at `/usr/local/bin/headscale`
+- **Config changes made:**
+  - Removed `randomize_client_port: false`
+  - Replaced `ephemeral_node_inactivity_timeout: 30m` with `node.ephemeral.inactivity_timeout: 30m`
+- **DB migration:** automatic, clean, no errors in logs
+- **Nodes after upgrade:** all 3 registered (burley-butte offline, mesh-bridge online, mt-isr offline) — same as before
+- **Backups (pre-upgrade):** `/etc/headscale/config.yaml.bak-pre029-20260621`, `/root/headscale-db-pre029-20260621.sqlite`, `/usr/local/bin/headscale.bak-v0.28.0-pre029`
+
+### CT 107 (mesh-bridge / Echo6 Headscale) upgrade checklist
+
+CT 107 runs Docker (not native binary). Steps will differ:
+
+1. Back up: `docker cp headscale-vanilla:/var/lib/headscale/db.sqlite /root/headscale-db-pre029-$(date +%Y%m%d).sqlite` and `cp /path/to/config.yaml config.yaml.bak-pre029`
+2. Apply the two config changes above (remove `randomize_client_port`, migrate ephemeral timeout key)
+3. Change the image tag in `docker-compose.yml` from `headscale/headscale:0.28.x` to `headscale/headscale:0.29.1`
+4. `docker compose up -d`
+5. Validate: `docker exec headscale-vanilla headscale version`, check `docker logs headscale-vanilla` for migration success, `docker exec headscale-vanilla headscale nodes list`
+
+---
+
+*Last updated: 2026-06-21*