echo6-docs/vault/runbooks/fleet-magicdns-resolved-migration.md

title	type	tags	related	updated
Fleet MagicDNS / systemd-resolved Migration	runbook	dns vpn		2026-06-22

Fleet MagicDNS / systemd-resolved Migration

Context / why

Fleet guests on Tailscale had a fragile DNS setup. On LXC CTs without systemd-resolved, Tailscale owns /etc/resolv.conf and points all queries (public included) at the MagicDNS proxy 100.100.100.100. If tailscaled loses its link, all DNS dies — including the lookup of the coordinator vpn.echo6.co needed to reconnect. That circular dependency is a hard brick.

This triggered during the June 2026 patch campaign: a CT went completely offline when tailscaled dropped and couldn't resolve its way back.

MagicDNS itself already works (Headscale base_domain: echo6.mesh, upstream 1.1.1.1) — the problem was fragility and lack of public-DNS fallback, not the feature itself.

Goal of this migration: bring every Tailscale guest to the robust split-DNS state the two VMs (recon-vm, arr) already had:

• systemd-resolved owns /etc/resolv.conf in stub mode
• Tailscale registers as a per-domain resolver for echo6.mesh
• A resolved global upstream (1.1.1.1/8.8.8.8) guarantees public DNS even when Tailscale is fully down
• Every guest uniform and documented

---

Gold-standard target state

Verified on recon-vm (VM 1130). After migration, each guest must satisfy all of these:

Check	Expected value
systemctl is-active systemd-resolved	active
systemctl is-enabled systemd-resolved	enabled
/etc/resolv.conf	symlink → ../run/systemd/resolve/stub-resolv.conf
resolvectl Global resolv.conf mode	stub
/etc/systemd/resolved.conf.d/upstream.conf	DNS=1.1.1.1 8.8.8.8
journalctl -u tailscaled | grep "dns: using"	dns: using "systemd-resolved" mode
journalctl -u tailscaled | grep "rc="	rc=resolved ... ret=systemd-resolved
resolvectl status tailscale0 DNS	100.100.100.100
resolvectl status tailscale0 Domains	echo6.mesh ~. (accept-dns/CorpDNS on)
Docker CTs: /etc/docker/daemon.json	"dns": ["1.1.1.1","8.8.8.8"]

Net result: *.echo6.mesh resolves via MagicDNS; public DNS is forwarded through Tailscale when it is up, and falls back to resolved's 1.1.1.1/8.8.8.8 when it is down.

---

Per-guest pre-flight checks

Run these before starting. They determine which steps apply.

1. PVE nameserver entry

# On the PVE host:
grep -i nameserver /etc/pve/lxc/<ID>.conf

If a nameserver: line is present, clear it before anything else:

sed -i "/^nameserver:/d" /etc/pve/lxc/<ID>.conf

Note: pct set <ID> --delete nameserver silently no-ops on PVE 9.x. Use the sed directly on the conf file.

2. Docker

pct exec <ID> -- bash -lc "which dockerd && cat /etc/docker/daemon.json 2>/dev/null || echo MISSING"

If dockerd is present and /etc/docker/daemon.json has no dns pin → apply Step 0 first.

3. NordVPN

pct exec <ID> -- bash -lc "which nordvpn && nordvpn settings 2>/dev/null || echo NOT_PRESENT"

If nordvpn is present, its kill-switch can block tailscaled from reaching the coordinator after a restart. Allowlist edge2 first:

pct exec <ID> -- bash -lc "
nordvpn allowlist add subnet 184.174.35.153/32
nordvpn allowlist add port 443
"

4. Bucket assignment

pct exec <ID> -- bash -lc "systemctl is-active systemd-resolved"

• active → Bucket R (already installed; reconfigure only — skip Step 1)
• inactive / not-found → Bucket I (needs install — run Step 1)

---

The recipe

Commands that run inside the CT are wrapped in pct exec <ID> -- bash -lc '...'; commands on the PVE host are noted explicitly.

Step 0 — Docker DNS pin (Docker CTs only)

If /etc/docker/daemon.json does not exist:

pct exec <ID> -- bash -lc "
cat > /etc/docker/daemon.json <<'EOF'
{\"dns\": [\"1.1.1.1\", \"8.8.8.8\"]}
EOF
systemctl restart docker
"

If it already exists with other keys, merge the dns field manually — do not overwrite blindly.

After systemctl restart docker, running containers will briefly restart. Expected and safe.

Note on container DNS: containers on bridge networks will show 127.0.0.11 in their own /etc/resolv.conf. That is Docker's embedded relay which forwards to the pinned upstream — expected behavior, not a stub.

---

Step 1 — Install systemd-resolved (Bucket I only)

pct exec <ID> -- bash -lc "
apt-get update &&
DEBIAN_FRONTEND=noninteractive apt-get install -y systemd-resolved &&
systemctl enable --now systemd-resolved
"

This pulls 3 new packages plus a routine systemd point-release bump. It does not disrupt DNS — Tailscale still owns /etc/resolv.conf at this point.

---

Step 2 — Resolved upstream drop-in

pct exec <ID> -- bash -lc "
mkdir -p /etc/systemd/resolved.conf.d
cat > /etc/systemd/resolved.conf.d/upstream.conf <<'EOF'
[Resolve]
DNS=1.1.1.1 8.8.8.8
EOF
systemctl restart systemd-resolved
"

---

Step 3 — Guard #1: tailscaled ordering

This guard stops tailscaled racing systemd-resolved's D-Bus at boot. Without it, tailscaled starts before resolved is ready and falls back to "direct" mode — breaking the split-DNS config.

pct exec <ID> -- bash -lc "
mkdir -p /etc/systemd/system/tailscaled.service.d
cat > /etc/systemd/system/tailscaled.service.d/resolved-ordering.conf <<'EOF'
[Unit]
Wants=systemd-resolved.service
After=systemd-resolved.service

[Service]
ExecStartPre=/bin/bash -c \"c=0; while ! busctl status org.freedesktop.resolve1 >/dev/null 2>&1; do c=\$((c+1)); [ \$c -ge 30 ] && break; sleep 0.5; done; true\"
EOF
systemctl daemon-reload
"

---

Step 4 — Guard #2: PVE resolv.conf overwrite protection

PVE rewrites /etc/resolv.conf from the host at pct start — and may mark it immutable with chattr +i. This service restores the stub symlink before any container services run.

pct exec <ID> -- bash -lc "
cat > /etc/systemd/system/fix-resolv-stub.service <<'EOF'
[Unit]
Description=Restore /etc/resolv.conf stub symlink (overwritten/immutable-flagged at container start)
DefaultDependencies=no
Before=systemd-resolved.service network-pre.target
After=local-fs-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c \"mkdir -p /run/systemd/resolve && if [ ! -L /etc/resolv.conf ]; then chattr -i /etc/resolv.conf 2>/dev/null; rm -f /etc/resolv.conf && ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf; fi\"

[Install]
WantedBy=sysinit.target
EOF
systemctl enable fix-resolv-stub.service
"

Why both guards?

• Guard #1 (Step 3): tailscaled races resolved's D-Bus on boot → would silently pick "direct" mode. The ExecStartPre probe waits for D-Bus to be ready.
• Guard #2 (Step 4): PVE restores its own static resolv.conf (possibly chattr +i) at every pct start, before systemd-resolved runs. Without this, the stub symlink is always gone on boot. Both were proven required on a clean guest.

---

Step 5 — Clear PVE nameserver (host, if still present)

# On the PVE host:
sed -i "/^nameserver:/d" /etc/pve/lxc/<ID>.conf

Safe to re-run if already done in pre-flight.

---

Step 6 — Live stub swap

Stop tailscaled first — while it is running it instantly rewrites /etc/resolv.conf the moment the symlink is removed.

pct exec <ID> -- bash -lc "
systemctl stop tailscaled
chattr -i /etc/resolv.conf 2>/dev/null
rm -f /etc/resolv.conf
ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
systemctl restart systemd-resolved
"

---

Step 7 — Restart tailscaled and verify mode

pct exec <ID> -- bash -lc "systemctl restart tailscaled"

Wait ~8 seconds, then check:

pct exec <ID> -- bash -lc "
journalctl -b -u tailscaled | grep -E 'rc=|dns: using'
"

Expected output includes both:

• dns: using "systemd-resolved" mode
• rc=resolved ... ret=systemd-resolved

Known pattern: if resolvectl status tailscale0 shows no DNS domain scopes after the first restart, restart tailscaled once more. The second restart reliably pushes the echo6.mesh ~. scope config. This was observed on all 3 baseline guests.

pct exec <ID> -- bash -lc "systemctl restart tailscaled && sleep 8 && resolvectl status tailscale0"

---

Step 8 — Enable accept-dns

pct exec <ID> -- bash -lc "
tailscale set --accept-dns=true
"

Verify:

pct exec <ID> -- bash -lc "
resolvectl status tailscale0
"

Expected: tailscale0 shows DNS Servers: 100.100.100.100 and Domains: echo6.mesh ~.. This confirms accept-dns/CorpDNS is active. (Note: tailscale status --json | jq .Self.CorpDNS returns null in this Tailscale/Headscale build — use resolvectl status tailscale0 as the authoritative check. Optionally: tailscale debug prefs | grep -i corp.)

---

Validation

Run after every guest. All checks must pass before moving on.

1. Resolver state

pct exec <ID> -- bash -lc "
resolvectl status
"

Confirm:

• Global resolv.conf mode: stub
• tailscale0 DNS: 100.100.100.100
• tailscale0 Domains: echo6.mesh ~.

2. Resolution checks

pct exec <ID> -- bash -lc "
getent hosts meshmonitor.echo6.mesh
getent hosts github.com
getent hosts mirror.gcr.io
"

• meshmonitor.echo6.mesh → a 100.64.x.x tailnet IP (MagicDNS working)
• github.com and mirror.gcr.io → public IPs (public DNS working)

3. Kill-test — the point of this whole migration

# Bring Tailscale down
pct exec <ID> -- bash -lc "tailscale down"

# Public DNS must still resolve (resolved fallback)
pct exec <ID> -- bash -lc "getent hosts github.com"

# Coordinator must still resolve — no circular brick
pct exec <ID> -- bash -lc "getent hosts vpn.echo6.co"

# Bring Tailscale back
pct exec <ID> -- bash -lc "tailscale up"

# After ~8s, MagicDNS must work again
sleep 8
pct exec <ID> -- bash -lc "getent hosts meshmonitor.echo6.mesh"

All four must succeed.

4. App health check

Verify the guest's primary application is healthy (method is per-guest).

5. Docker CTs

pct exec <ID> -- bash -lc "
docker exec <container> getent hosts github.com
"

Must resolve. Proves the DNS pin insulates containers from any Tailscale-related resolver changes.

6. Reboot persistence (mandatory)

# From PVE host:
pct reboot <ID>

After ~85 seconds:

pct exec <ID> -- bash -lc "
ls -la /etc/resolv.conf
resolvectl status | head -20
journalctl -b -u tailscaled | grep -E 'rc=|dns: using'
getent hosts meshmonitor.echo6.mesh
getent hosts github.com
"

All must pass. This is the check the two boot-ordering guards exist to satisfy: stub symlink intact, mode: stub, tailscaled in systemd-resolved mode, both DNS paths working.

---

Rollback

Per guest — returns to the known-good static state (public DNS works, no MagicDNS). No reboot required.

On the PVE host:

pct set <ID> --nameserver "1.1.1.1 8.8.8.8"

Inside the CT:

pct exec <ID> -- bash -lc "
chattr -i /etc/resolv.conf 2>/dev/null
rm -f /etc/resolv.conf
printf 'nameserver 1.1.1.1\nnameserver 8.8.8.8\n' > /etc/resolv.conf
tailscale set --accept-dns=false
systemctl restart tailscaled
"

To also disable the guards (optional):

pct exec <ID> -- bash -lc "
systemctl disable fix-resolv-stub.service
rm -f /etc/systemd/system/tailscaled.service.d/resolved-ordering.conf
systemctl daemon-reload
"

---

How to use MagicDNS

Any tailnet node resolves at <hostname>.echo6.mesh → its 100.64.0.x tailnet IP.

Examples:

• meshmonitor.echo6.mesh
• recon-vm.echo6.mesh
• cortex.echo6.mesh

List node names:

# On edge2 (CT 107, mesh-bridge):
headscale nodes list

Base domain echo6.mesh is set in Headscale config. Use these names instead of hardcoding tailnet IPs in service configs and cron jobs.

---

Fleet rollout status

Migration complete 2026-06-22 — all 19 Tailscale guests + 2 VMs on systemd-resolved split-DNS.

Across all 19 guests the recipe held with no failures. The "two tailscaled restarts" pattern (Step 7) was sometimes but not always needed — Bucket I install-path guests often received the echo6.mesh ~. domain scope on the first restart. Both boot guards (resolved-ordering.conf + fix-resolv-stub.service) proved necessary and sufficient for reboot persistence across every guest tested.

Completed — 2026-06-22

CT	Host	Bucket	Docker	Notes
CT108 meshai	utility	R	yes	—
CT102 searxng	utility	I	yes	—
CT112 cobalt	utility	R	yes	No containers deployed; daemon.json DNS pin applied
CT100 meshmonitor	utility	I	yes	meshai (CT108) stayed healthy throughout; MagicDNS canary validated via media.echo6.mesh
CT104 central	utility	I	no	—
CT101 caddy	utility	I	no	—
CT107 mesh-bridge	utility	I	no	Dual-tailnet bridge (echo6 tailscale0 + IdahoMesh tailscale1); both daemons migrated
CT110 peertube	media	I	no	NordVPN allowlist applied
CT111 mcc	media	I	no	Caddy on :80; healthy before+after
CT103 argus	cloud	R	yes	daemon.json merged (had runtime keys, no dns pin); argus-app stack (grafana/postgres) healthy; pre-existing: argus-app compose lacks restart:always — needs manual docker compose up -d after reboot (not a DNS issue)
CT120 immich	cloud	R	yes	daemon.json pre-pinned; machine_learning unhealthy pre-existing (self-cleared after reboot)
CT121 nextcloud	cloud	R	yes	daemon.json created (missing); 12-container AIO stack (apache/app/db/redis/collabora/…) all healthy before+after; CorpDNS null (expected for this build) — validated via resolvectl tailscale0 echo6.mesh ~.; reboot-persistent
CT101 wordpress	edge2	I	yes	—
CT100 pdm	edge2	I	no	pdm-enterprise apt repo 401s without subscription — disable during install, restore after
CT102 vaultwarden	edge2	I	no	—
CT103 forgejo	edge2	I	no	—
CT104 livesync	edge2	I	no	—
CT105 authentik	edge2	I	yes	—
CT106 matrix	edge2	I	no	—

Already compliant (gold-standard VMs)

Guest	Notes
recon-vm (VM 1130)	Reference state; validated
arr (VM 105)	Compliant

Pending

None — fleet migration complete.