auto: docs sync 2026-07-13T12:00:23+00:00

Files changed: engine/.embcache.json engine/changelog.md engine/lint-report.md vault/.trash/2026-06-19.md vault/docs/hardware/environment.md vault/docs/hardware/ip-allocation.md vault/docs/matrix/archivist.md vault/docs/matrix/matrix_host.md vault/docs/matrix/mautrix_signal.md vault/docs/matrix/synapse.md vault/docs/matrix/synapse_retention_discovery.md vault/docs/navi/cc-rules.md vault/docs/navi/deployment.md vault/docs/navi/themes.md vault/docs/services/ots-setup.md vault/docs/services/services.md vault/docs/services/usenet.md vault/docs/software/authentik.md vault/docs/software/caddy.md vault/docs/software/central.md vault/docs/software/dns.md vault/docs/software/geo-tools.md vault/docs/software/navi.md vault/docs/software/recon.md vault/docs/software/searxng.md vault/glossary.md vault/notes/echo6-landing-page-data-export.md vault/notes/ia-download-queue.md vault/projects/advbbs-project.md vault/projects/argus.md vault/projects/deploy-livesync.md vault/projects/fleet-patch-audit.md vault/projects/fleet-platform-baseline.md vault/projects/matrix-synapse-deployment.md vault/projects/meshai-config-hot-apply.md vault/projects/meshai-region-routing-plan.md vault/projects/meshai.md vault/projects/meshcore-transport.md vault/projects/meshtastic-headscale-runbook.md vault/projects/mmud-project.md vault/projects/nominatim-v5-reimport.md vault/runbooks/add-peertube-channel.md vault/runbooks/authentik-access-groups.md vault/runbooks/authentik-create-invitation.md vault/runbooks/authentik-oidc-application.md vault/runbooks/authentik-upgrade.md vault/runbooks/central-deploy-cutover.md vault/runbooks/ct-runbook.md vault/runbooks/edge2-access-reference.md vault/runbooks/expose-service-contabo.md vault/runbooks/expose-service-edge2.md vault/runbooks/expose-service-home.md vault/runbooks/fleet-magicdns-resolved-migration.md vault/runbooks/headless-browser-page-verification.md vault/runbooks/headscale-oidc-boot-order.md vault/runbooks/headscale-onboard-node.md vault/runbooks/ia-cli-reference.md vault/runbooks/ia-download-mirror.md vault/runbooks/idahomesh-bridge-setup.md vault/runbooks/idahomesh-vpn-device-setup.md vault/runbooks/lxc-service-migration.md vault/runbooks/mailcow-create-mailbox.md vault/runbooks/meshai-prod-compose-override.md vault/runbooks/meshmonitor-password-reset.md vault/runbooks/meshtastic-sidecar-node.md vault/runbooks/meshtasticd-sim-nodes-runbook.md vault/runbooks/nordvpn-lxc.md vault/runbooks/peertube-remote-runner.md vault/runbooks/pg-backup.md vault/runbooks/pi-nas-omv-runbook.md vault/runbooks/pipeline-patterns.md vault/runbooks/proxmox-create-ubuntu-vm.md vault/runbooks/proxmox-onboard-node.md vault/runbooks/pymc-repeater-kiss-tnc-reenumeration.md vault/runbooks/recon-operations.md vault/runbooks/recon-service-integration.md vault/runbooks/syncthing-add-node.md vault/runbooks/toc-cortex-pve9.2-update.md vault/session-resume/SESSION-HANDOFF-meshai-test.md
This commit is contained in:
echo6-autocommit 2026-07-13 12:00:23 +00:00
commit ef8b1e0bd9
79 changed files with 448 additions and 360 deletions

View file

@ -10,7 +10,7 @@ related:
- [[recon-service-integration]]
- [[proxmox-onboard-node]]
- [[ct-runbook]]
updated: 2026-07-11
updated: 2026-07-13
---
# Add PeerTube Channel

View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[authentik-oidc-application]]
- [[authentik]]
- [[authentik-create-invitation]]
- [[authentik]]
- [[authentik-upgrade]]
- [[deploy-livesync]]
- [[proxmox-onboard-node]]
updated: 2026-07-11
updated: 2026-07-13
---
# Authentik Access Groups
@ -24,7 +24,7 @@ Manage group-based application access via the [[authentik]] API. No web UI inter
## How It Works
By default, any authenticated Authentik user can access any application. Adding a **policy binding** that ties a **group** to an **application** restricts that app to group members only (plus superusers).
By default, any authenticated [[authentik]] user can access any application. Adding a **policy binding** that ties a **group** to an **application** restricts that app to group members only (plus superusers).
- One binding per group-application pair
- An app can have multiple group bindings (policy_engine_mode=`any` means membership in ANY bound group grants access)

View file

@ -5,12 +5,12 @@ tags:
- auth
aliases: []
related:
- [[authentik-oidc-application]]
- [[authentik-access-groups]]
- [[authentik]]
- [[authentik-oidc-application]]
- [[authentik-upgrade]]
- [[mailcow-create-mailbox]]
updated: 2026-07-11
updated: 2026-07-13
---
# Authentik: Create Invitation
@ -26,7 +26,7 @@ Any time a new user needs to be invited to Echo6 services. Invitations create a
## Prerequisites
- Authentik admin access at https://auth.echo6.co
- [[authentik]] admin access at https://auth.echo6.co
- For email mode: SMTP must be configured and working (no-reply@echo6.co via Mailcow)
---

View file

@ -8,9 +8,9 @@ related:
- [[authentik]]
- [[authentik-access-groups]]
- [[authentik-upgrade]]
- [[mailcow-create-mailbox]]
- [[expose-service-home]]
updated: 2026-07-11
- [[headscale-oidc-boot-order]]
- [[authentik-create-invitation]]
updated: 2026-07-13
---
# Add Authentik OIDC to an Application
@ -18,7 +18,7 @@ Fully automated via [[authentik]] API. No web UI interaction required.
**Prerequisite:** [[dns]] must already exist for the service (run expose-service-edge2.md or expose-service-home.md first).
**Authentik instance:** https://auth.echo6.co (edge2 CT 105, 100.64.0.36)
**[[authentik]] instance:** https://auth.echo6.co (edge2 CT 105, 100.64.0.36)
---

View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[authentik-oidc-application]]
- [[lxc-service-migration]]
- [[authentik]]
- [[headscale-oidc-boot-order]]
- [[lxc-service-migration]]
- [[authentik-create-invitation]]
- [[ct-runbook]]
updated: 2026-07-11
updated: 2026-07-13
---
# Authentik: Major Version Upgrade
@ -22,7 +22,7 @@ Upgrade [[authentik]] between major versions on edge2 CT 105. Covers backup, upg
## When to Use This
Any time Authentik is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.
Any time [[authentik]] is upgraded across major versions (e.g., 2024.12 → 2025.6 → 2025.12). Minor patch upgrades within the same major (e.g., 2025.12.3 → 2025.12.4) are lower risk but should still follow the backup steps.
---

View file

@ -1,9 +1,16 @@
---
title: "central — Deploy & Cutover Runbook"
type: runbook
tags: [recon]
related: ["[[central]]"]
updated: 2026-06-28
tags:
- mesh
aliases: []
related:
- [[central]]
- [[recon-operations]]
- [[lxc-service-migration]]
- [[deployment]]
- [[syncthing-add-node]]
updated: 2026-07-13
---
# central — Deploy & Cutover Runbook
@ -34,7 +41,7 @@ Use this runbook whenever you need to deploy a new release or roll back.
## Pre-flight (ALWAYS run before any deploy)
**1. Confirm services are healthy now:**
**1. Confirm [[services]] are healthy now:**
```bash
systemctl is-active central-supervisor central-archive central-gui

View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
- [[meshtasticd-sim-nodes-runbook]]
- [[proxmox-create-ubuntu-vm]]
- [[ots-setup]]
updated: 2026-06-18
- [[headscale-onboard-node]]
- [[nordvpn-lxc]]
- [[meshtasticd-sim-nodes-runbook]]
updated: 2026-07-13
---
# Proxmox CT/LXC Provisioning Runbook

View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[expose-service-edge2]]
- [[proxmox-onboard-node]]
- [[vaultwarden-plan]]
- [[lxc-service-migration]]
- [[authentik]]
- [[headscale-onboard-node]]
updated: 2026-06-18
updated: 2026-07-13
---
# edge2 Access Reference
@ -113,4 +113,4 @@ ssh edge2 'sudo pct stop <CTID>'
**Root cause:** PVE `root@pam` password and the system root SSH password are managed separately. On edge2, the system root password was set by cloud-init at provisioning and may differ. Additionally, root SSH login is disabled entirely.
**Prevention:** Document both auth paths (SSH user + PVE API) separately in credentials and environment docs.
**Prevention:** Document both auth paths (SSH user + PVE API) separately in credentials and [[environment]] docs.

View file

@ -8,13 +8,13 @@ related:
- [[expose-service-edge2]]
- [[expose-service-home]]
- [[lxc-service-migration]]
- [[headscale-onboard-node]]
- [[caddy]]
updated: 2026-07-11
- [[services]]
updated: 2026-07-13
---
# Expose Service on Contabo
> SUPERSEDED — Contabo was decommissioned 2026-06-19. Use [[expose-service-edge2]] (services) or [[expose-service-contabo]]→edge1 for mail. This doc is kept for history only.
> SUPERSEDED — Contabo was decommissioned 2026-06-19. Use [[expose-service-edge2]] ([[services]]) or [[expose-service-contabo]]→edge1 for mail. This doc is kept for history only.
## Prerequisites
- Service running in Docker on Contabo

View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[lxc-service-migration]]
- [[edge2-access-reference]]
- [[expose-service-contabo]]
- [[expose-service-home]]
- [[vaultwarden-plan]]
updated: 2026-06-18
- [[edge2-access-reference]]
- [[caddy]]
updated: 2026-07-13
---
# Expose Service on edge2 (Contabo Cloud VPS)
@ -67,7 +67,7 @@ sudo pct push <CTID> /path/on/host /path/in/ct
### 3. Add Caddy site block on edge2 host
Caddy runs on the edge2 host and terminates TLS.
[[caddy]] runs on the edge2 host and terminates TLS.
**For Cloudflare-proxied domains** (orange cloud / Full SSL mode):
```bash
@ -153,7 +153,7 @@ curl -I https://<domain>/
| edge2 Tailscale | 100.64.0.26 |
| Internal bridge | vmbr0, 10.10.10.0/24 |
| Gateway | 10.10.10.1 (edge2 host) |
| DNS in CTs | 1.1.1.1 |
| [[dns]] in CTs | 1.1.1.1 |
| CT IP range | 10.10.10.10+ (10=pdm, 11=wordpress) |
## CT Creation via PVE API (alternative)

View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[expose-service-edge2]]
- [[expose-service-contabo]]
- [[caddy]]
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
- [[ct-runbook]]
updated: 2026-06-18
updated: 2026-07-13
---
# Expose Service on Home Network
@ -25,8 +25,8 @@ updated: 2026-06-18
| Has OIDC? | Proxy to | Why |
|-----------|----------|-----|
| YES | Local IP (192.168.1.x:port) | Authentik SSO protects access |
| NO | Tailscale IP (100.64.0.x:port) | Only Caddy can reach backend |
| YES | Local IP (192.168.1.x:port) | [[authentik]] SSO protects access |
| NO | Tailscale IP (100.64.0.x:port) | Only [[caddy]] can reach backend |
If no OIDC, service MUST have Tailscale installed and registered with Headscale first.

View file

@ -1,16 +1,23 @@
---
title: Fleet MagicDNS / systemd-resolved Migration
type: runbook
tags: [dns, vpn]
related: []
updated: 2026-06-22
tags:
- dns
aliases: []
related:
- [[headscale-onboard-node]]
- [[caddy]]
- [[meshtastic-headscale-runbook]]
- [[headscale-oidc-boot-order]]
- [[lxc-service-migration]]
updated: 2026-07-13
---
# Fleet MagicDNS / systemd-resolved Migration
## Context / why
Fleet guests on Tailscale had a fragile DNS setup. On LXC CTs without systemd-resolved, Tailscale owns `/etc/resolv.conf` and points **all** queries (public included) at the MagicDNS proxy `100.100.100.100`. If tailscaled loses its link, all DNS dies — including the lookup of the coordinator `vpn.echo6.co` needed to reconnect. That circular dependency is a hard brick.
Fleet guests on Tailscale had a fragile [[dns]] setup. On LXC CTs without systemd-resolved, Tailscale owns `/etc/resolv.conf` and points **all** queries (public included) at the MagicDNS proxy `100.100.100.100`. If tailscaled loses its link, all DNS dies — including the lookup of the coordinator `vpn.echo6.co` needed to reconnect. That circular dependency is a hard brick.
This triggered during the June 2026 patch campaign: a CT went completely offline when tailscaled dropped and couldn't resolve its way back.
@ -176,7 +183,7 @@ systemctl daemon-reload
### Step 4 — Guard #2: PVE resolv.conf overwrite protection
PVE rewrites `/etc/resolv.conf` from the host at `pct start` — and may mark it immutable with `chattr +i`. This service restores the stub symlink before any container services run.
PVE rewrites `/etc/resolv.conf` from the host at `pct start` — and may mark it immutable with `chattr +i`. This service restores the stub symlink before any container [[services]] run.
```bash
pct exec <ID> -- bash -lc "
@ -431,16 +438,16 @@ Across all 19 guests the recipe held with no failures. The "two tailscaled resta
| CT | Host | Bucket | Docker | Notes |
|---|---|---|---|---|
| CT108 meshai | utility | R | yes | — |
| CT102 searxng | utility | I | yes | — |
| CT108 [[meshai]] | utility | R | yes | — |
| CT102 [[searxng]] | utility | I | yes | — |
| CT112 cobalt | utility | R | yes | No containers deployed; daemon.json DNS pin applied |
| CT100 meshmonitor | utility | I | yes | meshai (CT108) stayed healthy throughout; MagicDNS canary validated via media.echo6.mesh |
| CT104 central | utility | I | no | — |
| CT101 caddy | utility | I | no | — |
| CT104 [[central]] | utility | I | no | — |
| CT101 [[caddy]] | utility | I | no | — |
| CT107 mesh-bridge | utility | I | no | Dual-tailnet bridge (echo6 tailscale0 + IdahoMesh tailscale1); both daemons migrated |
| CT110 peertube | media | I | no | NordVPN allowlist applied |
| CT111 mcc | media | I | no | Caddy on :80; healthy before+after |
| CT103 argus | cloud | R | yes | daemon.json merged (had runtime keys, no dns pin); argus-app stack (grafana/postgres) healthy; pre-existing: argus-app compose lacks restart:always — needs manual `docker compose up -d` after reboot (not a DNS issue) |
| CT103 [[argus]] | cloud | R | yes | daemon.json merged (had runtime keys, no dns pin); argus-app stack (grafana/postgres) healthy; pre-existing: argus-app compose lacks restart:always — needs manual `docker compose up -d` after reboot (not a DNS issue) |
| CT120 immich | cloud | R | yes | daemon.json pre-pinned; machine_learning unhealthy pre-existing (self-cleared after reboot) |
| CT121 nextcloud | cloud | R | yes | daemon.json created (missing); 12-container AIO stack (apache/app/db/redis/collabora/…) all healthy before+after; CorpDNS null (expected for this build) — validated via resolvectl tailscale0 echo6.mesh ~.; reboot-persistent |
| CT101 wordpress | edge2 | I | yes | — |
@ -448,7 +455,7 @@ Across all 19 guests the recipe held with no failures. The "two tailscaled resta
| CT102 vaultwarden | edge2 | I | no | — |
| CT103 forgejo | edge2 | I | no | — |
| CT104 livesync | edge2 | I | no | — |
| CT105 authentik | edge2 | I | yes | — |
| CT105 [[authentik]] | edge2 | I | yes | — |
| CT106 matrix | edge2 | I | no | — |
### Already compliant (gold-standard VMs)

View file

@ -1,16 +1,23 @@
---
title: "Headless Browser — Visual Page Verification"
title: Headless Browser — Visual Page Verification
type: runbook
tags: [tooling]
related: ["[[central]]", "[[central-deploy-cutover]]"]
updated: 2026-06-29
tags:
- tooling
aliases: []
related:
- [[peertube-remote-runner]]
- [[central-deploy-cutover]]
- [[toc-cortex-pve9.2-update]]
- [[syncthing-add-node]]
- [[headscale-oidc-boot-order]]
updated: 2026-07-13
---
# Headless Browser — Visual Page Verification
Drive a real headless browser (Playwright + Chromium) from **cortex** to log in and **screenshot a deployed web page**, then view the screenshot. Use this to *actually look* at a page after a deploy — it catches render/layout/styling bugs that `curl` status codes and unit tests sail right past (a route can return `200`/`302` and still look broken).
Works for **any** web UI reachable from cortex (central, navi, Authentik, PeerTube, Mailcow, etc.) — just change the base URL, path, and credentials.
Works for **any** web UI reachable from cortex ([[central]], [[navi]], [[authentik]], PeerTube, Mailcow, etc.) — just change the base URL, path, and credentials.
> Reusing this in a prompt: tell Claude *"follow the headless-browser-page-verification runbook to screenshot `<service> <path>`"* and point it at the creds. Everything needed is self-contained below.
@ -18,7 +25,7 @@ Works for **any** web UI reachable from cortex (central, navi, Authentik, PeerTu
## Prerequisites
- Runs on **cortex** (where Claude Code executes). The target must be reachable from cortex — most fleet UIs are on the mesh (e.g. `http://100.64.0.12:8000`) or via a Caddy host.
- Runs on **cortex** (where Claude Code executes). The target must be reachable from cortex — most fleet UIs are on the mesh (e.g. `http://100.64.0.12:8000`) or via a [[caddy]] host.
- `python3` available.
- Credentials for auth-gated pages come from **`.ref/credentials`** (e.g. `CENTRAL_OPERATOR_USER`/`CENTRAL_OPERATOR_PASS`). Public/auth-exempt paths (`/login`, `/health`) need none.
- **Installing Playwright + Chromium is a package install** — get Matt's OK first per host policy (he authorized it 2026-06-29). No `apt`/system-dep install is needed on cortex; the browser is a self-contained download to `~/.cache/ms-playwright`.

View file

@ -2,17 +2,19 @@
title: Headscale — OIDC Disabled at Boot (Authentik Boot-Order Dependency)
type: runbook
tags:
- vpn
- auth
aliases: []
related:
- [[headscale-onboard-node]]
- [[authentik-oidc-application]]
- [[edge2-access-reference]]
updated: 2026-06-30
- [[headscale-onboard-node]]
- [[authentik]]
- [[authentik-upgrade]]
- [[fleet-magicdns-resolved-migration]]
updated: 2026-07-13
---
# Headscale — OIDC Disabled at Boot (Authentik Boot-Order Dependency)
Headscale wires up its OIDC provider **once, at process startup**, by fetching Authentik's discovery document. If Authentik is unreachable at that moment, Headscale silently falls back to CLI-only auth and runs **OIDC-disabled until it is restarted**. This is a boot-ordering hazard: the fleet Headscale (edge2 **CT107**) and Authentik (edge2 **CT105**, `auth.echo6.co`) live on the same host, so an edge2 reboot — or the DNS-bootstrap window after one — can bring Headscale up before Authentik is serving.
Headscale wires up its OIDC provider **once, at process startup**, by fetching [[authentik]]'s discovery document. If Authentik is unreachable at that moment, Headscale silently falls back to CLI-only auth and runs **OIDC-disabled until it is restarted**. This is a boot-ordering hazard: the fleet Headscale (edge2 **CT107**) and Authentik (edge2 **CT105**, `auth.echo6.co`) live on the same host, so an edge2 reboot — or the DNS-bootstrap window after one — can bring Headscale up before Authentik is serving.
## Symptom

View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[proxmox-onboard-node]]
- [[ct-runbook]]
- [[caddy]]
- [[meshtastic-headscale-runbook]]
- [[lxc-service-migration]]
updated: 2026-06-30
- [[fleet-magicdns-resolved-migration]]
- [[idahomesh-vpn-device-setup]]
updated: 2026-07-13
---
# Headscale / Tailscale — Onboard a New Node

View file

@ -10,7 +10,7 @@ related:
- [[idahomesh-vpn-device-setup]]
- [[archivist]]
- [[usenet]]
updated: 2026-06-18
updated: 2026-07-13
---
# Internet Archive CLI Reference

View file

@ -7,10 +7,10 @@ aliases: []
related:
- [[ia-cli-reference]]
- [[ia-download-queue]]
- [[recon]]
- [[pipeline-patterns]]
- [[syncthing-add-node]]
- [[idahomesh-vpn-device-setup]]
updated: 2026-06-18
updated: 2026-07-13
---
# Download & Mirror from Internet Archive

View file

@ -5,12 +5,12 @@ tags:
- mesh
aliases: []
related:
- [[meshtastic-headscale-runbook]]
- [[idahomesh-vpn-device-setup]]
- [[meshtastic-headscale-runbook]]
- [[meshtastic-sidecar-node]]
- [[headscale-onboard-node]]
- [[caddy]]
updated: 2026-06-18
- [[fleet-magicdns-resolved-migration]]
updated: 2026-07-13
---
# IdahoMesh Bridge Setup

View file

@ -9,8 +9,8 @@ related:
- [[meshtastic-headscale-runbook]]
- [[meshtastic-sidecar-node]]
- [[headscale-onboard-node]]
- [[caddy]]
updated: 2026-06-18
- [[fleet-magicdns-resolved-migration]]
updated: 2026-07-13
---
# IdahoMesh VPN — Device Setup

View file

@ -6,16 +6,15 @@ tags:
aliases: []
related:
- [[expose-service-edge2]]
- [[vaultwarden-plan]]
- [[headscale-onboard-node]]
- [[caddy]]
- [[expose-service-contabo]]
- [[edge2-access-reference]]
updated: 2026-07-11
- [[services]]
- [[matrix_host]]
updated: 2026-07-13
---
# LXC Service Migration — Contabo → edge2
> **Note on the source host references below:** Contabo (`100.64.0.1`) was the migration **source** host during the 2026-06 service evacuation and was decommissioned/rebuilt as **edge1** (mail-only) on 2026-06-19 — it no longer exists at that tailnet address. The `ssh root@100.64.0.1` commands throughout this runbook are illustrative of "the source host you are migrating from"; for any future migration, substitute the actual current source host and its real access pattern. edge2 targets always use `ssh edge2` + `sudo pct exec` — never `ssh root@<edge2-IP>` (root SSH is refused on edge2). See [[edge2-access-reference]]. The migration **pattern** itself (phases, gates, rollback structure) remains valid regardless of which host is the source.
> **Note on the source host references below:** Contabo (`100.64.0.1`) was the migration **source** host during the 2026-06 service evacuation and was decommissioned/rebuilt as **edge1** (mail-only) on [[2026-06-19]] — it no longer exists at that tailnet address. The `ssh root@100.64.0.1` commands throughout this runbook are illustrative of "the source host you are migrating from"; for any future migration, substitute the actual current source host and its real access pattern. edge2 targets always use `ssh edge2` + `sudo pct exec` — never `ssh root@<edge2-IP>` (root SSH is refused on edge2). See [[edge2-access-reference]]. The migration **pattern** itself (phases, gates, rollback structure) remains valid regardless of which host is the source.
> Proven pilots: **Vaultwarden → edge2 CT 102** (SQLite, 2026-06-16), **Forgejo → edge2 CT 103** (PostgreSQL + non-Caddy SSH port, 2026-06-16), **LiveSync (CouchDB) → edge2 CT 104** (cold named-volume tar + bind-mounted config, 2026-06-16), **[[authentik]] (PostgreSQL keystone) → edge2 CT 105** (SECRET_KEY-must-travel, multi-block [[caddy]] cutover across 2 site blocks, reboot tailscale-before-docker race, 2026-06-18), **Matrix stack → edge2 CT 106** (multi-DB Postgres + stateful Signal bridge, 5 containers, 2026-06-18), and **Headscale → edge2 CT 107** (tailnet control plane, noise_private.key must travel, 2026-06-19). This runbook generalizes these patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.
@ -23,7 +22,7 @@ updated: 2026-07-11
## Overview
Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo Caddy frontend unchanged (public [[dns]] never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.
Move a Docker service from the main Contabo VPS into an LXC on edge2, with the Contabo [[caddy]] frontend unchanged (public [[dns]] never moves; only the upstream token in the Caddyfile changes). Rollback is a single line.
**Architecture after migration:**
@ -79,7 +78,7 @@ ssh edge2 "printf 'lxc.cgroup2.devices.allow: c 10:200 rwm\nlxc.mount.entry: /de
ssh edge2 "sudo pct start <CTID>"
```
Verify: internet access via NAT, DNS resolution.
Verify: internet access via NAT, [[dns]] resolution.
**Rollback:** `ssh edge2 'sudo pct stop <CTID> && sudo pct destroy <CTID>'`
@ -333,7 +332,7 @@ vault.echo6.co {
}
```
**Multi-token cutover example (LiveSync, 2026-06-16):** LiveSync exposes TWO upstream ports (5984 for CouchDB, 5985 for provisioner) within a single Caddy site block. Both tokens were changed from `127.0.0.1:598x``100.64.0.35:598x` in one edit. A third upstream in the same block — the Authentik outpost at `127.0.0.1:9000` (used for `forward_auth` on `/_provision`) — was left **untouched** because it stays on Contabo. Change only the tokens that move; never touch the Authentik outpost address.
**Multi-token cutover example (LiveSync, 2026-06-16):** LiveSync exposes TWO upstream ports (5984 for CouchDB, 5985 for provisioner) within a single Caddy site block. Both tokens were changed from `127.0.0.1:598x``100.64.0.35:598x` in one edit. A third upstream in the same block — the [[authentik]] outpost at `127.0.0.1:9000` (used for `forward_auth` on `/_provision`) — was left **untouched** because it stays on Contabo. Change only the tokens that move; never touch the Authentik outpost address.
**Multi-block cutover example (Authentik, 2026-06-18):** Authentik appeared in 4 places across 2 site blocks — `auth.echo6.co` (outpost path matcher + catch-all both pointing to `127.0.0.1:9000`) and `notes.echo6.co` (outpost path matcher + `forward_auth` directive both pointing to `127.0.0.1:9000`). All 4 occurrences were updated to `100.64.0.36:9000` in one edit. Grep the entire Caddyfile for the service's port before cutting over — do not assume a service lives in only one block. See also G15 (dnsmasq must NOT be repointed) and G16 (SECRET_KEY must travel).
@ -443,7 +442,7 @@ Mailcow cannot be "moved" to a different host IP via the one-token Caddy approac
The rebuild window requires a temporary front-door path so tailnet clients don't lose access while edge1's OS is gone:
1. **Pre-stage on edge2:** Add a temporary host-Caddy block on edge2 for any service that must stay live during the rebuild (in this case: all the already-migrated services were already on edge2; no outage for those).
1. **Pre-stage on edge2:** Add a temporary host-Caddy block on edge2 for any service that must stay live during the rebuild (in this case: all the already-migrated [[services]] were already on edge2; no outage for those).
2. **DNS during rebuild:** echo6.co resolves via public GoDaddy DNS. During the rebuild the `mail.*` records still pointed at 5.189.158.149 — accept a brief mail outage, or pre-bump the TTL to 60s and use a temporary MX fallback.
3. **Headscale pre-auth key:** Generate before the rebuild starts (`headscale preauthkeys create`). After OS install, register edge1's Tailscale with the pre-auth key and the new 100.64.0.40 IP is assigned.
4. **Tailscale bootstrap DNS:** The tailnet uses `vpn.echo6.co` for its login-server (Headscale on edge2). Since edge2 is up throughout, the tailnet stays operational.

View file

@ -9,8 +9,8 @@ related:
- [[authentik-oidc-application]]
- [[caddy]]
- [[authentik-create-invitation]]
- [[proxmox-onboard-node]]
updated: 2026-07-11
- [[expose-service-edge2]]
updated: 2026-07-13
---
# Mailcow: Create Mailbox
@ -112,7 +112,7 @@ Regular user accounts can leave all access flags at their defaults (all enabled)
### The Problem
Mailcow domains configured with OIDC authentication (like `echo6.co` with Authentik SSO) set `authsource=generic-oidc` on **every new mailbox by default**. This tells Dovecot to authenticate the account through the OIDC provider instead of the local password hash.
Mailcow domains configured with OIDC authentication (like `echo6.co` with [[authentik]] SSO) set `authsource=generic-oidc` on **every new mailbox by default**. This tells Dovecot to authenticate the account through the OIDC provider instead of the local password hash.
For service accounts that log in via SMTP with a username and password, this means:
@ -265,7 +265,7 @@ cd /opt/mailcow-dockerized && docker compose restart netfilter-mailcow
|---------|---------|-----------|---------|
| no-reply@echo6.co | Authentik | mailcow | SSO invitation emails, notifications |
| cipher@echo6.co | CIPHER | generic-oidc | Daily intelligence briefs |
| recon@echo6.co | RECON | generic-oidc | Pipeline notifications |
| [[recon]]@echo6.co | RECON | generic-oidc | Pipeline notifications |
| fulcrum@echo6.co | Fulcrum | generic-oidc | Hub notifications |
**Note:** cipher, recon, and fulcrum currently use `generic-oidc`. If any of these need to send mail via SMTP (not through the SSO web UI), their authsource must be changed to `mailcow` per Step 2.

View file

@ -3,10 +3,14 @@ title: meshai prod compose override (cutover state + MeshCore radio)
type: runbook
tags:
- mesh
aliases: []
related:
- [[meshai]]
- [[meshai-region-routing-plan]]
- [[meshcore-transport]]
updated: 2026-07-07
- [[meshai]]
- [[meshai-config-hot-apply]]
- [[SESSION-HANDOFF-meshai-test]]
updated: 2026-07-13
---
# meshai prod compose override
@ -49,7 +53,7 @@ services:
the utility host into the container at a stable udev symlink path. Without it
the container cannot see the MeshCore hardware (see [[meshcore-transport]]).
Both are deployment state (cutover progress + host-specific device path), which
Both are [[deployment]] state (cutover progress + host-specific device path), which
is why they live in an override rather than the committed public compose file.
## Restore

View file

@ -9,8 +9,8 @@ related:
- [[recon-service-integration]]
- [[ct-runbook]]
- [[headscale-onboard-node]]
- [[recon-operations]]
updated: 2026-06-18
- [[pg-backup]]
updated: 2026-07-13
---
# MeshMonitor Admin Password Reset

View file

@ -9,8 +9,8 @@ related:
- [[meshtastic-headscale-runbook]]
- [[idahomesh-bridge-setup]]
- [[headscale-onboard-node]]
- [[advbbs-project]]
updated: 2026-07-11
- [[meshtasticd-sim-nodes-runbook]]
updated: 2026-07-13
---
# Meshtastic Sidecar Node — Modular Deployment Runbook

View file

@ -8,9 +8,9 @@ related:
- [[ct-runbook]]
- [[meshtastic-sidecar-node]]
- [[headscale-onboard-node]]
- [[meshtastic-headscale-runbook]]
- [[proxmox-onboard-node]]
- [[ip-allocation]]
updated: 2026-07-11
updated: 2026-07-13
---
# Meshtasticd SIM Node Runbook — LXC Deployment
@ -415,7 +415,7 @@ pct exec <CTID> -- systemctl status meshtasticd # Check without entering
## Container Inventory Template
Track your deployment:
Track your [[deployment]]:
| CTID | Hostname | MAC Address | Service | Port | Notes |
|------|-------------|---------------------|------------|------|-----------------|

View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[ct-runbook]]
- [[meshtasticd-sim-nodes-runbook]]
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
- [[peertube-remote-runner]]
updated: 2026-06-18
- [[meshtasticd-sim-nodes-runbook]]
- [[headscale-onboard-node]]
updated: 2026-07-13
---
# NordVPN / WireGuard in LXC
@ -151,7 +151,7 @@ Endpoint = <server-ip>:51820
PersistentKeepalive = 25
```
**Critical for LXC:** If the container runs services that must stay reachable on the local network (e.g., PeerTube on port 9000), you need split tunneling. Replace `AllowedIPs = 0.0.0.0/0` with specific routes that exclude your LAN:
**Critical for LXC:** If the container runs [[services]] that must stay reachable on the local network (e.g., PeerTube on port 9000), you need split tunneling. Replace `AllowedIPs = 0.0.0.0/0` with specific routes that exclude your LAN:
```ini
# Route everything EXCEPT local network through VPN

View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[add-peertube-channel]]
- [[headless-browser-page-verification]]
- [[nordvpn-lxc]]
- [[ct-runbook]]
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
updated: 2026-06-18
- [[recon-service-integration]]
updated: 2026-07-13
---
# PeerTube Remote Runner — GPU Transcoding

View file

@ -5,12 +5,12 @@ tags:
- storage
aliases: []
related:
- [[ct-runbook]]
- [[recon-operations]]
- [[meshmonitor-password-reset]]
- [[matrix-synapse-deployment]]
- [[ct-runbook]]
- [[synapse]]
updated: 2026-06-18
- [[meshmonitor-password-reset]]
- [[recon-operations]]
updated: 2026-07-13
---
# PostgreSQL Backup (Docker)

View file

@ -8,9 +8,9 @@ related:
- [[ct-runbook]]
- [[proxmox-onboard-node]]
- [[proxmox-create-ubuntu-vm]]
- [[headscale-onboard-node]]
- [[environment]]
updated: 2026-06-18
- [[toc-cortex-pve9.2-update]]
- [[nordvpn-lxc]]
updated: 2026-07-13
---
# Pi 5 NAS — OMV Provisioning Runbook
@ -126,7 +126,7 @@ Each drive is used individually — no RAID array.
### Enable SMB (Windows Shares)
1. **[[services]] → SMB/CIFS → Settings** — toggle **Enabled**, click **Save**
2. **Services → SMB/CIFS → Shares** — click **Create** for each shared folder you want accessible from Windows:
2. **[[services]] → SMB/CIFS → Shares** — click **Create** for each shared folder you want accessible from Windows:
- Select the shared folder
- **Public:** No
- **Browseable:** Yes

View file

@ -1,15 +1,16 @@
---
title: "Pipeline & Wrapper Patterns"
type: runbook
tags: [tooling]
tags:
- tooling
aliases: []
related:
- [[meshtastic-sidecar-node]]
- [[meshtastic-headscale-runbook]]
- [[headscale-onboard-node]]
- [[idahomesh-vpn-device-setup]]
- [[syncthing-add-node]]
updated: 2026-06-18
- [[idahomesh-bridge-setup]]
- [[headscale-onboard-node]]
updated: 2026-07-13
---
# Pipeline & Wrapper Patterns

View file

@ -5,12 +5,12 @@ tags:
- proxmox
aliases: []
related:
- [[proxmox-onboard-node]]
- [[ct-runbook]]
- [[environment]]
- [[proxmox-onboard-node]]
- [[headscale-onboard-node]]
- [[pi-nas-omv-runbook]]
updated: 2026-07-11
updated: 2026-07-13
---
# Proxmox — Create Ubuntu VM (Cloud-Init)

View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[ct-runbook]]
- [[headscale-onboard-node]]
- [[proxmox-create-ubuntu-vm]]
- [[edge2-access-reference]]
- [[headscale-onboard-node]]
- [[expose-service-home]]
updated: 2026-06-18
- [[edge2-access-reference]]
updated: 2026-07-13
---
# Runbook: Onboard a Proxmox Node

View file

@ -3,9 +3,14 @@ title: pymc-repeater KISS TNC Re-enumeration Fix
type: runbook
tags:
- mesh
aliases: []
related:
- [[meshtastic-sidecar-node]]
updated: 2026-07-01
- [[recon-operations]]
- [[SESSION-HANDOFF-meshai-test]]
- [[meshtasticd-sim-nodes-runbook]]
- [[meshtastic-headscale-runbook]]
- [[syncthing-add-node]]
updated: 2026-07-13
---
# pymc-repeater KISS TNC Re-enumeration Fix

View file

@ -6,11 +6,11 @@ tags:
aliases: []
related:
- [[recon]]
- [[deployment]]
- [[caddy]]
- [[recon-service-integration]]
- [[services]]
updated: 2026-07-11
- [[central-deploy-cutover]]
- [[deployment]]
- [[pymc-repeater-kiss-tnc-reenumeration]]
updated: 2026-07-13
---
# RECON Operations Runbook

View file

@ -5,12 +5,12 @@ tags:
- recon
aliases: []
related:
- [[proxmox-onboard-node]]
- [[recon-operations]]
- [[headscale-onboard-node]]
- [[proxmox-onboard-node]]
- [[lxc-service-migration]]
- [[caddy]]
updated: 2026-07-11
- [[headscale-onboard-node]]
- [[expose-service-home]]
updated: 2026-07-13
---
# RECON Dashboard Service Integration

View file

@ -7,14 +7,14 @@ aliases: []
related:
- [[ct-runbook]]
- [[proxmox-onboard-node]]
- [[central-deploy-cutover]]
- [[meshtasticd-sim-nodes-runbook]]
- [[idahomesh-vpn-device-setup]]
- [[headscale-onboard-node]]
updated: 2026-07-11
- [[matrix-synapse-deployment]]
updated: 2026-07-13
---
# Syncthing: Add a New Node to the Project Sync Cluster
> **Syncthing on Contabo was decommissioned 2026-06-19** with the edge1 rebuild (state removed; Forge is now the durable backup via the `echo6-docs-autocommit` cron). The `contabo` row below and its device ID are historical — do not treat it as a live cluster member. Any new-node onboarding should reassess whether this cluster still has a live counterpart before assuming `contabo` is reachable.
> **Syncthing on Contabo was decommissioned [[2026-06-19]]** with the edge1 rebuild (state removed; Forge is now the durable backup via the `echo6-docs-autocommit` cron). The `contabo` row below and its device ID are historical — do not treat it as a live cluster member. Any new-node onboarding should reassess whether this cluster still has a live counterpart before assuming `contabo` is reachable.
## Overview

View file

@ -3,9 +3,14 @@ title: toc + cortex PVE 9.2 / GPU update — run from matt-desktop
type: runbook
tags:
- proxmox
- ai
related: []
updated: 2026-06-22
aliases: []
related:
- [[fleet-platform-baseline]]
- [[environment]]
- [[ct-runbook]]
- [[fleet-patch-audit]]
- [[pi-nas-omv-runbook]]
updated: 2026-07-13
status: active
---