auto: docs sync 2026-07-12T00:00:23+00:00
Files changed: ", c.get(k))\nPY\n\\\"\n\"" engine/config.yaml engine/lib/lint.py engine/lib/vocab_gen.py engine/lint-report.md engine/sweep.sh vault/.obsidian/workspace.json vault/archive/projects/meshai-native-fire-severity-audit-cc-handoff.md vault/archive/projects/vaultwarden-plan.md vault/docs/matrix/matrix_host.md vault/docs/matrix/synapse.md vault/docs/services/services.md vault/docs/software/authentik.md vault/docs/software/caddy.md vault/docs/software/dns.md vault/docs/software/recon.md vault/docs/software/searxng.md vault/glossary.md vault/notes/echo6-landing-page-data-export.md vault/projects/matrix-synapse-deployment.md vault/projects/meshai.md vault/projects/meshtastic-headscale-runbook.md vault/projects/mmud-project.md vault/runbooks/add-peertube-channel.md vault/runbooks/authentik-access-groups.md vault/runbooks/authentik-create-invitation.md vault/runbooks/authentik-oidc-application.md vault/runbooks/authentik-upgrade.md vault/runbooks/expose-service-contabo.md vault/runbooks/lxc-service-migration.md vault/runbooks/mailcow-create-mailbox.md vault/runbooks/meshtastic-sidecar-node.md vault/runbooks/meshtasticd-sim-nodes-runbook.md vault/runbooks/proxmox-create-ubuntu-vm.md vault/runbooks/recon-operations.md vault/runbooks/recon-service-integration.md vault/runbooks/syncthing-add-node.md
This commit is contained in:
echo6-autocommit
parent
4a7a41df93
commit
cce29c595d
37 changed files with 336 additions and 262 deletions
|
|
@ -10,7 +10,7 @@ related:
|
|||
- [[recon-service-integration]]
|
||||
- [[proxmox-onboard-node]]
|
||||
- [[ct-runbook]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Add PeerTube Channel
|
||||
|
||||
|
|
@ -157,7 +157,7 @@ curl -s http://192.168.1.130:8420/api/peertube/channels \
|
|||
|
||||
If `tee` race condition empties the file:
|
||||
|
||||
1. Check Contabo backup: `ssh root@100.64.0.1 ls -la /opt/backups/recon/`
|
||||
1. Check edge1 backup (was Contabo pre-2026-06-19 rebuild): `ssh root@100.64.0.40 ls -la /opt/backups/recon/`
|
||||
2. Or rebuild from PeerTube DB:
|
||||
```bash
|
||||
ssh zvx@192.168.1.170 "sudo -u peertube psql peertube_prod -t -A -c \
|
||||
|
|
@ -178,7 +178,7 @@ If `tee` race condition empties the file:
|
|||
| Issue | Cause | Fix |
|
||||
|-------|-------|-----|
|
||||
| yt-dlp "Join this channel" error | Members-only first video | API auto-retries with `/videos` tab. CLI: add `--ignore-errors --playlist-items 1:5` and use `/videos` URL |
|
||||
| channel-map.json empty (0 bytes) | tee race condition | Always write to temp file first, then tee. Restore from backup or Contabo |
|
||||
| channel-map.json empty (0 bytes) | tee race condition | Always write to temp file first, then tee. Restore from backup or edge1 |
|
||||
| sudo: password required | Sudoers not set up | Create `/etc/sudoers.d/recon-mgmt` via `pct exec 110` from root@192.168.1.243 |
|
||||
| PeerTube "actor name already exists" | Channel exists in PeerTube but not in JSON | Add entry to JSON manually with correct `peertube_channel_id` |
|
||||
|
||||
|
|
|
|||
|
|
@ -10,13 +10,13 @@ related:
|
|||
- [[authentik-create-invitation]]
|
||||
- [[deploy-livesync]]
|
||||
- [[proxmox-onboard-node]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Authentik Access Groups
|
||||
|
||||
Manage group-based application access via the [[authentik]] API. No web UI interaction required.
|
||||
|
||||
**[[authentik]] instance:** https://auth.echo6.co (Contabo, 100.64.0.1)
|
||||
**[[authentik]] instance:** https://auth.echo6.co (edge2 CT 105, 100.64.0.36)
|
||||
|
||||
**Key behavior:** Users in `authentik Admins` (is_superuser=true) bypass ALL policy checks automatically. Group bindings only restrict non-superuser access.
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ related:
|
|||
- [[authentik-oidc-application]]
|
||||
- [[authentik-upgrade]]
|
||||
- [[mailcow-create-mailbox]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Authentik: Create Invitation
|
||||
|
||||
|
|
@ -156,13 +156,12 @@ See the [Access Groups runbook](authentik-access-groups.md) for detailed group m
|
|||
1. **Check custom attributes** — the `email` field must be present and correctly formatted
|
||||
2. **Check SMTP** — verify Authentik can send email:
|
||||
```bash
|
||||
ssh root@100.64.0.1
|
||||
docker exec authentik-server ak test_email matt@echo6.co
|
||||
ssh edge2 'sudo pct exec 105 -- docker exec authentik-server ak test_email matt@echo6.co'
|
||||
```
|
||||
3. **Check Mailcow authsource** — if SMTP auth fails, the no-reply@echo6.co mailbox may have reverted to `generic-oidc`. See [Mailcow Create Mailbox runbook](mailcow-create-mailbox.md), Step 2
|
||||
4. **Check Authentik logs**:
|
||||
```bash
|
||||
docker compose -f /opt/authentik/docker-compose.yml logs server --since 5m 2>&1 | grep -i email
|
||||
ssh edge2 'sudo pct exec 105 -- docker compose -f /opt/authentik/docker-compose.yml logs server --since 5m 2>&1 | grep -i email'
|
||||
```
|
||||
|
||||
### "Invalid invite/invite not found" when clicking link
|
||||
|
|
|
|||
|
|
@ -10,15 +10,15 @@ related:
|
|||
- [[authentik-upgrade]]
|
||||
- [[mailcow-create-mailbox]]
|
||||
- [[expose-service-home]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Add Authentik OIDC to an Application
|
||||
|
||||
Fully automated via [[authentik]] API. No web UI interaction required.
|
||||
|
||||
**Prerequisite:** [[dns]] must already exist for the service (run expose-service-contabo.md or expose-service-home.md first).
|
||||
**Prerequisite:** [[dns]] must already exist for the service (run expose-service-edge2.md or expose-service-home.md first).
|
||||
|
||||
**Authentik instance:** https://auth.echo6.co (Contabo, 100.64.0.1)
|
||||
**Authentik instance:** https://auth.echo6.co (edge2 CT 105, 100.64.0.36)
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -54,7 +54,7 @@ These conflict with Authentik's internal OAuth2 endpoints and **cannot be used**
|
|||
Create an API token from the Authentik admin account. This only needs to happen once — reuse the token across all OIDC setups.
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 "docker exec authentik-server \
|
||||
ssh edge2 "sudo pct exec 105 -- docker exec authentik-server \
|
||||
ak create_token --user akadmin --identifier oidc-automation --expiring 2>/dev/null \
|
||||
|| echo 'Token may already exist — check credentials file'"
|
||||
```
|
||||
|
|
@ -77,7 +77,7 @@ The API requires UUIDs for flows, scope mappings, and signing keys. These are st
|
|||
### Authorization flow
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 "curl -s \
|
||||
ssh edge2 "sudo pct exec 105 -- curl -s \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
'$AK_API/flows/instances/?slug=default-provider-authorization-implicit-consent' \
|
||||
| jq -r '.results[0].pk'"
|
||||
|
|
@ -89,7 +89,7 @@ Store as `AUTH_FLOW_PK`.
|
|||
|
||||
```bash
|
||||
# Get all scope mapping UUIDs at once
|
||||
ssh root@100.64.0.1 "curl -s \
|
||||
ssh edge2 "sudo pct exec 105 -- curl -s \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
'$AK_API/propertymappings/provider/scope/?ordering=scope_name' \
|
||||
| jq -r '.results[] | select(.scope_name == \"openid\" or .scope_name == \"email\" or .scope_name == \"profile\" or .scope_name == \"offline_access\") | \"\(.scope_name): \(.pk)\"'"
|
||||
|
|
@ -100,7 +100,7 @@ Store each UUID: `SCOPE_OPENID_PK`, `SCOPE_EMAIL_PK`, `SCOPE_PROFILE_PK`, `SCOPE
|
|||
### Signing key
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 "curl -s \
|
||||
ssh edge2 "sudo pct exec 105 -- curl -s \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
'$AK_API/crypto/certificatekeypairs/?name=authentik+Self-signed+Certificate&has_key=true' \
|
||||
| jq -r '.results[0].pk'"
|
||||
|
|
@ -131,7 +131,7 @@ fi
|
|||
Create the provider:
|
||||
|
||||
```bash
|
||||
PROVIDER_RESPONSE=$(ssh root@100.64.0.1 "curl -s \
|
||||
PROVIDER_RESPONSE=$(ssh edge2 "sudo pct exec 105 -- curl -s \
|
||||
-X POST '$AK_API/providers/oauth2/' \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
-H 'Content-Type: application/json' \
|
||||
|
|
@ -172,7 +172,7 @@ echo "Client Secret: $CLIENT_SECRET"
|
|||
## Step 4: Create the Application
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 "curl -s \
|
||||
ssh edge2 "sudo pct exec 105 -- curl -s \
|
||||
-X POST '$AK_API/core/applications/' \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
-H 'Content-Type: application/json' \
|
||||
|
|
@ -296,7 +296,7 @@ Check in order:
|
|||
Debug via API:
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 "curl -s \
|
||||
ssh edge2 "sudo pct exec 105 -- curl -s \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
'$AK_API/providers/oauth2/?search=$SERVICE_NAME' \
|
||||
| jq '.results[0] | {name, client_id, signing_key, access_token_validity, refresh_token_validity, property_mappings}'"
|
||||
|
|
@ -305,7 +305,7 @@ ssh root@100.64.0.1 "curl -s \
|
|||
Or via ak shell:
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 "docker exec authentik-server ak shell -c \"
|
||||
ssh edge2 "sudo pct exec 105 -- docker exec authentik-server ak shell -c \"
|
||||
from authentik.providers.oauth2.models import OAuth2Provider
|
||||
p = OAuth2Provider.objects.get(name='$SERVICE_NAME')
|
||||
print(f'Access Token: {p.access_token_validity}')
|
||||
|
|
@ -330,7 +330,7 @@ The redirect URI in the app config must **exactly** match what's in Authentik
|
|||
User isn't authorized for the application. By default all authenticated users have access. If you've added group restrictions via policy bindings, verify the user is in the correct group:
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 "curl -s \
|
||||
ssh edge2 "sudo pct exec 105 -- curl -s \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
'$AK_API/core/applications/$SERVICE_SLUG/' \
|
||||
| jq '{name, slug, policy_engine_mode}'"
|
||||
|
|
@ -344,12 +344,12 @@ Missing `offline_access` scope. Without refresh tokens, sessions only last as lo
|
|||
|
||||
```bash
|
||||
# Delete application first (it references the provider)
|
||||
ssh root@100.64.0.1 "curl -s -X DELETE \
|
||||
ssh edge2 "sudo pct exec 105 -- curl -s -X DELETE \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
'$AK_API/core/applications/$SERVICE_SLUG/'"
|
||||
|
||||
# Then delete provider
|
||||
ssh root@100.64.0.1 "curl -s -X DELETE \
|
||||
ssh edge2 "sudo pct exec 105 -- curl -s -X DELETE \
|
||||
-H 'Authorization: Bearer $AK_TOKEN' \
|
||||
'$AK_API/providers/oauth2/$PROVIDER_PK/'"
|
||||
```
|
||||
|
|
|
|||
|
|
@ -10,11 +10,13 @@ related:
|
|||
- [[authentik]]
|
||||
- [[authentik-create-invitation]]
|
||||
- [[ct-runbook]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Authentik: Major Version Upgrade
|
||||
|
||||
Upgrade [[authentik]] between major versions on Contabo. Covers backup, upgrade, verification, and rollback.
|
||||
> Migrated off Contabo → edge2 CT 105 on 2026-06-18.
|
||||
|
||||
Upgrade [[authentik]] between major versions on edge2 CT 105. Covers backup, upgrade, verification, and rollback.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -26,7 +28,7 @@ Any time Authentik is upgraded across major versions (e.g., 2024.12 → 2025.6
|
|||
|
||||
## Prerequisites
|
||||
|
||||
- SSH access to Contabo (`ssh root@100.64.0.1`)
|
||||
- SSH access to edge2 CT 105 via edge2 host: `ssh edge2 'sudo pct exec 105 -- bash'` (interactive shell inside the CT; root SSH direct to edge2/the CT is refused — see [[edge2-access-reference]])
|
||||
- Authentik compose directory: `/opt/authentik/`
|
||||
- Current version: check with `docker exec authentik-server ak --version`
|
||||
|
||||
|
|
@ -68,14 +70,14 @@ Look for:
|
|||
|
||||
## Step 2: Backup
|
||||
|
||||
### 2a. Snapshot Contabo (if Proxmox-managed)
|
||||
### 2a. Snapshot edge2 CT 105 (Proxmox-managed LXC)
|
||||
|
||||
If Contabo were a Proxmox VM, take a snapshot. Since it's a bare-metal VPS, skip this and rely on the file-level backups below.
|
||||
CT 105 is an LXC on edge2 — take a Proxmox snapshot before upgrading: `ssh edge2 "sudo pct snapshot 105 pre-upgrade-$(date +%Y%m%d)"`. Also rely on the file-level backups below.
|
||||
|
||||
### 2b. PostgreSQL Dump
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1
|
||||
ssh edge2 'sudo pct exec 105 -- bash' # interactive shell inside CT 105; run the rest of this section inside it
|
||||
|
||||
cd /opt/authentik
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
|
|
|
|||
|
|
@ -10,10 +10,12 @@ related:
|
|||
- [[lxc-service-migration]]
|
||||
- [[headscale-onboard-node]]
|
||||
- [[caddy]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Expose Service on Contabo
|
||||
|
||||
> SUPERSEDED — Contabo was decommissioned 2026-06-19. Use [[expose-service-edge2]] (services) or [[expose-service-contabo]]→edge1 for mail. This doc is kept for history only.
|
||||
|
||||
## Prerequisites
|
||||
- Service running in Docker on Contabo
|
||||
- Port bound to `127.0.0.1` only (never `0.0.0.0`)
|
||||
|
|
|
|||
|
|
@ -10,10 +10,13 @@ related:
|
|||
- [[headscale-onboard-node]]
|
||||
- [[caddy]]
|
||||
- [[expose-service-contabo]]
|
||||
updated: 2026-06-19
|
||||
- [[edge2-access-reference]]
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# LXC Service Migration — Contabo → edge2
|
||||
|
||||
> **Note on the source host references below:** Contabo (`100.64.0.1`) was the migration **source** host during the 2026-06 service evacuation and was decommissioned/rebuilt as **edge1** (mail-only) on 2026-06-19 — it no longer exists at that tailnet address. The `ssh root@100.64.0.1` commands throughout this runbook are illustrative of "the source host you are migrating from"; for any future migration, substitute the actual current source host and its real access pattern. edge2 targets always use `ssh edge2` + `sudo pct exec` — never `ssh root@<edge2-IP>` (root SSH is refused on edge2). See [[edge2-access-reference]]. The migration **pattern** itself (phases, gates, rollback structure) remains valid regardless of which host is the source.
|
||||
|
||||
> Proven pilots: **Vaultwarden → edge2 CT 102** (SQLite, 2026-06-16), **Forgejo → edge2 CT 103** (PostgreSQL + non-Caddy SSH port, 2026-06-16), **LiveSync (CouchDB) → edge2 CT 104** (cold named-volume tar + bind-mounted config, 2026-06-16), **[[authentik]] (PostgreSQL keystone) → edge2 CT 105** (SECRET_KEY-must-travel, multi-block [[caddy]] cutover across 2 site blocks, reboot tailscale-before-docker race, 2026-06-18), **Matrix stack → edge2 CT 106** (multi-DB Postgres + stateful Signal bridge, 5 containers, 2026-06-18), and **Headscale → edge2 CT 107** (tailnet control plane, noise_private.key must travel, 2026-06-19). This runbook generalizes these patterns into a reusable template for evacuating any Contabo-Caddy-fronted service to an edge2 LXC.
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -10,11 +10,13 @@ related:
|
|||
- [[caddy]]
|
||||
- [[authentik-create-invitation]]
|
||||
- [[proxmox-onboard-node]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Mailcow: Create Mailbox
|
||||
|
||||
Create a new mailbox in Mailcow on the Contabo VPS. Covers both interactive (UI) and API-driven creation, with the critical authsource fix for service accounts.
|
||||
> Migrated off Contabo → edge1 (mail-only) on 2026-06-19. Mailcow now lives in edge1 CT 101 (internal 10.10.10.2); public IP 5.189.158.149 is unchanged, but the tailscale address is now 100.64.0.40 (not 100.64.0.1).
|
||||
|
||||
Create a new mailbox in Mailcow on edge1 (Mailcow CT 101). Covers both interactive (UI) and API-driven creation, with the critical authsource fix for service accounts.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -26,7 +28,7 @@ Any time a new mailbox is created in Mailcow, but **especially** for service/sys
|
|||
|
||||
## Prerequisites
|
||||
|
||||
- SSH access to Contabo (`ssh root@100.64.0.1`)
|
||||
- SSH access to edge1 (`ssh edge1` or `ssh root@100.64.0.40`)
|
||||
- Mailcow API key (stored in Mailcow admin UI under System → Configuration → API)
|
||||
- Mailcow DB password: source from `/opt/mailcow-dockerized/.env` (`DBPASS`)
|
||||
|
||||
|
|
@ -51,7 +53,7 @@ MAILCOW_API_KEY=<api-key> # From Mailcow admin UI
|
|||
### Option A: Via Mailcow API
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1
|
||||
ssh edge1
|
||||
|
||||
curl -sk -X POST "https://127.0.0.1:8443/api/v1/add/mailbox" \
|
||||
-H "X-API-Key: ${MAILCOW_API_KEY}" \
|
||||
|
|
@ -128,7 +130,7 @@ The failure message gives no indication that OIDC is the cause. The password is
|
|||
Change the authsource from `generic-oidc` to `mailcow` in the database:
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1
|
||||
ssh edge1
|
||||
|
||||
# Source the DB password
|
||||
DBPASS=$(grep ^DBPASS /opt/mailcow-dockerized/.env | cut -d= -f2)
|
||||
|
|
@ -161,7 +163,7 @@ Rule of thumb: if the account will ever authenticate with a username + password
|
|||
Wait a few seconds after the authsource fix, then test:
|
||||
|
||||
```bash
|
||||
# From the Contabo host
|
||||
# From edge1
|
||||
python3 -c "
|
||||
import smtplib
|
||||
s = smtplib.SMTP('mail.echo6.co', 587, timeout=10)
|
||||
|
|
@ -236,7 +238,7 @@ Too many failed SMTP login attempts can trigger Mailcow's brute-force protection
|
|||
docker logs mailcowdockerized-netfilter-mailcow-1 --since 10m 2>&1 | grep -i ban
|
||||
```
|
||||
|
||||
If the Contabo IP (5.189.158.149) is banned, restart the netfilter container:
|
||||
If the edge1 public IP (5.189.158.149) is banned, restart the netfilter container:
|
||||
|
||||
```bash
|
||||
cd /opt/mailcow-dockerized && docker compose restart netfilter-mailcow
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ related:
|
|||
- [[idahomesh-bridge-setup]]
|
||||
- [[headscale-onboard-node]]
|
||||
- [[advbbs-project]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Meshtastic Sidecar Node — Modular Deployment Runbook
|
||||
|
||||
|
|
@ -127,7 +127,7 @@ ping -c 3 192.168.1.1 # Gateway reachable
|
|||
|
||||
| Tailnet | Headscale URL | Prefix | Key generation |
|
||||
|---------|---------------|--------|----------------|
|
||||
| Echo6 | `https://vpn.echo6.co` | 100.64.0.0/10 | On Contabo |
|
||||
| Echo6 | `https://vpn.echo6.co` | 100.64.0.0/10 | On edge2 CT 107 |
|
||||
| IdahoMesh | `https://vpn.idahomesh.com` | 100.100.0.0/16 | On CT 106 |
|
||||
|
||||
### Install Tailscale
|
||||
|
|
@ -138,10 +138,10 @@ curl -fsSL https://tailscale.com/install.sh | sh
|
|||
|
||||
### Generate preauthkey
|
||||
|
||||
**Echo6** (from cortex or any machine with Tailscale access to Contabo):
|
||||
**Echo6** (from cortex or any machine with Tailscale access to edge2):
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
|
||||
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h"
|
||||
```
|
||||
|
||||
**IdahoMesh** (from utility Proxmox host):
|
||||
|
|
@ -179,7 +179,7 @@ sudo systemctl daemon-reload
|
|||
```bash
|
||||
tailscale status # Should show connected
|
||||
tailscale ip -4 # Should show 100.64.x.x or 100.100.x.x
|
||||
ping -c 3 100.64.0.1 # Echo6: ping Contabo
|
||||
ping -c 3 100.64.0.26 # Echo6: ping edge2 (Headscale control now on edge2 CT 107, 100.64.0.38)
|
||||
ping -c 3 100.100.0.1 # IdahoMesh: ping Headscale
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ related:
|
|||
- [[headscale-onboard-node]]
|
||||
- [[proxmox-onboard-node]]
|
||||
- [[ip-allocation]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Meshtasticd SIM Node Runbook — LXC Deployment
|
||||
|
||||
|
|
@ -241,10 +241,10 @@ pct exec <CTID> -- bash -c "
|
|||
"
|
||||
```
|
||||
|
||||
Generate a preauth key on Contabo (user ID 1 = echo6):
|
||||
Generate a preauth key on edge2 CT 107 (user ID 1 = echo6):
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 'docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h'
|
||||
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale preauthkeys create --user 1 --reusable --expiration 1h"
|
||||
```
|
||||
|
||||
Register the node:
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ related:
|
|||
- [[proxmox-onboard-node]]
|
||||
- [[headscale-onboard-node]]
|
||||
- [[pi-nas-omv-runbook]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Proxmox — Create Ubuntu VM (Cloud-Init)
|
||||
|
||||
|
|
@ -19,7 +19,7 @@ Automated VM creation using Ubuntu cloud images. No interactive installer needed
|
|||
## Prerequisites
|
||||
|
||||
- SSH access to the target Proxmox host (directly or via jump box)
|
||||
- Headscale running on Contabo with a valid preauth key
|
||||
- Headscale running on edge2 CT 107 with a valid preauth key
|
||||
- Target Proxmox host has sufficient resources (check with `pvesm status`, `free -h`, `nproc`)
|
||||
|
||||
## Variables — Prompt the User
|
||||
|
|
@ -253,10 +253,10 @@ ssh zvx@$VM_IP 'curl -fsSL https://deb.nodesource.com/setup_22.x | sudo bash - &
|
|||
|
||||
## Step 12 — Tailscale Registration
|
||||
|
||||
Generate a preauth key on Contabo first:
|
||||
Generate a preauth key on edge2 CT 107 first:
|
||||
|
||||
```bash
|
||||
docker exec headscale headscale preauthkeys create --user echo6 --reusable --expiration 72h
|
||||
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale preauthkeys create --user echo6 --reusable --expiration 72h"
|
||||
```
|
||||
|
||||
Then register the VM:
|
||||
|
|
@ -268,7 +268,7 @@ ssh zvx@$VM_IP "sudo tailscale up --login-server https://vpn.echo6.co --auth-key
|
|||
|
||||
# Verify
|
||||
ssh zvx@$VM_IP 'tailscale status'
|
||||
docker exec headscale headscale nodes list
|
||||
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale nodes list"
|
||||
```
|
||||
|
||||
## Step 13 — Final Verification
|
||||
|
|
@ -287,7 +287,7 @@ ssh zvx@$VM_IP "
|
|||
echo '=== Disk ===' && df -h /
|
||||
"
|
||||
|
||||
docker exec headscale headscale nodes list
|
||||
ssh edge2 "sudo pct exec 107 -- docker exec headscale headscale nodes list"
|
||||
```
|
||||
|
||||
## Post-Creation
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ related:
|
|||
- [[caddy]]
|
||||
- [[recon-service-integration]]
|
||||
- [[services]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# RECON Operations Runbook
|
||||
|
||||
|
|
@ -87,11 +87,11 @@ python3 recon.py queue
|
|||
| Ollama | cortex | 11434 | Chat model for Aurora RAG |
|
||||
| NFS | pi-nas | — | /mnt/library (PDF source) |
|
||||
| Gemini API | Google | — | Enrichment + vision OCR (4 keys in .env) |
|
||||
| Contabo VPS | 100.64.0.1 | — | Backup destination |
|
||||
| edge1 | 100.64.0.40 | — | Backup destination (was Contabo 100.64.0.1 pre-2026-06-19 rebuild) |
|
||||
|
||||
## Backups
|
||||
|
||||
- **Destination:** `root@100.64.0.1:/opt/backups/recon/`
|
||||
- **Destination:** `root@100.64.0.40:/opt/backups/recon/` (edge1; was `root@100.64.0.1` on Contabo pre-2026-06-19 rebuild)
|
||||
- **Full sync (concepts, text, DB, config):** every 6 hours via cron
|
||||
- **DB snapshot only:** every 2 hours via cron
|
||||
- **Script:** `/opt/recon/scripts/backup.sh`
|
||||
|
|
@ -99,7 +99,7 @@ python3 recon.py queue
|
|||
### Verify backups
|
||||
|
||||
```bash
|
||||
ssh root@100.64.0.1 'ls -lh /opt/backups/recon/recon_*.db && du -sh /opt/backups/recon/'
|
||||
ssh root@100.64.0.40 'ls -lh /opt/backups/recon/recon_*.db && du -sh /opt/backups/recon/'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
|
@ -156,15 +156,15 @@ sqlite3 data/recon.db "UPDATE documents SET status='extracted' WHERE status='enr
|
|||
sqlite3 data/recon.db "UPDATE documents SET status='enriched' WHERE status='embedding';"
|
||||
```
|
||||
|
||||
### Full recovery from Contabo backup
|
||||
### Full recovery from edge1 backup
|
||||
|
||||
```bash
|
||||
ssh zvx@100.64.0.24
|
||||
sudo systemctl stop recon
|
||||
rsync -av root@100.64.0.1:/opt/backups/recon/concepts/ /opt/recon/data/concepts/
|
||||
rsync -av root@100.64.0.1:/opt/backups/recon/text/ /opt/recon/data/text/
|
||||
rsync -av root@100.64.0.40:/opt/backups/recon/concepts/ /opt/recon/data/concepts/
|
||||
rsync -av root@100.64.0.40:/opt/backups/recon/text/ /opt/recon/data/text/
|
||||
# Pick the latest DB backup
|
||||
rsync -av root@100.64.0.1:/opt/backups/recon/recon_latest.db /opt/recon/data/recon.db
|
||||
rsync -av root@100.64.0.40:/opt/backups/recon/recon_latest.db /opt/recon/data/recon.db
|
||||
cd /opt/recon && source venv/bin/activate
|
||||
python3 recon.py rebuild # Rebuilds Qdrant from concept JSONs
|
||||
sudo systemctl start recon
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ related:
|
|||
- [[headscale-onboard-node]]
|
||||
- [[lxc-service-migration]]
|
||||
- [[caddy]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# RECON Dashboard Service Integration
|
||||
|
||||
|
|
@ -22,7 +22,7 @@ Use this when you have a service running on a remote LXC/VM that needs a web man
|
|||
|
||||
## Prerequisites
|
||||
|
||||
- A running Flask or FastAPI dashboard (e.g., [[recon]] on VM 1130, WATCHTOWER on Contabo)
|
||||
- A running Flask or FastAPI dashboard (e.g., [[recon]] on VM 1130; WATCHTOWER on Contabo formerly served as an example here — **WATCHTOWER was decommissioned 2026-06-16** and is no longer a live integration target)
|
||||
- The target service running on a reachable host (LXC, VM, or bare metal)
|
||||
- SSH access from the dashboard host to the target host
|
||||
- The dashboard runs as a known user (e.g., `zvx`, `recon`, `watchtower`)
|
||||
|
|
@ -456,10 +456,12 @@ API endpoints:
|
|||
Dashboard panel: green/red dot + Restart/Stop/Start/Logs buttons + feedback box
|
||||
```
|
||||
|
||||
### WATCHTOWER monitoring remote services (Contabo → multiple hosts)
|
||||
### WATCHTOWER monitoring remote services (Contabo → multiple hosts) — HISTORICAL, DECOMMISSIONED
|
||||
|
||||
> **WATCHTOWER was decommissioned 2026-06-16** (archived to forge.echo6.co/matt/archive-watchtower). The example below reflects the pattern as it existed on the old Contabo VPS and is kept for reference only — it is not a live integration target and should not be used to route new work.
|
||||
|
||||
```
|
||||
DASHBOARD_HOST=5.189.158.149 (Contabo)
|
||||
DASHBOARD_HOST=5.189.158.149 (Contabo — decommissioned 2026-06-16)
|
||||
DASHBOARD_USER=root
|
||||
|
||||
Services managed:
|
||||
|
|
|
|||
|
|
@ -10,20 +10,22 @@ related:
|
|||
- [[meshtasticd-sim-nodes-runbook]]
|
||||
- [[idahomesh-vpn-device-setup]]
|
||||
- [[headscale-onboard-node]]
|
||||
updated: 2026-06-18
|
||||
updated: 2026-07-11
|
||||
---
|
||||
# Syncthing: Add a New Node to the Project Sync Cluster
|
||||
|
||||
> **Syncthing on Contabo was decommissioned 2026-06-19** with the edge1 rebuild (state removed; Forge is now the durable backup via the `echo6-docs-autocommit` cron). The `contabo` row below and its device ID are historical — do not treat it as a live cluster member. Any new-node onboarding should reassess whether this cluster still has a live counterpart before assuming `contabo` is reachable.
|
||||
|
||||
## Overview
|
||||
|
||||
Adds a new machine to the Syncthing `projects` folder mesh. All nodes sync bidirectionally — new files merge, nothing is overwritten or deleted.
|
||||
|
||||
**Current cluster:**
|
||||
**Current cluster (as of last update — `contabo` decommissioned 2026-06-19, kept below for history):**
|
||||
|
||||
| Node | Device ID (short) | Path | OS |
|
||||
|------|--------------------|------|----|
|
||||
| cortex | `6VP7KIB` | `/home/zvx/projects` | Ubuntu 24.04 |
|
||||
| contabo | `SBYGD4P` | `/home/zvx/projects` | Ubuntu 24.04 |
|
||||
| ~~contabo~~ (decommissioned 2026-06-19) | `SBYGD4P` | `/home/zvx/projects` | Ubuntu 24.04 |
|
||||
| bluefin | `5ZTWIXM` | `/var/home/malice/projects` | Fedora Atomic |
|
||||
| matt-desktop | `GCH6AAG` | `E:\Documents\projects` | Windows |
|
||||
|
||||
|
|
@ -164,9 +166,11 @@ MY_DEVICE_ID = "<new node device ID>"
|
|||
PROJECTS_PATH = "<local path to projects folder>" # e.g. /home/zvx/projects
|
||||
|
||||
# All cluster nodes — add the new node's ID to this list when updating existing nodes
|
||||
# NOTE: "contabo" is decommissioned (2026-06-19, edge1 rebuild) and kept here only for history.
|
||||
# Do not register new nodes against it; remove once the cluster config is next touched for real.
|
||||
DEVICES = {
|
||||
"cortex": {"id": "6VP7KIB-ZHBI3AT-XO5FMY2-LFAZYM6-UMAV75U-MZZADW3-ZOBHJXY-GF26DAC", "addr": "tcp://100.64.0.14:22000"},
|
||||
"contabo": {"id": "SBYGD4P-BUWMWRQ-JJYYG75-YBR4WOO-OH42WH4-IAAO33D-STJZX6O-SZA2SQ4", "addr": "tcp://100.64.0.1:22000"},
|
||||
# "contabo": {"id": "SBYGD4P-BUWMWRQ-JJYYG75-YBR4WOO-OH42WH4-IAAO33D-STJZX6O-SZA2SQ4", "addr": "tcp://100.64.0.1:22000"}, # DECOMMISSIONED 2026-06-19
|
||||
"bluefin": {"id": "5ZTWIXM-XNBUEW5-XWJM7PG-FJDMX5H-YMXM3CC-ZVS2PNO-NG2E3KJ-D5HXKQB", "addr": "dynamic"},
|
||||
"matt-desktop": {"id": "GCH6AAG-IWPH6TR-7GI7THZ-DIVXRRQ-EQMRBNN-IZG7Y2F-HM6BRLX-AC3MIQ6", "addr": "dynamic"},
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue