Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
89834796ff
commit
e9231ac24a
93 changed files with 51223 additions and 254 deletions
272
runbooks/syncthing-add-node.md
Normal file
272
runbooks/syncthing-add-node.md
Normal file
|
|
@ -0,0 +1,272 @@
|
|||
# Syncthing: Add a New Node to the Project Sync Cluster
|
||||
|
||||
## Overview
|
||||
|
||||
Adds a new machine to the Syncthing `projects` folder mesh. All nodes sync bidirectionally — new files merge, nothing is overwritten or deleted.
|
||||
|
||||
**Current cluster:**
|
||||
|
||||
| Node | Device ID (short) | Path | OS |
|
||||
|------|--------------------|------|----|
|
||||
| cortex | `6VP7KIB` | `/home/zvx/projects` | Ubuntu 24.04 |
|
||||
| contabo | `SBYGD4P` | `/home/zvx/projects` | Ubuntu 24.04 |
|
||||
| bluefin | `5ZTWIXM` | `/var/home/malice/projects` | Fedora Atomic |
|
||||
| matt-desktop | `GCH6AAG` | `E:\Documents\projects` | Windows |
|
||||
|
||||
**Syncthing version:** v2.0.15 (all nodes must run v2.x — v1.x is incompatible)
|
||||
|
||||
**Config API:** All configuration changes are done via the REST API at `http://127.0.0.1:8384/rest/config` using the API key from each node's config XML.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- New node has network access to at least one existing node (Tailscale preferred)
|
||||
- SSH access to the new node and at least one existing node
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Install Syncthing v2
|
||||
|
||||
### Linux (apt-based)
|
||||
|
||||
Download the binary directly — the apt repo may only have v1.x:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://github.com/syncthing/syncthing/releases/download/v2.0.15/syncthing-linux-amd64-v2.0.15.tar.gz -o /tmp/syncthing.tar.gz
|
||||
tar -xzf /tmp/syncthing.tar.gz -C /tmp
|
||||
sudo cp /tmp/syncthing-linux-amd64-v2.0.15/syncthing /usr/bin/syncthing
|
||||
syncthing --version # verify v2.x
|
||||
```
|
||||
|
||||
### Linux (Homebrew — Fedora Atomic/Bluefin)
|
||||
|
||||
```bash
|
||||
brew install syncthing
|
||||
# Creates ~/.local/state/syncthing/ for config
|
||||
```
|
||||
|
||||
### Windows
|
||||
|
||||
```powershell
|
||||
New-Item -ItemType Directory -Force -Path "$HOME\syncthing"
|
||||
Invoke-WebRequest -Uri "https://github.com/syncthing/syncthing/releases/download/v2.0.15/syncthing-windows-amd64-v2.0.15.zip" -OutFile "$HOME\syncthing\st.zip"
|
||||
Expand-Archive -Path "$HOME\syncthing\st.zip" -DestinationPath "$HOME\syncthing" -Force
|
||||
Copy-Item "$HOME\syncthing\syncthing-windows-amd64-v2.0.15\syncthing.exe" "$HOME\syncthing\syncthing.exe" -Force
|
||||
Remove-Item -Recurse -Force "$HOME\syncthing\syncthing-windows-amd64-v2.0.15"
|
||||
Remove-Item "$HOME\syncthing\st.zip"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Generate Config and Get Device ID
|
||||
|
||||
```bash
|
||||
syncthing generate
|
||||
# Output includes: device=XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX
|
||||
```
|
||||
|
||||
Save the full device ID — you'll need it for all other nodes.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Set Up Auto-Start
|
||||
|
||||
### Linux (systemd service — existing unit)
|
||||
|
||||
If `syncthing@<user>.service` exists (apt installs it):
|
||||
|
||||
```bash
|
||||
sudo systemctl enable --now syncthing@zvx
|
||||
```
|
||||
|
||||
### Linux (systemd user service — manual)
|
||||
|
||||
Create `~/.config/systemd/user/syncthing.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Syncthing - Open Source Continuous File Synchronization
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
ExecStart=/path/to/syncthing serve --no-browser --no-restart --logflags=0
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
SuccessExitStatus=3 4
|
||||
RestartForceExitStatus=3 4
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
```
|
||||
|
||||
```bash
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now syncthing
|
||||
```
|
||||
|
||||
### Windows (Scheduled Task)
|
||||
|
||||
```powershell
|
||||
schtasks /create /tn Syncthing /tr "C:\Users\administrator\syncthing\syncthing.exe serve --no-browser --no-restart" /sc onlogon /rl highest /f
|
||||
```
|
||||
|
||||
Then start it for the current session:
|
||||
|
||||
```powershell
|
||||
Start-Process -FilePath "$HOME\syncthing\syncthing.exe" -ArgumentList "serve","--no-browser","--no-restart" -WindowStyle Hidden
|
||||
```
|
||||
|
||||
### Windows Firewall
|
||||
|
||||
Required — syncthing won't accept connections without this:
|
||||
|
||||
```powershell
|
||||
netsh advfirewall firewall add rule name="Syncthing" dir=in action=allow program="C:\Users\administrator\syncthing\syncthing.exe" enable=yes
|
||||
netsh advfirewall firewall add rule name="Syncthing-Out" dir=out action=allow program="C:\Users\administrator\syncthing\syncthing.exe" enable=yes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Configure the New Node via REST API
|
||||
|
||||
Wait for syncthing to start (~5 seconds), then get the API key:
|
||||
|
||||
```bash
|
||||
# Linux
|
||||
grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//'
|
||||
|
||||
# Windows (PowerShell)
|
||||
([xml](Get-Content "$env:LOCALAPPDATA\Syncthing\config.xml")).configuration.gui.apikey
|
||||
```
|
||||
|
||||
Use the API to add devices and the projects folder. This Python snippet does it all — run it on the **new node**:
|
||||
|
||||
```python
|
||||
import json, urllib.request
|
||||
|
||||
API_KEY = "<apikey from above>"
|
||||
MY_DEVICE_ID = "<new node device ID>"
|
||||
PROJECTS_PATH = "<local path to projects folder>" # e.g. /home/zvx/projects
|
||||
|
||||
# All cluster nodes — add the new node's ID to this list when updating existing nodes
|
||||
DEVICES = {
|
||||
"cortex": {"id": "6VP7KIB-ZHBI3AT-XO5FMY2-LFAZYM6-UMAV75U-MZZADW3-ZOBHJXY-GF26DAC", "addr": "tcp://100.64.0.14:22000"},
|
||||
"contabo": {"id": "SBYGD4P-BUWMWRQ-JJYYG75-YBR4WOO-OH42WH4-IAAO33D-STJZX6O-SZA2SQ4", "addr": "tcp://100.64.0.1:22000"},
|
||||
"bluefin": {"id": "5ZTWIXM-XNBUEW5-XWJM7PG-FJDMX5H-YMXM3CC-ZVS2PNO-NG2E3KJ-D5HXKQB", "addr": "dynamic"},
|
||||
"matt-desktop": {"id": "GCH6AAG-IWPH6TR-7GI7THZ-DIVXRRQ-EQMRBNN-IZG7Y2F-HM6BRLX-AC3MIQ6", "addr": "dynamic"},
|
||||
}
|
||||
|
||||
def api(method, path, data=None):
|
||||
url = f"http://127.0.0.1:8384{path}"
|
||||
body = json.dumps(data).encode() if data else None
|
||||
req = urllib.request.Request(url, data=body, method=method,
|
||||
headers={"X-API-Key": API_KEY, "Content-Type": "application/json"})
|
||||
return json.loads(urllib.request.urlopen(req).read())
|
||||
|
||||
cfg = api("GET", "/rest/config")
|
||||
existing_ids = [d["deviceID"] for d in cfg["devices"]]
|
||||
|
||||
# Add all peer devices
|
||||
for name, dev in DEVICES.items():
|
||||
if dev["id"] not in existing_ids and dev["id"] != MY_DEVICE_ID:
|
||||
cfg["devices"].append({
|
||||
"deviceID": dev["id"], "name": name,
|
||||
"addresses": [dev["addr"]], "compression": "metadata",
|
||||
"paused": False, "autoAcceptFolders": False
|
||||
})
|
||||
|
||||
# Add projects folder if missing
|
||||
folder_ids = [f["id"] for f in cfg["folders"]]
|
||||
if "projects" not in folder_ids:
|
||||
all_device_ids = [d["id"] for d in DEVICES.values()] + [MY_DEVICE_ID]
|
||||
cfg["folders"].append({
|
||||
"id": "projects", "label": "projects",
|
||||
"path": PROJECTS_PATH, "type": "sendreceive",
|
||||
"rescanIntervalS": 60, "fsWatcherEnabled": True, "fsWatcherDelayS": 10,
|
||||
"devices": [{"deviceID": did} for did in set(all_device_ids)]
|
||||
})
|
||||
|
||||
api("PUT", "/rest/config", cfg)
|
||||
print("New node configured")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Add the New Node to ALL Existing Nodes
|
||||
|
||||
For **each** existing node, run the following (substituting the new node's device ID and name):
|
||||
|
||||
```bash
|
||||
APIKEY=$(grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//')
|
||||
NEW_ID="<new node device ID>"
|
||||
NEW_NAME="<new node name>"
|
||||
|
||||
CONFIG=$(curl -s -H "X-API-Key: $APIKEY" http://127.0.0.1:8384/rest/config)
|
||||
CONFIG=$(echo "$CONFIG" | python3 -c "
|
||||
import json,sys
|
||||
c = json.load(sys.stdin)
|
||||
did = '$NEW_ID'
|
||||
ids = [d['deviceID'] for d in c['devices']]
|
||||
if did not in ids:
|
||||
c['devices'].append({'deviceID': did, 'name': '$NEW_NAME', 'addresses': ['dynamic'], 'compression': 'metadata', 'paused': False, 'autoAcceptFolders': False})
|
||||
for f in c['folders']:
|
||||
if f['id'] == 'projects':
|
||||
fids = [d['deviceID'] for d in f['devices']]
|
||||
if did not in fids:
|
||||
f['devices'].append({'deviceID': did})
|
||||
json.dump(c, sys.stdout)
|
||||
")
|
||||
curl -s -X PUT -H "X-API-Key: $APIKEY" -H 'Content-Type: application/json' -d "$CONFIG" http://127.0.0.1:8384/rest/config
|
||||
```
|
||||
|
||||
> **Important:** The CLI (`syncthing cli config devices add` / `syncthing cli config folders <id> devices add`) panics on v2.0.15 with a `reflect.Value.Elem on slice Value` bug. Always use the REST API instead.
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Verify
|
||||
|
||||
Check connections from the new node:
|
||||
|
||||
```bash
|
||||
syncthing cli show connections
|
||||
```
|
||||
|
||||
Check sync status:
|
||||
|
||||
```bash
|
||||
APIKEY=$(grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//')
|
||||
curl -s -H "X-API-Key: $APIKEY" http://127.0.0.1:8384/rest/db/status?folder=projects | python3 -m json.tool
|
||||
```
|
||||
|
||||
Key fields: `state` should be `syncing` then `idle`, `needFiles` should reach `0`.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Problem | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| Connections establish then drop with "reading length: EOF" | Version mismatch (v1 vs v2) | Upgrade all nodes to v2.x |
|
||||
| Node shows as device but never connects | Firewall blocking port 22000 | Open inbound/outbound for syncthing binary (Windows) or port 22000 (Linux) |
|
||||
| `syncthing cli config ... add` panics | Known bug in v2.0.15 CLI | Use REST API at `http://127.0.0.1:8384/rest/config` instead |
|
||||
| Windows: syncthing not listening after start | Process started but exited silently | Check `%LOCALAPPDATA%\Syncthing\` for config issues; restart with `--logfile` flag |
|
||||
| SSH to Windows mangles backslashes | Bash SSH escaping | Use PowerShell scripts via SCP, or use `$HOME\` which expands server-side |
|
||||
|
||||
---
|
||||
|
||||
## Config File Locations
|
||||
|
||||
| OS | Config XML | Data/Index |
|
||||
|----|------------|------------|
|
||||
| Linux (apt) | `~/.local/state/syncthing/config.xml` | `~/.local/state/syncthing/` |
|
||||
| Linux (brew) | `~/.local/state/syncthing/config.xml` | `~/.local/state/syncthing/` |
|
||||
| Windows | `%LOCALAPPDATA%\Syncthing\config.xml` | `%LOCALAPPDATA%\Syncthing\` |
|
||||
|
||||
## API Reference
|
||||
|
||||
- **Get config:** `GET http://127.0.0.1:8384/rest/config`
|
||||
- **Set config:** `PUT http://127.0.0.1:8384/rest/config` (full config JSON)
|
||||
- **Connections:** `GET http://127.0.0.1:8384/rest/system/connections`
|
||||
- **Folder status:** `GET http://127.0.0.1:8384/rest/db/status?folder=projects`
|
||||
- **Header:** `X-API-Key: <apikey>`
|
||||
Loading…
Add table
Add a link
Reference in a new issue