echo6-docs/runbooks/syncthing-add-node.md

# Syncthing: Add a New Node to the Project Sync Cluster

## Overview

Adds a new machine to the Syncthing `projects` folder mesh. All nodes sync bidirectionally — new files merge, nothing is overwritten or deleted.

**Current cluster:**

| Node | Device ID (short) | Path | OS |
|------|--------------------|------|----|
| cortex | `6VP7KIB` | `/home/zvx/projects` | Ubuntu 24.04 |
| contabo | `SBYGD4P` | `/home/zvx/projects` | Ubuntu 24.04 |
| bluefin | `5ZTWIXM` | `/var/home/malice/projects` | Fedora Atomic |
| matt-desktop | `GCH6AAG` | `E:\Documents\projects` | Windows |

**Syncthing version:** v2.0.15 (all nodes must run v2.x — v1.x is incompatible)

**Config API:** All configuration changes are done via the REST API at `http://127.0.0.1:8384/rest/config` using the API key from each node's config XML.

---

## Prerequisites

- New node has network access to at least one existing node (Tailscale preferred)
- SSH access to the new node and at least one existing node

---

## Step 1: Install Syncthing v2

### Linux (apt-based)

Download the binary directly — the apt repo may only have v1.x:

```bash
curl -fsSL https://github.com/syncthing/syncthing/releases/download/v2.0.15/syncthing-linux-amd64-v2.0.15.tar.gz -o /tmp/syncthing.tar.gz
tar -xzf /tmp/syncthing.tar.gz -C /tmp
sudo cp /tmp/syncthing-linux-amd64-v2.0.15/syncthing /usr/bin/syncthing
syncthing --version  # verify v2.x
```

### Linux (Homebrew — Fedora Atomic/Bluefin)

```bash
brew install syncthing
# Creates ~/.local/state/syncthing/ for config
```

### Windows

```powershell
New-Item -ItemType Directory -Force -Path "$HOME\syncthing"
Invoke-WebRequest -Uri "https://github.com/syncthing/syncthing/releases/download/v2.0.15/syncthing-windows-amd64-v2.0.15.zip" -OutFile "$HOME\syncthing\st.zip"
Expand-Archive -Path "$HOME\syncthing\st.zip" -DestinationPath "$HOME\syncthing" -Force
Copy-Item "$HOME\syncthing\syncthing-windows-amd64-v2.0.15\syncthing.exe" "$HOME\syncthing\syncthing.exe" -Force
Remove-Item -Recurse -Force "$HOME\syncthing\syncthing-windows-amd64-v2.0.15"
Remove-Item "$HOME\syncthing\st.zip"
```

---

## Step 2: Generate Config and Get Device ID

```bash
syncthing generate
# Output includes: device=XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX
```

Save the full device ID — you'll need it for all other nodes.

---

## Step 3: Set Up Auto-Start

### Linux (systemd service — existing unit)

If `syncthing@<user>.service` exists (apt installs it):

```bash
sudo systemctl enable --now syncthing@zvx
```

### Linux (systemd user service — manual)

Create `~/.config/systemd/user/syncthing.service`:

```ini
[Unit]
Description=Syncthing - Open Source Continuous File Synchronization
After=network.target

[Service]
ExecStart=/path/to/syncthing serve --no-browser --no-restart --logflags=0
Restart=on-failure
RestartSec=10
SuccessExitStatus=3 4
RestartForceExitStatus=3 4

[Install]
WantedBy=default.target
```

```bash
systemctl --user daemon-reload
systemctl --user enable --now syncthing
```

### Windows (Scheduled Task)

```powershell
schtasks /create /tn Syncthing /tr "C:\Users\administrator\syncthing\syncthing.exe serve --no-browser --no-restart" /sc onlogon /rl highest /f
```

Then start it for the current session:

```powershell
Start-Process -FilePath "$HOME\syncthing\syncthing.exe" -ArgumentList "serve","--no-browser","--no-restart" -WindowStyle Hidden
```

### Windows Firewall

Required — syncthing won't accept connections without this:

```powershell
netsh advfirewall firewall add rule name="Syncthing" dir=in action=allow program="C:\Users\administrator\syncthing\syncthing.exe" enable=yes
netsh advfirewall firewall add rule name="Syncthing-Out" dir=out action=allow program="C:\Users\administrator\syncthing\syncthing.exe" enable=yes
```

---

## Step 4: Configure the New Node via REST API

Wait for syncthing to start (~5 seconds), then get the API key:

```bash
# Linux
grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//'

# Windows (PowerShell)
([xml](Get-Content "$env:LOCALAPPDATA\Syncthing\config.xml")).configuration.gui.apikey
```

Use the API to add devices and the projects folder. This Python snippet does it all — run it on the **new node**:

```python
import json, urllib.request

API_KEY = "<apikey from above>"
MY_DEVICE_ID = "<new node device ID>"
PROJECTS_PATH = "<local path to projects folder>"  # e.g. /home/zvx/projects

# All cluster nodes — add the new node's ID to this list when updating existing nodes
DEVICES = {
    "cortex": {"id": "6VP7KIB-ZHBI3AT-XO5FMY2-LFAZYM6-UMAV75U-MZZADW3-ZOBHJXY-GF26DAC", "addr": "tcp://100.64.0.14:22000"},
    "contabo": {"id": "SBYGD4P-BUWMWRQ-JJYYG75-YBR4WOO-OH42WH4-IAAO33D-STJZX6O-SZA2SQ4", "addr": "tcp://100.64.0.1:22000"},
    "bluefin": {"id": "5ZTWIXM-XNBUEW5-XWJM7PG-FJDMX5H-YMXM3CC-ZVS2PNO-NG2E3KJ-D5HXKQB", "addr": "dynamic"},
    "matt-desktop": {"id": "GCH6AAG-IWPH6TR-7GI7THZ-DIVXRRQ-EQMRBNN-IZG7Y2F-HM6BRLX-AC3MIQ6", "addr": "dynamic"},
}

def api(method, path, data=None):
    url = f"http://127.0.0.1:8384{path}"
    body = json.dumps(data).encode() if data else None
    req = urllib.request.Request(url, data=body, method=method,
        headers={"X-API-Key": API_KEY, "Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())

cfg = api("GET", "/rest/config")
existing_ids = [d["deviceID"] for d in cfg["devices"]]

# Add all peer devices
for name, dev in DEVICES.items():
    if dev["id"] not in existing_ids and dev["id"] != MY_DEVICE_ID:
        cfg["devices"].append({
            "deviceID": dev["id"], "name": name,
            "addresses": [dev["addr"]], "compression": "metadata",
            "paused": False, "autoAcceptFolders": False
        })

# Add projects folder if missing
folder_ids = [f["id"] for f in cfg["folders"]]
if "projects" not in folder_ids:
    all_device_ids = [d["id"] for d in DEVICES.values()] + [MY_DEVICE_ID]
    cfg["folders"].append({
        "id": "projects", "label": "projects",
        "path": PROJECTS_PATH, "type": "sendreceive",
        "rescanIntervalS": 60, "fsWatcherEnabled": True, "fsWatcherDelayS": 10,
        "devices": [{"deviceID": did} for did in set(all_device_ids)]
    })

api("PUT", "/rest/config", cfg)
print("New node configured")
```

---

## Step 5: Add the New Node to ALL Existing Nodes

For **each** existing node, run the following (substituting the new node's device ID and name):

```bash
APIKEY=$(grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//')
NEW_ID="<new node device ID>"
NEW_NAME="<new node name>"

CONFIG=$(curl -s -H "X-API-Key: $APIKEY" http://127.0.0.1:8384/rest/config)
CONFIG=$(echo "$CONFIG" | python3 -c "
import json,sys
c = json.load(sys.stdin)
did = '$NEW_ID'
ids = [d['deviceID'] for d in c['devices']]
if did not in ids:
    c['devices'].append({'deviceID': did, 'name': '$NEW_NAME', 'addresses': ['dynamic'], 'compression': 'metadata', 'paused': False, 'autoAcceptFolders': False})
for f in c['folders']:
    if f['id'] == 'projects':
        fids = [d['deviceID'] for d in f['devices']]
        if did not in fids:
            f['devices'].append({'deviceID': did})
json.dump(c, sys.stdout)
")
curl -s -X PUT -H "X-API-Key: $APIKEY" -H 'Content-Type: application/json' -d "$CONFIG" http://127.0.0.1:8384/rest/config
```

> **Important:** The CLI (`syncthing cli config devices add` / `syncthing cli config folders <id> devices add`) panics on v2.0.15 with a `reflect.Value.Elem on slice Value` bug. Always use the REST API instead.

---

## Step 6: Verify

Check connections from the new node:

```bash
syncthing cli show connections
```

Check sync status:

```bash
APIKEY=$(grep apikey ~/.local/state/syncthing/config.xml | sed 's/.*<apikey>//' | sed 's/<\/apikey.*//')
curl -s -H "X-API-Key: $APIKEY" http://127.0.0.1:8384/rest/db/status?folder=projects | python3 -m json.tool
```

Key fields: `state` should be `syncing` then `idle`, `needFiles` should reach `0`.

---

## Troubleshooting

| Problem | Cause | Fix |
|---------|-------|-----|
| Connections establish then drop with "reading length: EOF" | Version mismatch (v1 vs v2) | Upgrade all nodes to v2.x |
| Node shows as device but never connects | Firewall blocking port 22000 | Open inbound/outbound for syncthing binary (Windows) or port 22000 (Linux) |
| `syncthing cli config ... add` panics | Known bug in v2.0.15 CLI | Use REST API at `http://127.0.0.1:8384/rest/config` instead |
| Windows: syncthing not listening after start | Process started but exited silently | Check `%LOCALAPPDATA%\Syncthing\` for config issues; restart with `--logfile` flag |
| SSH to Windows mangles backslashes | Bash SSH escaping | Use PowerShell scripts via SCP, or use `$HOME\` which expands server-side |

---

## Config File Locations

| OS | Config XML | Data/Index |
|----|------------|------------|
| Linux (apt) | `~/.local/state/syncthing/config.xml` | `~/.local/state/syncthing/` |
| Linux (brew) | `~/.local/state/syncthing/config.xml` | `~/.local/state/syncthing/` |
| Windows | `%LOCALAPPDATA%\Syncthing\config.xml` | `%LOCALAPPDATA%\Syncthing\` |

## API Reference

- **Get config:** `GET http://127.0.0.1:8384/rest/config`
- **Set config:** `PUT http://127.0.0.1:8384/rest/config` (full config JSON)
- **Connections:** `GET http://127.0.0.1:8384/rest/system/connections`
- **Folder status:** `GET http://127.0.0.1:8384/rest/db/status?folder=projects`
- **Header:** `X-API-Key: <apikey>`