diff --git a/.obsidian/plugins/obsidian-livesync/data.json b/.obsidian/plugins/obsidian-livesync/data.json index 475d9f9..ff00659 100644 --- a/.obsidian/plugins/obsidian-livesync/data.json +++ b/.obsidian/plugins/obsidian-livesync/data.json @@ -5,7 +5,7 @@ "couchDB_USER": "", "couchDB_PASSWORD": "", "couchDB_DBNAME": "cc-db", - "liveSync": false, + "liveSync": true, "syncOnSave": false, "syncOnStart": true, "savingDelay": 200, diff --git a/.obsidian/workspace.json b/.obsidian/workspace.json index 83b9500..e3417d9 100644 --- a/.obsidian/workspace.json +++ b/.obsidian/workspace.json @@ -189,6 +189,8 @@ }, "active": "ea4cc678c44e8b67", "lastOpenFiles": [ + "docs/services/ots-setup.md", + "docs/hardware/ip-allocation.md", "CLAUDE-baseline.md", "INDEX.md", "notes/ia-download-queue.md", @@ -216,8 +218,6 @@ "docs/matrix/synapse_retention_discovery.md", "docs/matrix/matrix_host.md", "docs/matrix/archivist.md", - "docs/matrix/mautrix_signal.md", - "docs/matrix/synapse.md", "notes", "archive/matrix", "docs/matrix", diff --git a/INDEX.md b/INDEX.md index d5bfa86..9d33b6e 100644 --- a/INDEX.md +++ b/INDEX.md @@ -16,8 +16,10 @@ The map of this vault — start here. `docs/` is current reference, `runbooks/` ## 🧭 Reference — current state (`docs/`) **Infrastructure** -- [[environment]] — Proxmox cluster, VMs, LXCs, IP allocation, Headscale nodes, SSH access +- [[environment]] — Proxmox cluster, VMs, LXCs, Headscale nodes, SSH access +- [[ip-allocation]] — full 192.168.1.x IP allocation table (LXCs, VMs, hosts, Meshtastic nodes, Tailscale IPs) - [[services]] — full services inventory by server (ports, compose paths, access URLs) +- [[ots-setup]] — OpenTAKServer on CT 109 (native install, MQTT gateway, Meshtastic integration, troubleshooting) - [[usenet]] — SABnzbd + Usenet providers & indexers **Software / platforms** (`docs/software/`) diff --git a/docs/hardware/environment.md b/docs/hardware/environment.md index 57bb505..18a71ef 100644 --- a/docs/hardware/environment.md +++ b/docs/hardware/environment.md @@ -115,7 +115,7 @@ Five nodes running Proxmox VE: | .240-.250 | Proxmox hosts + bare metal | | .251-.254 | Meshtastic nodes | -Full details: `/home/zvx/projects/utility/ip-allocation.md` +Full details: [[ip-allocation]] ## Headscale Node List diff --git a/docs/hardware/ip-allocation.md b/docs/hardware/ip-allocation.md new file mode 100755 index 0000000..712affa --- /dev/null +++ b/docs/hardware/ip-allocation.md @@ -0,0 +1,108 @@ +# Echo6 Network IP Allocation + +## IP Ranges + +| Range | Purpose | +|-------|---------| +| .1-.10 | Network Infrastructure (router, switches, APs) | +| .11-.99 | DHCP (phones, laptops, guests) | +| .100-.149 | LXC Containers | +| .150-.199 | VMs | +| .200-.224 | Security Cameras | +| .225-.239 | IoT/Embedded Devices | +| .240-.250 | Proxmox Hosts + Bare Metal Servers | +| .251-.254 | Meshtastic Nodes | + +--- + +## Current Allocations + +### Network Infrastructure (.1-.10) +| IP | Device | Notes | +|----|--------|-------| +| .1 | Router/Gateway | | +| .2-.10 | (available) | Switches, APs | + +### LXC Containers (.100-.149) +| IP | Container | Host | Purpose | +|----|-----------|------|---------| +| .100 | meshmonitor (CT 100) | utility | MeshMonitor web UI | +| .101 | caddy (CT 101) | utility | Home reverse proxy | +| .102 | searxng (CT 102) | utility | Echo6 Search (SearXNG) | +| .103 | argus (CT 103) | utility | ARGUS OSINT platform | +| .104 | meshing-around (CT 104) | utility | Mesh bot + WebGUI | +| .106 | meshtastic-hs (CT 106) | utility | IdahoMesh Headscale | +| .107 | mesh-bridge (CT 107) | utility | Dual-tailscaled bridge | +| .109 | mmud (CT 109) | utility | MMUD game daemon + web dashboard | +| .111 | mmud-embr (CT 111) | utility | MMUD SIM: EMBR (game server) | +| .112 | mmud-dcrg (CT 112) | utility | MMUD SIM: DCRG (broadcast) | +| .113 | mmud-grst (CT 113) | utility | MMUD SIM: GRST (Grist barkeep) | +| .114 | mmud-mrn (CT 114) | utility | MMUD SIM: MRN (Maren healer) | +| .115 | mmud-trvl (CT 115) | utility | MMUD SIM: TRVL (Torval merchant) | +| .116 | mmud-wspr (CT 116) | utility | MMUD SIM: WSPR (Whisper sage) | +| .118 | archivist (CT 118) | utility | Signal/Matrix archive bot | +| .130 | recon (VM 1130) | data | RECON pipeline (migrated from CT 130) | +| .144 | meshai (CT 108) | utility | MeshAI assistant | +| .170 | peertube (CT 110) | media | PeerTube streaming | +| .182 | immich (CT 120) | cloud | Immich photos | +| .183 | nextcloud (CT 121) | cloud | Nextcloud AIO | + +### VMs (.150-.199) +| IP | VM | Host | Purpose | +|----|-----|------|---------| +| .150-.199 | (available) | | | + +### Security Cameras (.200-.224) +| IP | Camera | Location | +|----|--------|----------| +| .200-.224 | (available) | | + +### IoT/Embedded (.225-.239) +| IP | Device | Notes | +|----|--------|-------| +| .225-.238 | (available) | | +| .239 | (reserved) | Former TOC IP | + +### Proxmox Hosts + Bare Metal (.240-.250) +| IP | Hostname | Role | Tailscale | +|----|----------|------|-----------| +| .240 | data | Proxmox - DB node | 100.64.0.20 | +| .241 | utility | Proxmox - Utility services | 100.64.0.19 | +| .242 | cloud | Proxmox - Cloud storage | 100.64.0.22 | +| .243 | media | Proxmox - Media server | 100.64.0.21 | +| .244 | echo6-toc | LLM/AI server | 100.64.0.5 | +| .245-.250 | (available) | | | + +### Meshtastic Nodes (.251-.254) +| IP | Node | Notes | +|----|------|-------| +| .251-.252 | (available) | | +| .253 | AIDA-NEBRA | Raspberry Pi running meshtasticd | +| .254 | (available) | | + +--- + +## Tailscale IPs (100.64.0.x) + +| Tailscale IP | Device | Local IP | +|--------------|--------|----------| +| 100.64.0.5 | echo6-toc | 192.168.1.244 | +| 100.64.0.6 | contabo-mailcow | 5.189.158.149 (external) | +| 100.64.0.18 | aida-nebra | 192.168.1.253 | +| 100.64.0.19 | utility | 192.168.1.241 | +| 100.64.0.20 | data | 192.168.1.240 | +| 100.64.0.21 | media | 192.168.1.243 | +| 100.64.0.22 | cloud | 192.168.1.242 | + +--- + +## Notes + +- DHCP range starts at .11 to leave room for network infra +- LXC containers get .100-.149 for easy identification +- Meshtastic nodes at .251-.254 (max 4 hardware nodes) +- AIDA-NEBRA at .253 is technically a Meshtastic node (runs meshtasticd) + +--- + +*Last updated: 2026-06-14* diff --git a/docs/services/ots-setup.md b/docs/services/ots-setup.md new file mode 100644 index 0000000..6dcbfd4 --- /dev/null +++ b/docs/services/ots-setup.md @@ -0,0 +1,571 @@ +# OpenTAKServer Setup Documentation + +## Installation Date +April 17, 2026 + +## Overview +OpenTAKServer installation on Echo6 infrastructure for TAK (Team Awareness Kit) server capabilities. + +--- + +## Container Specifications + +**VMID:** 109 +**Hostname:** opentakserver +**Host:** utility Proxmox (192.168.1.241) +**IP Address:** 192.168.1.109/24 +**Gateway:** 192.168.1.1 +**OS:** Ubuntu 24.04 LTS +**Resources:** 2 cores, 4GB RAM, 8GB disk +**Storage:** local-lvm +**Features:** nesting=1 +**Auto-start:** enabled + +--- + +## Access Information + +### External Access +- **URL:** https://ots.k7zvx.com +- **Admin Username:** administrator +- **Admin Password:** password (**CHANGE THIS IMMEDIATELY**) + +### SSH Access +```bash +# Via Proxmox host +ssh root@192.168.1.241 'pct exec 109 -- bash' + +# As zvx user +ssh root@192.168.1.241 'pct exec 109 -- su - zvx' + +# Direct container entry +pct enter 109 +``` + +**Container User:** +- Username: zvx +- Password: 7redditGold +- Sudo: NOPASSWD enabled + +--- + +## Network Configuration + +### DNS +- **Domain:** ots.k7zvx.com +- **DNS Provider:** GoDaddy +- **Record Type:** A +- **Points to:** 199.6.36.163 (home external IP) +- **TTL:** 3600 seconds + +### SSL Certificate +- **Provider:** Let's Encrypt +- **Method:** acme.sh with GoDaddy DNS validation +- **Location:** /etc/caddy/certs/ots.k7zvx.com.* (on CT 101) +- **Auto-renewal:** Configured via acme.sh + +### Reverse Proxy (Caddy on CT 101) +``` +ots.k7zvx.com { + tls /etc/caddy/certs/ots.k7zvx.com.fullchain.crt /etc/caddy/certs/ots.k7zvx.com.key + reverse_proxy https://192.168.1.109:443 { + transport http { + tls_insecure_skip_verify + } + } +} +``` + +### Port Forwarding +- **Router:** Ubiquiti firewall (192.168.1.28) +- **External ports:** 80/443 → 192.168.1.101 (Caddy CT) +- **Internal proxy:** Caddy → 192.168.1.109:443 (OpenTAKServer) + +--- + +## OpenTAKServer Configuration + +### Installation Details +- **Installation Method:** Official installer script +- **Install User:** zvx +- **Install Path:** /home/zvx/ots/ +- **Virtual Environment:** /home/zvx/.opentakserver_venv/ +- **Version:** 1.7.10 + +### Services +```bash +# Main services +systemctl status opentakserver +systemctl status eud_handler +systemctl status eud_handler_ssl +systemctl status cot_parser +systemctl status nginx + +# Service logs +tail -f /home/zvx/ots/logs/opentakserver.log +tail -f /home/zvx/ots/logs/eud_handler_tcp.log +tail -f /home/zvx/ots/logs/eud_handler_ssl.log +tail -f /home/zvx/ots/logs/cot_parser.log +``` + +### Listening Ports +- **80:** nginx (HTTP, redirects to HTTPS) +- **443:** nginx (HTTPS, web UI) +- **8080:** nginx (HTTP Marti API) +- **8088:** eud_handler (TCP CoT) +- **8089:** eud_handler (TLS CoT) +- **8443:** nginx (HTTPS Marti API) +- **8446:** nginx (Admin API) +- **1936:** nginx +- **8322:** nginx +- **8883:** nginx + +### Database Configuration +- **Type:** PostgreSQL +- **Database name:** ots +- **Username:** ots +- **Password:** eCt8HwnBFlZJ6K1arTn4 +- **Connection string:** postgresql+psycopg2://ots:eCt8HwnBFlZJ6K1arTn4@127.0.0.1/ots +- **Service:** postgresql (systemd) + +### Configuration File +- **Location:** /home/zvx/ots/config.yml +- **Backup:** /home/zvx/ots/config.yml.bak (if needed) + +--- + +## Meshtastic MQTT Gateway + +### Overview +OpenTAKServer's native Meshtastic integration allows remote gateway radios to connect via MQTT over TLS. This enables: +- Meshtastic mesh positions to appear as TAK contacts on the map +- Text messages from mesh to flow into TAK chat +- Remote gateway deployment over Starlink, cell hotspots, or any internet connection + +**Setup Date:** April 17, 2026 +**Status:** ✅ Fully Operational + +### Architecture +``` +Meshtastic Radio (GPS) + └─→ MQTT TLS (port 8883) via ots.k7zvx.com + └─→ Nginx Stream Proxy (TLS termination) + └─→ RabbitMQ MQTT (localhost:1883) + └─→ OpenTAKServer (HTTP auth backend) + └─→ PostgreSQL (user database) + └─→ TAK Clients (ATAK/iTAK/WinTAK) +``` + +### Server Configuration + +#### RabbitMQ MQTT Settings +**Location:** `/etc/rabbitmq/rabbitmq.conf` + +```conf +# MQTT listener bound to loopback only (security) +mqtt.listeners.tcp.default = 127.0.0.1:1883 + +# Authentication via OTS HTTP backend +auth_backends.1 = internal +auth_backends.2 = http +auth_http.http_method = post +auth_http.user_path = http://127.0.0.1:8081/api/rabbitmq/auth +auth_http.vhost_path = http://127.0.0.1:8081/api/rabbitmq/vhost +auth_http.resource_path = http://127.0.0.1:8081/api/rabbitmq/resource +auth_http.topic_path = http://127.0.0.1:8081/api/rabbitmq/topic + +# AMQP also restricted to loopback +listeners.tcp.default = 127.0.0.1:5672 +``` + +**Security:** Ports 1883 (MQTT) and 5672 (AMQP) are bound to localhost only. External access is via nginx TLS proxy on port 8883. + +#### Nginx Stream Configuration +**Location:** `/etc/nginx/streams-available/rabbitmq` + +```nginx +server { + listen 8883 ssl; + proxy_pass 127.0.0.1:1883; + + ssl_certificate /home/zvx/ots/ca/certs/opentakserver/opentakserver.pem; + ssl_certificate_key /home/zvx/ots/ca/certs/opentakserver/opentakserver.nopass.key; +} +``` + +**Note:** Currently uses self-signed OTS CA certificate. Meshtastic radios must disable TLS certificate verification ("allow insecure"). + +#### OTS Meshtastic Configuration +**Location:** `/home/zvx/ots/config.yml` + +```yaml +OTS_ENABLE_MESHTASTIC: true +OTS_MESHTASTIC_GROUP: __ANON__ +OTS_MESHTASTIC_TOPIC: opentakserver +OTS_MESHTASTIC_PUBLISH_INTERVAL: 30 +OTS_MESHTASTIC_NODEINFO_INTERVAL: 3 +OTS_MESHTASTIC_DOWNLINK_CHANNELS: [] +``` + +- **Group `__ANON__`:** Broadcasts Meshtastic positions to all connected TAK clients +- **Topic `opentakserver`:** MQTT topic root for all Meshtastic messages +- **Publish interval:** OTS publishes updates to TAK clients every 30 seconds + +### Client Configuration (Meshtastic Radio) + +#### Connection Settings +- **Server/Address:** `ots.k7zvx.com` +- **Port:** Auto (uses 8883 when TLS enabled) +- **Enable TLS/SSL:** ✅ **REQUIRED** +- **TLS Certificate Verification:** ❌ **DISABLED** ("Allow Insecure") *— server uses self-signed cert* + +#### Authentication +- **Username:** `[unique_username]` — created in OTS web UI (Settings → Users) +- **Password:** `[user_password]` — set in OTS web UI + +#### MQTT Settings +- **Topic:** `opentakserver` +- **Encryption Enabled:** ❌ **DISABLED** — **CRITICAL:** OTS cannot decode PSK-encrypted MQTT payloads. The LoRa RF hop between radios is still PSK-encrypted regardless of this setting. This toggle only controls whether the radio re-encrypts before publishing to MQTT. TLS on port 8883 provides wire security instead. +- **JSON Output:** ❌ **DISABLED** — OTS expects protobuf format, not JSON. Leave this off. +- **Proxy to Client:** ❌ **DISABLED** — required per OTS docs; enabling this causes presence/LWT issues with gateway operation +- **Uplink Enabled:** ✅ **REQUIRED** — configured per-channel when scanning QR code from OTS web UI + +#### Position Settings +- **Map Publish:** ❌ **DISABLED** — prevents public map sharing (map.meshtastic.org) +- **Position Enabled:** ✅ **REQUIRED** +- **Position Precision:** Any — full precision is sent to private server regardless of this setting + +**Important:** "Map Publish" only affects the public Meshtastic map. Position data still flows to custom MQTT servers (ots.k7zvx.com) with full precision when disabled. + +### User Setup Process + +1. **Create OTS user account:** + ```bash + # Via OTS Web UI + https://ots.k7zvx.com → Settings → Users → Create User + + # Set username (e.g., "matt", "jeepnjonny") + # Set password + ``` + +2. **Configure Meshtastic radio** with connection settings above + +3. **Test connection:** + - Send a text message from the radio + - Message should appear in iTAK/ATAK within seconds (will appear in triplicate due to mesh retry logic) + +4. **Verify position data:** + - Position updates send based on mesh broadcast interval (typically 15-30 minutes) + - Node should appear as contact on TAK map with accurate location + - Updates continue at configured interval + +### Monitoring & Verification + +#### Check MQTT Connection +```bash +# View RabbitMQ logs for connection activity +sudo tail -f /var/log/rabbitmq/rabbit@opentakserver.log | grep MQTT + +# List active MQTT connections +sudo rabbitmqctl list_connections protocol user peer_host peer_port + +# Monitor MQTT traffic (subscribe to all topics) +mosquitto_sub -h 127.0.0.1 -p 1883 -u [username] -P [password] -t "#" -v +``` + +#### Check OTS Processing +```bash +# Monitor Meshtastic controller logs +sudo tail -f /home/zvx/ots/logs/opentakserver.log | grep -i mesh + +# Check for position data +sudo grep "latitude_i" /home/zvx/ots/logs/opentakserver.log | tail -20 + +# View Meshtastic EUDs in database +sudo -u postgres psql -d ots -c "SELECT callsign, uid, meshtastic_id, last_event_time FROM euds WHERE meshtastic_id IS NOT NULL;" +``` + +### Troubleshooting + +#### Connection Refused +**Symptom:** Radio shows "connection refused" error + +**Causes & Fixes:** +1. **TLS not enabled** — Radio connecting to port 1883 (blocked externally). Enable TLS on radio to use port 8883. +2. **Wrong server** — Verify `ots.k7zvx.com` (not `mqtt.k7zvx.com`) +3. **Firewall** — Verify port 8883 is accessible: `telnet ots.k7zvx.com 8883` + +#### Authentication Failed (CONNACK code 4) +**Symptom:** RabbitMQ logs show "access refused for user" + +**Causes & Fixes:** +1. **User doesn't exist** — Create user in OTS web UI first +2. **Wrong password** — Verify credentials match OTS user settings +3. **Check logs:** + ```bash + sudo grep "MQTT connection failed" /var/log/rabbitmq/rabbit@opentakserver.log + ``` + +#### TLS Certificate Verification Failed +**Symptom:** Connection fails with SSL/certificate errors + +**Fix:** Disable TLS certificate verification on radio ("Allow Insecure" or "Skip Verify") + +**Reason:** OTS uses self-signed certificate. Option to install Let's Encrypt cert exists but not currently implemented. + +#### Radio Connected But No Data +**Symptom:** MQTT connection shows as active but no messages/position in OTS + +**Causes & Fixes:** +1. **Uplink disabled** — Ensure uplink is enabled on the channel (set via QR code from OTS Meshtastic web UI page) +2. **No GPS lock** — Verify radio has satellite fix and coordinates displayed +3. **Wrong topic** — Ensure radio is publishing to `opentakserver` topic +4. **Test with text message** — Send message from radio to verify MQTT publishing works + +#### Node Appears But No Position on Map +**Symptom:** Contact exists in iTAK but no location shown + +**Causes & Fixes:** +1. **No GPS lock** — Radio must have satellite fix to send position +2. **Position broadcasts disabled** — Enable "Position Enabled" on radio +3. **Waiting for interval** — Position updates send less frequently than text messages (typically 15-30 min) +4. **Check logs:** + ```bash + sudo grep "latitude_i\|position" /home/zvx/ots/logs/opentakserver.log | tail -20 + ``` + +#### Unicode Encoding Errors +**Symptom:** OTS logs show `UnicodeEncodeError` related to emojis + +**Fix:** Remove emojis from Meshtastic node name (long name and short name). PostgreSQL database cannot handle emojis with current encoding. + +**Example error:** +``` +UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f3f4' in position 0 +``` + +### Known Behaviors + +#### Triplicate Messages +**Behavior:** Each text message appears 3 times in iTAK + +**Reason:** Meshtastic retry logic sends messages 3 times for reliability over mesh network. OTS forwards all copies. + +**Impact:** Normal behavior, ensures messages get through despite poor mesh coverage. Not configurable without modifying OTS source. + +#### Position Update Frequency +**Mesh vs MQTT:** Meshtastic has separate intervals: +- **Mesh broadcasts:** Position sent to local mesh (15-30 min typical) +- **MQTT publishing:** Position sent to MQTT server (can be same or different) + +Radio may show position locally more often than it publishes to MQTT server. Configure "MQTT Position Interval" separately from mesh broadcast interval. + +#### Map Publish vs Custom MQTT +**Map Publish:** Controls publishing to public Meshtastic map (map.meshtastic.org) +- Has 1-hour minimum interval due to privacy concerns +- Shows degraded precision on public map + +**Custom MQTT (OTS):** Unaffected by map publish settings +- Receives full precision position data +- Updates at configured mesh/MQTT interval +- Completely private — only visible to your TAK clients + +Disabling "Map Publish" does NOT affect position data sent to custom MQTT servers like OTS. + +### Active Gateways + +| Node ID | Callsign | Owner | Channel | Status | +|---------|----------|-------|---------|--------| +| !f861851e | Meshtastic 851e | matt | LFtrackers | Active | + +### Port Summary + +| Port | Service | Bind Address | External Access | Purpose | +|------|---------|--------------|-----------------|---------| +| 1883 | RabbitMQ MQTT | 127.0.0.1 | ❌ No (loopback only) | MQTT plaintext (internal) | +| 5672 | RabbitMQ AMQP | 127.0.0.1 | ❌ No (loopback only) | Internal OTS ↔ RabbitMQ | +| 8883 | Nginx Stream | 0.0.0.0 | ✅ Yes (via ots.k7zvx.com) | MQTT TLS (Meshtastic radios) | + +--- + +## Critical Fixes Applied + +### 1. Ubuntu Mirror Issues +**Problem:** Canonical's GeoDNS is broken for Filer Telephone ISP (Idaho). All DNS servers (local, Google, Cloudflare) returned only UK mirror IPs (91.189.x.x range) which had severe packet loss and timeouts. + +**Root Cause:** +- Your ISP has poor peering with Canonical's UK infrastructure +- GeoDNS should return US mirrors but only returned UK for this region +- 30-50% packet loss between ISP and Canonical's network + +**Fix Applied:** +```bash +# Changed /etc/apt/sources.list to use Oregon mirror +sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list + +# Results: +# - Before: Timeouts (10+ seconds) +# - After: 11 MB/s download speed +``` + +**Testing:** +- archive.ubuntu.com: TIMEOUT +- mirrors.kernel.org: 0.16s response time +- mirror.arizona.edu: 0.31s response time + +### 2. PostgreSQL Driver Compatibility +**Problem:** OpenTAKServer installer uses psycopg3 by default, which has a compatibility bug with SQLAlchemy's PostgreSQL version detection. This causes the service to crash-loop with: +``` +TypeError: cannot use a string pattern on a bytes-like object +``` + +**Fix Applied:** +```bash +# Install psycopg2-binary (stable driver) +pip install psycopg2-binary + +# Update connection string in /home/zvx/ots/config.yml +# Changed: postgresql+psycopg:// +# To: postgresql+psycopg2:// +``` + +**Impact:** Without this fix, the backend API doesn't run and login fails silently. + +--- + +## Troubleshooting + +### Service Won't Start +```bash +# Check service status +systemctl status opentakserver + +# Check logs for errors +tail -50 /home/zvx/ots/logs/opentakserver.log + +# Common issues: +# 1. Database connection errors → check PostgreSQL is running +# 2. Port conflicts → check nginx/other services +# 3. psycopg errors → ensure psycopg2-binary is installed +``` + +### Login Fails +```bash +# Verify backend API is running +systemctl status opentakserver + +# Check if service is crash-looping +journalctl -u opentakserver -f + +# Test local API access +curl -I http://localhost:8080 +curl -I https://localhost:443 +``` + +### Slow Package Updates +```bash +# Verify using fast mirror +grep "URIs:" /etc/apt/sources.list + +# Should show: http://mirrors.kernel.org/ubuntu +# If not, run: +sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list +apt update +``` + +### External Access Issues +```bash +# Test from outside network +curl -I https://ots.k7zvx.com + +# Check DNS resolution +dig +short ots.k7zvx.com @8.8.8.8 +# Should return: 199.6.36.163 + +# Check Caddy is proxying correctly (on CT 101) +systemctl status caddy +journalctl -u caddy -f +``` + +--- + +## Maintenance + +### Update OpenTAKServer +```bash +# Stop services +systemctl stop opentakserver eud_handler eud_handler_ssl cot_parser + +# Activate virtualenv +su - zvx +source ~/.opentakserver_venv/bin/activate + +# Update via pip (if available) +pip install --upgrade opentakserver + +# Or follow official upgrade docs: +# https://docs.opentakserver.io/ + +# Restart services +sudo systemctl start opentakserver eud_handler eud_handler_ssl cot_parser +``` + +### Backup +```bash +# Database backup +sudo -u postgres pg_dump ots > /tmp/ots_backup_$(date +%Y%m%d).sql + +# Configuration backup +cp /home/zvx/ots/config.yml /home/zvx/ots/config.yml.backup + +# Certificate backup (from CT 101) +tar -czf /tmp/ots_certs_$(date +%Y%m%d).tar.gz /etc/caddy/certs/ots.k7zvx.com.* +``` + +### Change Admin Password +1. Log in to https://ots.k7zvx.com with administrator/password +2. Navigate to Settings → Users +3. Select administrator user +4. Change password +5. Save changes + +--- + +## Next Steps + +1. ✅ Change default administrator password +2. ⏳ Configure TAK client connections +3. ⏳ Set up user certificates for client authentication +4. ⏳ Configure data packages and plugins +5. ⏳ Set up monitoring/alerting for the service +6. ⏳ Document backup/restore procedures +7. ⏳ Test failover scenarios + +--- + +## Reference Links + +- **OpenTAKServer Documentation:** https://docs.opentakserver.io/ +- **OpenTAKServer GitHub:** https://github.com/brian7704/OpenTAKServer +- **TAK.gov Resources:** https://tak.gov/ +- **Echo6 Infrastructure Docs:** /home/zvx/projects/.ref/docs/ + +--- + +## Notes + +### Why Not Docker? +Previous attempts to install OpenTAKServer via Docker encountered persistent authentication issues. The native installation method worked successfully. + +### ISP-Specific Issue +The Ubuntu mirror issue is specific to Filer Telephone Company's network peering. This may affect other services in the future. Consider: +- Using mirrors.kernel.org for all Ubuntu-based containers +- Setting up a local apt-cacher-ng if this becomes widespread +- Monitoring ISP routing changes + +### Tailscale Not Involved +Initial suspicion was that Tailscale exit nodes or DNS were causing the mirror issues. Testing confirmed this was false - the issue persists even with local DNS and no Tailscale routing. + +--- + +*Last updated: 2026-04-17* diff --git a/docs/services/services.md b/docs/services/services.md index 06de62f..e7496cb 100644 --- a/docs/services/services.md +++ b/docs/services/services.md @@ -45,6 +45,7 @@ | LiveSync | Contabo | 127.0.0.1:5984 | https://notes.echo6.co | Obsidian sync (CouchDB + provisioner, Docker, JWT auth) | | TAK Server | Contabo | 100.64.0.1:8446/8443/8089 | https://tak.echo6.co | TAK Server (Docker, Authentik forward auth on admin portal) | | SIGIL | Contabo | 100.64.0.1:8990 | https://tak.echo6.co/sigil | TAK management console (Docker, Authentik forward auth) | +| OpenTAKServer (OTS) | utility (CT 109) | 192.168.1.109:443 | https://ots.k7zvx.com | Live TAK server (native install, nginx+RabbitMQ+PostgreSQL, Meshtastic MQTT gateway on port 8883) — see [[ots-setup]] | | Echo6 Cortex Agent | cortex (VM 150) | N/A (Matrix bot) | #cortex:echo6.co in echo6-ops space | Claude Code bridge — @cortex:echo6.co, session continuity, E2EE (systemd) | | Echo6 Contabo Agent | Contabo | N/A (Matrix bot) | #contabo:echo6.co in echo6-ops space | Claude Code bridge — @contabo:echo6.co, session continuity, E2EE (systemd) | | mautrix-signal | Contabo | 29328 (internal) | Internal (matrix-net) | Signal bridge — @signalbot:echo6.co, E2BE, MSC4190, auto-portals |