echo6-docs/vault/docs/services/ots-setup.md

# OpenTAKServer Setup Documentation

## Installation Date
April 17, 2026

## Overview
OpenTAKServer installation on Echo6 infrastructure for TAK (Team Awareness Kit) server capabilities.

---

## Container Specifications

**VMID:** 109
**Hostname:** opentakserver
**Host:** utility Proxmox (192.168.1.241)
**IP Address:** 192.168.1.109/24
**Gateway:** 192.168.1.1
**OS:** Ubuntu 24.04 LTS
**Resources:** 2 cores, 4GB RAM, 8GB disk
**Storage:** local-lvm
**Features:** nesting=1
**Auto-start:** enabled

---

## Access Information

### External Access
- **URL:** https://ots.k7zvx.com
- **Admin Username:** administrator
- **Admin Password:** password (**CHANGE THIS IMMEDIATELY**)

### SSH Access
```bash
# Via Proxmox host
ssh root@192.168.1.241 'pct exec 109 -- bash'

# As zvx user
ssh root@192.168.1.241 'pct exec 109 -- su - zvx'

# Direct container entry
pct enter 109
```

**Container User:**
- Username: zvx
- Password: 7redditGold
- Sudo: NOPASSWD enabled

---

## Network Configuration

### DNS
- **Domain:** ots.k7zvx.com
- **DNS Provider:** GoDaddy
- **Record Type:** A
- **Points to:** 199.6.36.163 (home external IP)
- **TTL:** 3600 seconds

### SSL Certificate
- **Provider:** Let's Encrypt
- **Method:** acme.sh with GoDaddy DNS validation
- **Location:** /etc/caddy/certs/ots.k7zvx.com.* (on CT 101)
- **Auto-renewal:** Configured via acme.sh

### Reverse Proxy (Caddy on CT 101)
```
ots.k7zvx.com {
    tls /etc/caddy/certs/ots.k7zvx.com.fullchain.crt /etc/caddy/certs/ots.k7zvx.com.key
    reverse_proxy https://192.168.1.109:443 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}
```

### Port Forwarding
- **Router:** Ubiquiti firewall (192.168.1.28)
- **External ports:** 80/443 → 192.168.1.101 (Caddy CT)
- **Internal proxy:** Caddy → 192.168.1.109:443 (OpenTAKServer)

---

## OpenTAKServer Configuration

### Installation Details
- **Installation Method:** Official installer script
- **Install User:** zvx
- **Install Path:** /home/zvx/ots/
- **Virtual Environment:** /home/zvx/.opentakserver_venv/
- **Version:** 1.7.10

### Services
```bash
# Main services
systemctl status opentakserver
systemctl status eud_handler
systemctl status eud_handler_ssl
systemctl status cot_parser
systemctl status nginx

# Service logs
tail -f /home/zvx/ots/logs/opentakserver.log
tail -f /home/zvx/ots/logs/eud_handler_tcp.log
tail -f /home/zvx/ots/logs/eud_handler_ssl.log
tail -f /home/zvx/ots/logs/cot_parser.log
```

### Listening Ports
- **80:** nginx (HTTP, redirects to HTTPS)
- **443:** nginx (HTTPS, web UI)
- **8080:** nginx (HTTP Marti API)
- **8088:** eud_handler (TCP CoT)
- **8089:** eud_handler (TLS CoT)
- **8443:** nginx (HTTPS Marti API)
- **8446:** nginx (Admin API)
- **1936:** nginx
- **8322:** nginx
- **8883:** nginx

### Database Configuration
- **Type:** PostgreSQL
- **Database name:** ots
- **Username:** ots
- **Password:** eCt8HwnBFlZJ6K1arTn4
- **Connection string:** postgresql+psycopg2://ots:eCt8HwnBFlZJ6K1arTn4@127.0.0.1/ots
- **Service:** postgresql (systemd)

### Configuration File
- **Location:** /home/zvx/ots/config.yml
- **Backup:** /home/zvx/ots/config.yml.bak (if needed)

---

## Meshtastic MQTT Gateway

### Overview
OpenTAKServer's native Meshtastic integration allows remote gateway radios to connect via MQTT over TLS. This enables:
- Meshtastic mesh positions to appear as TAK contacts on the map
- Text messages from mesh to flow into TAK chat
- Remote gateway deployment over Starlink, cell hotspots, or any internet connection

**Setup Date:** April 17, 2026
**Status:** ✅ Fully Operational

### Architecture
```
Meshtastic Radio (GPS)
  └─→ MQTT TLS (port 8883) via ots.k7zvx.com
      └─→ Nginx Stream Proxy (TLS termination)
          └─→ RabbitMQ MQTT (localhost:1883)
              └─→ OpenTAKServer (HTTP auth backend)
                  └─→ PostgreSQL (user database)
                      └─→ TAK Clients (ATAK/iTAK/WinTAK)
```

### Server Configuration

#### RabbitMQ MQTT Settings
**Location:** `/etc/rabbitmq/rabbitmq.conf`

```conf
# MQTT listener bound to loopback only (security)
mqtt.listeners.tcp.default = 127.0.0.1:1883

# Authentication via OTS HTTP backend
auth_backends.1 = internal
auth_backends.2 = http
auth_http.http_method   = post
auth_http.user_path     = http://127.0.0.1:8081/api/rabbitmq/auth
auth_http.vhost_path    = http://127.0.0.1:8081/api/rabbitmq/vhost
auth_http.resource_path = http://127.0.0.1:8081/api/rabbitmq/resource
auth_http.topic_path    = http://127.0.0.1:8081/api/rabbitmq/topic

# AMQP also restricted to loopback
listeners.tcp.default = 127.0.0.1:5672
```

**Security:** Ports 1883 (MQTT) and 5672 (AMQP) are bound to localhost only. External access is via nginx TLS proxy on port 8883.

#### Nginx Stream Configuration
**Location:** `/etc/nginx/streams-available/rabbitmq`

```nginx
server {
    listen     8883 ssl;
    proxy_pass 127.0.0.1:1883;

    ssl_certificate /home/zvx/ots/ca/certs/opentakserver/opentakserver.pem;
    ssl_certificate_key /home/zvx/ots/ca/certs/opentakserver/opentakserver.nopass.key;
}
```

**Note:** Currently uses self-signed OTS CA certificate. Meshtastic radios must disable TLS certificate verification ("allow insecure").

#### OTS Meshtastic Configuration
**Location:** `/home/zvx/ots/config.yml`

```yaml
OTS_ENABLE_MESHTASTIC: true
OTS_MESHTASTIC_GROUP: __ANON__
OTS_MESHTASTIC_TOPIC: opentakserver
OTS_MESHTASTIC_PUBLISH_INTERVAL: 30
OTS_MESHTASTIC_NODEINFO_INTERVAL: 3
OTS_MESHTASTIC_DOWNLINK_CHANNELS: []
```

- **Group `__ANON__`:** Broadcasts Meshtastic positions to all connected TAK clients
- **Topic `opentakserver`:** MQTT topic root for all Meshtastic messages
- **Publish interval:** OTS publishes updates to TAK clients every 30 seconds

### Client Configuration (Meshtastic Radio)

#### Connection Settings
- **Server/Address:** `ots.k7zvx.com`
- **Port:** Auto (uses 8883 when TLS enabled)
- **Enable TLS/SSL:** ✅ **REQUIRED**
- **TLS Certificate Verification:** ❌ **DISABLED** ("Allow Insecure") *— server uses self-signed cert*

#### Authentication
- **Username:** `[unique_username]` — created in OTS web UI (Settings → Users)
- **Password:** `[user_password]` — set in OTS web UI

#### MQTT Settings
- **Topic:** `opentakserver`
- **Encryption Enabled:** ❌ **DISABLED** — **CRITICAL:** OTS cannot decode PSK-encrypted MQTT payloads. The LoRa RF hop between radios is still PSK-encrypted regardless of this setting. This toggle only controls whether the radio re-encrypts before publishing to MQTT. TLS on port 8883 provides wire security instead.
- **JSON Output:** ❌ **DISABLED** — OTS expects protobuf format, not JSON. Leave this off.
- **Proxy to Client:** ❌ **DISABLED** — required per OTS docs; enabling this causes presence/LWT issues with gateway operation
- **Uplink Enabled:** ✅ **REQUIRED** — configured per-channel when scanning QR code from OTS web UI

#### Position Settings
- **Map Publish:** ❌ **DISABLED** — prevents public map sharing (map.meshtastic.org)
- **Position Enabled:** ✅ **REQUIRED**
- **Position Precision:** Any — full precision is sent to private server regardless of this setting

**Important:** "Map Publish" only affects the public Meshtastic map. Position data still flows to custom MQTT servers (ots.k7zvx.com) with full precision when disabled.

### User Setup Process

1. **Create OTS user account:**
   ```bash
   # Via OTS Web UI
   https://ots.k7zvx.com → Settings → Users → Create User

   # Set username (e.g., "matt", "jeepnjonny")
   # Set password
   ```

2. **Configure Meshtastic radio** with connection settings above

3. **Test connection:**
   - Send a text message from the radio
   - Message should appear in iTAK/ATAK within seconds (will appear in triplicate due to mesh retry logic)

4. **Verify position data:**
   - Position updates send based on mesh broadcast interval (typically 15-30 minutes)
   - Node should appear as contact on TAK map with accurate location
   - Updates continue at configured interval

### Monitoring & Verification

#### Check MQTT Connection
```bash
# View RabbitMQ logs for connection activity
sudo tail -f /var/log/rabbitmq/rabbit@opentakserver.log | grep MQTT

# List active MQTT connections
sudo rabbitmqctl list_connections protocol user peer_host peer_port

# Monitor MQTT traffic (subscribe to all topics)
mosquitto_sub -h 127.0.0.1 -p 1883 -u [username] -P [password] -t "#" -v
```

#### Check OTS Processing
```bash
# Monitor Meshtastic controller logs
sudo tail -f /home/zvx/ots/logs/opentakserver.log | grep -i mesh

# Check for position data
sudo grep "latitude_i" /home/zvx/ots/logs/opentakserver.log | tail -20

# View Meshtastic EUDs in database
sudo -u postgres psql -d ots -c "SELECT callsign, uid, meshtastic_id, last_event_time FROM euds WHERE meshtastic_id IS NOT NULL;"
```

### Troubleshooting

#### Connection Refused
**Symptom:** Radio shows "connection refused" error

**Causes & Fixes:**
1. **TLS not enabled** — Radio connecting to port 1883 (blocked externally). Enable TLS on radio to use port 8883.
2. **Wrong server** — Verify `ots.k7zvx.com` (not `mqtt.k7zvx.com`)
3. **Firewall** — Verify port 8883 is accessible: `telnet ots.k7zvx.com 8883`

#### Authentication Failed (CONNACK code 4)
**Symptom:** RabbitMQ logs show "access refused for user"

**Causes & Fixes:**
1. **User doesn't exist** — Create user in OTS web UI first
2. **Wrong password** — Verify credentials match OTS user settings
3. **Check logs:**
   ```bash
   sudo grep "MQTT connection failed" /var/log/rabbitmq/rabbit@opentakserver.log
   ```

#### TLS Certificate Verification Failed
**Symptom:** Connection fails with SSL/certificate errors

**Fix:** Disable TLS certificate verification on radio ("Allow Insecure" or "Skip Verify")

**Reason:** OTS uses self-signed certificate. Option to install Let's Encrypt cert exists but not currently implemented.

#### Radio Connected But No Data
**Symptom:** MQTT connection shows as active but no messages/position in OTS

**Causes & Fixes:**
1. **Uplink disabled** — Ensure uplink is enabled on the channel (set via QR code from OTS Meshtastic web UI page)
2. **No GPS lock** — Verify radio has satellite fix and coordinates displayed
3. **Wrong topic** — Ensure radio is publishing to `opentakserver` topic
4. **Test with text message** — Send message from radio to verify MQTT publishing works

#### Node Appears But No Position on Map
**Symptom:** Contact exists in iTAK but no location shown

**Causes & Fixes:**
1. **No GPS lock** — Radio must have satellite fix to send position
2. **Position broadcasts disabled** — Enable "Position Enabled" on radio
3. **Waiting for interval** — Position updates send less frequently than text messages (typically 15-30 min)
4. **Check logs:**
   ```bash
   sudo grep "latitude_i\|position" /home/zvx/ots/logs/opentakserver.log | tail -20
   ```

#### Unicode Encoding Errors
**Symptom:** OTS logs show `UnicodeEncodeError` related to emojis

**Fix:** Remove emojis from Meshtastic node name (long name and short name). PostgreSQL database cannot handle emojis with current encoding.

**Example error:**
```
UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f3f4' in position 0
```

### Known Behaviors

#### Triplicate Messages
**Behavior:** Each text message appears 3 times in iTAK

**Reason:** Meshtastic retry logic sends messages 3 times for reliability over mesh network. OTS forwards all copies.

**Impact:** Normal behavior, ensures messages get through despite poor mesh coverage. Not configurable without modifying OTS source.

#### Position Update Frequency
**Mesh vs MQTT:** Meshtastic has separate intervals:
- **Mesh broadcasts:** Position sent to local mesh (15-30 min typical)
- **MQTT publishing:** Position sent to MQTT server (can be same or different)

Radio may show position locally more often than it publishes to MQTT server. Configure "MQTT Position Interval" separately from mesh broadcast interval.

#### Map Publish vs Custom MQTT
**Map Publish:** Controls publishing to public Meshtastic map (map.meshtastic.org)
- Has 1-hour minimum interval due to privacy concerns
- Shows degraded precision on public map

**Custom MQTT (OTS):** Unaffected by map publish settings
- Receives full precision position data
- Updates at configured mesh/MQTT interval
- Completely private — only visible to your TAK clients

Disabling "Map Publish" does NOT affect position data sent to custom MQTT servers like OTS.

### Active Gateways

| Node ID | Callsign | Owner | Channel | Status |
|---------|----------|-------|---------|--------|
| !f861851e | Meshtastic 851e | matt | LFtrackers | Active |

### Port Summary

| Port | Service | Bind Address | External Access | Purpose |
|------|---------|--------------|-----------------|---------|
| 1883 | RabbitMQ MQTT | 127.0.0.1 | ❌ No (loopback only) | MQTT plaintext (internal) |
| 5672 | RabbitMQ AMQP | 127.0.0.1 | ❌ No (loopback only) | Internal OTS ↔ RabbitMQ |
| 8883 | Nginx Stream | 0.0.0.0 | ✅ Yes (via ots.k7zvx.com) | MQTT TLS (Meshtastic radios) |

---

## Critical Fixes Applied

### 1. Ubuntu Mirror Issues
**Problem:** Canonical's GeoDNS is broken for Filer Telephone ISP (Idaho). All DNS servers (local, Google, Cloudflare) returned only UK mirror IPs (91.189.x.x range) which had severe packet loss and timeouts.

**Root Cause:**
- Your ISP has poor peering with Canonical's UK infrastructure
- GeoDNS should return US mirrors but only returned UK for this region
- 30-50% packet loss between ISP and Canonical's network

**Fix Applied:**
```bash
# Changed /etc/apt/sources.list to use Oregon mirror
sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list

# Results:
# - Before: Timeouts (10+ seconds)
# - After: 11 MB/s download speed
```

**Testing:**
- archive.ubuntu.com: TIMEOUT
- mirrors.kernel.org: 0.16s response time
- mirror.arizona.edu: 0.31s response time

### 2. PostgreSQL Driver Compatibility
**Problem:** OpenTAKServer installer uses psycopg3 by default, which has a compatibility bug with SQLAlchemy's PostgreSQL version detection. This causes the service to crash-loop with:
```
TypeError: cannot use a string pattern on a bytes-like object
```

**Fix Applied:**
```bash
# Install psycopg2-binary (stable driver)
pip install psycopg2-binary

# Update connection string in /home/zvx/ots/config.yml
# Changed: postgresql+psycopg://
# To:      postgresql+psycopg2://
```

**Impact:** Without this fix, the backend API doesn't run and login fails silently.

---

## Troubleshooting

### Service Won't Start
```bash
# Check service status
systemctl status opentakserver

# Check logs for errors
tail -50 /home/zvx/ots/logs/opentakserver.log

# Common issues:
# 1. Database connection errors → check PostgreSQL is running
# 2. Port conflicts → check nginx/other services
# 3. psycopg errors → ensure psycopg2-binary is installed
```

### Login Fails
```bash
# Verify backend API is running
systemctl status opentakserver

# Check if service is crash-looping
journalctl -u opentakserver -f

# Test local API access
curl -I http://localhost:8080
curl -I https://localhost:443
```

### Slow Package Updates
```bash
# Verify using fast mirror
grep "URIs:" /etc/apt/sources.list

# Should show: http://mirrors.kernel.org/ubuntu
# If not, run:
sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list
apt update
```

### External Access Issues
```bash
# Test from outside network
curl -I https://ots.k7zvx.com

# Check DNS resolution
dig +short ots.k7zvx.com @8.8.8.8
# Should return: 199.6.36.163

# Check Caddy is proxying correctly (on CT 101)
systemctl status caddy
journalctl -u caddy -f
```

---

## Maintenance

### Update OpenTAKServer
```bash
# Stop services
systemctl stop opentakserver eud_handler eud_handler_ssl cot_parser

# Activate virtualenv
su - zvx
source ~/.opentakserver_venv/bin/activate

# Update via pip (if available)
pip install --upgrade opentakserver

# Or follow official upgrade docs:
# https://docs.opentakserver.io/

# Restart services
sudo systemctl start opentakserver eud_handler eud_handler_ssl cot_parser
```

### Backup
```bash
# Database backup
sudo -u postgres pg_dump ots > /tmp/ots_backup_$(date +%Y%m%d).sql

# Configuration backup
cp /home/zvx/ots/config.yml /home/zvx/ots/config.yml.backup

# Certificate backup (from CT 101)
tar -czf /tmp/ots_certs_$(date +%Y%m%d).tar.gz /etc/caddy/certs/ots.k7zvx.com.*
```

### Change Admin Password
1. Log in to https://ots.k7zvx.com with administrator/password
2. Navigate to Settings → Users
3. Select administrator user
4. Change password
5. Save changes

---

## Next Steps

1. ✅ Change default administrator password
2. ⏳ Configure TAK client connections
3. ⏳ Set up user certificates for client authentication
4. ⏳ Configure data packages and plugins
5. ⏳ Set up monitoring/alerting for the service
6. ⏳ Document backup/restore procedures
7. ⏳ Test failover scenarios

---

## Reference Links

- **OpenTAKServer Documentation:** https://docs.opentakserver.io/
- **OpenTAKServer GitHub:** https://github.com/brian7704/OpenTAKServer
- **TAK.gov Resources:** https://tak.gov/
- **Echo6 Infrastructure Docs:** /home/zvx/projects/.ref/docs/

---

## Notes

### Why Not Docker?
Previous attempts to install OpenTAKServer via Docker encountered persistent authentication issues. The native installation method worked successfully.

### ISP-Specific Issue
The Ubuntu mirror issue is specific to Filer Telephone Company's network peering. This may affect other services in the future. Consider:
- Using mirrors.kernel.org for all Ubuntu-based containers
- Setting up a local apt-cacher-ng if this becomes widespread
- Monitoring ISP routing changes

### Tailscale Not Involved
Initial suspicion was that Tailscale exit nodes or DNS were causing the mirror issues. Testing confirmed this was false - the issue persists even with local DNS and no Tailscale routing.

---

*Last updated: 2026-04-17*