echo6-docs/vault/docs/services/ots-setup.md

title	type	tags	aliases	related	updated
OpenTAKServer Setup Documentation	reference	proxmox		ct-runbook caddy ip-allocation environment synapse	2026-06-18

OpenTAKServer Setup Documentation

Installation Date

April 17, 2026

Overview

OpenTAKServer installation on Echo6 infrastructure for TAK (Team Awareness Kit) server capabilities.

---

Container Specifications

VMID: 109 Hostname: opentakserver Host: utility Proxmox (192.168.1.241) IP Address: 192.168.1.109/24 Gateway: 192.168.1.1 OS: Ubuntu 24.04 LTS Resources: 2 cores, 4GB RAM, 8GB disk Storage: local-lvm Features: nesting=1 Auto-start: enabled

---

Access Information

External Access

• URL: https://ots.k7zvx.com
• Admin Username: administrator
• Admin Password: password (CHANGE THIS IMMEDIATELY)

SSH Access

# Via Proxmox host
ssh root@192.168.1.241 'pct exec 109 -- bash'

# As zvx user
ssh root@192.168.1.241 'pct exec 109 -- su - zvx'

# Direct container entry
pct enter 109

Container User:

• Username: zvx
• Password: 7redditGold
• Sudo: NOPASSWD enabled

---

Network Configuration

DNS

• Domain: ots.k7zvx.com
• dns Provider: GoDaddy
• Record Type: A
• Points to: 199.6.36.163 (home external IP)
• TTL: 3600 seconds

SSL Certificate

• Provider: Let's Encrypt
• Method: acme.sh with GoDaddy DNS validation
• Location: /etc/caddy/certs/ots.k7zvx.com.* (on CT 101)
• Auto-renewal: Configured via acme.sh

Reverse Proxy (Caddy on CT 101)

ots.k7zvx.com {
    tls /etc/caddy/certs/ots.k7zvx.com.fullchain.crt /etc/caddy/certs/ots.k7zvx.com.key
    reverse_proxy https://192.168.1.109:443 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

Port Forwarding

• Router: Ubiquiti firewall (192.168.1.28)
• External ports: 80/443 → 192.168.1.101 (Caddy CT)
• Internal proxy: Caddy → 192.168.1.109:443 (OpenTAKServer)

---

OpenTAKServer Configuration

Installation Details

• Installation Method: Official installer script
• Install User: zvx
• Install Path: /home/zvx/ots/
• Virtual environment: /home/zvx/.opentakserver_venv/
• Version: 1.7.10

Services

# Main services
systemctl status opentakserver
systemctl status eud_handler
systemctl status eud_handler_ssl
systemctl status cot_parser
systemctl status nginx

# Service logs
tail -f /home/zvx/ots/logs/opentakserver.log
tail -f /home/zvx/ots/logs/eud_handler_tcp.log
tail -f /home/zvx/ots/logs/eud_handler_ssl.log
tail -f /home/zvx/ots/logs/cot_parser.log

Listening Ports

• 80: nginx (HTTP, redirects to HTTPS)
• 443: nginx (HTTPS, web UI)
• 8080: nginx (HTTP Marti API)
• 8088: eud_handler (TCP CoT)
• 8089: eud_handler (TLS CoT)
• 8443: nginx (HTTPS Marti API)
• 8446: nginx (Admin API)
• 1936: nginx
• 8322: nginx
• 8883: nginx

Database Configuration

• Type: PostgreSQL
• Database name: ots
• Username: ots
• Password: eCt8HwnBFlZJ6K1arTn4
• Connection string: postgresql+psycopg2://ots:eCt8HwnBFlZJ6K1arTn4@127.0.0.1/ots
• Service: postgresql (systemd)

Configuration File

• Location: /home/zvx/ots/config.yml
• Backup: /home/zvx/ots/config.yml.bak (if needed)

---

Meshtastic MQTT Gateway

Overview

OpenTAKServer's native Meshtastic integration allows remote gateway radios to connect via MQTT over TLS. This enables:

• Meshtastic mesh positions to appear as TAK contacts on the map
• Text messages from mesh to flow into TAK chat
• Remote gateway deployment over Starlink, cell hotspots, or any internet connection

Setup Date: April 17, 2026 Status: ✅ Fully Operational

Architecture

Meshtastic Radio (GPS)
  └─→ MQTT TLS (port 8883) via ots.k7zvx.com
      └─→ Nginx Stream Proxy (TLS termination)
          └─→ RabbitMQ MQTT (localhost:1883)
              └─→ OpenTAKServer (HTTP auth backend)
                  └─→ PostgreSQL (user database)
                      └─→ TAK Clients (ATAK/iTAK/WinTAK)

Server Configuration

RabbitMQ MQTT Settings

Location: /etc/rabbitmq/rabbitmq.conf

# MQTT listener bound to loopback only (security)
mqtt.listeners.tcp.default = 127.0.0.1:1883

# Authentication via OTS HTTP backend
auth_backends.1 = internal
auth_backends.2 = http
auth_http.http_method   = post
auth_http.user_path     = http://127.0.0.1:8081/api/rabbitmq/auth
auth_http.vhost_path    = http://127.0.0.1:8081/api/rabbitmq/vhost
auth_http.resource_path = http://127.0.0.1:8081/api/rabbitmq/resource
auth_http.topic_path    = http://127.0.0.1:8081/api/rabbitmq/topic

# AMQP also restricted to loopback
listeners.tcp.default = 127.0.0.1:5672

Security: Ports 1883 (MQTT) and 5672 (AMQP) are bound to localhost only. External access is via nginx TLS proxy on port 8883.

Nginx Stream Configuration

Location: /etc/nginx/streams-available/rabbitmq

server {
    listen     8883 ssl;
    proxy_pass 127.0.0.1:1883;

    ssl_certificate /home/zvx/ots/ca/certs/opentakserver/opentakserver.pem;
    ssl_certificate_key /home/zvx/ots/ca/certs/opentakserver/opentakserver.nopass.key;
}

Note: Currently uses self-signed OTS CA certificate. Meshtastic radios must disable TLS certificate verification ("allow insecure").

OTS Meshtastic Configuration

Location: /home/zvx/ots/config.yml

OTS_ENABLE_MESHTASTIC: true
OTS_MESHTASTIC_GROUP: __ANON__
OTS_MESHTASTIC_TOPIC: opentakserver
OTS_MESHTASTIC_PUBLISH_INTERVAL: 30
OTS_MESHTASTIC_NODEINFO_INTERVAL: 3
OTS_MESHTASTIC_DOWNLINK_CHANNELS: []

• Group __ANON__: Broadcasts Meshtastic positions to all connected TAK clients
• Topic opentakserver: MQTT topic root for all Meshtastic messages
• Publish interval: OTS publishes updates to TAK clients every 30 seconds

Client Configuration (Meshtastic Radio)

Connection Settings

• Server/Address: ots.k7zvx.com
• Port: Auto (uses 8883 when TLS enabled)
• Enable TLS/SSL: ✅ REQUIRED
• TLS Certificate Verification: ❌ DISABLED ("Allow Insecure") — server uses self-signed cert

Authentication

• Username: [unique_username] — created in OTS web UI (Settings → Users)
• Password: [user_password] — set in OTS web UI

MQTT Settings

• Topic: opentakserver
• Encryption Enabled: ❌ DISABLED — CRITICAL: OTS cannot decode PSK-encrypted MQTT payloads. The LoRa RF hop between radios is still PSK-encrypted regardless of this setting. This toggle only controls whether the radio re-encrypts before publishing to MQTT. TLS on port 8883 provides wire security instead.
• JSON Output: ❌ DISABLED — OTS expects protobuf format, not JSON. Leave this off.
• Proxy to Client: ❌ DISABLED — required per OTS docs; enabling this causes presence/LWT issues with gateway operation
• Uplink Enabled: ✅ REQUIRED — configured per-channel when scanning QR code from OTS web UI

Position Settings

• Map Publish: ❌ DISABLED — prevents public map sharing (map.meshtastic.org)
• Position Enabled: ✅ REQUIRED
• Position Precision: Any — full precision is sent to private server regardless of this setting

Important: "Map Publish" only affects the public Meshtastic map. Position data still flows to custom MQTT servers (ots.k7zvx.com) with full precision when disabled.

User Setup Process

1. Create OTS user account:

   # Via OTS Web UI
   https://ots.k7zvx.com → Settings → Users → Create User
   
   # Set username (e.g., "matt", "jeepnjonny")
   # Set password
   

2. Configure Meshtastic radio with connection settings above

3. Test connection:

   • Send a text message from the radio
   • Message should appear in iTAK/ATAK within seconds (will appear in triplicate due to mesh retry logic)

4. Verify position data:

   • Position updates send based on mesh broadcast interval (typically 15-30 minutes)
   • Node should appear as contact on TAK map with accurate location
   • Updates continue at configured interval

Monitoring & Verification

Check MQTT Connection

# View RabbitMQ logs for connection activity
sudo tail -f /var/log/rabbitmq/rabbit@opentakserver.log | grep MQTT

# List active MQTT connections
sudo rabbitmqctl list_connections protocol user peer_host peer_port

# Monitor MQTT traffic (subscribe to all topics)
mosquitto_sub -h 127.0.0.1 -p 1883 -u [username] -P [password] -t "#" -v

Check OTS Processing

# Monitor Meshtastic controller logs
sudo tail -f /home/zvx/ots/logs/opentakserver.log | grep -i mesh

# Check for position data
sudo grep "latitude_i" /home/zvx/ots/logs/opentakserver.log | tail -20

# View Meshtastic EUDs in database
sudo -u postgres psql -d ots -c "SELECT callsign, uid, meshtastic_id, last_event_time FROM euds WHERE meshtastic_id IS NOT NULL;"

Troubleshooting

Connection Refused

Symptom: Radio shows "connection refused" error

Causes & Fixes:

1. TLS not enabled — Radio connecting to port 1883 (blocked externally). Enable TLS on radio to use port 8883.
2. Wrong server — Verify ots.k7zvx.com (not mqtt.k7zvx.com)
3. Firewall — Verify port 8883 is accessible: telnet ots.k7zvx.com 8883

Authentication Failed (CONNACK code 4)

Symptom: RabbitMQ logs show "access refused for user"

Causes & Fixes:

1. User doesn't exist — Create user in OTS web UI first
2. Wrong password — Verify credentials match OTS user settings
3. Check logs:

   sudo grep "MQTT connection failed" /var/log/rabbitmq/rabbit@opentakserver.log
   

TLS Certificate Verification Failed

Symptom: Connection fails with SSL/certificate errors

Fix: Disable TLS certificate verification on radio ("Allow Insecure" or "Skip Verify")

Reason: OTS uses self-signed certificate. Option to install Let's Encrypt cert exists but not currently implemented.

Radio Connected But No Data

Symptom: MQTT connection shows as active but no messages/position in OTS

Causes & Fixes:

1. Uplink disabled — Ensure uplink is enabled on the channel (set via QR code from OTS Meshtastic web UI page)
2. No GPS lock — Verify radio has satellite fix and coordinates displayed
3. Wrong topic — Ensure radio is publishing to opentakserver topic
4. Test with text message — Send message from radio to verify MQTT publishing works

Node Appears But No Position on Map

Symptom: Contact exists in iTAK but no location shown

Causes & Fixes:

1. No GPS lock — Radio must have satellite fix to send position
2. Position broadcasts disabled — Enable "Position Enabled" on radio
3. Waiting for interval — Position updates send less frequently than text messages (typically 15-30 min)
4. Check logs:

   sudo grep "latitude_i\|position" /home/zvx/ots/logs/opentakserver.log | tail -20
   

Unicode Encoding Errors

Symptom: OTS logs show UnicodeEncodeError related to emojis

Fix: Remove emojis from Meshtastic node name (long name and short name). PostgreSQL database cannot handle emojis with current encoding.

Example error:

UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f3f4' in position 0

Known Behaviors

Triplicate Messages

Behavior: Each text message appears 3 times in iTAK

Reason: Meshtastic retry logic sends messages 3 times for reliability over mesh network. OTS forwards all copies.

Impact: Normal behavior, ensures messages get through despite poor mesh coverage. Not configurable without modifying OTS source.

Position Update Frequency

Mesh vs MQTT: Meshtastic has separate intervals:

• Mesh broadcasts: Position sent to local mesh (15-30 min typical)
• MQTT publishing: Position sent to MQTT server (can be same or different)

Radio may show position locally more often than it publishes to MQTT server. Configure "MQTT Position Interval" separately from mesh broadcast interval.

Map Publish vs Custom MQTT

Map Publish: Controls publishing to public Meshtastic map (map.meshtastic.org)

• Has 1-hour minimum interval due to privacy concerns
• Shows degraded precision on public map

Custom MQTT (OTS): Unaffected by map publish settings

• Receives full precision position data
• Updates at configured mesh/MQTT interval
• Completely private — only visible to your TAK clients

Disabling "Map Publish" does NOT affect position data sent to custom MQTT servers like OTS.

Active Gateways

Node ID	Callsign	Owner	Channel	Status
!f861851e	Meshtastic 851e	matt	LFtrackers	Active

Port Summary

Port	Service	Bind Address	External Access	Purpose
1883	RabbitMQ MQTT	127.0.0.1	❌ No (loopback only)	MQTT plaintext (internal)
5672	RabbitMQ AMQP	127.0.0.1	❌ No (loopback only)	Internal OTS ↔ RabbitMQ
8883	Nginx Stream	0.0.0.0	✅ Yes (via ots.k7zvx.com)	MQTT TLS (Meshtastic radios)

---

Critical Fixes Applied

1. Ubuntu Mirror Issues

Problem: Canonical's GeoDNS is broken for Filer Telephone ISP (Idaho). All DNS servers (local, Google, Cloudflare) returned only UK mirror IPs (91.189.x.x range) which had severe packet loss and timeouts.

Root Cause:

• Your ISP has poor peering with Canonical's UK infrastructure
• GeoDNS should return US mirrors but only returned UK for this region
• 30-50% packet loss between ISP and Canonical's network

Fix Applied:

# Changed /etc/apt/sources.list to use Oregon mirror
sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list

# Results:
# - Before: Timeouts (10+ seconds)
# - After: 11 MB/s download speed

Testing:

• archive.ubuntu.com: TIMEOUT
• mirrors.kernel.org: 0.16s response time
• mirror.arizona.edu: 0.31s response time

2. PostgreSQL Driver Compatibility

Problem: OpenTAKServer installer uses psycopg3 by default, which has a compatibility bug with SQLAlchemy's PostgreSQL version detection. This causes the service to crash-loop with:

TypeError: cannot use a string pattern on a bytes-like object

Fix Applied:

# Install psycopg2-binary (stable driver)
pip install psycopg2-binary

# Update connection string in /home/zvx/ots/config.yml
# Changed: postgresql+psycopg://
# To:      postgresql+psycopg2://

Impact: Without this fix, the backend API doesn't run and login fails silently.

---

Troubleshooting

Service Won't Start

# Check service status
systemctl status opentakserver

# Check logs for errors
tail -50 /home/zvx/ots/logs/opentakserver.log

# Common issues:
# 1. Database connection errors → check PostgreSQL is running
# 2. Port conflicts → check nginx/other services
# 3. psycopg errors → ensure psycopg2-binary is installed

Login Fails

# Verify backend API is running
systemctl status opentakserver

# Check if service is crash-looping
journalctl -u opentakserver -f

# Test local API access
curl -I http://localhost:8080
curl -I https://localhost:443

Slow Package Updates

# Verify using fast mirror
grep "URIs:" /etc/apt/sources.list

# Should show: http://mirrors.kernel.org/ubuntu
# If not, run:
sed -i "s|http://archive.ubuntu.com/ubuntu|http://mirrors.kernel.org/ubuntu|g" /etc/apt/sources.list
apt update

External Access Issues

# Test from outside network
curl -I https://ots.k7zvx.com

# Check DNS resolution
dig +short ots.k7zvx.com @8.8.8.8
# Should return: 199.6.36.163

# Check Caddy is proxying correctly (on CT 101)
systemctl status caddy
journalctl -u caddy -f

---

Maintenance

Update OpenTAKServer

# Stop services
systemctl stop opentakserver eud_handler eud_handler_ssl cot_parser

# Activate virtualenv
su - zvx
source ~/.opentakserver_venv/bin/activate

# Update via pip (if available)
pip install --upgrade opentakserver

# Or follow official upgrade docs:
# https://docs.opentakserver.io/

# Restart services
sudo systemctl start opentakserver eud_handler eud_handler_ssl cot_parser

Backup

# Database backup
sudo -u postgres pg_dump ots > /tmp/ots_backup_$(date +%Y%m%d).sql

# Configuration backup
cp /home/zvx/ots/config.yml /home/zvx/ots/config.yml.backup

# Certificate backup (from CT 101)
tar -czf /tmp/ots_certs_$(date +%Y%m%d).tar.gz /etc/caddy/certs/ots.k7zvx.com.*

Change Admin Password

1. Log in to https://ots.k7zvx.com with administrator/password
2. Navigate to Settings → Users
3. Select administrator user
4. Change password
5. Save changes

---

Next Steps

1. ✅ Change default administrator password
2. ⏳ Configure TAK client connections
3. ⏳ Set up user certificates for client authentication
4. ⏳ Configure data packages and plugins
5. ⏳ Set up monitoring/alerting for the service
6. ⏳ Document backup/restore procedures
7. ⏳ Test failover scenarios

---

Reference Links

• OpenTAKServer Documentation: https://docs.opentakserver.io/
• OpenTAKServer GitHub: https://github.com/brian7704/OpenTAKServer
• TAK.gov Resources: https://tak.gov/
• Echo6 Infrastructure Docs: /home/zvx/projects/.ref/docs/

---

Notes

Why Not Docker?

Previous attempts to install OpenTAKServer via Docker encountered persistent authentication issues. The native installation method worked successfully.

ISP-Specific Issue

The Ubuntu mirror issue is specific to Filer Telephone Company's network peering. This may affect other services in the future. Consider:

• Using mirrors.kernel.org for all Ubuntu-based containers
• Setting up a local apt-cacher-ng if this becomes widespread
• Monitoring ISP routing changes

Tailscale Not Involved

Initial suspicion was that Tailscale exit nodes or DNS were causing the mirror issues. Testing confirmed this was false - the issue persists even with local DNS and no Tailscale routing.

---

Last updated: 2026-04-17