Migration: consolidate Echo6 docs to cortex with full infrastructure cleanup sync
- Documents recent infrastructure cleanup (8 CTs destroyed, 35 DNS records removed, Headscale cleanup) - Adds 24 new runbooks covering Authentik, PeerTube, Meshtastic, RECON, Proxmox, Mailcow, Internet Archive, GPU routing - Adds project documentation for headscale, vaultwarden, peertube, matrix, mmud, advbbs, arr stack - Updates services.md, environment.md, caddy.md, authentik.md to match live infrastructure - Removes 4 deprecated runbook duplicates (canonical versions live in projects/) - Adds .gitignore for binary archives and editor temp files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
89834796ff
commit
e9231ac24a
93 changed files with 51223 additions and 254 deletions
245
projects/advbbs-project.md
Normal file
245
projects/advbbs-project.md
Normal file
|
|
@ -0,0 +1,245 @@
|
|||
# advBBS — Claude Code Project Context
|
||||
|
||||
## Source of Truth
|
||||
|
||||
**GitHub repo**: https://github.com/zvx-echo6/advbbs (always pull latest before working)
|
||||
|
||||
## What is advBBS?
|
||||
|
||||
A federated, encryption-first BBS for Meshtastic mesh radio networks. Users interact by sending text DMs to a Meshtastic node running the BBS. Multi-hop mail routing between BBS nodes over LoRa radio. Runs on Raspberry Pi Zero 2 W (~100MB RAM).
|
||||
|
||||
Built with Python 3.11, SQLite (WAL mode), Meshtastic Python API. Docker-deployed. This is a "vibe-coded" project built with AI assistance — functional but may have rough edges.
|
||||
|
||||
---
|
||||
|
||||
## Package Structure
|
||||
|
||||
```
|
||||
advbbs/
|
||||
├── __init__.py
|
||||
├── __main__.py # Entry point
|
||||
├── config.py # TOML config loading, dataclasses
|
||||
├── cli/
|
||||
│ ├── config_rich.py # Rich-based interactive config TUI
|
||||
├── commands/
|
||||
│ ├── dispatcher.py # Command parser + all !commands
|
||||
├── core/
|
||||
│ ├── bbs.py # Main BBS class, event loop, session mgmt
|
||||
│ ├── boards.py # Board service (CRUD, access control)
|
||||
│ ├── crypto.py # Argon2id + ChaCha20-Poly1305 encryption
|
||||
│ ├── mail.py # Mail service (inbox, send, read, delete)
|
||||
│ ├── maintenance.py # Scheduled cleanup tasks
|
||||
│ ├── rate_limiter.py # Per-node rate limiting
|
||||
├── db/
|
||||
│ ├── connection.py # SQLite connection, schema, migrations
|
||||
│ ├── models.py # Dataclasses (User, Message, Board, etc.)
|
||||
│ ├── messages.py # MessageRepository (CRUD)
|
||||
│ ├── users.py # UserRepository, NodeRepository, UserNodeRepository
|
||||
├── mesh/
|
||||
│ ├── interface.py # Meshtastic radio interface, send/receive DMs
|
||||
├── sync/
|
||||
│ ├── manager.py # SyncManager — federation orchestrator (RAP, mail routing, retry logic)
|
||||
│ ├── compat/
|
||||
│ │ ├── advbbs_native.py # Wire protocol handler (HELLO, SYNC_ACK, bulletin format)
|
||||
├── utils/
|
||||
│ ├── formatting.py # Text formatting helpers
|
||||
│ ├── pagination.py # Message pagination for mesh constraints
|
||||
tests/
|
||||
├── test_boards.py
|
||||
├── test_crypto.py
|
||||
├── test_mail.py
|
||||
├── test_maintenance.py
|
||||
├── test_pagination.py
|
||||
├── test_sync.py
|
||||
docs/
|
||||
├── commands.md
|
||||
├── mail.md
|
||||
├── boards.md
|
||||
├── sync.md # Federation + RAP protocol docs
|
||||
├── configuration.md
|
||||
├── deployment.md
|
||||
├── security.md
|
||||
├── rap-testing.md # Multi-hop RAP test procedures
|
||||
├── migration.md # fq51bbs → advBBS migration
|
||||
├── quickstart.md
|
||||
├── USER-QUICKSTART.md
|
||||
├── ELI5.md
|
||||
```
|
||||
|
||||
### Non-obvious file placements
|
||||
|
||||
- `crypto.py` and `rate_limiter.py` → `core/` (not root)
|
||||
- `interface.py` (Meshtastic mesh interface) → `mesh/` (not root)
|
||||
- `advbbs_native.py` (wire protocol) → `sync/compat/` (not root)
|
||||
- `dispatcher.py` (all user commands) → `commands/` (not root)
|
||||
- `config_rich.py` (TUI config) → `cli/` (not root)
|
||||
- `formatting.py`, `pagination.py` → `utils/`
|
||||
|
||||
---
|
||||
|
||||
## Database
|
||||
|
||||
SQLite with WAL mode, autocommit, `check_same_thread=False`. Row factory enabled for dict-like access.
|
||||
|
||||
### Schema (3 migrations)
|
||||
|
||||
**Migration 001 — Core tables:**
|
||||
- `users` — id, username, password_hash, salt, encryption_key, recovery_key_enc, is_admin, is_banned, ban fields
|
||||
- `nodes` — Meshtastic nodes (node_id like `!abcdef12`, short_name, long_name, SNR/RSSI)
|
||||
- `user_nodes` — Multi-node identity (user_id ↔ node_id, is_primary)
|
||||
- `messages` — uuid (UNIQUE), msg_type (`mail`/`bulletin`/`system`), board_id, sender/recipient user/node IDs, subject_enc, body_enc (BLOB NOT NULL), timestamps, origin_bbs, forwarded_to, hop_count, delivery_attempts
|
||||
- `boards` — name, description, board_type, board_key_enc
|
||||
- `board_access` — Per-user restricted board access
|
||||
- `board_states` — Per-user read position
|
||||
- `bbs_peers` — node_id, bbs_name, protocol, sync_enabled, trust_level
|
||||
- `sync_log` — message_uuid, peer_id, direction, status, attempts
|
||||
|
||||
**Migration 002 — Settings/maintenance:**
|
||||
- Added columns: `messages.deleted_at_us`, `bbs_peers.callsign/name/capabilities/last_seen_us`
|
||||
- New tables: `bbs_settings` (KV store), `board_read_positions`
|
||||
|
||||
**Migration 003 — RAP:**
|
||||
- Added peer columns: `health_status`, `failed_heartbeats`, `last_heartbeat_us`, `last_pong_us`, `quality_score`
|
||||
- New tables: `rap_routes` (dest_bbs, via_peer_id, hop_count, quality_score, expires_at_us), `rap_pending_mail` (queued mail for offline routes)
|
||||
|
||||
### Timestamps
|
||||
|
||||
All timestamps are microseconds since epoch (`int(time.time() * 1_000_000)`), stored as INTEGER. Column suffix `_us`.
|
||||
|
||||
---
|
||||
|
||||
## Wire Protocol
|
||||
|
||||
All inter-BBS messages sent as Meshtastic DMs. Format: `advBBS|1|<MSG_TYPE>|<payload>`
|
||||
|
||||
### RAP Messages (Route Announcement Protocol)
|
||||
|
||||
| Message | Purpose | Payload |
|
||||
|---------|---------|---------|
|
||||
| `RAP_PING` | Heartbeat | `timestamp_us` |
|
||||
| `RAP_PONG` | Response + routes | `timestamp_us\|route_table` |
|
||||
| `RAP_ROUTES` | Route table broadcast | `route_table` |
|
||||
|
||||
Route table format: `BBS1:hop:quality;BBS2:hop:quality` (e.g., `MV51:0:1.0;J51B:1:1.00`)
|
||||
|
||||
### Mail Protocol Messages
|
||||
|
||||
| Message | Format | Purpose |
|
||||
|---------|--------|---------|
|
||||
| `MAILREQ` | `MAILREQ\|uuid\|from_user\|from_bbs\|to_user\|to_bbs\|hop\|num_parts\|route` | Request delivery |
|
||||
| `MAILACK` | `MAILACK\|uuid\|OK` | Accept, ready for chunks |
|
||||
| `MAILNAK` | `MAILNAK\|uuid\|reason` | Reject (NOUSER, NOROUTE, MAXHOPS, LOOP) |
|
||||
| `MAILDAT` | `MAILDAT\|uuid\|part/total\|data` | Message chunk (max 150 chars × 3) |
|
||||
| `MAILDLV` | `MAILDLV\|uuid\|OK\|user@BBS` | Delivery confirmation |
|
||||
|
||||
### Mail Flow
|
||||
|
||||
```
|
||||
Sender BBS Destination BBS
|
||||
│ │
|
||||
│── MAILREQ ────────────▶│ (pre-flight: user exists?)
|
||||
│◀── MAILACK ────────────│ (ready for chunks)
|
||||
│── MAILDAT 1/1 ────────▶│ (body chunk)
|
||||
│ │ (store in DB)
|
||||
│◀── MAILDLV ────────────│ (confirmed)
|
||||
```
|
||||
|
||||
Multi-hop: intermediate BBS relays MAILREQ/MAILDAT, tracked via `_relay_mail` dict. Max 5 hops. Route list in MAILREQ prevents loops.
|
||||
|
||||
---
|
||||
|
||||
## Key Architecture Patterns
|
||||
|
||||
### Threading Model
|
||||
|
||||
- **Main thread**: asyncio event loop (`bbs._loop`) — runs tick(), scheduled tasks
|
||||
- **Meshtastic callback thread**: `on_receive` fires from Meshtastic library thread
|
||||
- **Bridge**: `_schedule_async(coro)` uses `asyncio.run_coroutine_threadsafe()` to schedule work from callback thread onto main loop
|
||||
|
||||
### Session Management
|
||||
|
||||
Sessions keyed by Meshtastic node_id. Login requires both password AND a registered node (node-based 2FA). Sessions expire after inactivity.
|
||||
|
||||
### Encryption
|
||||
|
||||
- **At rest**: All message bodies encrypted with user-derived keys (Argon2id KDF → ChaCha20-Poly1305)
|
||||
- **Transport**: Meshtastic PSK encryption (AES-256) recommended
|
||||
- **Remote mail**: Stored plaintext on receiving BBS (encrypted at read time by recipient's key)
|
||||
|
||||
### Message Constraints
|
||||
|
||||
- LoRa max ~150 bytes usable per packet
|
||||
- Remote mail body max 450 chars (3 chunks × 150)
|
||||
- Pagination helper chunks long responses for mesh delivery
|
||||
- TX queue collision avoidance: 2.5s delay between protocol DMs
|
||||
|
||||
### Peer Security
|
||||
|
||||
Federation traffic whitelisted by peer — only configured peers accepted. Non-peer protocol messages rejected.
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
TOML config file. Key sections: `[bbs]`, `[database]`, `[meshtastic]`, `[crypto]`, `[features]`, `[operating_mode]`, `[sync]`, `[rate_limits]`, `[web_reader]`, `[cli_config]`, `[logging]`.
|
||||
|
||||
Operating modes: `full`, `mail_only`, `boards_only`, `repeater`.
|
||||
|
||||
Peers configured as `[[sync.peers]]` arrays with `node_id`, `name`, `protocol`, `enabled`.
|
||||
|
||||
RAP timing defaults are conservative for mesh (12h heartbeat, 36h route expiry, 24h route share).
|
||||
|
||||
---
|
||||
|
||||
## Current Live Federation Topology
|
||||
|
||||
```
|
||||
MV51 (Old Man Malice / Matt) ◀──▶ J51B (JeepnJonny)
|
||||
node: !00ff0001 node: !60a43e58
|
||||
```
|
||||
|
||||
Both running Docker containers. Meshtastic simulator (meshtasticd) for testing.
|
||||
|
||||
---
|
||||
|
||||
## Known Bug: Federation Mail Delivery Failure
|
||||
|
||||
### Symptom
|
||||
```
|
||||
[ERROR] advbbs.sync.manager: DELIVER b8e78195: Failed to store in database
|
||||
```
|
||||
|
||||
### Root Cause
|
||||
`create_incoming_remote_mail()` in `db/messages.py` returns `None` for both duplicates (logged at DEBUG — invisible) and real DB errors. Mesh radio retransmissions deliver the same MAILDAT twice, triggering duplicate detection, but the caller can't distinguish this from a real failure.
|
||||
|
||||
Additionally, when a duplicate IS detected, no MAILDLV confirmation is sent back, causing the sender to retry indefinitely.
|
||||
|
||||
### Fix Required (3 changes)
|
||||
|
||||
1. **`db/messages.py`** — `create_incoming_remote_mail`: Return `"duplicate"` sentinel instead of `None` for duplicate UUID. Promote log from DEBUG → INFO. Add traceback to exception path.
|
||||
|
||||
2. **`sync/manager.py`** — `_deliver_remote_mail`: Handle `"duplicate"` return: log at INFO, still send MAILDLV confirmation, clean up state. Improve error message for real failures.
|
||||
|
||||
3. **`sync/manager.py`** — `handle_maildat` (the `_handle_maildat` section around line 1068): Add `delivering` flag guard to prevent double scheduling from mesh retransmissions.
|
||||
|
||||
---
|
||||
|
||||
## Development Notes
|
||||
|
||||
- Tests: `pytest tests/` — unit tests for crypto, mail, boards, maintenance, pagination, sync
|
||||
- Docker build: `docker compose build` (can take 10-15 min on Pi, may need swap)
|
||||
- Config TUI: `advbbs-config` or `python -m advbbs.cli.config_rich`
|
||||
- Logs: `docker compose logs -f`
|
||||
- DB inspection: `sqlite3 /data/advbbs.db ".tables"` inside container
|
||||
|
||||
---
|
||||
|
||||
## Style / Conventions
|
||||
|
||||
- Logging: `logger = logging.getLogger(__name__)` per module
|
||||
- DB access: Repository pattern (MessageRepository, UserRepository, etc.) wrapping Database methods
|
||||
- All DB timestamps: microseconds (`_us` suffix)
|
||||
- UUIDs: `str(uuid.uuid4())` for message dedup
|
||||
- Meshtastic node IDs: hex string with `!` prefix (e.g., `!00ff0001`)
|
||||
- Commands: `!` prefix, case-insensitive, short aliases
|
||||
- Config: TOML with dataclass parsing in `config.py`
|
||||
Loading…
Add table
Add a link
Reference in a new issue