auto: docs sync 2026-06-15T21:21:42+00:00
Files changed: .obsidian/appearance.json .obsidian/graph.json .obsidian/plugins/obsidian-livesync/data.json .obsidian/workspace.json INDEX.md archive/matrix/PHASE6_DECISION.md archive/matrix/PLAN.md archive/matrix/appservices.md archive/matrix/archive_receiver_discovery.md archive/matrix/archivist_discovery.md archive/matrix/hookshot_deployment_discovery.md archive/matrix/hookshot_e2ee_discovery.md archive/matrix/hookshot_payload_discovery.md archive/matrix/hookshot_room_targeting_discovery.md docs/matrix/archivist.md docs/matrix/matrix_host.md docs/matrix/mautrix_signal.md docs/matrix/synapse.md docs/matrix/synapse_homeserver.yaml.sanitized docs/matrix/synapse_retention_discovery.md docs/services/services.md notes/echo6-landing-page-data-export.md notes/ia-download-queue.md projects/argus.md
This commit is contained in:
parent
133639ede2
commit
e5da5f023d
24 changed files with 85 additions and 51 deletions
476
notes/echo6-landing-page-data-export.md
Normal file
476
notes/echo6-landing-page-data-export.md
Normal file
|
|
@ -0,0 +1,476 @@
|
|||
# Echo6 Landing Page — Data Export
|
||||
## Echo6 Platform Reference — Infrastructure, Services & Brand Identity
|
||||
|
||||
---
|
||||
|
||||
## 1. Brand / Identity
|
||||
|
||||
- **Domain:** echo6.co
|
||||
- **Admin email:** admin@echo6.co
|
||||
- **Auth provider:** Authentik (auth.echo6.co) — OIDC SSO across all services
|
||||
|
||||
### Brand Colors (extracted from logo)
|
||||
|
||||
| Color | Hex | RGB | Usage |
|
||||
|-------|-----|-----|-------|
|
||||
| Echo6 Cyan | `#28C0E8` | rgb(40, 192, 232) | Primary accent, buttons, links, focus states |
|
||||
| Echo6 Cyan Light | `#5DD4F5` | rgb(93, 212, 245) | Hover states |
|
||||
| Echo6 Yellow | `#F0D848` | rgb(240, 216, 72) | Secondary accent, warnings |
|
||||
| Background Primary | `#0a0e17` | rgb(10, 14, 23) | Page backgrounds |
|
||||
| Background Secondary | `#111827` | rgb(17, 24, 39) | Cards, sidebars, inputs |
|
||||
| Background Tertiary | `#1a2332` | rgb(26, 35, 50) | Hover states |
|
||||
| Border | `#1e3a5f` | rgb(30, 58, 95) | All borders |
|
||||
| Text Primary | `#e0e6ed` | rgb(224, 230, 237) | Main text |
|
||||
| Text Muted | `#7a8ca0` | rgb(122, 140, 160) | Secondary text |
|
||||
|
||||
### Design Language
|
||||
|
||||
- **Aesthetic:** Cyberpunk — dark, sharp, clean. No excessive glow or neon blur.
|
||||
- **Font:** JetBrains Mono (monospace) throughout all Echo6-branded interfaces
|
||||
- **Text:** All lowercase in navigation, menus, and footer elements
|
||||
- **Effects:** Subtle — search bar focus shadow at 0.12 opacity max, no colored glows
|
||||
|
||||
---
|
||||
|
||||
## 2. Public-Facing Services
|
||||
|
||||
| Service | URL | Description | Auth |
|
||||
|---------|-----|-------------|------|
|
||||
| Echo6 Search (Homepage) | https://echo6.co | SearXNG search — branded cyberpunk homepage, Google-style layout | Public (SearXNG) |
|
||||
| Aurora (AI Assistant) | https://ai.echo6.co | RAG-augmented LLM chat — locally-hosted, queries a 95K+ vector knowledge base | Authentik OIDC |
|
||||
| PeerTube (Video) | https://stream.echo6.co | Self-hosted video platform — 99 curated YouTube channels mirrored, GPU-transcoded | Authentik OIDC |
|
||||
| File Server | https://files.echo6.co | PDF/document library — ~13,239 documents (military doctrine, survival, comms, trades) | Public |
|
||||
| Photos (Immich) | https://immich.echo6.co | Self-hosted photo management | Authentik OIDC |
|
||||
| Mail (Mailcow) | https://mail.echo6.co | Email — Mailcow webmail | Authentik OIDC |
|
||||
| Cloud (Nextcloud) | https://nextcloud.echo6.co | Self-hosted cloud storage | Authentik OIDC |
|
||||
| Jellyfin | https://jellyfin.echo6.co | Media streaming | Authentik OIDC |
|
||||
| Requests (Jellyseerr) | https://requests.echo6.co | Media request management | Authentik OIDC |
|
||||
| WATCHTOWER | https://wt.echo6.co | Infrastructure monitoring dashboard | Authentik forward auth |
|
||||
| Auth (Authentik) | https://auth.echo6.co | SSO identity provider — user portal and admin | Authentik native |
|
||||
|
||||
---
|
||||
|
||||
## 3. Echo6 Homepage — SearXNG Custom Theme
|
||||
|
||||
### Overview
|
||||
The Echo6 homepage at `echo6.co` is a customized SearXNG instance (not a standalone static page). The default SearXNG UI is reskinned via CSS overlay, template overrides, and settings.yml changes to match the Echo6 cyberpunk brand.
|
||||
|
||||
### Implementation Approach
|
||||
- **CSS overlay** (`echo6-custom.css`) — all color, font, layout, and component overrides
|
||||
- **Logo replacement** — Echo6 logo replaces SearXNG default logo via volume mount or direct file copy
|
||||
- **Template override** (`base.html`) — injects the top navigation bar, waffle menu JS, and custom footer
|
||||
- **settings.yml** — instance name, theme, base URL configuration
|
||||
|
||||
### SearXNG Settings
|
||||
|
||||
```yaml
|
||||
general:
|
||||
instance_name: "Echo6"
|
||||
|
||||
ui:
|
||||
default_theme: simple
|
||||
center_alignment: false # results stretch to fill viewport
|
||||
theme_args:
|
||||
simple_style: dark
|
||||
|
||||
server:
|
||||
base_url: "https://echo6.co/"
|
||||
|
||||
brand:
|
||||
issue_url: false
|
||||
docs_url: false
|
||||
public_instances: false
|
||||
wiki_url: false
|
||||
```
|
||||
|
||||
### Homepage Layout (Google.com-inspired)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ .//photos .//mail [⠿] [👤] │ ← top nav (right-aligned)
|
||||
│ │
|
||||
│ [ECHO6 LOGO] │ ← centered ~35-40% from top
|
||||
│ │
|
||||
│ ┌─────────────────────┐ │
|
||||
│ │ 🔍 search input │ │ ← pill search bar
|
||||
│ └─────────────────────┘ │
|
||||
│ │
|
||||
├─────────────────────────────────────────────────────────┤
|
||||
│ echo6.co │
|
||||
│ preferences · about │ ← footer (minimal)
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- Homepage locked to viewport (no scroll, `overflow: hidden` on body.index)
|
||||
- Logo + search bar vertically centered at ~35-40% from top using flexbox
|
||||
- Default SearXNG text/wordmark hidden via CSS
|
||||
- Results pages scroll normally (only homepage is viewport-locked)
|
||||
|
||||
### Top Navigation
|
||||
|
||||
| Element | Style | Target |
|
||||
|---------|-------|--------|
|
||||
| .//photos | Cyan monospace link | `https://auth.echo6.co/application/launch/<immich-slug>/` |
|
||||
| .//mail | Cyan monospace link | `https://auth.echo6.co/application/launch/<mailcow-slug>/` |
|
||||
| ⠿ (waffle) | Circle button, 36px | Opens waffle menu dropdown |
|
||||
| 👤 (login) | Circle button, 32px, border | `https://auth.echo6.co/` |
|
||||
|
||||
### Waffle Menu (11 items, 3-column grid)
|
||||
|
||||
| Label | Target |
|
||||
|-------|--------|
|
||||
| aurora | `https://auth.echo6.co/application/launch/<openwebui-slug>/` |
|
||||
| stream | `https://auth.echo6.co/application/launch/<peertube-slug>/` |
|
||||
| files | `https://files.echo6.co` (public — no launch URL) |
|
||||
| watchtower | `https://auth.echo6.co/application/launch/<watchtower-slug>/` |
|
||||
| photos | `https://auth.echo6.co/application/launch/<immich-slug>/` |
|
||||
| mail | `https://auth.echo6.co/application/launch/<mailcow-slug>/` |
|
||||
| cloud | `https://auth.echo6.co/application/launch/<nextcloud-slug>/` |
|
||||
| jellyfin | `https://auth.echo6.co/application/launch/<jellyfin-slug>/` |
|
||||
| requests | `https://auth.echo6.co/application/launch/<jellyseerr-slug>/` |
|
||||
| admin | `https://auth.echo6.co/` |
|
||||
| preferences | `https://echo6.co/preferences` |
|
||||
|
||||
All authenticated service links use Authentik launch URLs (`/application/launch/<slug>/`) for seamless SSO pass-through.
|
||||
|
||||
### CSS Architecture (14 sections)
|
||||
|
||||
1. CSS variables (brand colors)
|
||||
2. Homepage viewport lock + centering
|
||||
3. SearXNG wordmark hiding
|
||||
4. Logo sizing
|
||||
5. Search bar (pill shape, cyan focus)
|
||||
6. Search buttons
|
||||
7. Top navigation bar (fixed, right-aligned)
|
||||
8. Waffle menu dropdown (grid, fade-in animation)
|
||||
9. Footer (minimal, lowercase)
|
||||
10. Search results page (stretched full-width, themed)
|
||||
11. Preferences page (dark forms, cyan accents)
|
||||
12. Scrollbar styling
|
||||
13. Selection highlight
|
||||
14. Responsive breakpoints (768px, 480px)
|
||||
|
||||
### SearXNG File Locations
|
||||
|
||||
| Asset | Path |
|
||||
|-------|------|
|
||||
| Logo | `/usr/local/searxng/searx/static/themes/simple/img/searxng.png` |
|
||||
| Favicon | `/usr/local/searxng/searx/static/themes/simple/img/favicon.png` |
|
||||
| Custom CSS | Injected via template override or volume mount |
|
||||
| Base template | `/usr/local/searxng/searx/templates/simple/base.html` |
|
||||
| Settings | `/etc/searxng/settings.yml` |
|
||||
|
||||
---
|
||||
|
||||
## 4. Authentik Theme — Echo6 Branding
|
||||
|
||||
### Overview
|
||||
The Authentik instance at `auth.echo6.co` is themed to match the Echo6 cyberpunk aesthetic. This covers the login flow, user dashboard, and admin interface.
|
||||
|
||||
### Brand Settings (Admin → System → Brands)
|
||||
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| Branding title | `echo6` |
|
||||
| Logo | Echo6 logo (uploaded via Customization → Files) |
|
||||
| Favicon | Echo6 favicon |
|
||||
| Theme | `dark` (forced — not automatic) |
|
||||
| Custom CSS | Echo6 Authentik CSS (see below) |
|
||||
|
||||
### Flow Text Customization
|
||||
|
||||
| Flow | Title |
|
||||
|------|-------|
|
||||
| Authentication | `echo6 // login` |
|
||||
| Invalidation | `echo6 // logout` |
|
||||
| Recovery | `echo6 // recovery` |
|
||||
| User settings | `echo6 // settings` |
|
||||
|
||||
### User Dashboard Layout
|
||||
|
||||
```yaml
|
||||
settings:
|
||||
theme:
|
||||
base: dark
|
||||
cardLayout: 3-column
|
||||
```
|
||||
|
||||
### CSS Customization Scope
|
||||
|
||||
The Authentik CSS overrides PatternFly (`pf-c-*`) components:
|
||||
- Login flow: dark gradient background, cyan buttons, JetBrains Mono font
|
||||
- Form inputs: dark backgrounds (`#111827`), cyan focus borders
|
||||
- User dashboard: dark app cards with cyan hover borders
|
||||
- Admin interface: dark sidebar, cyan active nav indicators
|
||||
- Buttons: primary = Echo6 cyan with dark text, secondary = border-only
|
||||
- Tables, tabs, modals, alerts, dropdowns — all themed
|
||||
- Global font override to JetBrains Mono
|
||||
|
||||
### Application Icons
|
||||
Each Authentik application has a matching service icon uploaded for the user dashboard tiles. Icons sourced from official project assets or cyberpunk-styled variants.
|
||||
|
||||
### Version Note
|
||||
Authentik 2025.12.4 — custom CSS is applied via the Brand CSS field in System → Brands → Edit. The Shadow DOM CSS injection bug from 2025.12.0 is fixed in this version. Logo/favicon use the Files system (Customization → Files) rather than the old `/media/` mount path.
|
||||
|
||||
---
|
||||
|
||||
## 5. Infrastructure Overview
|
||||
|
||||
### Nodes (all connected via Tailscale / self-hosted Headscale)
|
||||
|
||||
| Node | Role | Key Services |
|
||||
|------|------|-------------|
|
||||
| data | Proxmox host | Hosts RECON VM (VM 1130) |
|
||||
| utility | Proxmox host | Caddy reverse proxy (CT 101), TLS termination |
|
||||
| cloud | Proxmox host | — |
|
||||
| media | Proxmox host | PeerTube LXC (CT 110) |
|
||||
| toc | Proxmox host | GPU passthrough host for cortex VM |
|
||||
| cortex | VM on toc | RTX A4000 16GB, Qdrant, Ollama, TEI, OpenWebUI, PeerTube runner |
|
||||
| pi-nas | NFS storage (OMV) | 18TB — PDF library + PeerTube video storage |
|
||||
| Contabo VPS | Remote | WATCHTOWER, DNS, automated backups (919GB available) |
|
||||
|
||||
### GPU — NVIDIA RTX A4000 (16GB VRAM, on cortex)
|
||||
|
||||
| Workload | VRAM | Silicon |
|
||||
|----------|------|---------|
|
||||
| TEI embeddings (bge-m3) | 1.3 GB | CUDA |
|
||||
| Aurora LLM (JOSIEFIED Qwen3 8B) | ~5 GB | CUDA/Tensor |
|
||||
| Whisper transcription (medium) | ~5 GB | CUDA (short videos) or CPU (long videos) |
|
||||
| NVENC video encoding | ~0.2-0.5 GB/stream | Dedicated ASIC (no CUDA contention) |
|
||||
| **Typical total** | **~11.5 GB / 16 GB** | |
|
||||
|
||||
---
|
||||
|
||||
## 6. RECON — Knowledge Extraction Pipeline
|
||||
|
||||
### What It Does
|
||||
Processes PDFs and web content → extracts structured knowledge concepts via Gemini AI → embeds into a vector database → provides RAG-augmented retrieval for Aurora.
|
||||
|
||||
### Pipeline Flow
|
||||
```
|
||||
PDF/Web Source → Extract Text → Gemini Enrichment → JSON (saved) → Embed (TEI) → Qdrant
|
||||
```
|
||||
|
||||
### Key Stats
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total documents catalogued | ~13,239 PDFs + web sources |
|
||||
| Documents complete | ~1,200+ (and growing) |
|
||||
| Vectors in Qdrant | 95,000+ |
|
||||
| Embedding throughput | 1,711 embeddings/sec (TEI) |
|
||||
| Enrichment rate | ~100 docs/hour |
|
||||
| Search latency | <10ms (HNSW index) |
|
||||
| Full catalogue reprocess time | ~50 hours |
|
||||
| Full catalogue cost | ~$135 (Gemini API) |
|
||||
|
||||
### Content Sources
|
||||
|
||||
| Source | Count | Description |
|
||||
|--------|-------|-------------|
|
||||
| Army Field Manuals | ~160 | Official Army doctrine (FM series) |
|
||||
| Survival Companion Library | ~6,600 | Comprehensive survival/preparedness collection |
|
||||
| Other documents | ~6,300 | Assorted reference, technical, trades |
|
||||
| meshtastic.org | ~247 pages | Mesh communications documentation |
|
||||
| ready.gov | ~5 pages | Emergency preparedness |
|
||||
|
||||
### Knowledge Domains (Aurora Taxonomy)
|
||||
|
||||
| Tier | Domain | Examples |
|
||||
|------|--------|----------|
|
||||
| 1 | Foundational Skills | Water, food, fire, shelter, first aid, navigation, security |
|
||||
| 2 | Sustainment Systems | Long-term food/water/energy, medical stockpiling, tool maintenance |
|
||||
| 3 | Defense & Military Tactics | Perimeter defense, small unit tactics, weapons, OPSEC |
|
||||
| 4 | Off-Grid Systems | Power generation, water independence, thermal, comms infrastructure |
|
||||
| 5 | Communications & Intelligence | Radio ops, mesh networking, SIGINT, OSINT, backup comms |
|
||||
| 6 | Scenario Playbooks | Urban collapse, rural homesteading, bug-out, governance |
|
||||
|
||||
### Scenario Timescales
|
||||
|
||||
| Tag | Scope |
|
||||
|-----|-------|
|
||||
| tuesday_prepper | Next-week disruption (power outage, storm) |
|
||||
| month_prepper | Extended disruption (supply chain, regional disaster) |
|
||||
| year_prepper | Long-term grid-down |
|
||||
| multi_year | Sustained collapse |
|
||||
| eotwawki | End of the world as we know it |
|
||||
|
||||
### Concept Schema (22 fields per concept)
|
||||
Core content, classification (domain/subdomain/skill level/scenario), provenance (source type, credibility score, verification status), reference (chapter, page, key facts), and technical metadata (deterministic IDs, embeddings, download URLs).
|
||||
|
||||
### Pipeline Architecture
|
||||
|
||||
| Stage | Workers | Bottleneck | Description |
|
||||
|-------|---------|------------|-------------|
|
||||
| Extract | 4 | CPU-bound | PyPDF2 → pdftotext → Tesseract → Gemini Vision (4-method fallback chain) |
|
||||
| Enrich | 16 | I/O-bound (Gemini API) | 10-page windows → Gemini 2.5 Flash Lite → structured JSON concepts |
|
||||
| Embed | batch | I/O-bound (TEI) | bge-m3 1024-dim → Qdrant insert, 128/batch |
|
||||
| Scanner | 1 | Hourly cron | Auto-discovers new PDFs from NFS mount |
|
||||
|
||||
### Resilience Features
|
||||
- JSON-first: all Gemini extractions saved to disk before any DB insert
|
||||
- Idempotent: content hashing prevents duplicate processing
|
||||
- Recoverable: full Qdrant rebuild from JSON files alone
|
||||
- Exponential backoff with jitter for API rate limits
|
||||
- Window-level failure tolerance (partial document processing)
|
||||
- Error classification (transient vs permanent)
|
||||
- Automated backup to Contabo VPS every 6 hours
|
||||
|
||||
---
|
||||
|
||||
## 7. Aurora — AI Assistant
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Interface | OpenWebUI at ai.echo6.co |
|
||||
| Model | JOSIEFIED Qwen3 8B (~55 tok/s, ~5GB VRAM) |
|
||||
| RAG source | RECON Qdrant (95K+ vectors, top-5 retrieval) |
|
||||
| Embedding model | bge-m3 (1024-dim via TEI) |
|
||||
| Score threshold | 0.3 cosine similarity |
|
||||
| Citations | Clickable badges linking to files.echo6.co PDFs or source web pages |
|
||||
| Dual-mode | Think Toggle filter for reasoning vs. fast response |
|
||||
| Auth | Authentik OIDC |
|
||||
|
||||
---
|
||||
|
||||
## 8. PeerTube — Video Platform
|
||||
|
||||
### Instance Details
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| URL | https://stream.echo6.co |
|
||||
| Version | PeerTube v8.0.2 |
|
||||
| Installation | Native (not Docker), Debian 12 |
|
||||
| Host | CT 110 on media node |
|
||||
| Storage | NFS from pi-nas (~18TB capacity) |
|
||||
| Database | PostgreSQL (peertube_prod) |
|
||||
| Auth | Authentik OIDC (signup disabled) |
|
||||
| Transcoding | 720p + 1080p HLS only, NVENC GPU via remote runner |
|
||||
| Transcription | Whisper medium (GPU for <1hr, CPU for >1hr) |
|
||||
|
||||
### Channel Library
|
||||
|
||||
| Category | Examples |
|
||||
|----------|----------|
|
||||
| Tactical/Firearms | Forgotten Weapons, Active Self Protection, Garand Thumb |
|
||||
| Preparedness/Homesteading | Canadian Prepper, City Prepping |
|
||||
| Trades — Electrical | — |
|
||||
| Trades — Plumbing | — |
|
||||
| Trades — HVAC | — |
|
||||
| Trades — Welding | — |
|
||||
| Trades — Automotive | — |
|
||||
| Trades — Woodworking | — |
|
||||
| Ham Radio | Dave Casler, Ham Radio Crash Course, DX Commander |
|
||||
| Comms/Meshtastic | Andreas Spiess, The Comms Channel, CommsPrepper |
|
||||
| Education — Math | — |
|
||||
| Education — Science | — |
|
||||
| Education — Engineering | — |
|
||||
| Education — History | — |
|
||||
| Medical/Fieldcraft | PrepMedic, Skinny Medic |
|
||||
| Lockpicking/Security | LockPickingLawyer, Bosnianbill |
|
||||
|
||||
### Key Stats
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Total channels | 99 curated YouTube channels |
|
||||
| Channels fully synced | 57 of 99 |
|
||||
| Estimated total library size | ~15.3 TB |
|
||||
| Transcoding output | 720p + 1080p HLS |
|
||||
| Transcoder throughput | ~67 videos/hr (NVENC) |
|
||||
| Transcode win rate | 83% (post-probe-gate optimization) |
|
||||
| Import method | Resumable chunked upload (10MB chunks) |
|
||||
|
||||
### Bulk Import Pipeline
|
||||
|
||||
```
|
||||
Downloader (CT 110, yt-dlp + VPN rotation)
|
||||
→ Transcoder (cortex, NVENC GPU, 4 workers)
|
||||
→ Importer (CT 110, PeerTube API, resumable upload)
|
||||
```
|
||||
|
||||
- Round-robin across 99 channels with sliding window (50 videos/batch)
|
||||
- VPN rotation on rate limit (NordVPN, 6 countries: US, CA, UK, DE, NL, SE)
|
||||
- Pre-encode probe gate skips already-efficient codecs (H.265/AV1/VP9) and low-bitrate H.264
|
||||
- Automatic dedup via download archive
|
||||
- All three stages run as systemd services
|
||||
|
||||
---
|
||||
|
||||
## 9. Networking & Security
|
||||
|
||||
| Layer | Technology |
|
||||
|-------|-----------|
|
||||
| Mesh VPN | Tailscale (self-hosted Headscale) |
|
||||
| Reverse proxy | Caddy (CT 101 on utility) — auto TLS |
|
||||
| DNS | GoDaddy (external), dnsmasq split DNS (internal) |
|
||||
| Authentication | Authentik OIDC SSO across all services |
|
||||
| SSO Launch URLs | `https://auth.echo6.co/application/launch/<slug>/` for seamless pass-through |
|
||||
| Backup transport | rsync over SSH (ed25519 keys) |
|
||||
|
||||
---
|
||||
|
||||
## 10. System Relationships
|
||||
|
||||
```
|
||||
RECON → Knowledge extraction & vector storage
|
||||
Aurora → LLM assistant, queries RECON for knowledge (RAG)
|
||||
ARGUS → Intelligence/OSINT platform, feeds intel into RECON
|
||||
PeerTube → Video library (curated educational/preparedness content)
|
||||
WATCHTOWER → Unified infrastructure monitoring
|
||||
echo6.co → SearXNG search homepage (branded, cyberpunk theme)
|
||||
auth.echo6.co → Authentik SSO (themed to match Echo6 brand)
|
||||
files.echo6.co → Document/PDF download server
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Technology Stack Summary
|
||||
|
||||
| Layer | Technologies |
|
||||
|-------|-------------|
|
||||
| Virtualization | Proxmox (5 nodes) |
|
||||
| Networking | Tailscale/Headscale, Caddy, nginx, dnsmasq |
|
||||
| GPU compute | NVIDIA RTX A4000 (CUDA, NVENC, Tensor) |
|
||||
| AI/ML | Gemini 2.5 Flash Lite, Ollama, TEI (bge-m3), JOSIEFIED Qwen3 8B |
|
||||
| Vector DB | Qdrant (HNSW index, cosine similarity) |
|
||||
| Databases | SQLite (RECON), PostgreSQL (PeerTube) |
|
||||
| Video | PeerTube v8, yt-dlp, ffmpeg/NVENC, Whisper |
|
||||
| Search | SearXNG (custom Echo6 theme) |
|
||||
| Auth | Authentik (OIDC, custom Echo6 theme) |
|
||||
| Storage | NFS (pi-nas, 18TB), Contabo VPS (919GB backup) |
|
||||
| Languages | Python (RECON pipeline), Node.js (PeerTube), Bash |
|
||||
| OS | Ubuntu 24.04 (RECON), Debian 12 (PeerTube) |
|
||||
|
||||
---
|
||||
|
||||
## 12. Design / Copy Hooks for Landing Page
|
||||
|
||||
### Tagline Ideas (raw material)
|
||||
- 13,000+ documents processed into searchable knowledge
|
||||
- 99 curated educational channels, GPU-transcoded and self-hosted
|
||||
- AI-powered knowledge retrieval across military doctrine, survival, trades, and communications
|
||||
- Fully self-hosted, zero cloud dependency for core services
|
||||
- From PDF to actionable knowledge in 50 hours
|
||||
|
||||
### Key Differentiators
|
||||
- **Self-hosted everything**: LLM, video, auth, DNS, monitoring — no SaaS dependencies
|
||||
- **Knowledge pipeline**: Not just storing PDFs — extracting, structuring, and making them searchable via AI
|
||||
- **GPU-accelerated**: Single RTX A4000 powers video transcoding, AI inference, embeddings, and transcription simultaneously (different silicon paths)
|
||||
- **Resilient by design**: JSON-first storage, automated backups, full rebuild capability, exponential backoff
|
||||
- **Curated content**: 99 hand-picked channels across tactical, trades, comms, medical, and education
|
||||
- **Cohesive brand**: Cyberpunk aesthetic applied consistently across search, auth, and all services
|
||||
|
||||
### Numbers That Pop
|
||||
- 95,000+ knowledge vectors
|
||||
- 13,239 documents
|
||||
- 99 video channels
|
||||
- 1,711 embeddings/second
|
||||
- <10ms search latency
|
||||
- 22-field concept schema
|
||||
- 6 knowledge domains
|
||||
- 5 Proxmox nodes
|
||||
- 18TB storage
|
||||
- ~15.3TB video library
|
||||
- 11 services in the waffle menu
|
||||
158
notes/ia-download-queue.md
Normal file
158
notes/ia-download-queue.md
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
# Internet Archive Download Queue
|
||||
|
||||
Tracked items to pull down to pi-nas (192.168.1.245).
|
||||
|
||||
---
|
||||
|
||||
## Queue
|
||||
|
||||
### 1. preppers_paradise — Appropriate Technology Reference Library
|
||||
- **IA Identifier:** `preppers_paradise`
|
||||
- **Path filter:** `preppers/Appropriate Technology Reference Library/`
|
||||
- **Files:** 991 PDFs (13.6 GB)
|
||||
- **Content:** Development-world practical tech — water/sanitation, agriculture, construction, energy, health, tools. Peace Corps style reference library.
|
||||
- **Status:** Pending
|
||||
|
||||
### 2. preppers_paradise — Tech Anarchist
|
||||
- **IA Identifier:** `preppers_paradise`
|
||||
- **Path filter:** `preppers/Tech Anarchist/`
|
||||
- **Files:** 362 PDFs (2.6 GB)
|
||||
- **Content:** Networking, mesh networks, cybersecurity, privacy, crypto, ML/AI, electronics.
|
||||
- **Status:** Pending
|
||||
|
||||
### 3. survivalist-pdf-archive
|
||||
- **IA Identifier:** `survivalist-pdf-archive`
|
||||
- **Files:** 76 new PDFs (643 MB)
|
||||
- **Content:** Military FMs (combatives, NBC, map reading, urban ops), medical references (Wilderness Medicine 5th ed, Ditch Medicine, Where There Is No Doctor/Dentist), nuclear survival (Kearny, FEMA), herbal medicine, checklists, SEAL survival handbook.
|
||||
- **Status:** Pending
|
||||
|
||||
### 4. 150 Survival Secrets
|
||||
- **IA Identifier:** `150-survival-secrets-advice-on-survival-kits-extreme-weather-rapid-evacuation-fo`
|
||||
- **Files:** 1 PDF (17.5 MB)
|
||||
- **Content:** Survival kits, extreme weather, rapid evacuation, food storage, active shooters, first aid.
|
||||
- **Status:** Pending
|
||||
|
||||
### 5. Prepping 101
|
||||
- **IA Identifier:** `prepping-101-40-steps-you-can-take-to-be-prepared-for-weather-disasters-2018-kathy-harrison`
|
||||
- **Files:** 1 PDF (26.6 MB)
|
||||
- **Content:** 40-step preparedness guide for weather disasters (Kathy Harrison, 2018).
|
||||
- **Status:** Pending
|
||||
|
||||
### 6. Build the Perfect Bug Out Bag
|
||||
- **IA Identifier:** `build-the-perfect-bug-out-bag-your-72-hour-disaster-survival-kit-creek-stewart`
|
||||
- **Files:** 1 PDF (13.8 MB)
|
||||
- **Content:** Creek Stewart's 72-hour disaster survival kit guide.
|
||||
- **Status:** Pending
|
||||
|
||||
### 7. Full Self Sufficiency (City Prepping)
|
||||
- **IA Identifier:** `full-self-sufficiency`
|
||||
- **Files:** 3 PDFs (20.7 MB)
|
||||
- **Content:** City Prepping guides — start preparing, bug-out guide, cyber attack protection.
|
||||
- **Status:** Pending
|
||||
|
||||
### 8. FM 21-76 Army Survival Handbook
|
||||
- **IA Identifier:** `FM2176NEWU.S.ArmySurvivalHandbook`
|
||||
- **Files:** 1 PDF (20.5 MB)
|
||||
- **Content:** 676-page U.S. Army survival handbook. May differ from Army_Pubs copy.
|
||||
- **Status:** Pending
|
||||
|
||||
### 9. Preppers Emergency First Aid
|
||||
- **IA Identifier:** `preppers-emergency-first-aid`
|
||||
- **Files:** 1 PDF (0.6 MB)
|
||||
- **Content:** Emergency first aid & survival medicine handbook.
|
||||
- **Status:** Pending
|
||||
|
||||
### 10. Civilian Defensive Tactics
|
||||
- **IA Identifier:** `final-draft-cdn`
|
||||
- **Files:** 1 PDF (3.3 MB)
|
||||
- **Content:** Civilian defensive tactics guidebook.
|
||||
- **Status:** Pending
|
||||
|
||||
### 11. All About Pandemics
|
||||
- **IA Identifier:** `VictoriaLacroixAllAboutPandemicsEpidemicOfInfectiousDisease`
|
||||
- **Files:** 1 PDF (2.1 MB)
|
||||
- **Content:** Introductory pandemic/H1N1 reference (91 pages).
|
||||
- **Status:** Pending
|
||||
|
||||
### 12. Cold War Culture
|
||||
- **IA Identifier:** `coldwar_culture_anthro_misc`
|
||||
- **Files:** 6 PDFs (2.6 MB)
|
||||
- **Content:** Academic papers on Cold War survivalist culture and civil defense.
|
||||
- **Status:** Pending
|
||||
|
||||
### 13. Preppers Checklist
|
||||
- **IA Identifier:** `preppers-checklist-and-a-basic-survival-guide-for-any-emergency`
|
||||
- **Files:** 1 PDF (0.4 MB)
|
||||
- **Content:** Emergency preparedness checklist + basic survival guide.
|
||||
- **Status:** Pending
|
||||
|
||||
### 14. Recession Proof Guide
|
||||
- **IA Identifier:** `recession-proof-guide-3`
|
||||
- **Files:** 1 PDF (1.5 MB)
|
||||
- **Content:** City Prepping recession preparedness guide.
|
||||
- **Status:** Pending
|
||||
|
||||
### 15. Facebook American Nationalist Movement
|
||||
- **IA Identifier:** `facebook-american-nationalist-movement-july-2020_202007`
|
||||
- **Files:** 3 PDFs (1.3 MB)
|
||||
- **Content:** Facebook group admins, members, and posts archive (July 2020).
|
||||
- **Status:** Pending
|
||||
|
||||
### 16. Prepping For What
|
||||
- **IA Identifier:** `prepping-for-what`
|
||||
- **Files:** PNGs + HTML (no PDFs) (~16 MB total)
|
||||
- **Content:** Reddit posts on nuclear war, climate change, disaster preparedness. Images and HTML.
|
||||
- **Note:** Download all files (no PDF filter).
|
||||
- **Status:** Pending
|
||||
|
||||
### 17. Preppers Home Defense
|
||||
- **IA Identifier:** `PreppersHomeDefense`
|
||||
- **Files:** 1 EPUB (3.3 MB)
|
||||
- **Content:** Home defense guide.
|
||||
- **Note:** EPUB format, not PDF.
|
||||
- **Status:** Pending
|
||||
|
||||
### 18. Expedient Hand Grenades
|
||||
- **IA Identifier:** `ExpedientHandGrenadesDmitrieffDesertPublications.pdf`
|
||||
- **Files:** 1 ZIP (3.2 MB)
|
||||
- **Content:** George Dmitrieff / Desert Publications reference (69 pages).
|
||||
- **Note:** PDF inside ZIP archive.
|
||||
- **Status:** Pending
|
||||
|
||||
### 19. Mockmill Grain Mill Instructions
|
||||
- **IA Identifier:** `mockmill-100-200-grain-mill-instructions-d.-d.-teoli-jr.-a.-c.`
|
||||
- **Files:** 1 PDF (0.5 MB)
|
||||
- **Content:** Mockmill 100/200 grain mill operating instructions.
|
||||
- **Status:** Pending
|
||||
|
||||
---
|
||||
|
||||
## Skipped
|
||||
|
||||
### preppers_paradise — Prepper_Ark
|
||||
- **Reason:** 94% overlap with local Survival-Companion-Library. IA versions are higher-res scans (3-15x larger) of the same PDFs. 2,449 of 2,611 titles already in local library.
|
||||
- **Unique files:** 162 PDFs (0.4 GB) — minor gaps in distillers, fire, herbs, primitive survival.
|
||||
- **Decision:** Skip unless we want the hi-res versions later.
|
||||
|
||||
### Lateral 7.2
|
||||
- **Reason:** Academic cultural studies journal — sociology papers, not prepper material.
|
||||
|
||||
### Ancapistao
|
||||
- **Reason:** Portuguese-language anarcho-capitalist manifesto. Not practical reference.
|
||||
|
||||
### De Google
|
||||
- **Reason:** Essentially empty (0.0 MB PDF).
|
||||
|
||||
### Navigating the Greatest Shift
|
||||
- **Reason:** New age astrology/predictions, not practical preparedness.
|
||||
|
||||
---
|
||||
|
||||
## Totals
|
||||
|
||||
- **Queued:** 1,453 PDFs + assorted files, ~16.9 GB
|
||||
- **Destination:** TBD (pi-nas /export/library/)
|
||||
|
||||
---
|
||||
|
||||
*Last updated: 2026-02-14*
|
||||
Loading…
Add table
Add a link
Reference in a new issue