docs: migrate Authentik (SSO keystone) to edge2 CT 105
- Authentik -> edge2 CT 105 (Postgres pg_dump/restore; SECRET_KEY carried verbatim; zero-downtime until ~2s cutover) - Multi-block Caddy cutover: auth.echo6.co + notes.echo6.co outpost/forward_auth -> 100.64.0.36:9000 - runbook: add reboot tailscale-before-docker gotcha; clarify dnsmasq must NOT be repointed (points at Caddy host) - source left stopped + intact on Contabo as cold rollback Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
24
engine/.githooks/pre-commit
Executable file
|
|
@ -0,0 +1,24 @@
|
|||
#!/usr/bin/env bash
|
||||
# .githooks/pre-commit — Vault lint report hook
|
||||
#
|
||||
# Runs lint.py in REPORT mode (never --strict) so it writes lint-report.md
|
||||
# and always exits 0. This hook MUST NOT block commits — the root autocommit
|
||||
# cron (echo6-agent) depends on commits succeeding unconditionally.
|
||||
#
|
||||
# To wire: git config core.hooksPath engine/.githooks
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(git rev-parse --show-toplevel)"
|
||||
LINT="${REPO_ROOT}/engine/lib/lint.py"
|
||||
|
||||
if [[ ! -f "${LINT}" ]]; then
|
||||
echo "[lint] WARNING: lint.py not found at ${LINT}, skipping." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "[lint] Running vault lint (report mode)..." >&2
|
||||
python3 "${LINT}" >&2 || true # || true: never fail the commit
|
||||
|
||||
# Always exit 0 — do NOT change this to exit 1 or add --strict.
|
||||
exit 0
|
||||
9
engine/Modelfile
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
# Echo6 Vault Tagger — pinned, reproducible. Build: ollama create vault-tagger -f Modelfile
|
||||
FROM qwen2.5:7b-instruct-q8_0
|
||||
|
||||
PARAMETER temperature 0.1
|
||||
PARAMETER top_p 0.9
|
||||
PARAMETER num_ctx 8192
|
||||
|
||||
# System prompt is kept in prompts/system.md (source of truth). Keep this in sync on bootstrap.
|
||||
SYSTEM """You are the Echo6 vault tagger. Given a markdown document and a controlled vocabulary (topic categories + an entity lexicon of this user's hosts, services, containers, projects, and acronyms), you classify the document and extract entities. You output ONLY valid JSON matching the requested schema. You never invent tags outside the provided vocabulary. You never expand acronyms you are unsure about — you flag them instead."""
|
||||
68
engine/README.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# Echo6 Vault Engine
|
||||
|
||||
Self-contained maintenance engine for the Obsidian knowledge vault at
|
||||
`/home/zvx/projects/.ref/vault/`. Runs entirely on cortex (RTX A4000).
|
||||
|
||||
## What it does
|
||||
|
||||
| Layer | Tool | Job |
|
||||
|-------|------|-----|
|
||||
| Tagger | Qwen2.5-7B-Instruct (Ollama) | Classifies docs; assigns topic tags, type, entity refs |
|
||||
| Embeddings | bge-m3 via TEI (reused service) | Embeds docs into Qdrant for similarity / related links |
|
||||
| Lint | `lib/lint.py` (deterministic) | Enforces frontmatter schema; fixes safe violations automatically |
|
||||
| Agent | `lib/agent.py` | Orchestrates tagger + embeddings over changed docs |
|
||||
| Sweep | `sweep.sh` | Daily entry point; GPU guard, lint, agent, changelog |
|
||||
|
||||
## File layout
|
||||
|
||||
```
|
||||
engine/
|
||||
config.yaml — single source of truth (endpoints, vocab, schema, schedule)
|
||||
Modelfile — pinned vault-tagger build (qwen2.5:7b-instruct-q8_0, temp 0.1)
|
||||
bootstrap.sh — idempotent setup: verify services, build model, install cron
|
||||
sweep.sh — daily maintenance sweep (called by cron)
|
||||
prompts/
|
||||
system.md — canonical system prompt (source of truth; synced into Modelfile)
|
||||
fewshot.md — tagged examples for prompt engineering (added in Step 5)
|
||||
lib/
|
||||
vocab_gen.py — generates vocab.json from live infra inventory (Step 3)
|
||||
lint.py — deterministic frontmatter linter (Step 4)
|
||||
agent.py — tagger + embeddings agent (Step 5)
|
||||
vocab.json — generated entity lexicon (not committed; built by vocab_gen.py)
|
||||
changelog.md — append-only audit log of all automated changes
|
||||
```
|
||||
|
||||
## How it runs
|
||||
|
||||
- **Daily cron** (`0 9 * * *` UTC): `sweep.sh` checks GPU VRAM, runs lint, runs agent
|
||||
over docs changed since last sweep, appends to `changelog.md`.
|
||||
- **Git pre-commit hook** (Step 4): runs lint against staged vault docs before commit.
|
||||
- **Manual**: `./sweep.sh` or `python3 lib/agent.py --full` to reprocess all docs.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
./bootstrap.sh
|
||||
```
|
||||
|
||||
`bootstrap.sh` is idempotent and documents every step. Base model weights (~8 GB) are
|
||||
pulled by bootstrap — they are not committed to the repo. Rebuilding from scratch:
|
||||
|
||||
```bash
|
||||
./bootstrap.sh # pulls qwen2.5:7b-instruct-q8_0, builds vault-tagger, installs cron
|
||||
```
|
||||
|
||||
## Vocabulary
|
||||
|
||||
- **Tier 1 — topic tags**: stable list in `config.yaml` under `topic_categories`
|
||||
- **Tier 2 — entity lexicon**: generated into `vocab.json` by `lib/vocab_gen.py`
|
||||
from live Proxmox, Docker, and Headscale inventory. Regenerate anytime with:
|
||||
`python3 lib/vocab_gen.py`
|
||||
|
||||
## Configuration
|
||||
|
||||
All tunables are in `config.yaml`. Key settings:
|
||||
|
||||
- `behavior.auto_apply` — write changes directly (true) or dry-run only (false)
|
||||
- `behavior.confidence_threshold` — below this, changes are flagged not applied
|
||||
- `schedule.defer_if_gpu_busy_mib` — skip sweep if GPU is already under load
|
||||
123
engine/bootstrap.sh
Executable file
|
|
@ -0,0 +1,123 @@
|
|||
#!/usr/bin/env bash
|
||||
# bootstrap.sh — Echo6 Vault Engine setup
|
||||
#
|
||||
# Idempotent setup script. Safe to re-run; each step is guarded.
|
||||
# Run from any directory. Script locates itself via SCRIPT_DIR.
|
||||
#
|
||||
# Steps:
|
||||
# 1. Verify services reachable (ollama, TEI, qdrant)
|
||||
# 2. [DISABLED] Pull base model (~8 GB) — enable when ready
|
||||
# 3. [DISABLED] Build vault-tagger modelfile — depends on step 2
|
||||
# 4. TODO: Install Python deps for lib/
|
||||
# 5. TODO: Generate initial vocab (lib/vocab_gen.py)
|
||||
# 6. TODO: Install git pre-commit hook
|
||||
# 7. TODO: Install cron job from config.yaml schedule
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
CONFIG="${SCRIPT_DIR}/config.yaml"
|
||||
MODELFILE="${SCRIPT_DIR}/Modelfile"
|
||||
|
||||
echo "==> Echo6 Vault Engine bootstrap"
|
||||
echo " engine_dir : ${SCRIPT_DIR}"
|
||||
echo " config : ${CONFIG}"
|
||||
echo ""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 1 — Verify services reachable
|
||||
# ---------------------------------------------------------------------------
|
||||
echo "[1/7] Checking service endpoints..."
|
||||
|
||||
check_endpoint() {
|
||||
local name="$1"
|
||||
local url="$2"
|
||||
if curl -sf --max-time 5 "$url" > /dev/null 2>&1; then
|
||||
echo " OK ${name} (${url})"
|
||||
else
|
||||
echo " FAIL ${name} (${url}) — is the service running?"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
check_endpoint "ollama" "http://localhost:11434"
|
||||
check_endpoint "TEI/bge-m3" "http://localhost:8090/health"
|
||||
check_endpoint "qdrant" "http://localhost:6333"
|
||||
|
||||
echo ""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 2 — Pull base model (DISABLED — enable when ready to pull ~8 GB)
|
||||
# ---------------------------------------------------------------------------
|
||||
# Uncomment the following block together with Step 3 when ready:
|
||||
#
|
||||
# echo "[2/7] Pulling base model qwen2.5:7b-instruct-q8_0..."
|
||||
# ollama pull qwen2.5:7b-instruct-q8_0
|
||||
# echo ""
|
||||
|
||||
echo "[2/7] SKIPPED — base model pull disabled. Uncomment in bootstrap.sh when ready."
|
||||
echo ""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 3 — Build vault-tagger from Modelfile (DISABLED — depends on step 2)
|
||||
# ---------------------------------------------------------------------------
|
||||
# Uncomment together with Step 2:
|
||||
#
|
||||
# echo "[3/7] Building vault-tagger model from Modelfile..."
|
||||
# ollama create vault-tagger -f "${MODELFILE}"
|
||||
# echo ""
|
||||
|
||||
echo "[3/7] SKIPPED — vault-tagger build disabled. Uncomment after pulling base model."
|
||||
echo ""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 4 — Python dependencies for lib/
|
||||
# ---------------------------------------------------------------------------
|
||||
# TODO (Step 2): Install Python deps once requirements.txt is written.
|
||||
# DO NOT run pip install without explicit approval from Matt.
|
||||
#
|
||||
# Example (do NOT uncomment without approval):
|
||||
# pip install -r "${SCRIPT_DIR}/requirements.txt"
|
||||
|
||||
echo "[4/7] TODO — Python deps not yet defined. See lib/ stubs. Do not pip install without approval."
|
||||
echo ""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 5 — Generate initial vocab (entity lexicon)
|
||||
# ---------------------------------------------------------------------------
|
||||
# TODO (Step 3): Calls lib/vocab_gen.py to query proxmox/docker/headscale
|
||||
# and write engine/vocab.json.
|
||||
#
|
||||
# Example:
|
||||
# python3 "${SCRIPT_DIR}/lib/vocab_gen.py"
|
||||
|
||||
echo "[5/7] TODO — vocab_gen.py not yet implemented (Step 3)."
|
||||
echo ""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 6 — Install git pre-commit hook
|
||||
# ---------------------------------------------------------------------------
|
||||
# TODO (Step 4): Symlink or copy a pre-commit hook that runs lint.py
|
||||
# against staged vault docs before commit.
|
||||
#
|
||||
# Example:
|
||||
# HOOK="${SCRIPT_DIR}/../../.git/hooks/pre-commit"
|
||||
# ln -sf "${SCRIPT_DIR}/hooks/pre-commit" "${HOOK}"
|
||||
|
||||
echo "[6/7] TODO — git pre-commit hook not yet implemented (Step 4)."
|
||||
echo ""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 7 — Install cron job
|
||||
# ---------------------------------------------------------------------------
|
||||
# TODO (Step 6): Install cron from schedule.cron in config.yaml.
|
||||
# Cron entry should call sweep.sh with appropriate guards.
|
||||
#
|
||||
# config.yaml schedule.cron: "0 9 * * *"
|
||||
# Example crontab line:
|
||||
# 0 9 * * * /home/zvx/projects/.ref/engine/sweep.sh >> /home/zvx/projects/.ref/engine/sweep.log 2>&1
|
||||
|
||||
echo "[7/7] TODO — cron job not yet installed (Step 6)."
|
||||
echo ""
|
||||
|
||||
echo "==> bootstrap.sh complete (partial — disabled steps noted above)."
|
||||
54
engine/config.yaml
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# Echo6 Vault Engine — configuration (single source of truth for the engine)
|
||||
vault_dir: /home/zvx/projects/.ref/vault
|
||||
engine_dir: /home/zvx/projects/.ref/engine
|
||||
|
||||
models:
|
||||
tagger:
|
||||
ollama_endpoint: http://localhost:11434
|
||||
model: vault-tagger # built by bootstrap from Modelfile
|
||||
base_model: qwen2.5:7b-instruct-q8_0
|
||||
temperature: 0.1
|
||||
format: json
|
||||
embeddings:
|
||||
tei_endpoint: http://localhost:8090 # existing bge-m3 service (reuse)
|
||||
model: bge-m3
|
||||
qdrant_endpoint: http://localhost:6333
|
||||
qdrant_collection: vault_docs
|
||||
|
||||
# Stable topic-tag vocabulary (tier 1). The entity lexicon (tier 2) is GENERATED into vocab.json.
|
||||
topic_categories:
|
||||
- mesh
|
||||
- matrix
|
||||
- recon
|
||||
- media
|
||||
- auth
|
||||
- dns
|
||||
- vpn
|
||||
- storage
|
||||
- proxmox
|
||||
- ai
|
||||
- mail
|
||||
|
||||
# Read-only inventory sources for the generated entity lexicon (tier 2)
|
||||
inventory:
|
||||
proxmox_nodes: [data, utility, cloud, media, toc] # pct list / qm list
|
||||
proxmox_vps: [contabo, edge2] # pct list
|
||||
docker_hosts: [cortex, utility, media, contabo] # docker ps
|
||||
headscale_host: contabo # headscale nodes list
|
||||
ssh_user: zvx
|
||||
|
||||
# Frontmatter property schema enforced by lint
|
||||
frontmatter_schema:
|
||||
required: [title, type, tags, updated]
|
||||
optional: [aliases, related, status]
|
||||
types: [reference, runbook, project, note, index, session]
|
||||
|
||||
behavior:
|
||||
auto_apply: true # write changes directly...
|
||||
log_changes: true # ...but log every change for audit/revert
|
||||
changelog: /home/zvx/projects/.ref/engine/changelog.md
|
||||
confidence_threshold: 0.6 # below this, flag in changelog instead of silent
|
||||
|
||||
schedule:
|
||||
cron: "0 9 * * *" # 09:00 UTC daily (off-peak); guard checks GPU before running
|
||||
defer_if_gpu_busy_mib: 6000 # skip/defer if >this much VRAM already in use
|
||||
0
engine/lib/__init__.py
Normal file
BIN
engine/lib/__pycache__/vocab_gen.cpython-312.pyc
Normal file
82
engine/lib/agent.py
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
"""
|
||||
agent.py — Vault Tagger + Embeddings Agent
|
||||
|
||||
Job:
|
||||
Run the vault-tagger LLM (Qwen2.5-7B via Ollama) over new or changed vault documents,
|
||||
update their frontmatter with corrected tags/type/entities, then re-embed them into
|
||||
Qdrant via the existing bge-m3 TEI service.
|
||||
|
||||
Pipeline per document:
|
||||
1. Read doc + current frontmatter
|
||||
2. Load vocab: topic_categories from config.yaml + entity lexicon from vocab.json
|
||||
3. Call vault-tagger via Ollama /api/generate (JSON mode, temp 0.1)
|
||||
4. Parse JSON response; validate fields against vocabulary
|
||||
5. If confidence >= threshold: apply tags/type to frontmatter (auto_apply)
|
||||
Else: flag in changelog, do not modify file
|
||||
6. Re-embed via TEI bge-m3 and upsert into Qdrant (collection: vault_docs)
|
||||
7. Update related wikilinks in frontmatter.related if embedding similarity > 0.85
|
||||
8. Append changelog entry (file, old tags, new tags, confidence, timestamp)
|
||||
|
||||
State tracking:
|
||||
Maintains engine/.last_sweep (ISO timestamp) to process only docs modified since last run.
|
||||
Pass --full to reprocess all docs.
|
||||
|
||||
Implemented in: Step 5
|
||||
"""
|
||||
|
||||
# TODO (Step 5): imports — pathlib, json, yaml, httpx or requests, datetime, argparse, logging
|
||||
|
||||
|
||||
def load_config(config_path: str) -> dict:
|
||||
"""Load and return parsed config.yaml."""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
def load_vocab(engine_dir: str) -> dict:
|
||||
"""Load topic_categories from config + entity lexicon from vocab.json."""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
def get_changed_docs(vault_dir: str, since: str) -> list:
|
||||
"""Return list of .md paths modified after `since` (ISO timestamp)."""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
def call_tagger(doc_text: str, vocab: dict, config: dict) -> dict:
|
||||
"""
|
||||
POST to Ollama /api/generate with vault-tagger model.
|
||||
Returns parsed JSON response dict.
|
||||
"""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
def embed_document(doc_text: str, tei_endpoint: str) -> list[float]:
|
||||
"""POST to TEI bge-m3 endpoint; return embedding vector."""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
def upsert_qdrant(doc_id: str, vector: list[float], payload: dict, config: dict) -> None:
|
||||
"""Upsert a document vector + metadata into Qdrant vault_docs collection."""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
def find_related(doc_id: str, vector: list[float], config: dict, threshold: float = 0.85) -> list[str]:
|
||||
"""Query Qdrant for nearest neighbours above threshold; return doc ids."""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
def apply_tagger_result(path, result: dict, config: dict) -> dict:
|
||||
"""
|
||||
Write tagger output back to doc frontmatter if confidence >= threshold.
|
||||
Returns summary of changes made.
|
||||
"""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Entry point. Parse args, load state, process changed docs, update state."""
|
||||
raise NotImplementedError("implemented in step 5")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
697
engine/lib/lint.py
Normal file
|
|
@ -0,0 +1,697 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
lint.py — Deterministic Vault Lint (no LLM, stdlib only)
|
||||
|
||||
Checks performed (all deterministic):
|
||||
1. Frontmatter schema — required keys, valid type, tags is a list
|
||||
2. Tag vocabulary — tags must be in topic_categories + {meta}
|
||||
3. Dead wikilinks — [[target]] resolves against note basenames + entity vocab
|
||||
4. Orphans — notes with zero incoming wikilinks (INFO, capped at 40)
|
||||
|
||||
Severity:
|
||||
ERROR — dead wikilinks
|
||||
WARN — schema violations, unknown tags
|
||||
INFO — orphan notes
|
||||
|
||||
Usage:
|
||||
python3 engine/lib/lint.py # report mode, always exit 0
|
||||
python3 engine/lib/lint.py --strict # exit 1 if any ERROR findings
|
||||
|
||||
Writes: engine/lint-report.md
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import NamedTuple
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Config / vocab loading (stdlib-only minimal YAML parser)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _parse_simple_yaml(text: str) -> dict:
|
||||
"""
|
||||
Minimal YAML parser: handles only the scalar/list constructs in config.yaml.
|
||||
Supports: key: value, key: [a, b], - item under a key, # comments.
|
||||
Does NOT handle nested dicts beyond top-level — enough for config.yaml.
|
||||
"""
|
||||
result: dict = {}
|
||||
current_key = None
|
||||
current_list: list | None = None
|
||||
|
||||
for raw_line in text.splitlines():
|
||||
line = raw_line.rstrip()
|
||||
stripped = line.lstrip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
# Flush pending list if indent drops
|
||||
if current_list is not None and not line.startswith(" "):
|
||||
result[current_key] = current_list
|
||||
current_list = None
|
||||
current_key = None
|
||||
continue
|
||||
|
||||
# List item under current key
|
||||
if stripped.startswith("- ") and current_list is not None:
|
||||
current_list.append(stripped[2:].strip().strip('"').strip("'"))
|
||||
continue
|
||||
|
||||
# End of list block
|
||||
if current_list is not None and not stripped.startswith("-"):
|
||||
result[current_key] = current_list
|
||||
current_list = None
|
||||
current_key = None
|
||||
|
||||
if ":" in stripped:
|
||||
key, _, value = stripped.partition(":")
|
||||
key = key.strip()
|
||||
value = value.strip()
|
||||
if value.startswith("[") and value.endswith("]"):
|
||||
# Inline list: [a, b, c]
|
||||
inner = value[1:-1]
|
||||
result[key] = [v.strip().strip('"').strip("'") for v in inner.split(",") if v.strip()]
|
||||
elif value == "":
|
||||
# May start a list block
|
||||
current_key = key
|
||||
current_list = []
|
||||
elif value.startswith("#"):
|
||||
result[key] = ""
|
||||
else:
|
||||
# Scalar — strip inline comment
|
||||
value = value.split(" #")[0].strip().strip('"').strip("'")
|
||||
result[key] = value
|
||||
|
||||
if current_list is not None and current_key:
|
||||
result[current_key] = current_list
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def load_config(engine_dir: Path) -> dict:
|
||||
"""Load config.yaml using minimal YAML parser."""
|
||||
config_path = engine_dir / "config.yaml"
|
||||
text = config_path.read_text(encoding="utf-8")
|
||||
return _parse_simple_yaml(text)
|
||||
|
||||
|
||||
def load_vocab(engine_dir: Path) -> dict:
|
||||
"""Load vocab.json."""
|
||||
vocab_path = engine_dir / "vocab.json"
|
||||
return json.loads(vocab_path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frontmatter parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_FM_FENCE = re.compile(r"^---\s*$")
|
||||
|
||||
|
||||
def parse_frontmatter(path: Path) -> tuple[dict | None, str]:
|
||||
"""
|
||||
Parse YAML frontmatter block from a markdown file.
|
||||
Returns (frontmatter_dict_or_None, body_text).
|
||||
body_text is the full file text if no frontmatter.
|
||||
"""
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
lines = text.splitlines(keepends=True)
|
||||
|
||||
if not lines or not _FM_FENCE.match(lines[0].rstrip()):
|
||||
return None, text
|
||||
|
||||
end_idx = None
|
||||
for i in range(1, len(lines)):
|
||||
if _FM_FENCE.match(lines[i].rstrip()):
|
||||
end_idx = i
|
||||
break
|
||||
|
||||
if end_idx is None:
|
||||
return None, text
|
||||
|
||||
fm_text = "".join(lines[1:end_idx])
|
||||
body = "".join(lines[end_idx + 1 :])
|
||||
fm = _parse_yaml_frontmatter(fm_text)
|
||||
return fm, body
|
||||
|
||||
|
||||
def _parse_yaml_frontmatter(text: str) -> dict:
|
||||
"""
|
||||
Parse simple flat YAML frontmatter. Handles:
|
||||
key: scalar
|
||||
key: [a, b, c]
|
||||
key:
|
||||
- a
|
||||
- b
|
||||
tags: [a, b]
|
||||
tags:
|
||||
- a
|
||||
"""
|
||||
result: dict = {}
|
||||
current_key: str | None = None
|
||||
current_list: list | None = None
|
||||
|
||||
for raw_line in text.splitlines():
|
||||
line = raw_line.rstrip()
|
||||
stripped = line.lstrip()
|
||||
|
||||
if not stripped:
|
||||
continue
|
||||
|
||||
# List item
|
||||
if stripped.startswith("- ") and current_list is not None:
|
||||
current_list.append(stripped[2:].strip().strip('"').strip("'"))
|
||||
continue
|
||||
|
||||
# New key — flush pending list
|
||||
if current_list is not None:
|
||||
result[current_key] = current_list
|
||||
current_list = None
|
||||
current_key = None
|
||||
|
||||
if ":" in stripped:
|
||||
key, _, value = stripped.partition(":")
|
||||
key = key.strip()
|
||||
value = value.strip()
|
||||
|
||||
if value.startswith("[") and value.endswith("]"):
|
||||
inner = value[1:-1]
|
||||
result[key] = [
|
||||
v.strip().strip('"').strip("'")
|
||||
for v in inner.split(",")
|
||||
if v.strip()
|
||||
]
|
||||
elif value == "":
|
||||
current_key = key
|
||||
current_list = []
|
||||
else:
|
||||
result[key] = value.strip('"').strip("'")
|
||||
|
||||
if current_list is not None and current_key:
|
||||
result[current_key] = current_list
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 1: Frontmatter schema
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
VALID_TYPES = {"reference", "runbook", "project", "note", "index", "session"}
|
||||
|
||||
|
||||
def check_frontmatter(fm: dict | None, config: dict) -> list[dict]:
|
||||
"""
|
||||
Returns list of findings dicts: {severity, check, message}
|
||||
WARN severity for all schema violations.
|
||||
"""
|
||||
findings = []
|
||||
|
||||
required_keys = ["title", "type", "tags", "updated"]
|
||||
|
||||
if fm is None:
|
||||
findings.append(
|
||||
{
|
||||
"severity": "WARN",
|
||||
"check": "frontmatter",
|
||||
"message": "missing frontmatter block entirely",
|
||||
}
|
||||
)
|
||||
return findings
|
||||
|
||||
for key in required_keys:
|
||||
if key not in fm or fm[key] is None or fm[key] == "":
|
||||
findings.append(
|
||||
{
|
||||
"severity": "WARN",
|
||||
"check": "frontmatter",
|
||||
"message": f"missing required key: {key!r}",
|
||||
}
|
||||
)
|
||||
|
||||
if "type" in fm and fm["type"]:
|
||||
if fm["type"] not in VALID_TYPES:
|
||||
findings.append(
|
||||
{
|
||||
"severity": "WARN",
|
||||
"check": "frontmatter",
|
||||
"message": f"invalid type {fm['type']!r} — must be one of {sorted(VALID_TYPES)}",
|
||||
}
|
||||
)
|
||||
|
||||
if "tags" in fm and fm["tags"] is not None:
|
||||
if not isinstance(fm["tags"], list):
|
||||
findings.append(
|
||||
{
|
||||
"severity": "WARN",
|
||||
"check": "frontmatter",
|
||||
"message": f"tags must be a list, got {type(fm['tags']).__name__}: {fm['tags']!r}",
|
||||
}
|
||||
)
|
||||
|
||||
return findings
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 2: Tag vocabulary
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
ALLOWED_EXTRA_TAGS = {"meta"}
|
||||
|
||||
|
||||
def check_tags(fm: dict | None, allowed_tags: set[str]) -> list[dict]:
|
||||
"""Returns WARN findings for tags outside the allowed vocabulary."""
|
||||
if fm is None:
|
||||
return []
|
||||
tags = fm.get("tags")
|
||||
if not tags or not isinstance(tags, list):
|
||||
return []
|
||||
|
||||
findings = []
|
||||
for tag in tags:
|
||||
if tag not in allowed_tags:
|
||||
findings.append(
|
||||
{
|
||||
"severity": "WARN",
|
||||
"check": "tag-vocab",
|
||||
"message": f"unknown tag {tag!r} (not in topic_categories or allowed extras)",
|
||||
}
|
||||
)
|
||||
return findings
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 3: Dead wikilinks
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# Matches [[target]], [[target|alias]], [[target#heading]]
|
||||
# But NOT bash [[ ... ]] test syntax.
|
||||
# Strategy: require the inner text to look like a note name:
|
||||
# - no spaces at start/end of target
|
||||
# - may contain word chars, hyphens, dots, spaces, #, | — but NOT operators like
|
||||
# -n, -f, ==, !, &&, ||, etc.
|
||||
# We detect shell syntax by checking for space-separated words starting with - or
|
||||
# containing shell operators.
|
||||
|
||||
_WIKILINK_RE = re.compile(
|
||||
r"""
|
||||
\[\[ # opening [[
|
||||
([^\[\]\n]+?) # capture: link target (non-greedy, no newlines)
|
||||
(?:\|[^\[\]\n]*)? # optional |alias
|
||||
\]\] # closing ]]
|
||||
""",
|
||||
re.VERBOSE,
|
||||
)
|
||||
|
||||
_FENCED_CODE_RE = re.compile(r"```.*?```", re.DOTALL)
|
||||
_INLINE_CODE_RE = re.compile(r"`[^`\n]+`")
|
||||
|
||||
|
||||
def _looks_like_shell(inner: str) -> bool:
|
||||
"""Return True if the wikilink inner text looks like bash test syntax."""
|
||||
# Shell: -n "$VAR", ! -f "$FILE", "$CODEC" == "value", -z, etc.
|
||||
s = inner.strip()
|
||||
# Contains bash operators or variable expansions
|
||||
if re.search(r'\$[{(A-Za-z_]', s):
|
||||
return True
|
||||
# Starts with - (flag) or ! (negation), or contains == / != / && / ||
|
||||
if re.match(r'^\s*[!-]', s):
|
||||
return True
|
||||
if re.search(r'==|!=|&&|\|\|', s):
|
||||
return True
|
||||
# Multiple space-separated tokens that look like args
|
||||
tokens = s.split()
|
||||
if len(tokens) > 1 and tokens[0].startswith("-"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _is_note_like(inner: str) -> bool:
|
||||
"""
|
||||
Return True if inner text looks like a note name:
|
||||
word chars, hyphens, dots, spaces, but no shell operators.
|
||||
"""
|
||||
if _looks_like_shell(inner):
|
||||
return False
|
||||
# Must match a reasonable note-name pattern
|
||||
# Allow: letters, digits, hyphens, underscores, dots, spaces
|
||||
return bool(re.match(r'^[\w\s.\-/]+$', inner.strip()))
|
||||
|
||||
|
||||
def _strip_code_blocks(text: str) -> str:
|
||||
"""Remove fenced code blocks and inline code to avoid false wikilink matches."""
|
||||
text = _FENCED_CODE_RE.sub("", text)
|
||||
text = _INLINE_CODE_RE.sub("", text)
|
||||
return text
|
||||
|
||||
|
||||
def _extract_wikilinks(body: str) -> list[str]:
|
||||
"""Extract note targets from wikilinks in body text, ignoring code blocks and shell syntax."""
|
||||
clean = _strip_code_blocks(body)
|
||||
targets = []
|
||||
for m in _WIKILINK_RE.finditer(clean):
|
||||
raw = m.group(1)
|
||||
# Strip heading: [[target#heading|alias]] -> target
|
||||
# Strip alias already handled by regex (group 1 = before |)
|
||||
target = raw.split("|")[0].split("#")[0].strip()
|
||||
if target and _is_note_like(target):
|
||||
targets.append(target)
|
||||
return targets
|
||||
|
||||
|
||||
def _note_basename(path: Path) -> str:
|
||||
"""Return the basename (without extension) of a vault note, lowercased."""
|
||||
return path.stem.lower()
|
||||
|
||||
|
||||
def _normalize_name(name: str) -> str:
|
||||
"""Normalize: lowercase, spaces↔hyphens."""
|
||||
return name.lower().replace(" ", "-").replace("_", "-")
|
||||
|
||||
|
||||
def build_note_index(vault_dir: Path) -> dict[str, Path]:
|
||||
"""
|
||||
Build a mapping of normalized name → Path for all vault notes.
|
||||
One entry per file, keyed by normalized basename.
|
||||
"""
|
||||
index: dict[str, Path] = {}
|
||||
for p in vault_dir.rglob("*.md"):
|
||||
key = _normalize_name(p.stem)
|
||||
index[key] = p
|
||||
return index
|
||||
|
||||
|
||||
def build_entity_names(vocab: dict) -> set[str]:
|
||||
"""
|
||||
Build a set of normalized entity names + aliases from vocab.json.
|
||||
These count as valid wikilink targets (they're in the knowledge graph).
|
||||
"""
|
||||
names: set[str] = set()
|
||||
for entity in vocab.get("entities", []):
|
||||
names.add(_normalize_name(entity["name"]))
|
||||
names.add(_normalize_name(entity["display"]))
|
||||
for alias in entity.get("aliases", []):
|
||||
names.add(_normalize_name(alias))
|
||||
for cat in vocab.get("topic_categories", []):
|
||||
names.add(_normalize_name(cat))
|
||||
return names
|
||||
|
||||
|
||||
def check_dead_links(
|
||||
path: Path,
|
||||
body: str,
|
||||
note_index: dict[str, Path],
|
||||
entity_names: set[str],
|
||||
) -> list[dict]:
|
||||
"""
|
||||
Returns ERROR findings for wikilinks that resolve to neither a vault note
|
||||
nor a known entity name.
|
||||
"""
|
||||
findings = []
|
||||
for target in _extract_wikilinks(body):
|
||||
norm = _normalize_name(target)
|
||||
if norm in note_index or norm in entity_names:
|
||||
continue
|
||||
findings.append(
|
||||
{
|
||||
"severity": "ERROR",
|
||||
"check": "dead-link",
|
||||
"message": f"dead wikilink [[{target}]]",
|
||||
}
|
||||
)
|
||||
return findings
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Check 4: Orphans (incoming wikilink count)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
ORPHAN_CAP = 40
|
||||
|
||||
|
||||
def build_backlink_counts(
|
||||
vault_dir: Path,
|
||||
note_index: dict[str, Path],
|
||||
entity_names: set[str],
|
||||
) -> dict[str, int]:
|
||||
"""
|
||||
Count how many other notes link TO each note.
|
||||
Returns {normalized_stem: count}.
|
||||
"""
|
||||
counts: dict[str, int] = {k: 0 for k in note_index}
|
||||
|
||||
for p in vault_dir.rglob("*.md"):
|
||||
try:
|
||||
_, body = parse_frontmatter(p)
|
||||
except Exception:
|
||||
continue
|
||||
for target in _extract_wikilinks(body):
|
||||
norm = _normalize_name(target)
|
||||
if norm in counts:
|
||||
counts[norm] += 1
|
||||
|
||||
return counts
|
||||
|
||||
|
||||
def find_orphans(
|
||||
vault_dir: Path,
|
||||
note_index: dict[str, Path],
|
||||
backlink_counts: dict[str, int],
|
||||
) -> list[dict]:
|
||||
"""
|
||||
Return INFO findings for notes with zero incoming wikilinks.
|
||||
Capped at ORPHAN_CAP entries.
|
||||
"""
|
||||
findings = []
|
||||
for key, path in sorted(note_index.items()):
|
||||
if backlink_counts.get(key, 0) == 0:
|
||||
rel = path.relative_to(vault_dir)
|
||||
findings.append(
|
||||
{
|
||||
"severity": "INFO",
|
||||
"check": "orphan",
|
||||
"message": f"no incoming links: {rel}",
|
||||
"_path": path,
|
||||
}
|
||||
)
|
||||
return findings[:ORPHAN_CAP]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Reporting
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class LintResult(NamedTuple):
|
||||
path: Path
|
||||
findings: list[dict]
|
||||
|
||||
|
||||
def format_report(
|
||||
results: list[LintResult],
|
||||
vault_dir: Path,
|
||||
total_docs: int,
|
||||
elapsed_s: float,
|
||||
) -> str:
|
||||
"""Format a human-readable + markdown report."""
|
||||
ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
errors = [f for r in results for f in r.findings if f["severity"] == "ERROR"]
|
||||
warns = [f for r in results for f in r.findings if f["severity"] == "WARN"]
|
||||
infos = [f for r in results for f in r.findings if f["severity"] == "INFO"]
|
||||
|
||||
# Count specific warn subcategories
|
||||
fm_missing = sum(
|
||||
1
|
||||
for r in results
|
||||
for f in r.findings
|
||||
if f["severity"] == "WARN"
|
||||
and f["check"] == "frontmatter"
|
||||
and "missing frontmatter block" in f["message"]
|
||||
)
|
||||
fm_invalid = sum(
|
||||
1
|
||||
for r in results
|
||||
for f in r.findings
|
||||
if f["severity"] == "WARN"
|
||||
and f["check"] == "frontmatter"
|
||||
and "missing frontmatter block" not in f["message"]
|
||||
)
|
||||
tag_warns = sum(
|
||||
1 for r in results for f in r.findings if f["severity"] == "WARN" and f["check"] == "tag-vocab"
|
||||
)
|
||||
|
||||
lines = [
|
||||
f"# Vault Lint Report",
|
||||
f"",
|
||||
f"Generated: {ts} | Docs scanned: {total_docs} | Elapsed: {elapsed_s:.1f}s",
|
||||
f"",
|
||||
f"## Summary",
|
||||
f"",
|
||||
f"| Severity | Count |",
|
||||
f"|----------|-------|",
|
||||
f"| ERROR (dead links) | {len(errors)} |",
|
||||
f"| WARN (schema) | {len(warns)} |",
|
||||
f"| INFO (orphans) | {len(infos)} |",
|
||||
f"",
|
||||
f"### WARN breakdown",
|
||||
f"- Missing frontmatter block: {fm_missing}",
|
||||
f"- Invalid/missing frontmatter fields: {fm_invalid}",
|
||||
f"- Unknown tags: {tag_warns}",
|
||||
f"",
|
||||
]
|
||||
|
||||
if errors:
|
||||
lines.append("## ERROR — Dead Wikilinks")
|
||||
lines.append("")
|
||||
for r in results:
|
||||
errs = [f for f in r.findings if f["severity"] == "ERROR"]
|
||||
if errs:
|
||||
rel = r.path.relative_to(vault_dir)
|
||||
for f in errs:
|
||||
lines.append(f"- `{rel}` — {f['message']}")
|
||||
lines.append("")
|
||||
else:
|
||||
lines.append("## ERROR — Dead Wikilinks")
|
||||
lines.append("")
|
||||
lines.append("_None. All wikilinks resolve._")
|
||||
lines.append("")
|
||||
|
||||
if warns:
|
||||
lines.append("## WARN — Schema & Tag Violations")
|
||||
lines.append("")
|
||||
for r in results:
|
||||
ws = [f for f in r.findings if f["severity"] == "WARN"]
|
||||
if ws:
|
||||
rel = r.path.relative_to(vault_dir)
|
||||
for f in ws:
|
||||
lines.append(f"- `{rel}` — {f['message']}")
|
||||
lines.append("")
|
||||
|
||||
if infos:
|
||||
lines.append(f"## INFO — Orphan Notes (no incoming links, capped at {ORPHAN_CAP})")
|
||||
lines.append("")
|
||||
for f in infos:
|
||||
lines.append(f"- {f['message']}")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Public API (importable by sweep.sh or other scripts)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def run_lint(vault_dir: Path, engine_dir: Path) -> tuple[list[LintResult], dict]:
|
||||
"""
|
||||
Run all lint checks. Returns (results, stats_dict).
|
||||
Importable entry point.
|
||||
"""
|
||||
import time
|
||||
|
||||
config = load_config(engine_dir)
|
||||
vocab = load_vocab(engine_dir)
|
||||
|
||||
# Build allowed tag set
|
||||
topic_cats = set(vocab.get("topic_categories", []))
|
||||
topic_cats |= ALLOWED_EXTRA_TAGS
|
||||
|
||||
note_index = build_note_index(vault_dir)
|
||||
entity_names = build_entity_names(vocab)
|
||||
|
||||
# Build backlink counts (single pass over all docs)
|
||||
backlink_counts = build_backlink_counts(vault_dir, note_index, entity_names)
|
||||
|
||||
all_paths = sorted(vault_dir.rglob("*.md"))
|
||||
results: list[LintResult] = []
|
||||
|
||||
t0 = time.monotonic()
|
||||
|
||||
for path in all_paths:
|
||||
findings: list[dict] = []
|
||||
try:
|
||||
fm, body = parse_frontmatter(path)
|
||||
except Exception as exc:
|
||||
findings.append(
|
||||
{"severity": "WARN", "check": "parse-error", "message": f"could not parse: {exc}"}
|
||||
)
|
||||
results.append(LintResult(path=path, findings=findings))
|
||||
continue
|
||||
|
||||
findings.extend(check_frontmatter(fm, config))
|
||||
findings.extend(check_tags(fm, topic_cats))
|
||||
findings.extend(check_dead_links(path, body, note_index, entity_names))
|
||||
results.append(LintResult(path=path, findings=findings))
|
||||
|
||||
# Orphan check (uses backlink counts already computed)
|
||||
orphan_findings = find_orphans(vault_dir, note_index, backlink_counts)
|
||||
# Attach orphan findings as a synthetic "vault" result
|
||||
if orphan_findings:
|
||||
results.append(LintResult(path=vault_dir / "_orphans_", findings=orphan_findings))
|
||||
|
||||
elapsed = time.monotonic() - t0
|
||||
|
||||
stats = {
|
||||
"total_docs": len(all_paths),
|
||||
"errors": sum(1 for r in results for f in r.findings if f["severity"] == "ERROR"),
|
||||
"warns": sum(1 for r in results for f in r.findings if f["severity"] == "WARN"),
|
||||
"infos": sum(1 for r in results for f in r.findings if f["severity"] == "INFO"),
|
||||
"elapsed_s": elapsed,
|
||||
}
|
||||
return results, stats
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
import time
|
||||
|
||||
strict = "--strict" in sys.argv
|
||||
|
||||
# Locate engine/vault dirs relative to this file
|
||||
this_file = Path(__file__).resolve()
|
||||
engine_dir = this_file.parent.parent # engine/lib/lint.py → engine/
|
||||
config = load_config(engine_dir)
|
||||
|
||||
vault_str = config.get("vault_dir", "")
|
||||
if not vault_str:
|
||||
# Fallback: sibling of engine_dir named "vault"
|
||||
vault_dir = engine_dir.parent / "vault"
|
||||
else:
|
||||
vault_dir = Path(vault_str)
|
||||
|
||||
if not vault_dir.exists():
|
||||
print(f"ERROR: vault_dir not found: {vault_dir}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
results, stats = run_lint(vault_dir, engine_dir)
|
||||
|
||||
report_text = format_report(
|
||||
results, vault_dir, stats["total_docs"], stats["elapsed_s"]
|
||||
)
|
||||
|
||||
# Write report
|
||||
report_path = engine_dir / "lint-report.md"
|
||||
report_path.write_text(report_text, encoding="utf-8")
|
||||
|
||||
# Print to stdout
|
||||
print(report_text)
|
||||
print(f"--- Wrote: {report_path} ---")
|
||||
|
||||
# Summary line
|
||||
print(
|
||||
f"\nSummary: {stats['total_docs']} docs | "
|
||||
f"{stats['errors']} ERRORs | {stats['warns']} WARNs | {stats['infos']} INFOs"
|
||||
)
|
||||
|
||||
if strict and stats["errors"] > 0:
|
||||
sys.exit(1)
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
1175
engine/lib/vocab_gen.py
Normal file
170
engine/lint-report.md
Normal file
|
|
@ -0,0 +1,170 @@
|
|||
# Vault Lint Report
|
||||
|
||||
Generated: 2026-06-18T05:41:36Z | Docs scanned: 98 | Elapsed: 0.0s
|
||||
|
||||
## Summary
|
||||
|
||||
| Severity | Count |
|
||||
|----------|-------|
|
||||
| ERROR (dead links) | 0 |
|
||||
| WARN (schema) | 106 |
|
||||
| INFO (orphans) | 38 |
|
||||
|
||||
### WARN breakdown
|
||||
- Missing frontmatter block: 95
|
||||
- Invalid/missing frontmatter fields: 3
|
||||
- Unknown tags: 8
|
||||
|
||||
## ERROR — Dead Wikilinks
|
||||
|
||||
_None. All wikilinks resolve._
|
||||
|
||||
## WARN — Schema & Tag Violations
|
||||
|
||||
- `CLAUDE-baseline.md` — missing frontmatter block entirely
|
||||
- `INDEX.md` — missing required key: 'tags'
|
||||
- `archive/AUDIT-2026-02-21.md` — missing frontmatter block entirely
|
||||
- `archive/README.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/PHASE6_DECISION.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/PLAN.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/appservices.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/archive_receiver_discovery.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/archivist_discovery.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/hookshot_deployment_discovery.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/hookshot_e2ee_discovery.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/hookshot_payload_discovery.md` — missing frontmatter block entirely
|
||||
- `archive/matrix/hookshot_room_targeting_discovery.md` — missing frontmatter block entirely
|
||||
- `archive/projects/DEPLOY-API-KEYS-TAB.md` — missing frontmatter block entirely
|
||||
- `archive/projects/arr-stack-runbook.md` — missing frontmatter block entirely
|
||||
- `archive/projects/arr-wiring-runbook.md` — missing frontmatter block entirely
|
||||
- `archive/projects/cc-deploy-watchtower-v2.md` — missing frontmatter block entirely
|
||||
- `archive/projects/headscale-full-deployment.md` — missing frontmatter block entirely
|
||||
- `archive/projects/last-ember-project.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-phase5-prompt.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-phase6-prompt.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-prompts/mmud-prompts/01-update-planned.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-prompts/mmud-prompts/02-npc-nodes.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-prompts/mmud-prompts/03-darkcragg.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-prompts/mmud-prompts/04-dcrg-node.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-prompts/mmud-prompts/05-phase5.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-prompts/mmud-prompts/06-phase6.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-prompts/mmud-prompts/README.md` — missing frontmatter block entirely
|
||||
- `archive/projects/mmud/mmud-prompts/mmud-prompts/mmud-project.md` — missing frontmatter block entirely
|
||||
- `archive/projects/openwebui-theme-deploy.md` — missing frontmatter block entirely
|
||||
- `archive/projects/peertube-phase2-project.md` — missing frontmatter block entirely
|
||||
- `archive/projects/peertube-rebuild.md` — missing frontmatter block entirely
|
||||
- `archive/projects/utility-caddy-initial-setup.md` — missing frontmatter block entirely
|
||||
- `archive/projects/vaultwarden-deployment.md` — missing frontmatter block entirely
|
||||
- `archive/reports/logistics_migration.md` — missing frontmatter block entirely
|
||||
- `archive/reports/post_validation_report.md` — missing frontmatter block entirely
|
||||
- `archive/reports/task_a_aurora_validation.md` — missing frontmatter block entirely
|
||||
- `archive/reports/task_c_watchdog_test.md` — missing frontmatter block entirely
|
||||
- `docs/hardware/environment.md` — missing frontmatter block entirely
|
||||
- `docs/hardware/ip-allocation.md` — missing frontmatter block entirely
|
||||
- `docs/matrix/archivist.md` — missing frontmatter block entirely
|
||||
- `docs/matrix/matrix_host.md` — missing frontmatter block entirely
|
||||
- `docs/matrix/mautrix_signal.md` — missing frontmatter block entirely
|
||||
- `docs/matrix/synapse.md` — missing frontmatter block entirely
|
||||
- `docs/matrix/synapse_retention_discovery.md` — missing frontmatter block entirely
|
||||
- `docs/navi/cc-rules.md` — missing frontmatter block entirely
|
||||
- `docs/navi/deployment.md` — missing frontmatter block entirely
|
||||
- `docs/navi/themes.md` — missing frontmatter block entirely
|
||||
- `docs/services/ots-setup.md` — missing frontmatter block entirely
|
||||
- `docs/services/services.md` — missing frontmatter block entirely
|
||||
- `docs/services/usenet.md` — missing frontmatter block entirely
|
||||
- `docs/software/authentik.md` — missing frontmatter block entirely
|
||||
- `docs/software/caddy.md` — missing frontmatter block entirely
|
||||
- `docs/software/dns.md` — missing frontmatter block entirely
|
||||
- `docs/software/geo-tools.md` — missing frontmatter block entirely
|
||||
- `docs/software/recon.md` — missing frontmatter block entirely
|
||||
- `docs/software/searxng.md` — missing frontmatter block entirely
|
||||
- `notes/echo6-landing-page-data-export.md` — missing frontmatter block entirely
|
||||
- `notes/ia-download-queue.md` — missing frontmatter block entirely
|
||||
- `plans/vaultwarden-plan.md` — missing frontmatter block entirely
|
||||
- `projects/advbbs-project.md` — missing frontmatter block entirely
|
||||
- `projects/argus.md` — missing frontmatter block entirely
|
||||
- `projects/deploy-livesync.md` — missing frontmatter block entirely
|
||||
- `projects/matrix-synapse-deployment.md` — missing frontmatter block entirely
|
||||
- `projects/meshtastic-headscale-runbook.md` — missing frontmatter block entirely
|
||||
- `projects/mmud-project.md` — missing frontmatter block entirely
|
||||
- `runbooks/add-peertube-channel.md` — missing frontmatter block entirely
|
||||
- `runbooks/authentik-access-groups.md` — missing frontmatter block entirely
|
||||
- `runbooks/authentik-create-invitation.md` — missing frontmatter block entirely
|
||||
- `runbooks/authentik-oidc-application.md` — missing frontmatter block entirely
|
||||
- `runbooks/authentik-upgrade.md` — missing frontmatter block entirely
|
||||
- `runbooks/ct-runbook.md` — missing frontmatter block entirely
|
||||
- `runbooks/edge2-access-reference.md` — missing frontmatter block entirely
|
||||
- `runbooks/expose-service-contabo.md` — missing frontmatter block entirely
|
||||
- `runbooks/expose-service-edge2.md` — missing frontmatter block entirely
|
||||
- `runbooks/expose-service-home.md` — missing frontmatter block entirely
|
||||
- `runbooks/headscale-onboard-node.md` — missing frontmatter block entirely
|
||||
- `runbooks/ia-cli-reference.md` — missing frontmatter block entirely
|
||||
- `runbooks/ia-download-mirror.md` — missing frontmatter block entirely
|
||||
- `runbooks/idahomesh-bridge-setup.md` — missing frontmatter block entirely
|
||||
- `runbooks/idahomesh-vpn-device-setup.md` — missing frontmatter block entirely
|
||||
- `runbooks/lxc-service-migration.md` — missing frontmatter block entirely
|
||||
- `runbooks/mailcow-create-mailbox.md` — missing frontmatter block entirely
|
||||
- `runbooks/meshmonitor-password-reset.md` — missing frontmatter block entirely
|
||||
- `runbooks/meshtastic-sidecar-node.md` — missing frontmatter block entirely
|
||||
- `runbooks/meshtasticd-sim-nodes-runbook.md` — missing frontmatter block entirely
|
||||
- `runbooks/nordvpn-lxc.md` — missing frontmatter block entirely
|
||||
- `runbooks/peertube-remote-runner.md` — missing frontmatter block entirely
|
||||
- `runbooks/pg-backup.md` — missing frontmatter block entirely
|
||||
- `runbooks/pi-nas-omv-runbook.md` — missing frontmatter block entirely
|
||||
- `runbooks/pipeline-patterns.md` — missing frontmatter block entirely
|
||||
- `runbooks/proxmox-create-ubuntu-vm.md` — missing frontmatter block entirely
|
||||
- `runbooks/proxmox-onboard-node.md` — missing frontmatter block entirely
|
||||
- `runbooks/recon-operations.md` — missing frontmatter block entirely
|
||||
- `runbooks/recon-service-integration.md` — missing frontmatter block entirely
|
||||
- `runbooks/syncthing-add-node.md` — missing frontmatter block entirely
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — missing required key: 'updated'
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — invalid type 'session-resume' — must be one of ['index', 'note', 'project', 'reference', 'runbook', 'session']
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — unknown tag 'session-resume' (not in topic_categories or allowed extras)
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — unknown tag 'handoff' (not in topic_categories or allowed extras)
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — unknown tag 'meshai' (not in topic_categories or allowed extras)
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — unknown tag 'meshtastic' (not in topic_categories or allowed extras)
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — unknown tag 'aida-nebra' (not in topic_categories or allowed extras)
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — unknown tag 'diagnostic' (not in topic_categories or allowed extras)
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — unknown tag 'resilience-test' (not in topic_categories or allowed extras)
|
||||
- `session-resume/SESSION-HANDOFF-meshai-test.md` — unknown tag 'open' (not in topic_categories or allowed extras)
|
||||
|
||||
## INFO — Orphan Notes (no incoming links, capped at 40)
|
||||
|
||||
- no incoming links: archive/projects/mmud/mmud-prompts/mmud-prompts/01-update-planned.md
|
||||
- no incoming links: archive/projects/mmud/mmud-prompts/mmud-prompts/02-npc-nodes.md
|
||||
- no incoming links: archive/projects/mmud/mmud-prompts/mmud-prompts/03-darkcragg.md
|
||||
- no incoming links: archive/projects/mmud/mmud-prompts/mmud-prompts/04-dcrg-node.md
|
||||
- no incoming links: archive/projects/mmud/mmud-prompts/mmud-prompts/05-phase5.md
|
||||
- no incoming links: archive/projects/mmud/mmud-prompts/mmud-prompts/06-phase6.md
|
||||
- no incoming links: archive/matrix/appservices.md
|
||||
- no incoming links: archive/matrix/archive_receiver_discovery.md
|
||||
- no incoming links: archive/matrix/archivist_discovery.md
|
||||
- no incoming links: archive/projects/arr-stack-runbook.md
|
||||
- no incoming links: archive/projects/arr-wiring-runbook.md
|
||||
- no incoming links: archive/AUDIT-2026-02-21.md
|
||||
- no incoming links: archive/projects/cc-deploy-watchtower-v2.md
|
||||
- no incoming links: archive/projects/DEPLOY-API-KEYS-TAB.md
|
||||
- no incoming links: runbooks/edge2-access-reference.md
|
||||
- no incoming links: runbooks/expose-service-edge2.md
|
||||
- no incoming links: glossary.md
|
||||
- no incoming links: archive/projects/headscale-full-deployment.md
|
||||
- no incoming links: archive/matrix/hookshot_deployment_discovery.md
|
||||
- no incoming links: archive/matrix/hookshot_e2ee_discovery.md
|
||||
- no incoming links: archive/matrix/hookshot_payload_discovery.md
|
||||
- no incoming links: archive/matrix/hookshot_room_targeting_discovery.md
|
||||
- no incoming links: INDEX.md
|
||||
- no incoming links: archive/projects/last-ember-project.md
|
||||
- no incoming links: archive/reports/logistics_migration.md
|
||||
- no incoming links: archive/projects/mmud/mmud-phase5-prompt.md
|
||||
- no incoming links: archive/projects/mmud/mmud-phase6-prompt.md
|
||||
- no incoming links: archive/projects/openwebui-theme-deploy.md
|
||||
- no incoming links: archive/projects/peertube-phase2-project.md
|
||||
- no incoming links: archive/projects/peertube-rebuild.md
|
||||
- no incoming links: archive/matrix/PLAN.md
|
||||
- no incoming links: archive/reports/post_validation_report.md
|
||||
- no incoming links: archive/projects/mmud/mmud-prompts/mmud-prompts/README.md
|
||||
- no incoming links: archive/reports/task_a_aurora_validation.md
|
||||
- no incoming links: archive/reports/task_c_watchdog_test.md
|
||||
- no incoming links: archive/projects/utility-caddy-initial-setup.md
|
||||
- no incoming links: archive/projects/vaultwarden-deployment.md
|
||||
- no incoming links: plans/vaultwarden-plan.md
|
||||
52
engine/prompts/fewshot.md
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
# Echo6 Vault Tagger — Few-Shot Examples
|
||||
|
||||
These examples will be populated from real vault documents during the tagger implementation
|
||||
step (Step 5). Each example should show an input document snippet and the exact JSON output
|
||||
the tagger should produce, demonstrating correct vocabulary usage.
|
||||
|
||||
TODO: Add 3–5 real examples drawn from actual vault docs (runbook, reference, project, note).
|
||||
|
||||
---
|
||||
|
||||
## Template: Example Structure
|
||||
|
||||
### Input (document snippet)
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Headscale Setup
|
||||
type: runbook
|
||||
tags: [mesh, vpn]
|
||||
updated: 2025-03-10
|
||||
---
|
||||
|
||||
# Headscale Setup
|
||||
|
||||
Steps to install and configure Headscale on the Contabo VPS...
|
||||
```
|
||||
|
||||
### Expected JSON output
|
||||
|
||||
```json
|
||||
{
|
||||
"tags": ["mesh", "vpn"],
|
||||
"entities": ["contabo", "headscale"],
|
||||
"glossary_proposals": [],
|
||||
"type": "runbook",
|
||||
"confidence": 0.95
|
||||
}
|
||||
```
|
||||
|
||||
### Notes on this example
|
||||
|
||||
- `tags` uses only values from `topic_categories`
|
||||
- `entities` matches names present in `entity_lexicon` (contabo is a proxmox_vps; headscale is a known service)
|
||||
- `confidence` is high because type is explicitly set in frontmatter and tags are unambiguous
|
||||
- If the frontmatter had said `type: guide` (not in the allowed list), the tagger would infer `runbook` from the content and note the discrepancy
|
||||
|
||||
---
|
||||
|
||||
TODO: Add example 2 — reference doc (host description, multiple entities)
|
||||
TODO: Add example 3 — note/session doc (low entity density, inferred type)
|
||||
TODO: Add example 4 — ambiguous doc with glossary_proposals populated
|
||||
TODO: Add example 5 — doc where confidence drops below 0.6
|
||||
67
engine/prompts/system.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# Echo6 Vault Tagger — System Prompt (Canonical)
|
||||
|
||||
## Role
|
||||
|
||||
You are the Echo6 vault tagger, a local AI assistant running on cortex (RTX A4000).
|
||||
Your sole job is to classify Obsidian markdown documents and extract structured metadata
|
||||
from them using a controlled vocabulary. You operate fully offline and deterministically.
|
||||
|
||||
## Inputs (provided per call)
|
||||
|
||||
- **document**: the full text of a markdown file (frontmatter + body)
|
||||
- **topic_categories**: a stable list of tier-1 topic tags (e.g. mesh, auth, proxmox, ai)
|
||||
- **entity_lexicon**: a generated JSON dictionary mapping known names to type
|
||||
(hosts, services, containers, projects, acronyms) — tier 2 vocabulary
|
||||
|
||||
## Output
|
||||
|
||||
Respond with ONLY a single valid JSON object. No prose, no markdown fences, no explanation.
|
||||
|
||||
```json
|
||||
{
|
||||
"tags": [ "string", "..." ],
|
||||
"entities": [ "string", "..." ],
|
||||
"glossary_proposals": [ "string", "..." ],
|
||||
"type": "string",
|
||||
"confidence": 0.0
|
||||
}
|
||||
```
|
||||
|
||||
Field definitions:
|
||||
- **tags**: tier-1 topic tags drawn exclusively from topic_categories
|
||||
- **entities**: known names matched from entity_lexicon
|
||||
- **glossary_proposals**: unknown acronyms or terms worth adding to the lexicon
|
||||
- **type**: one of reference | runbook | project | note | index | session
|
||||
- **confidence**: float 0.0–1.0, your overall confidence in this classification
|
||||
|
||||
## Rules — follow exactly
|
||||
|
||||
1. **Only use provided vocabulary.** `tags` must be a subset of `topic_categories`.
|
||||
`entities` must be a subset of the keys in `entity_lexicon`. Never invent new tags.
|
||||
|
||||
2. **Strict JSON only.** The output must parse with `json.loads()` with no preprocessing.
|
||||
No trailing commas. No comments. No markdown code fences around the JSON.
|
||||
|
||||
3. **Low confidence — flag, do not guess.** If `confidence < 0.6`, still emit valid JSON
|
||||
but keep `tags` and `entities` conservative — only include what you are sure of.
|
||||
Add uncertain terms to `glossary_proposals` instead.
|
||||
|
||||
4. **Never hallucinate expansions.** If you encounter an acronym not in `entity_lexicon`,
|
||||
do NOT guess its expansion. Add the raw acronym to `glossary_proposals`.
|
||||
|
||||
5. **Never fabricate wikilinks or related files.** You output metadata only.
|
||||
|
||||
6. **Type inference.** Use the document frontmatter `type` field if present and valid.
|
||||
Otherwise infer from content: runbooks have steps/commands; references describe systems;
|
||||
projects track work; sessions are journal/meeting notes; index files link to others.
|
||||
|
||||
7. **Tags are used as-is** from the vocab list — do not pluralize or alter them.
|
||||
|
||||
## Confidence scoring guide
|
||||
|
||||
| Range | Meaning |
|
||||
|-----------|----------------------------------------------------------------------|
|
||||
| 0.9–1.0 | Clear topic, entities all recognized, type obvious |
|
||||
| 0.7–0.89 | Good confidence; minor ambiguity in one dimension |
|
||||
| 0.6–0.69 | Borderline; result written but flagged in changelog |
|
||||
| below 0.6 | Do not apply silently; flag for human review |
|
||||
66
engine/sweep.sh
Executable file
|
|
@ -0,0 +1,66 @@
|
|||
#!/usr/bin/env bash
|
||||
# sweep.sh — Echo6 Vault Engine daily maintenance sweep
|
||||
#
|
||||
# Invoked by cron (schedule: "0 9 * * *" from config.yaml).
|
||||
# Also callable manually: ./sweep.sh
|
||||
#
|
||||
# What this does (when fully implemented):
|
||||
# 1. GPU-busy guard: check VRAM usage; defer if > defer_if_gpu_busy_mib (6000 MiB default)
|
||||
# 2. Run lint (lib/lint.py) over all vault docs — fix or flag frontmatter issues
|
||||
# 3. Run agent (lib/agent.py) over changed/new docs since last run — tag + embed
|
||||
# 4. Append a summary entry to changelog.md
|
||||
#
|
||||
# Configuration is read from config.yaml (engine_dir, vault_dir, thresholds, changelog path).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
CONFIG="${SCRIPT_DIR}/config.yaml"
|
||||
|
||||
echo "==> Echo6 vault sweep — $(date -u '+%Y-%m-%dT%H:%M:%SZ')"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 1 — GPU-busy guard
|
||||
# ---------------------------------------------------------------------------
|
||||
# TODO: Query nvidia-smi for used VRAM; compare to defer_if_gpu_busy_mib from config.yaml.
|
||||
# If busy, log a deferred entry to changelog and exit 0 (not an error, just deferred).
|
||||
#
|
||||
# Example skeleton:
|
||||
# USED_MIB=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -1)
|
||||
# THRESHOLD=6000 # read from config.yaml
|
||||
# if [ "$USED_MIB" -gt "$THRESHOLD" ]; then
|
||||
# echo "GPU busy (${USED_MIB} MiB > ${THRESHOLD} MiB threshold) — deferring sweep."
|
||||
# exit 0
|
||||
# fi
|
||||
|
||||
echo "[1/4] TODO — GPU-busy guard not yet implemented."
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 2 — Run lint
|
||||
# ---------------------------------------------------------------------------
|
||||
# TODO: Call lib/lint.py to validate/fix frontmatter across vault docs.
|
||||
# Lint should be idempotent and log all changes to changelog.
|
||||
#
|
||||
# python3 "${SCRIPT_DIR}/lib/lint.py" --config "${CONFIG}"
|
||||
|
||||
echo "[2/4] TODO — lint.py not yet implemented (Step 2)."
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 3 — Run agent over changed/new docs
|
||||
# ---------------------------------------------------------------------------
|
||||
# TODO: Call lib/agent.py to tag + embed documents modified since last sweep.
|
||||
# Agent tracks last-run timestamp in a state file (e.g. engine/.last_sweep).
|
||||
#
|
||||
# python3 "${SCRIPT_DIR}/lib/agent.py" --config "${CONFIG}"
|
||||
|
||||
echo "[3/4] TODO — agent.py not yet implemented (Step 5)."
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 4 — Append changelog summary
|
||||
# ---------------------------------------------------------------------------
|
||||
# TODO: agent.py and lint.py both append to changelog.md directly.
|
||||
# This step adds a sweep-level summary entry.
|
||||
|
||||
echo "[4/4] TODO — changelog summary not yet implemented."
|
||||
|
||||
echo "==> sweep.sh done."
|
||||
1215
engine/vocab.json
Normal file
0
.gitignore → vault/.gitignore
vendored
|
|
@ -17,6 +17,6 @@
|
|||
"repelStrength": 10,
|
||||
"linkStrength": 1,
|
||||
"linkDistance": 250,
|
||||
"scale": 2.25,
|
||||
"scale": 1,
|
||||
"close": false
|
||||
}
|
||||
|
|
@ -11,10 +11,14 @@
|
|||
"id": "ea4cc678c44e8b67",
|
||||
"type": "leaf",
|
||||
"state": {
|
||||
"type": "graph",
|
||||
"state": {},
|
||||
"icon": "lucide-git-fork",
|
||||
"title": "Graph view"
|
||||
"type": "markdown",
|
||||
"state": {
|
||||
"file": "glossary.md",
|
||||
"mode": "source",
|
||||
"source": false
|
||||
},
|
||||
"icon": "lucide-file",
|
||||
"title": "glossary"
|
||||
}
|
||||
}
|
||||
]
|
||||
|
|
@ -185,7 +189,18 @@
|
|||
},
|
||||
"active": "ea4cc678c44e8b67",
|
||||
"lastOpenFiles": [
|
||||
"docs/hardware/environment.md.tmp.40509.b6a12d701b09",
|
||||
"credentials.tmp.40509.595364788ca8",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.ac2c03680b76",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.c3f4d9e4365e",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.ae9f0d9aaaea",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.f6c568f75061",
|
||||
"docs/hardware/ip-allocation.md.tmp.40509.03b7ba9c244f",
|
||||
"docs/hardware/ip-allocation.md.tmp.40509.a068767a4b20",
|
||||
"docs/hardware/ip-allocation.md.tmp.40509.da0228cd9f66",
|
||||
"docs/hardware/environment.md.tmp.40509.e6282fa31980",
|
||||
"docs/hardware/environment.md.tmp.40509.4bb7ceaced2b",
|
||||
"docs/hardware/environment.md.tmp.40509.a6b0a267d126",
|
||||
"glossary.md",
|
||||
"docs/hardware/environment.md",
|
||||
"nodes/ots-ct.md",
|
||||
"mocs/mesh.md",
|
||||
|
|
@ -210,17 +225,6 @@
|
|||
"nodes/meshtastic-hs-ct.md",
|
||||
"nodes/meshai-ct.md",
|
||||
"nodes/meshmonitor-ct.md",
|
||||
"nodes/utility.md",
|
||||
"mocs",
|
||||
"hardware",
|
||||
"services",
|
||||
"nodes",
|
||||
"session-resume",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.ede2941a9a27",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.f5a73384917a",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.0b5e082dca64",
|
||||
"runbooks/lxc-service-migration.md.tmp.40509.1dd3d6e8eb64",
|
||||
"docs/hardware/ip-allocation.md.tmp.40509.ca654fd93971",
|
||||
"assets/echo6yellow_logo_422x422_square.png",
|
||||
"assets/echo6yellow_logo_422x81.png",
|
||||
"assets/echo6_logo.png",
|
||||
|
Before Width: | Height: | Size: 1.7 KiB After Width: | Height: | Size: 1.7 KiB |
|
Before Width: | Height: | Size: 1.9 KiB After Width: | Height: | Size: 1.9 KiB |
|
Before Width: | Height: | Size: 69 KiB After Width: | Height: | Size: 69 KiB |
|
Before Width: | Height: | Size: 5 KiB After Width: | Height: | Size: 5 KiB |
|
Before Width: | Height: | Size: 64 KiB After Width: | Height: | Size: 64 KiB |
|
Before Width: | Height: | Size: 18 KiB After Width: | Height: | Size: 18 KiB |
|
|
@ -10,7 +10,7 @@ ROOT_PASSWORD=7redditGold
|
|||
GODADDY_API_KEY=dKiSoC24ZLTR_3KkMjFyYrzgChk1JHjNLnU
|
||||
GODADDY_API_SECRET=ExCGBASgSrN4A2mP4daU4G
|
||||
|
||||
# Authentik SSO Platform (https://auth.echo6.co)
|
||||
# Authentik SSO Platform (https://auth.echo6.co) — migrated to edge2 CT 105 / 10.10.10.23 / tailnet 100.64.0.36 on 2026-06-18
|
||||
AUTHENTIK_API_TOKEN=YG24Zu7c7JNhrfC564N2NvJt2HmIr6Jyi9BgV629XGAZC70hvGbyNz8i4l7w
|
||||
AUTHENTIK_URL=https://auth.echo6.co
|
||||
# Admin credentials for web UI/API access
|
||||
|
|
@ -86,7 +86,7 @@ Five nodes running Proxmox VE:
|
|||
| Contabo Server | 5.189.158.149 | 100.64.0.1 | External VPS: Mail, Authentik, Headscale, Forge, Matrix |
|
||||
| edge2 | 184.174.35.153 | 100.64.0.26 | Contabo Cloud VPS 30 NVMe — Proxmox VE 8.4.19 (LXC-only), 8c/24GB/400GB |
|
||||
|
||||
*Last updated: 2026-06-17 — Added edge2 CT 104 (livesync, 10.10.10.22, 100.64.0.35, migrated 2026-06-16); previously added CT 103 (forgejo), CT 102 (vaultwarden), pdm CT 100, wordpress CT 101*
|
||||
*Last updated: 2026-06-18 — Added edge2 CT 105 (authentik, 10.10.10.23, 100.64.0.36, node 48, migrated 2026-06-18); previously added CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden), pdm CT 100, wordpress CT 101*
|
||||
|
||||
## LXC Containers
|
||||
|
||||
|
|
@ -108,6 +108,7 @@ Five nodes running Proxmox VE:
|
|||
| vaultwarden | edge2 (CT 102) | 10.10.10.20 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
|
||||
| forgejo | edge2 (CT 103) | 10.10.10.21 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
|
||||
| livesync | edge2 (CT 104) | 10.10.10.22 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
|
||||
| authentik | edge2 (CT 105) | 10.10.10.23 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
|
||||
|
||||
## IP Allocation Scheme
|
||||
|
||||
|
|
@ -158,6 +159,7 @@ Current registered nodes (26 total):
|
|||
| vaultwarden | 100.64.0.33 | LXC (edge2 CT 102) |
|
||||
| forgejo | 100.64.0.34 | LXC (edge2 CT 103) — node id 46 |
|
||||
| livesync | 100.64.0.35 | LXC (edge2 CT 104) — migrated 2026-06-16 |
|
||||
| authentik | 100.64.0.36 | LXC (edge2 CT 105) — node id 48, migrated 2026-06-18 |
|
||||
|
||||
## IdahoMesh Headscale Node List
|
||||
|
||||
|
|
@ -58,6 +58,7 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
|
|||
| 10.10.10.20 | vaultwarden | CT 102 | 100.64.0.33 | Vaultwarden password manager (migrated from Contabo 2026-06-16) |
|
||||
| 10.10.10.21 | forgejo | CT 103 | 100.64.0.34 | Forgejo git server (migrated from Contabo 2026-06-16) |
|
||||
| 10.10.10.22 | livesync | CT 104 | 100.64.0.35 | LiveSync Obsidian sync (CouchDB + provisioner; migrated from Contabo 2026-06-16) |
|
||||
| 10.10.10.23 | authentik | CT 105 | 100.64.0.36 | Authentik SSO platform (migrated from Contabo 2026-06-18) |
|
||||
|
||||
### VMs (.150-.199)
|
||||
| IP | VM | Host | Purpose |
|
||||
|
|
@ -110,6 +111,7 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
|
|||
| 100.64.0.33 | vaultwarden (CT 102 on edge2) — node id 45 | 10.10.10.20 (vmbr0) |
|
||||
| 100.64.0.34 | forgejo (CT 103 on edge2) — node id 46 | 10.10.10.21 (vmbr0) |
|
||||
| 100.64.0.35 | livesync (CT 104 on edge2) — hostname `livesync` | 10.10.10.22 (vmbr0) |
|
||||
| 100.64.0.36 | authentik (CT 105 on edge2) — node id 48 | 10.10.10.23 (vmbr0) |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -122,4 +124,4 @@ edge2 (Contabo Cloud VPS 184.174.35.153 / Tailscale 100.64.0.26) uses a separate
|
|||
|
||||
---
|
||||
|
||||
*Last updated: 2026-06-17 — Added edge2 CT 104 (livesync) at 10.10.10.22 / 100.64.0.35; previously added CT 103 (forgejo), CT 102 (vaultwarden)*
|
||||
*Last updated: 2026-06-18 — Added edge2 CT 105 (authentik) at 10.10.10.23 / 100.64.0.36 (node id 48); previously added CT 104 (livesync), CT 103 (forgejo), CT 102 (vaultwarden)*
|
||||
|
|
@ -14,7 +14,7 @@
|
|||
| mesh-bridge | utility (CT 107) | 192.168.1.107 | Internal | Dual-tailscaled bridge (echo6 ↔ idahomesh) |
|
||||
| MeshAI | utility (CT 108) | 192.168.1.144:4403 | Internal | LLM-powered Meshtastic assistant (Docker, Gemini Flash, Google grounding) |
|
||||
| ARGUS | utility (CT 103) | 192.168.1.103 | Internal | OSINT intelligence gathering platform (Docker, SearXNG + local LLM analysis) |
|
||||
| Authentik | Contabo | 5.189.158.149:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) |
|
||||
| Authentik | edge2 (CT 105) | 100.64.0.36:9000 | https://auth.echo6.co | SSO provider (Echo6 branded, custom CSS, dark theme) — fronted by Contabo Caddy (reverse_proxy 100.64.0.36:9000); **migrated from Contabo 2026-06-18** |
|
||||
| Forge (Forgejo) | edge2 (CT 103) | 100.64.0.34:3001 HTTP / :2222 SSH (via Contabo DNAT) | https://forge.echo6.co | Git server — fronted by Contabo Caddy (reverse_proxy 100.64.0.34:3001); git SSH via iptables DNAT on Contabo (forgejo-ssh-dnat.service) — **migrated from Contabo 2026-06-16** |
|
||||
| Headscale | Contabo | 5.189.158.149 | https://vpn.echo6.co | Tailscale coordination (OIDC enabled) |
|
||||
| Headplane | Contabo | 127.0.0.1:3100 | https://vpn.echo6.co/admin | Headscale web UI (OIDC via Authentik) |
|
||||
|
|
@ -280,6 +280,21 @@
|
|||
- Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-prelivesync` exists
|
||||
- **Resources:** 2 cores / 1024 MB RAM / 512 MB swap / 8 GB rootfs on `local`; unprivileged; onboot; Docker
|
||||
|
||||
### edge2 - CT 105 (10.10.10.23 / Tailscale: 100.64.0.36, node 48 `authentik`)
|
||||
- Authentik SSO platform (https://auth.echo6.co — **migrated from Contabo 2026-06-18**)
|
||||
- Headscale node id 48, hostname `authentik`, tailnet IP 100.64.0.36
|
||||
- Compose path: `/opt/authentik/docker-compose.yml`
|
||||
- Containers: `authentik-server` + `authentik-worker` (ghcr.io/goauthentik/server:2025.12.4) + `authentik-postgres` (postgres:16); NO Redis
|
||||
- Worker runs as user:root and has docker.sock bind-mount (manages embedded outposts)
|
||||
- Binds to `100.64.0.36:9000`; Contabo Caddy proxies here over tailnet for both `auth.echo6.co` (catch-all + outpost path matcher) and `notes.echo6.co` outpost/forward_auth references
|
||||
- `AUTHENTIK_SECRET_KEY` carried byte-for-byte (sessions stayed valid across cutover — users dropped straight in)
|
||||
- Bind-mounts (data/media, branding, certs, custom-templates) migrated intact
|
||||
- Email dep: mail.echo6.co (unchanged)
|
||||
- DB: PostgreSQL 16 (`authentik-postgres` container); ~705 MB (~18 MB pg_dump)
|
||||
- Source on Contabo STOPPED but intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-authentik` exists on Contabo
|
||||
- Reboot-survival fix: systemd unit on CT 105 gates `docker compose up` on `tailscale-online` (Docker was racing Tailscale on boot, failing the bind to the tailnet IP)
|
||||
- **Resources:** 2 cores / 4096 MB RAM / 512 MB swap / 20 GB rootfs on `local`; unprivileged; onboot; Docker
|
||||
|
||||
### edge2 - CT 102 (10.10.10.20 / Tailscale: 100.64.0.33, node 45 `vaultwarden`)
|
||||
- Vaultwarden password manager (port 8086, https://vault.echo6.co, Docker)
|
||||
- Headscale node id 45, name `vaultwarden`, user `echo6`
|
||||
|
|
@ -290,7 +305,7 @@
|
|||
- **Migrated from Contabo to edge2 CT 102 on 2026-06-16**
|
||||
|
||||
### Contabo VPS (5.189.158.149 / Tailscale: 100.64.0.1)
|
||||
- Authentik (SSO, Echo6 branded — custom CSS, dark theme, logo, favicon, flow titles)
|
||||
- ~~Authentik~~ — **migrated to edge2 CT 105 on 2026-06-18** (Caddy now proxies auth.echo6.co + notes.echo6.co outpost/forward_auth → 100.64.0.36:9000; source STOPPED at `/opt/authentik`, intact as cold rollback; `/etc/caddy/Caddyfile.bak-pre-authentik` exists)
|
||||
- Forge (Git) — **migrated to edge2 CT 103 on 2026-06-16** (Caddy now proxies to 100.64.0.34:3001; SSH DNAT via forgejo-ssh-dnat.service)
|
||||
- Headscale (mesh VPN)
|
||||
- Mailcow (email)
|
||||
159
vault/glossary.md
Normal file
|
|
@ -0,0 +1,159 @@
|
|||
---
|
||||
title: Glossary & Vocabulary
|
||||
type: reference
|
||||
tags: [meta]
|
||||
updated: 2026-06-18
|
||||
---
|
||||
# Glossary & Vocabulary
|
||||
|
||||
> Auto-generated by the vault engine (`engine/lib/vocab_gen.py`).
|
||||
> Acronym expansions marked _(unconfirmed)_ need a human pass.
|
||||
|
||||
## Topic categories
|
||||
|
||||
mesh · matrix · recon · media · auth · dns · vpn · storage · proxmox · ai · mail
|
||||
|
||||
## Acronyms
|
||||
|
||||
| Acronym | Expansion |
|
||||
|---|---|
|
||||
| MMUD | Mesh Multi-User Dungeon |
|
||||
| AIDA | _(unconfirmed)_ |
|
||||
|
||||
## Entities
|
||||
|
||||
### Hosts / Proxmox nodes
|
||||
|
||||
- **argus** — aliases: 100.64.0.25
|
||||
- **bluefin** — aliases: 100.64.0.30
|
||||
- **cloud** (Cloud) — aliases: 192.168.1.242, 100.64.0.4
|
||||
- **contabo** — aliases: 100.64.0.1
|
||||
- **data** (Data) — aliases: 192.168.1.240, 100.64.0.6
|
||||
- **edge2** — aliases: 184.174.35.153, 100.64.0.26
|
||||
- **forgejo** — aliases: 100.64.0.34
|
||||
- **iphone-eud** — aliases: 100.64.0.16
|
||||
- **media** (Media) — aliases: 192.168.1.243, 100.64.0.3
|
||||
- **mesh-bridge** — aliases: 100.100.0.3, 100.64.0.22
|
||||
- **meshai** — aliases: 100.64.0.32
|
||||
- **meshmonitor** — aliases: 100.64.0.7
|
||||
- **meshmonitor-dev** — aliases: 100.64.0.27
|
||||
- **nextcloud** — aliases: 100.64.0.11
|
||||
- **peertube** — aliases: 100.64.0.23
|
||||
- **toc** (Toc) — aliases: 192.168.1.244, 100.64.0.13
|
||||
- **utility** (Utility) — aliases: 192.168.1.241, 100.64.0.5
|
||||
|
||||
### Virtual Machines
|
||||
|
||||
- **arr** — aliases: 192.168.1.160, 100.64.0.18, VM 105 — on: media
|
||||
- **cortex** — aliases: 192.168.1.150, 100.64.0.14, VM 150 — on: toc
|
||||
- **recon** — aliases: 100.64.0.24
|
||||
- **recon-vm** — aliases: 192.168.1.130, 100.64.0.24, VM 1130 — on: data
|
||||
|
||||
### LXC Containers
|
||||
|
||||
- **archivist** — aliases: 192.168.1.118, CT 118 — on: utility
|
||||
- **caddy** — aliases: 192.168.1.101, CT 101, 100.64.0.8 — on: utility
|
||||
- **immich** — aliases: 192.168.1.182, CT 120, 100.64.0.2 — on: cloud
|
||||
- **livesync** — aliases: 10.10.10.22, CT 104, 100.64.0.35 — on: edge2
|
||||
- **meshtastic-hs** — aliases: 192.168.1.106, CT 106 — on: utility
|
||||
- **pdm** — aliases: 10.10.10.10, CT 100, 100.64.0.28 — on: edge2
|
||||
- **running** — aliases: CT 100 — on: utility _(live)_
|
||||
- **searxng** — aliases: 192.168.1.102, CT 102, 100.64.0.15 — on: utility
|
||||
- **vaultwarden** — aliases: 10.10.10.20, CT 102, 100.64.0.33 — on: edge2
|
||||
- **wordpress** — aliases: 10.10.10.11, CT 101, 100.64.0.31 — on: edge2
|
||||
|
||||
### Raspberry Pi / Edge nodes
|
||||
|
||||
- **aida-nebra** — aliases: 192.168.1.253, 100.64.0.9, !27780c47
|
||||
- **burley-butte** — aliases: 100.100.0.1
|
||||
- **mt-burleybutte** — aliases: 192.168.1.185
|
||||
- **mt-isr** — aliases: 192.168.1.141, 100.100.0.5 (IdahoMesh), 100.100.0.5
|
||||
- **pi-nas** — aliases: 192.168.1.245, 100.64.0.21
|
||||
|
||||
### Desktops
|
||||
|
||||
- **matt-desktop** — aliases: 192.168.1.111, 100.64.0.10
|
||||
|
||||
### Routers / Network devices
|
||||
|
||||
- **gl-a1300** — aliases: 100.64.0.29
|
||||
|
||||
### Services
|
||||
|
||||
- **authentik** (Authentik) — aliases: Authentik, auth — on: contabo
|
||||
- **authentik-postgres** — aliases: authentik-postgres — on: contabo _(live)_
|
||||
- **authentik-server** — aliases: authentik-server — on: contabo _(live)_
|
||||
- **authentik-worker** — aliases: authentik-worker — on: contabo _(live)_
|
||||
- **echo6-agent** — aliases: echo6-agent — on: 2026-06-16
|
||||
- **echo6-contabo-agent** (Echo6 Contabo Agent) — aliases: Echo6 Contabo Agent — on: contabo
|
||||
- **echo6-cortex-agent** (Echo6 Cortex Agent) — aliases: Echo6 Cortex Agent — on: cortex
|
||||
- **echo6-search-searxng** (Echo6 Search (SearXNG)) — aliases: Echo6 Search (SearXNG) — on: utility
|
||||
- **element-web** (Element Web) — aliases: Element Web, element — on: contabo
|
||||
- **files** (Files) — aliases: Files, files — on: data
|
||||
- **forge-forgejo** (Forge (Forgejo)) — aliases: Forge (Forgejo), forge — on: edge2
|
||||
- **headplane** (Headplane) — aliases: Headplane, vpn — on: contabo
|
||||
- **headscale** (Headscale) — aliases: Headscale, vpn — on: contabo
|
||||
- **idahomesh-headscale** (IdahoMesh Headscale) — aliases: IdahoMesh Headscale — on: utility
|
||||
- **jellyfin** (Jellyfin) — aliases: Jellyfin, jellyfin — on: media
|
||||
- **jellyseer** (Jellyseer) — aliases: Jellyseer, requests — on: media
|
||||
- **lidarr** — aliases: lidarr — on: media _(live)_
|
||||
- **mailcow** (Mailcow) — aliases: Mailcow, mail — on: contabo
|
||||
- **mailcowdockerized-acme-mailcow-1** — aliases: mailcowdockerized-acme-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-clamd-mailcow-1** — aliases: mailcowdockerized-clamd-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-dockerapi-mailcow-1** — aliases: mailcowdockerized-dockerapi-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-dovecot-mailcow-1** — aliases: mailcowdockerized-dovecot-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-memcached-mailcow-1** — aliases: mailcowdockerized-memcached-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-mysql-mailcow-1** — aliases: mailcowdockerized-mysql-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-netfilter-mailcow-1** — aliases: mailcowdockerized-netfilter-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-nginx-mailcow-1** — aliases: mailcowdockerized-nginx-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-ofelia-mailcow-1** — aliases: mailcowdockerized-ofelia-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-olefy-mailcow-1** — aliases: mailcowdockerized-olefy-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-php-fpm-mailcow-1** — aliases: mailcowdockerized-php-fpm-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-postfix-mailcow-1** — aliases: mailcowdockerized-postfix-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-postfix-tlspol-mailcow-1** — aliases: mailcowdockerized-postfix-tlspol-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-redis-mailcow-1** — aliases: mailcowdockerized-redis-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-rspamd-mailcow-1** — aliases: mailcowdockerized-rspamd-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-sogo-mailcow-1** — aliases: mailcowdockerized-sogo-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-unbound-mailcow-1** — aliases: mailcowdockerized-unbound-mailcow-1 — on: contabo _(live)_
|
||||
- **mailcowdockerized-watchdog-mailcow-1** — aliases: mailcowdockerized-watchdog-mailcow-1 — on: contabo _(live)_
|
||||
- **matrix-element** — aliases: matrix-element — on: contabo _(live)_
|
||||
- **matrix-mas** (Matrix MAS) — aliases: Matrix MAS — on: contabo
|
||||
- **matrix-postgres** — aliases: matrix-postgres — on: contabo _(live)_
|
||||
- **matrix-synapse** (Matrix Synapse) — aliases: Matrix Synapse, matrix — on: contabo
|
||||
- **mautrix-signal** — aliases: mautrix-signal — on: contabo
|
||||
- **meshtastic-cli** (Meshtastic CLI) — aliases: Meshtastic CLI — on: mt-isr
|
||||
- **meshtasticd** — aliases: meshtasticd — on: mt-burleybutte
|
||||
- **meshtasticd-aida-n2** (meshtasticd (AIDA-N2)) — aliases: meshtasticd (AIDA-N2) — on: aida-nebra
|
||||
- **navidrome** — aliases: navidrome — on: media _(live)_
|
||||
- **nexus-agent** — aliases: nexus-agent — on: 2026-06-16
|
||||
- **nexus-hub** — aliases: nexus-hub — on: 2026-06-16
|
||||
- **obsidian-remote** — aliases: obsidian-remote — on: cortex _(live)_
|
||||
- **ollama** — aliases: ollama — on: cortex _(live)_
|
||||
- **open-webui** (Open WebUI) — aliases: Open WebUI, ai — on: cortex
|
||||
- **opentakserver-ots** (OpenTAKServer (OTS)) — aliases: OpenTAKServer (OTS) — on: utility
|
||||
- **prowlarr** (Prowlarr) — aliases: Prowlarr — on: media
|
||||
- **proxmox-ve** (Proxmox VE) — aliases: Proxmox VE, proxmox — on: data
|
||||
- **pt-transcoder** — aliases: pt-transcoder — on: cortex
|
||||
- **qdrant** (Qdrant) — aliases: Qdrant — on: cortex
|
||||
- **radarr** (Radarr) — aliases: Radarr — on: media
|
||||
- **recon-sparse** — aliases: recon-sparse — on: cortex
|
||||
- **sabnzbd** (SABnzbd) — aliases: SABnzbd — on: media
|
||||
- **samba** (Samba) — aliases: Samba — on: data
|
||||
- **sigil** (SIGIL) — aliases: SIGIL — on: 2026-06-16
|
||||
- **sonarr** (Sonarr) — aliases: Sonarr — on: media
|
||||
- **syncthing** (Syncthing) — aliases: Syncthing — on: contabo
|
||||
- **tak-server** (TAK Server) — aliases: TAK Server — on: 2026-06-16
|
||||
- **tei** (TEI) — aliases: TEI — on: cortex
|
||||
- **termix** (Termix) — aliases: Termix — on: contabo
|
||||
- **utility-caddy** (Utility Caddy) — aliases: Utility Caddy — on: utility
|
||||
- **watchtower** (WATCHTOWER) — aliases: WATCHTOWER — on: 2026-06-16
|
||||
|
||||
### Projects
|
||||
|
||||
- **advbbs-project** — aliases: advbbs-project
|
||||
- **argus** — aliases: argus
|
||||
- **deploy-livesync** — aliases: deploy-livesync
|
||||
- **matrix-synapse-deployment** — aliases: matrix-synapse-deployment
|
||||
- **meshtastic-headscale-runbook** — aliases: meshtastic-headscale-runbook
|
||||
- **mmud-project** — aliases: mmud-project
|
||||
|
||||