echo6-docs/projects/matrix-synapse-deployment.md

# Matrix Synapse Deployment

**Status:** Deployed 2026-02-15, migrated to Contabo 2026-02-15
**Target:** Contabo VPS (5.189.158.149 / 100.64.0.1)
**URLs:** https://matrix.echo6.co (Synapse), https://element.echo6.co (Element Web)
**Server Name:** echo6.co (federated identity: @user:echo6.co)

---

## Architecture

| Component | Detail |
|-----------|--------|
| Host | Contabo VPS (5.189.158.149 / 100.64.0.1) |
| Docker services | Synapse (127.0.0.1:8008), Element Web (127.0.0.1:8088), PostgreSQL 16 |
| Reverse proxy | Contabo Caddy (auto ACME certs) |
| SSO | Authentik OIDC → communication-users group |
| Federation | Well-known delegation on echo6.co base domain (served by utility Caddy) |
| Compose path | `/opt/matrix/docker-compose.yml` |
| Backup | Daily at 3AM, 14-day retention, `/opt/matrix/backups/` |

The server name is `echo6.co` (not `matrix.echo6.co`) so federated user IDs are `@user:echo6.co`. The Synapse instance lives at `matrix.echo6.co` and delegation is handled via `.well-known` endpoints on the base domain.

---

## Step 1: Provision LXC Container

Run **ct-runbook.md** on the utility node with these parameters:

```
CTID=108
HOSTNAME=matrix
STORAGE=local-lvm
DISK_SIZE=16
MEMORY=2048
CORES=2
BRIDGE=vmbr0
```

After the runbook completes (user, SSH, Docker, Tailscale all verified), continue here.

---

## Step 2: Create Project Structure

SSH into CT 108:

```bash
CT_IP=$(ssh root@192.168.1.241 "pct exec 108 -- hostname -I | awk '{print \$1}'")
sshpass -p '7redditGold' ssh zvx@$CT_IP
```

Create directories:

```bash
sudo mkdir -p /opt/matrix/{synapse,postgres,element,backups,scripts}
sudo chown -R zvx:zvx /opt/matrix
```

---

## Step 3: Create Docker Compose

Create `/opt/matrix/docker-compose.yml`:

```yaml
services:
  postgres:
    image: postgres:16-alpine
    container_name: matrix-postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: synapse
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_INITDB_ARGS: "--encoding=UTF8 --lc-collate=C --lc-ctype=C"
    volumes:
      - ./postgres:/var/lib/postgresql/data
    networks:
      - matrix-net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U synapse -d synapse"]
      interval: 10s
      timeout: 5s
      retries: 5

  synapse:
    image: matrixdotorg/synapse:latest
    container_name: matrix-synapse
    restart: unless-stopped
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      SYNAPSE_CONFIG_PATH: /data/homeserver.yaml
    volumes:
      - ./synapse:/data
    ports:
      - "8008:8008"
    networks:
      - matrix-net

  element:
    image: vectorim/element-web:latest
    container_name: matrix-element
    restart: unless-stopped
    volumes:
      - ./element/config.json:/app/config.json:ro
    ports:
      - "8080:80"
    networks:
      - matrix-net

networks:
  matrix-net:
    driver: bridge
```

Create `/opt/matrix/.env`:

```bash
POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=' | head -c 32)
echo "POSTGRES_PASSWORD=$POSTGRES_PASSWORD" > /opt/matrix/.env
chmod 600 /opt/matrix/.env
echo "Save this password to /home/zvx/projects/.ref/credentials"
cat /opt/matrix/.env
```

---

## Step 4: Generate Synapse Config

```bash
cd /opt/matrix
docker run -it --rm \
  -v ./synapse:/data \
  -e SYNAPSE_SERVER_NAME=echo6.co \
  -e SYNAPSE_REPORT_STATS=no \
  matrixdotorg/synapse:latest generate
```

Edit `synapse/homeserver.yaml` — replace the full `database` section and add OIDC config:

```yaml
server_name: "echo6.co"
public_baseurl: "https://matrix.echo6.co/"

listeners:
  - port: 8008
    type: http
    tls: false
    x_forwarded: true
    bind_addresses: ['0.0.0.0']
    resources:
      - names: [client, federation]
        compress: false

database:
  name: psycopg2
  args:
    user: synapse
    password: <POSTGRES_PASSWORD from .env>
    database: synapse
    host: matrix-postgres
    port: 5432
    cp_min: 5
    cp_max: 10

media_store_path: /data/media_store
enable_registration: false
url_preview_enabled: true

# Authentik OIDC — fill client_id and client_secret after running authentik-oidc-application.md
oidc_providers:
  - idp_id: authentik
    idp_name: "Echo6 SSO"
    discover: true
    issuer: "https://auth.echo6.co/application/o/matrix/"
    client_id: "<from authentik-oidc-application.md>"
    client_secret: "<from authentik-oidc-application.md>"
    scopes: ["openid", "profile", "email"]
    user_mapping_provider:
      config:
        localpart_template: "{{ user.preferred_username }}"
        display_name_template: "{{ user.name }}"
```

---

## Step 5: Configure Element Web

Create `/opt/matrix/element/config.json`:

```json
{
  "default_server_config": {
    "m.homeserver": {
      "base_url": "https://matrix.echo6.co",
      "server_name": "echo6.co"
    }
  },
  "brand": "Echo6 Chat",
  "disable_guests": true,
  "disable_3pid_login": false
}
```

---

## Step 6: Start Services

```bash
cd /opt/matrix
docker compose up -d
docker compose ps
```

Wait for Synapse to initialize the database (watch logs):

```bash
docker compose logs -f synapse
# Wait for "Synapse now listening on TCP port 8008"
```

---

## Step 7: Expose via Caddy and DNS

Run **expose-service-home.md** twice — once for `matrix.echo6.co` and once for `element.echo6.co`.

This service has OIDC, so use local IP per the runbook's decision table.

### matrix.echo6.co

- Backend: `192.168.1.108:8008` (local IP, has OIDC)
- Issue cert, install cert, add Caddy site block, add GoDaddy DNS

Caddy site block (note the path-based routing for Matrix):

```caddyfile
matrix.echo6.co {
    tls /etc/caddy/certs/matrix.echo6.co.fullchain.crt /etc/caddy/certs/matrix.echo6.co.key
    reverse_proxy /_matrix/* 192.168.1.108:8008
    reverse_proxy /_synapse/* 192.168.1.108:8008
}
```

### element.echo6.co

- Backend: `192.168.1.108:8080` (local IP)
- Issue cert, install cert, add Caddy site block, add GoDaddy DNS

```caddyfile
element.echo6.co {
    tls /etc/caddy/certs/element.echo6.co.fullchain.crt /etc/caddy/certs/element.echo6.co.key
    reverse_proxy 192.168.1.108:8080
}
```

### Well-known delegation (federation)

This must go on the `echo6.co` base domain. Check if there's already an `echo6.co` block in the Utility Caddy Caddyfile — if so, merge these `handle` directives into it. If not, add a new block:

```caddyfile
echo6.co {
    tls /etc/caddy/certs/echo6.co.fullchain.crt /etc/caddy/certs/echo6.co.key

    handle /.well-known/matrix/server {
        header Content-Type application/json
        respond `{"m.server": "matrix.echo6.co:443"}`
    }
    handle /.well-known/matrix/client {
        header Content-Type application/json
        header Access-Control-Allow-Origin *
        respond `{"m.homeserver": {"base_url": "https://matrix.echo6.co"}}`
    }

    # ... any existing handlers for echo6.co ...
}
```

If `echo6.co` doesn't have a cert yet, issue one via acme.sh following the same pattern in expose-service-home.md.

### dnsmasq split DNS

Add to `/etc/dnsmasq.d/tailscale-dns.conf` on Contabo:

```
address=/matrix.echo6.co/100.64.0.8
address=/element.echo6.co/100.64.0.8
```

Both point to the Utility Caddy Tailscale IP (100.64.0.8), which proxies to CT 108.

Restart dnsmasq:

```bash
ssh root@100.64.0.1 "systemctl restart dnsmasq"
```

---

## Step 8: Configure Authentik SSO

Run **authentik-oidc-application.md** with these inputs:

```
SERVICE_NAME=Matrix
SERVICE_SLUG=matrix
SERVICE_URL=https://matrix.echo6.co
OIDC_CALLBACK_PATH=/_synapse/client/oidc/callback
NEEDS_OFFLINE_ACCESS=no
CLIENT_TYPE=confidential
```

After completing the runbook, take the Client ID and Client Secret and update `synapse/homeserver.yaml` (Step 4) with the real values. Then restart Synapse:

```bash
cd /opt/matrix && docker compose restart synapse
```

---

## Step 9: Bind Access Group

Run **authentik-access-groups.md** Procedure B to bind the `matrix` application to the `communication-users` group.

```
APP_SLUG=matrix
GROUP_PK=31bce176-cd86-4aea-8db3-a57e03d5c2d1   # communication-users
```

This shares the same access group as Mailcow.

---

## Step 10: Create Admin User

```bash
docker exec -it matrix-synapse register_new_matrix_user \
  -u matt \
  -p <secure-password> \
  -a \
  -c /data/homeserver.yaml \
  http://localhost:8008
```

---

## Step 11: Schedule PostgreSQL Backups

Run **pg-backup.md** with these inputs:

```
CONTAINER_NAME=matrix-postgres
DB_NAME=synapse
DB_USER=synapse
BACKUP_DIR=/opt/matrix/backups
RETENTION_DAYS=14
CRON_SCHEDULE="0 3 * * *"
```

---

## Verification

### Internal (from CT 108)

```bash
curl -s http://localhost:8008/_matrix/client/versions | jq .
curl -s http://localhost:8008/_matrix/federation/v1/version | jq .
curl -s http://localhost:8080 | head -5
```

### External (from cortex or any tailnet device)

```bash
curl -s https://matrix.echo6.co/_matrix/client/versions | jq .
curl -s https://matrix.echo6.co/_matrix/federation/v1/version | jq .
curl -sI https://element.echo6.co | head -5
curl -s https://echo6.co/.well-known/matrix/server | jq .
curl -s https://echo6.co/.well-known/matrix/client | jq .
```

### Federation

```bash
curl -s "https://federationtester.matrix.org/api/report?server_name=echo6.co" | jq '.FederationOK'
```

Must return `true`.

### SSO

1. Open https://element.echo6.co
2. Click SSO login
3. Should redirect to auth.echo6.co → authenticate → redirect back to Element
4. Verify user identity matches Authentik profile

---

## Troubleshooting

### Synapse won't start

```bash
docker compose logs synapse 2>&1 | tail -50
```

Common causes: bad YAML indentation in homeserver.yaml, wrong PostgreSQL password, database not ready.

### Federation test fails

Check in order:
1. `.well-known/matrix/server` returns `{"m.server": "matrix.echo6.co:443"}`
2. `/_matrix/federation/v1/version` is accessible from the public internet
3. Caddy is routing `/_matrix/*` paths correctly (not just root)
4. GoDaddy DNS for `echo6.co` points to 199.6.36.163

### SSO login loop

See troubleshooting in authentik-oidc-application.md. Most common cause: missing signing key on the Authentik provider, or wrong callback path.

### Element can't connect

Verify Element's `config.json` has `base_url` set to `https://matrix.echo6.co` (not `http://`, not `localhost`).

---

## Runbook References

| Step | Runbook | Purpose |
|------|---------|---------|
| 1 | ct-runbook.md | LXC provisioning, Docker, user, SSH, Tailscale |
| 7 | expose-service-home.md | SSL cert, Caddy site block, GoDaddy DNS |
| 8 | authentik-oidc-application.md | Create OIDC provider + application |
| 9 | authentik-access-groups.md | Bind communication-users group |
| 11 | pg-backup.md | Scheduled PostgreSQL backup with retention |

---

## Credentials Reference

Store in `/home/zvx/projects/.ref/credentials`:

```
# Matrix Synapse
MATRIX_POSTGRES_PASSWORD=<from .env>
MATRIX_OIDC_CLIENT_ID=<from authentik-oidc-application.md>
MATRIX_OIDC_CLIENT_SECRET=<from authentik-oidc-application.md>
MATRIX_OIDC_ISSUER=https://auth.echo6.co/application/o/matrix/
MATRIX_ADMIN_USER=matt
```

---

## Post-Deploy Updates

After deployment, update these docs:

- `docs/services/services.md` — add Matrix entry
- `docs/software/caddy.md` — add matrix.echo6.co and element.echo6.co site blocks
- `docs/software/dns.md` — note well-known delegation on echo6.co
- `docs/hardware/environment.md` — add CT 108 to LXC table and Headscale node list
- `runbooks/authentik-access-groups.md` — add Matrix to application bindings table

---

*Created: 2026-02-15*