| title |
type |
tags |
aliases |
related |
updated |
status |
| toc + cortex PVE 9.2 / GPU update — run from matt-desktop |
runbook |
|
|
|
2026-07-13 |
active |
toc + cortex update — run from matt-desktop
The other 4 cluster nodes (data, utility, cloud, media) were upgraded to PVE 9.2.3 / kernel 7.0.12 on 2026-06-21/22. toc + cortex were deliberately left because rebooting toc drops cortex (the usual Claude Code host) — so this pair is driven from matt-desktop instead.
How to run
- On matt-desktop (Windows 11), open a terminal (PowerShell or cmd).
- Run
claude (Claude Code 2.1.185 is installed).
- Paste the prompt below. Fill in the toc root password where marked (it's in
.ref/credentials).
- It's safe to run now — all 4 other cluster nodes are up and quorate, so toc rebooting alone keeps the cluster at 4/5.
Prompt (paste into Claude Code on matt-desktop)
You're on matt-desktop (Windows). Task: update + reboot two Linux Proxmox machines (toc, a PVE host, and cortex, the GPU VM on it) over SSH. These were left for you because rebooting toc kills the host the rest of the automation runs on. Verify each step; STOP and ask me if a check fails.
MACHINES
- toc — Proxmox VE host (GPU server). SSH: ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@100.64.0.13 (PASSWORD auth, password: <PASTE TOC ROOT PASSWORD>). toc is node 5 of a 5-node cluster "echo6-cluster"; the other 4 nodes are already upgraded.
- cortex — VM 150 on toc. An NVIDIA RTX A4000 is PCI-passed-through from toc into cortex. cortex runs AI containers (ollama, tei, qdrant, open-webui). Reach it via: ssh zvx@100.64.0.14 if a key/password works, else run commands inside it from toc with: qm guest exec 150 -- bash -lc '<cmd>'
PART A — cortex GPU driver update FIRST (toc stays up):
1. On toc, snapshot cortex: qm snapshot 150 pre-gpu-update
2. In cortex, baseline: nvidia-smi (driver ~580.159.03), dkms status, docker ps (4 AI containers up).
3. In cortex, update: sudo apt-get update && sudo apt-get upgrade -y (pulls NVIDIA driver 580.159->580.167 + DKMS rebuild + nvidia-container-toolkit 1.18->1.19). Watch the NVIDIA DKMS module rebuild succeed.
4. Reboot cortex: on toc qm reboot 150 (wait ~2-3 min).
5. VERIFY cortex: nvidia-smi shows 580.167; dkms status shows nvidia module built for the running kernel; docker ps (containers back); ollama+tei on GPU (docker exec ollama ollama ps shows GPU; tei answers an embed). If nvidia-smi fails / GPU gone -> roll back on toc: qm rollback 150 pre-gpu-update , then tell me.
PART B — toc PVE 9.2 upgrade + reboot (AFTER cortex verified):
6. On toc: pvecm status -> must be Quorate: Yes. If not, STOP and tell me.
7. On toc: apt-get update && DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade -y -o Dpkg::Options::="--force-confold" -o Dpkg::Options::="--force-confdef" (brings PVE 9.1->9.2, QEMU 10->11, LXC 6->7, and a NEW DEFAULT KERNEL 7.0.x — the other nodes booted proxmox-kernel 7.0.12). If it errors mid-way, STOP (don't reboot half-upgraded).
8. Cleanly stop cortex: qm shutdown 150 (wait until qm status 150 = stopped).
9. Reboot toc: reboot . Wait ~3-5 min.
10. After toc back: pveversion = 9.2; pvecm status Quorate Yes 5/5; then the CRITICAL check — qm start 150 (if not auto-started), wait, then in cortex run nvidia-smi. The passed-through GPU MUST be visible on toc's new 7.0 kernel. If nvidia-smi fails in cortex after toc's reboot -> vfio/passthrough broke on kernel 7.0; STOP and tell me (don't keep rebooting; we check toc's vfio config).
11. FINAL: cortex up + GPU + 4 AI containers healthy; toc on 9.2, 5/5 quorate.
SAFETY: snapshot cortex before driver work; roll back that snapshot on any GPU failure. Do NOT touch the other 4 nodes. If any verify fails, STOP and tell me.
Report: cortex driver before->after + DKMS + containers-on-GPU; toc pveversion before->after + kernel 7.0; cluster quorate; and explicit confirmation the GPU still passes through to cortex after toc's reboot.
Notes / gotchas (from the other 4 nodes)
- PVE 9.2 ships kernel 7.0 as the new default — toc will boot 7.0.12, NOT 6.17.13. The GPU passthrough (vfio) must survive the 7.0 kernel — that's the single highest-risk check (step 10). If the GPU doesn't appear in cortex after toc's reboot, stop and check toc's vfio / kernel-cmdline / blacklist config.
- cortex's NVIDIA driver upgrade triggers a DKMS rebuild against cortex's kernel — confirm
dkms status shows it built.
- The cortex VM snapshot (
qm snapshot 150 pre-gpu-update) is the rollback for the driver work.
- Per memory, the Obsidian LiveSync bridge depends on cortex's live Obsidian process — confirm Obsidian/sync is running on cortex after it comes back.
ef8b1e0bd9