Security, upgrades and troubleshooting
Security and monitoring · upgrade runbook and troubleshooting
TL;DR — the 30-second summarytldr
The core of BOMLLM security in one sentence: everything binds to localhost or Tailscale, the only thing open to the world is the single Cloudflare Tunnel gate, every API has a key, every key has a budget, and no secret is ever committed to the repo
The final chapter is the "survivor guide" — every other chapter points back here when trouble hits. It has two parts: upgrade discipline that keeps the system from breaking during upgrades, and the fix recipe for when it breaks at 2 AM
When to usewhen to use
This chapter is not a "read once" — come back to it every time you (1) add a new channel, (2) open a new port/service, (3) before a big upgrade
Come back to this chapter whenever: you are about to upgrade any component, you hit an odd symptom, or you read the words "if you get stuck anywhere" in another chapter
How-tohow to
The bind table of every service (actual policy from ss -tlnp):
| Service | binds to | why |
|---|---|---|
| LiteLLM :4000 | 127.0.0.1 (+ Tailscale via a forward for probing) | the public path is the tunnel only |
| Open WebUI :3000 | 127.0.0.1 | sits behind the tunnel |
| SearXNG :8888 | 127.0.0.1 | internal only |
| Qdrant :6333 | 127.0.0.1 | no public vectors |
| Postgres / Valkey | container network only | no host port opened at all |
| Ollama (Mac) :11434 | Tailscale IP only | mesh only + firewall open |
| SSH | non-standard port + key only | ufw defaults to DROP |
Properties re-checked on every audit round:
# Basic checks you can run yourself (run on the VPS)
curl -o /dev/null -w "%{http_code}" -X POST https://llm.example.com/v1/chat/completions
# must return 401 — no key, no entry (checked every round)
curl -o /dev/null -w "%{http_code}" https://chat.example.com/api/models
# must return 401 — no session, no model list
ss -tlnp | grep -v 127.0.0.1 # see if anything outside localhost is listening that should not beIf any result differs from this = stop everything and fix it back to correct first
Secret hygiene (recap of chapter 4): the master key lives only in .env on the VPS, per-channel keys can rotate without touching the master, .env is chmod 600 + gitignored, logs/reports only ever say "key loaded, length N", and n8n keeps secrets in the credential store only
Monitoring: Uptime Kuma on the NAS probes every 5 minutes (firing GET /api/version through the mesh) and sends a Telegram alert when a status flips — monitoring status is summarized on an internal board
If you get stuck anywhere → Chapter 9
The 6 rules of upgrading (none negotiable):
- Pin the version of everything — latest is absolutely forbidden (we have been bitten by latest changing behavior under us)
- One component per time window — upgrade two things at once and break, and you learn nothing at all
- Back up before touching — dump Postgres + snapshot the config, and it must be "really restorable" (a backup never restored is a rumor)
- Write the rollback plan before typing the upgrade command
- Run the 6-probe check after every change — all six green or roll back
- Outside peak hours — after 23:00 every day, avoiding our own automatic restart window
The 6-probe check set (run after every change):
The full original lives in docs/upgrade.md in the repo — copy it into your own script freely
Common symptoms with hands-on fixes:
| Symptom | Common cause | Fix |
|---|---|---|
| Bot answers blank | think enabled + low num_predict | pin think:false on the route (chapter 4) |
| Text streams slowly, in bursts | two models resident until swap kicks in | evict the secondary model (keep_alive 0), then see chapter 2 |
| Cannot log into the chat site, rapid 401s | key/session expired | check the master key in .env, then up -d again |
| Thai answers start showing Chinese characters | system prompt lost / the new route is missing Config G | go back and check the route + chapter 6 |
| Today's cloud bill abnormally high | the Mac was down long enough that Tier 3 worked hard | check the hourly spend log + find the cause on the Mac |
| Updated and broken | floating version / params silently dropped | roll back to the old tag first, find the cause afterwards |
The 2 AM recipe (broken at night): (1) do not debug deep while sleepy — roll back to the old tag first (2) save the last 30 minutes of logs into an incident folder (3) sleep (4) write a one-page post-mortem within 48 hours naming the root cause + which rule, had it been followed, would have prevented this incident, then add that rule to the manual
Config exampleconfig example
Threats vs countermeasures table (abridged from docs/security.md):
| Threat | What the system does |
|---|---|
| Leaked key (git/image/log) | gitignore + pre-push checks + a per-key budget limits the damage |
| Internet scanners | exposed through the tunnel only + 401 without a key |
| VPS breached | no models/heavy data on the VPS; the Mac sits in the mesh; off-site backups every night |
| Prompt injection via web results | search results are context, not commands + the system prompt pins behavior + bots have post-processing |
| Runaway costs | daily budget per key + $5 total cap + alerts |
| Supply (image) changed under you | every dependency pinned to a version tag, latest forbidden |
The popular upgrade steps in short (full version in docs/upgrade.md):
# --- Ollama on the Mac ---
brew upgrade ollama # then restart the service
ollama --version # client == server must match
ollama ps # the main model must be resident again
# the first warm-up chat must be < 5 s to count as done
# --- LiteLLM / WebUI on the VPS ---
docker compose pull litellm # the pinned tag
docker compose up -d litellm # recreate just this one
docker compose logs -f litellm # watch boot: parse config + DB migrate
# after upgrade: diff /model/info against the pre-upgrade snapshot + run the 6-probe smokeThe top risk = database migrations, so dump Postgres every time before pulling
Pro tip — field-tested advicepro tip
FAQfaq
Q: Does any data leave the house? — No, except when the Tier 3 cloud is pulled in (it can be disabled entirely in chapter 4) and web searches that go online to fetch results (the search engine runs locally, but the target website sees the VPS IP as usual)
Q: What if I want it locked down as tight as possible? — close signup on the WebUI + wrap every admin page in Cloudflare Access + turn off Tier 3 + set ufw to DROP all inbound except SSH/Tailscale — the system goes completely silent to the outside world
For other questions see the FAQ
Q: How often should I update? — an urgent CVE = immediately but by the 6 rules; normal = once a month in a night window; macOS = only in a quiet week
Q: So broken you do not know what to do? — roll the whole stack back to its previous tags (possible because everything is version-pinned + you have dumps), then hunt the cause on a copy, not on the live system customers are using
See alsosee also
- Chapter 2 · Prepare the network — where this architecture comes from
- docs/upgrade.md — the full English version
Figuresfigures
Real screen recording for this lesson · click play to hear the summary · not investment advice
▶ Watch the manual video: https://youtu.be/O4hM39cgDCY