Highly Available AI Inference Cluster

Mon, 17 Aug 2026 00:00:00 +0000

What this proves: Resilience — enterprise-grade AI infrastructure at a single operator’s desk. Three servers, three GPUs, one routing plane that switches models and fails over without the agent ever noticing.

Overview

I started with the single-box build: a dual-RTX-3090 server running one model. It worked. Then I hit the two problems that kill single-box AI in practice: it’s a single point of failure that interrupts the agent mid-task, and one GPU can’t honestly serve three very different workloads.

So this became what any serious database deployment would be: a fleet with a control plane in front of it. Not three Ollama instances. A tiered compute setup — a heavy compute node, a production node running a vision model, and a dedicated control plane — with a custom routing layer that picks the model, translates the API, and moves traffic when something’s down. If a tier dies, the plane routes around it. My tools don’t know the difference.

The routing layer didn’t exist off the shelf, so I built it: a Rust proxy that speaks both OpenAI and Anthropic protocols, routes by capability and by cost profile, and meters every request.

Architecture

 Claude Code / OpenCode / OpenWebUI / scripts
 |
 switchyard :4000
 (Rust routing plane, control box)
 capability routing / staged escalation /
 A-B splits / failover / per-request metering
 |
 +---------------+----------------+
 | | |
 VADER VAULTTEC (wicket: monitoring,
 dual 3090s Arc B70 telemetry + Grafana)
 vLLM 27B llama.cpp Gemma
 :4242 :4243

The Fleet

Vader — heavy compute. Ryzen 9 5950X, 128GB RAM, dual RTX 3090 (48GB combined). Runs vLLM with a 27B MoE flagship: tensor-parallel across both GPUs, FP8 KV cache, 100K context, speculative decoding. This is the workhorse for daily agentic coding — the one that reads a whole repo and doesn’t blink.

VaultTec — production. EPYC 4345P, 64GB ECC, Unraid, a 12-bay hoard, and an Intel Arc B70 with 32GB. Runs llama.cpp locally with Gemma 4 26B — a vision model that powers my family’s daily chat in OpenWebUI. It’s been the most stable thing on the cluster: load and unload, long conversations, no drama. The production box also hosts the next home of the control plane; the fleet is consolidating, not spreading.

Wicket — control plane. Ryzen 9 5900XT, 64GB, NVMe. Currently hosts the routing plane and the monitoring stack (Prometheus + Grafana) that give me per-request visibility into every target. It also runs the retired 35B tier — gone now. Models earn their slot or they get cut. That’s the rule that keeps the fleet honest.

The Routing Plane

One endpoint for everything. The layer in front does the thinking:

Protocol translation. OpenAI and Anthropic clients both hit the same URL. Claude Code, Codex, OpenWebUI, and scripts all work unchanged.
Capability routing. A classifier picks the model that fits the request — agentic tool-calling goes to the flagship, casual conversation goes to the smaller one.
Staged escalation. Cheap-and-fast first. When tool-use signals show up in the conversation, it escalates to the capable tier. You pay for intelligence only when the request actually needs it.
A/B splits. Random routing between local and capable targets with per-target telemetry, so I can watch quality and cost side by side instead of guessing.
Failover and key rotation. Per-target health, rotating API-key pools, and load-aware routing. A dead or busy node stops receiving traffic; nobody upstream notices.

Built in Rust as a standalone binary, deployed as a container, monitored end to end.

The Economics

Last real session on the cluster, two hours of active agentic CLI work: 12.1M input tokens, 84.8K output, 174 requests, 88.5% cache hit rate. At frontier list pricing, a month of that pace is easily five figures a year in cloud spend — zero on local. And none of it leaves the house.

Private, fast, metered, and mine. That’s the whole pitch.

Philosophy

The goal was never to run the most impressive AI stack. It’s to run an AI stack that behaves like the boring production infrastructure I already respect: several boxes, each with one job, a control plane in front, telemetry on everything, and a rule that nothing stays in the fleet unless it earns its slot.

If the 2024 build was “I put a big GPU in a box,” this one is what happens when you treat AI like a real service. Up, load-balanced, observable, and switchable — at someone else’s cost, hyper-efficient, with my hands on every layer.