I Built an LLM Traffic Router for My Homelab — Here is What Happened

Wed, 12 Aug 2026 00:00:00 +0000

I have three GPUs between two machines in my homelab. Two RTX 3090s on one server, an Intel Battlemage B70 on another. Between them sits a mess of llama.cpp, Ollama, and vLLM instances, each running a different model. Half a dozen endpoints. No consistency. Picking models was manual.

I was routing requests by muscle memory. Simple question? Hit the lighter model. Complex task? Switch to the heavier one. The mental overhead of loading up OpenCode, changing models between sessions, or writing API code that guesses which endpoint to call was eating time. I needed all endpoints to disappear behind a single URL so the client just sends a request and the infrastructure figures out the rest.

That is where NVIDIA NeMo Switchyard came in.

Switchyard is a Rust proxy that sits between AI clients and model backends. It translates between OpenAI and Anthropic API formats, applies routing rules, and logs everything. One URL replaces all your model endpoints. The proxy decides which backend handles each request without any client configuration.

I deployed it on my homelab “wicket” box — AMD Ryzen 9 5900XT, 64GB RAM, that B70 GPU. The vLLM instances on my other machines handle the heavy inference. Switchyard orchestrates the whole setup.

The Setup

There are two kinds of routing Switchyard supports out of the box. You can route by algorithm or by complexity.

Algorithm routing (random split, round-robin, stage-based escalation) is useful for A/B testing or load balancing. I set up both a random split and a stage router that starts with the cheapest model and escalates if it detects tool-use signals.

Complexity routing is where it gets interesting. The llm_classifier mode climbs the request, analyzes it, and routes to a lightweight model or a capable model depending on what the request actually demands. This is what you end up using for real work.

I built a Gemma-4 E4B classifier on the B70 to handle the verdicts. Here is the thing that surprised me: a 4B model running at 90 tokens per second makes a better router than a 35B model chugging at 24 tokens per second. The E4B is lighter, faster, and still smart enough to tell the difference between “translate this sentence” and “write a Rust parsing library in idiomatic code.”

The classifier sits alongside a Qwen3.6-35B model on the same B70 GPU. The 35B takes about 20GB of VRAM. The E4B takes 5GB. Both fit. The B70 has 32GB prefetchable memory. With 8GB of headroom left, hitting the classifier fast enough was not a problem.

How the Classifier Actually Behaves

I expected the classifier to pass everything to the heavy model for anything remotely complex. The logs showed something different.

It routes about 80% of requests to the lightweight tier. The heavy model only gets the genuinely complex tasks. That is the opposite of what I assumed when I first started. Most of what I do is simple text — queries, reformatting, explaining things — and the proxy knows enough to not bother waking up the big model for that.

The classifier uses a model like the E4B to analyze prompt complexity. If the classification hits above a threshold, it routes to the capable model. My base_threshold is set to 0.5, meaning the request has to register as moderately complex or higher before the heavier model activates.

The benefit is immediate cost savings on GPU memory utilization. I avoid running the 35B model 80% of the time. The B70 handles the light work instantly.

Integration Was Easy

Pointing clients at Switchyard takes three environment variables:

export OPENAI_API_BASE="http://YOUR_ROUTER_IP:4000"
export OPENAI_API_KEY="your-api-key-here"
export OPENAI_MODEL="smart"

The smart model name is the route. Switchyard receives the request, analyzes it, routes to the appropriate backend, and translates the response back to the client. The client never knows it sent the request to three different GPUs on different machines. It sends one request and gets one response. The switching happens transparently behind the proxy.

I wired OpenCode to use the Smart route. switchyard/smart is now one of my OpenCode model choices. The UI has the additional routes, including a stage-based router and passthrough paths for when I want to bypass routing entirely.

Why This Matters

Every setup with more than one backend runs into this problem. You have the capable model for complex work, the lighter model for speed, and the expensive cloud model for when local hardware struggles. Switching between them manually is tedious. Setting up dashboards to monitor the expensive model is a burden.

A traffic router handles the switching. It routes by complexity, creates the log file, and translates the API formats. If you add a new backend, you edit one config file and restart the proxy. The clients never change.

I am running a Prometheus stack alongside the proxy for observability. The dashboard tracks request counts, success rates, and per-model latency. When routing patterns shift or the classifier makes a mistake, you see it in the logs. When the heavy model is hit more than usual, you know there is an actual reason for it instead of guessing.

It is infrastructure doing what monitors are supposed to do — telling you when stuff is slow or broken. The proxy replaces a manual switch with a self-service one.

What Hurts

Two things.

First, running multiple models on one GPU eats VRAM fast. The B70 fits both the 35B weights and the E4B weights, but the total consumption gets tight if both are making heavy context passes. I settled on a 35B/4B split. The 35B is reserved for the capable endpoints, and the E4B is reserved for the classifier that runs lightweight routes. It works, but it means I cannot run another model on that GPU without killing something.

Second, the classifier is occasionally overly conservative. When the threshold is set to 0.5, it leans toward the heavier model. I am running the E4B classifier with a higher threshold to reduce false positives. The goal is 80/20 lightweight/complex split. If I am hitting 60/40, I adjust the threshold. If I am hitting 90/10, I lower it. This is not a plug-and-play fix — it takes tuning.

The Bottom Line

If you are running more than one local model backend, a traffic router is worth the setup cost. It removes the manual switching overhead, translates API formats, and makes inference behavior observable. The heavy lifting stays on the servers that have the memory, and the simple tasks stay on the lighter model without waking up the big one.

You set up infrastructure once. You do not want to keep switching GPUs or reconfiguring API endpoints every time you change tasks. A traffic router handles the switching automatically.

This is a project on my homelab using NVIDIA NeMo Switchyard on a Battlemage B70 GPU.