Self-Hosted AI Inference Cluster

What this proves: AI — I build and operate AI infrastructure right now: dual 3090s serving Qwen 3.6 35B to real workflows, zero cloud dependency.

This was the v1 single-box build. The current state is a three-server high-availability cluster with a custom routing plane — Highly Available AI Inference Cluster.

Overview

Running local LLMs isn’t a hobby — it’s a production system. This cluster powers everything from agentic coding workflows to personal AI assistance, running entirely on hardware I own with zero cloud dependency.

The core setup: dual NVIDIA RTX 3090s (48GB VRAM combined) running vLLM with tensor parallelism to serve Qwen 3.6 35B. 100K context window, 8 concurrent sequences, and the kind of throughput that actually makes local models usable for real work.

Hardware

• GPU: 2x NVIDIA RTX 3090 (24GB VRAM each, 48GB combined via NVLink)
• CPU: AMD Ryzen 9 5950X (16 cores)
• RAM: 128GB DDR4
• Storage: NVMe for model weights and cache

Software Stack

• Inference: vLLM with tensor parallelism
• Model: Qwen 3.6 35B (quantized)
• Orchestration: Docker containers
• API: OpenAI-compatible endpoint
• Context: 100K token context window

What It Powers

• Agentic coding workflows (this conversation, for example)
• Code review and analysis
• Research and documentation assistance
• Personal AI tasks across the household
• Model evaluation and benchmarking

Why Self-Hosted

Privacy, cost, and control. Every API call to a cloud LLM is a data leak waiting to happen. Every token costs money. Every rate limit is a deadline on your productivity. Self-hosted means your AI works when you need it, handles whatever you throw at it, and never charges per token.

The initial hardware investment pays for itself in a few months compared to equivalent cloud API usage. After that, it’s just electricity.