Qwen3.8 27B on Dual RTX 3090s: A vLLM Field Report

There is a gap between a model announcing itself and a model surviving a week of production sessions. Between those two moments is where the interesting stuff happens: the flag that OOMs at full VRAM, the default setting that quietly makes the model overthink, the architecture change you end up reverse-engineering from the logs because the release notes underplay it.

Qwen 3.8 dropped on the 14th. I swapped it in as my daily local coding driver that same day — same two RTX 3090s, same 128 GB box, same role 3.6 had been filling for the week before. It works. It is a step up on real work. And the first two days were rough in ways nobody warned me about, because the failure didn’t show up as the model being wrong. It showed up as the model refusing to start, and an error message that lied about why.

Here is what three days of hammering it into a daily driver actually told me. The specific knob values are the part I’m keeping back — DM me if you want the exact config — but the diagnosis, the reasoning, and the shape of what the logs showed are all here, and that’s the part that actually transfers.

The Setup

Hardware is unchanged from the 3.6 post: two 24 GB RTX 3090s (no FP8 tensor cores, no cheap tensor-parallelism niceties a 4090 would hand you), 128 GB of DDR4, and a 5950X. Same box, same role, same two cards. What changed is the model, and underneath it, the architecture.

If you read the 3.6 post, the baseline launch shape for this rig is already out there and it has not changed. This post is the delta: what actually happened the moment the 3.8 build hit the same two cards, the three things that broke or surprised me, and the reasoning that matters when someone else is running a different version.

The Model: What’s Actually New

3.8 keeps the 27 B size and stays dense — no MoE, no “active parameter” trick to hide behind. It is, plainly, the next generation of the model I was already running. The native context is 262k tokens (extendable toward 1M if you want to chase it), it is Apache 2.0, and it is genuinely multimodal now — image and video in on the same endpoint.

The part that matters for how it runs, and for why the first-day problems behaved the way they did, is the architecture. That is worth its own section, because it is the thing the release notes underplay and the thing that changes what 48 GB can do.

The Architecture That Changes Your KV Math

One line of the 3.8 spec reads like a marginal footnote — “hybrid linear-attention” — but on two 3090s it is the single most important detail. I will call it the way the model card does: a hybrid of linear-attention layers (Gated DeltaNet) and standard full-attention layers.

The mechanism I actually care about is this: in a plain transformer, every token you keep in context earns a KV cache entry in every layer, and that cache grows linearly with context length. In a linear-attention layer, the running state is a fixed-size recurrent matrix instead — it does not stack up per token. 3.8 is mostly linear-attention layers with a handful of full-attention layers sprinkled through, so only that fraction of the layers grow your KV cache as the context stretches. I read the layer breakdown as: most of your KV pressure just went away.

I want to be careful here. I am inferring this from observed behavior — flatter KV growth than 3.6 at comparable context — plus the architecture description, not from a line-by-line trace. But the direction is clear, and it is the same reason 3.8 feels comfortable in context windows where 3.6 was already tight on this hardware. On 48 GB, where KV is usually the first thing to squeeze you, that is a real win, and it is not a marketing sentence about parameter counts.

Serving on vLLM 0.26: Same Bones, New Model

No full config dump here. The general launch shape for this rig is in the 3.6 post and it has not changed. What I am putting in this post is the 3.8-specific stuff: the decisions that mattered and the reasons, not the numbers.

At a high level, the same set of levers is doing the work: tensor parallelism across the two cards, quantized weights (AWQ-INT4), FP8 KV cache to keep the cache at half its size, chunked prefill and prefix caching, FlashInfer as the backend, the tool-call parser for structured output, and MTP. All standard shims you would expect for a 48 GB hybrid build.

The one 3.8-specific wrinkle worth naming is block alignment. The hybrid attention implementation pads context to multiples of 16, so if you are capping effective context, you want that cap to line up with the pad. Order it correctly relative to the serving flags or the framework will silently revert it to the model max. Small thing, but it is the difference between “my context ceiling works” and “it quietly ignored me.”

The OOM That Wasn’t a Compatibility Error

The first real problem reminded me of something I have hit before with newer hybrid backends: an OOM that announces itself as a “compatibility” or “not implemented” error, not a memory error. When 3.8 first came up, at the top of the VRAM ceiling I had it set at, a long prefill would die — and the message pointed at the attention backend, not at exhaustion. I took it at face value the first time and started looking for a version patch.

It was not a version problem. It was out of memory. The linear-attention (Gated DeltaNet) prefill path allocates a scratch tensor that scales with the chunk size and the head count, and on a card that is simultaneously running the attention path, enough heap gets stranded that the failure never shows up cleanly in the usual OOM trace. What I actually did, in plain terms: I backed the VRAM ceiling off from the top, and I turned on the CUDA “expandable segments” allocator so the runtime extends existing memory blocks instead of minting new ones. That combination cleared it and long prefills stopped dropping.

I am flagging the scratch-tensor sizing as my read of the behavior against the architecture, not something I have derived from the source. What I can stand fully behind is the diagnosis and the fix: the error is a memory problem dressed up as a backend error, and a ceiling plus an expanded allocator is what unblocks the prefill.

The Context Cliff

The second surprise was behavioral, not startup. 3.8 ships with its highest reasoning effort (the top of the setting) on by default. On a short, well-bounded task that is fine — the model thinks hard and delivers. At the longer end, though, the combination of maxed-out reasoning plus a big context window makes the model spend a lot of its effort re-deriving things it already knows and re-planning work that is already done. It overthinks.

The fix is not a flag; it is a discipline decision. You do not get 262k tokens of usefulness on 48 GB — a 262k context you actually fill will crush your KV budget and your attention quality at the same time. So I leashed the effective context to what my real sessions use — a few tens of thousands of tokens, a small fraction of the theoretical max — and left the default reasoning effort alone, bounding the work in the prompt instead of fiddling with the model config.

The two trade against each other, and the coupling is what bites. What the hybrid architecture hands you in KV headroom, the default max-reasoning chew eats back unless you actually bound the work. Leash the context, bound the effort in the prompt, and 3.8 behaves like the benchmarks say it should. Push either one and it starts to feel like a smaller model on a smaller box.

MTP: Close to Free

The 3.6 post ran MTP with a single speculative slot. 3.8’s MTP is meaningfully better — in my runs it was accepting blocks of about two new tokens per drafted block, which is the “close to free” regime: the model gets extra tokens per forward pass it would otherwise have paid full price for. I left it on without tinkering.

I have not done an ablation to pin down the exact gain, so treat this as directional. The check I ran is that it was stable — not churning, not retracting rewrites — and that it pulled ahead of a no-MTP reference. That is enough for a daily driver; I am not running a proper benchmark to argue about the number here.

Concurrency Is Where the Win Is

The third thing — and the one that most changes how I actually use it — is concurrency. vLLM is a continuous-batching engine: it interleaves decode steps across multiple requests. On 3.8 that becomes a real win, not a theoretical one.

My own single stream runs in the low-to-mid tens of tokens per second — a fine daily-driver speed. The interesting part is what happened when five concurrent sessions hit the same endpoint: the aggregate throughput roughly tripled the single-stream rate, and the KV peak at five concurrent never climbed past about a third of the cache. That is the signature you want: headroom for batching, and a system that actually uses it.

This is also where the Ollama comparison I almost wrote got dropped, and I will say why in a couple of lines. I ran an Ollama battery against the same model and it uniformly fell apart under concurrent load — not because Ollama is bad, but because the hybrid build is hard-capped at a single parallel slot on that stack (the parallel setting is effectively ignored for this class of model). One session at a time is where that stack wants to live, and for a daily driver that is a hard ceiling. vLLM has no such ceiling, and that single line is the whole reason this post exists.

What I’d Actually Tell You

A daily driver is a specific job. It has to run all day, stay out of the way, handle a bloated context, and do a second-day refactor without drifting. For that job, the store is:

• Bind context to what you use. The hybrid architecture earns you a lot of KV headroom. Your job is to not waste it on a context window you do not need. Leash it.
• Understand your max reasoning effort. The default is the top of the setting. That is a config choice, and on long context it is a behavior change. Bound the effort in the prompt, not in the model config.
• Treat “compatibility” OOMs as memory OOMs. When a hybrid backend refuses a long prefill, the first thing to check is not version compatibility — it is heap. You usually do not need a patch; you need a ceiling and an expanded allocator.
• Use concurrency. Single-stream tok/s is a vanity metric for a daily driver. The win is how many concurrent sessions the system comfortably holds, and vLLM gives you that on two 3090s when the cache has headroom.
• Leave MTP on. On 3.8 it is cheap enough that the right answer is almost always “just turn it on.”

Skip the rest. You do not need someone’s exact tokenizer or knob value to make 3.8 a good farm. The diagnosis is above, and the diagnosis is the part that actually transfers.

What I’m Still Testing

Two things are on the bench this week:

• Leash the client, not just the model. I want to pass the effective context budget directly to the client (opencode, the IDE agent) so it stops padding the context and re-reading whole files. Right now the model’s context is leashed server-side, but the client still does not know where the shelf edge is. A context-length flag the toolchain honors would fix the overthinking I get on long sessions. I have not decided whether that lives in the serving config or on the client side.
• vLLM’s large workspace. 3.6 had a single-slot limitation for really long single completions. The vLLM 0.26 notes call that out as a “large workspace” config. I have not figured out the right shape yet, but if it works the way the notes suggest, one big single completion becomes a clean answer instead of a hack.

Resources

• Qwen3.8 27B (model card)
• Qwen3.8 release
• Ollama GDN hybrid single-slot limitation: issue #14879
• vLLM documentation
• The 3.6 rig (full baseline config lives here): running-qwen36-27b-dual-rtx-3090-vllm-v019

Key Takeaways

• 3.8 is a denser, sharper 27 B. The same 48 GB, the same two 3090s. The model moved; the constraints did not.
• The hybrid architecture is the biggest change. Roughly half the KV cost, more room for batching, and a wide gap between the 260k context and what 48 GB actually supports.
• A “compatibility” OOM on a long prefill is a memory problem more often than not. Check heap before you start version-matching.
• The max-reasoning default is a behavior change, not a knob. Bound the work, not just the context.
• Single-stream tok/s is a vanity metric for a daily driver. The real win is how many concurrent sessions the box holds before the KV cache starts to press back.
• MTP is cheap enough now that leaving it on is the correct default.

If you are running a hybrid-attention model on consumer Ampere cards and want to swap notes — the exact VRAM ceiling that unblocked your prefills, whether block-size padding bit you, how MTP actually behaves on your setup — I read every one of them.

Connect on LinkedIn.