Qwen3.8-27B: a frontier-class model you can actually own

On 14 August 2026 Alibaba published the open weights for Qwen3.8-27B under Apache 2.0. 27 billion parameters, natively multimodal, 262K context out of the box. That is a model good enough for daily work and small enough to run on a single machine you own, which is a combination we have not had before.

Thanks​

Thanks to Selim Kirbiyik, who provided the RTX PRO 6000 that the GPU benchmarks were run on. The Dell DGX Spark reached us through Fraiss and Cancom Austria, who put the machine in our hands for these measurements.

What this release actually changes​

Most of the interesting open-weight models of the last two years were either too small to be useful or too large to self-host. A Mixture-of-Experts model with hundreds of billions of parameters is technically "open", but you still need a rack to run it. Qwen3.8-27B is a dense model, so every parameter is active on every token, and 27B of them fit into a machine that costs less than a company car.

Two details matter more than the parameter count. The context window is 262,144 tokens natively, extendable to roughly a million with YaRN scaling, which is enough to put an entire mid-sized repository in front of the model. And the licence is Apache 2.0, not one of the "open but" community licences. You can run it, modify it, and use it commercially, on your own hardware, without asking anyone.

That is the whole point for us. Companies come to us because their legal department blocked cloud AI, not because they wanted to build a GPU cluster. A capable model under a permissive licence, running on hardware they already own, is exactly what KI on-premise is supposed to look like.

One honest caveat: every launch benchmark you have read so far comes from Qwen itself, and several of the suites are in-house or modified. That is not an accusation, it is just how model launches work. It is also why we ran our own.

Deploying the llama.cpp server​

In the Framework Desktop article we were still using Ollama. Since then we have moved day-to-day inference to llama.cpp as well. The reason is the same one that applies to this test: Ollama hides exactly the knobs we wanted to turn. Which quantisation file is loaded, how many layers go to the GPU, what precision the KV cache uses, and how much context is actually allocated.

llama-server \
  -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL \
  --alias qwen3.8-27b \
  --n-gpu-layers 99 \
  --ctx-size 262144 \
  --flash-attn on \
  --parallel 1 \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  --temp 0.7 --top-p 0.8 --presence-penalty 1.5 \
  --host 0.0.0.0 --port 8080
# srv  llama_server: model loaded

That is the entire configuration. --n-gpu-layers 99 puts every layer on the GPU, --ctx-size 262144 reserves the model's full native context, --flash-attn on enables flash attention, and --parallel 1 keeps the server to a single slot.

The sampling values are the ones Qwen recommends for non-thinking mode. In thinking mode they want --temp 1.0 --top-p 0.95 instead, and the difference is not cosmetic: thinking mode at 0.7 makes the model terse and noticeably worse at multi-step reasoning.

Note what is not in there. There is no --cache-type-k or --cache-type-v, so the KV cache runs at full precision. More on why in a moment.

llama-server speaks the OpenAI API, so everything that used to point at Ollama kept working unchanged. That is what made the switch painless.

The quants come from Unsloth​

We did not quantise the model ourselves. The GGUF files are from unsloth/Qwen3.8-27B-GGUF, built with Unsloth Dynamic V3.0.

The short version of why that matters: a flat 4-bit quant treats every layer the same, which is wasteful in some places and destructive in others. A dynamic quant measures which layers are sensitive and keeps those at higher precision while pushing the rest down harder. You end up at roughly the same file size with a measurably smaller quality drop, and for a 27B model at 4-bit that difference is the line between "useful for coding" and "close, but I would have written it faster myself".

The full BF16 weights are about 55 GB. The file we run, UD-Q4_K_XL, is 16.68 GiB. Worth saying clearly, because it trips people up: the weight file fitting in VRAM does not mean the model fits. KV cache, activations and the vision encoder all want memory on top, and at the full 262,144-token context the cache alone dwarfs the quantised weights. With 96 GB on the card there was room for all of it at full precision, so we left the cache alone. On a smaller GPU that is the first lever to reach for: --cache-type-k q8_0 --cache-type-v q8_0 roughly halves it, at some cost in quality at long context.

Multi-Token Prediction​

The two --spec- flags in the command above are the single biggest speed lever in this setup, and they cost nothing.

Qwen3.8 was trained with a Multi-Token Prediction head, and that head ships inside the GGUF. llama.cpp can use it as a built-in draft model for speculative decoding: the MTP head guesses the next few tokens, the full model verifies the whole batch in one forward pass, and every guess that survives is a token you got for free. The usual speculative-decoding setup needs a second, smaller model loaded alongside the first. Here there is no second model, and the draft context llama.cpp spins up alongside it costs a small fraction of what the model itself takes.

--spec-type draft-mtp turns it on and --spec-draft-n-max sets how many tokens the head may guess ahead. We ran with 4. Deeper drafting is not automatically better: every rejected guess is work the GPU did for nothing, so the sweet spot moves with memory bandwidth and with how predictable the text is. Two is the conservative starting point most write-ups suggest. On a card this size, four paid off.

How much it pays off is measurable, and the server tells you directly. Over one long generation it reported 72% of drafted tokens accepted with a mean accepted run of 3.89 tokens. Across a full working session that acceptance rate swung between 0.39 and 0.94, averaging 0.59. What that does to actual throughput is further down.

--parallel 1 is there because the draft path wants a single slot. With multiple slots the acceptance rate drops.

Three very different machines​

We tested on two systems that solve the same problem from opposite directions, and a third that arrived late.

The first is a workstation with an NVIDIA RTX PRO 6000 Blackwell Workstation Edition, the card Selim provided. 96 GB of dedicated VRAM, compute capability 12.0. The model, its KV cache at the full native context and everything else fit on the card at once with room to spare. Here llama.cpp runs on its CUDA backend.

The second is the Framework Desktop from the FPGA article: a Ryzen AI Max+ 395 (Strix Halo) with 128 GB of LPDDR5X-8000 shared between CPU and GPU. On paper that is 256 GB/s across a 256-bit bus, and around 215 GB/s in practice. The measurements below show what that means for output speed very directly: decode here runs about six times slower than on the dedicated card. In exchange you get more usable memory and a fraction of the power draw and noise. On this machine we built llama.cpp twice, once against ROCm and once against Vulkan. Both put the work on the integrated RDNA 3.5 GPU, and which of the two comes out ahead depends on context depth.

A third machine joined late and gets a short section of its own below: a DGX Spark, built around NVIDIA's GB10 Grace Blackwell superchip with 128 GB of unified LPDDR5X. We have only run the standard benchmark on it so far, so it stays out of the rest of this article.

Three backends, then: CUDA, ROCm and Vulkan, driven by the same flags. The interesting question is not which machine wins, it is whether the cheap one is fast enough to be useful.

Benchmarks​

llama-bench gives the clean baseline. No speculative decoding, no chat template, no client. Just prefill and decode, measured at a set of context depths.

The -d list is the trick worth stealing. Each value is a context depth to pre-fill before measuring, so a single run gives you the whole curve from an empty context to a nearly full one instead of one number that only describes the best case.

pp512 is prefill, how fast the model reads. tg128 is decode, how fast it writes.

RTX PRO 6000, CUDA​

llama-bench -m Qwen3.8-27B-UD-Q4_K_XL.gguf -d 0,512,10000,100000,250000
# ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97285 MiB):
#   Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes

Context depth	Prefill pp512 (t/s)	Decode tg128 (t/s)
0	4126.92 ± 104.05	74.73 ± 0.14
512	4131.61 ± 124.87	74.79 ± 0.12
10,000	3780.64 ± 99.17	72.13 ± 0.28
100,000	1955.39 ± 28.07	56.97 ± 0.03
250,000	1049.22 ± 9.96	42.29 ± 0.04

Decode barely notices the first ten thousand tokens, then starts paying for the cache: 57 t/s at 100K, 42 t/s at 250K. Prefill falls by a factor of four across the same range. The honest read is that the long context is not free, but even the worst number here is well past reading speed while the machine holds a quarter of a million tokens of context.

Framework Desktop, Vulkan versus ROCm​

On the Framework Desktop we ran the same benchmark twice, once against Vulkan and once against ROCm. Same quant file, same depths.

./llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -d 0,512,10000,100000
# ggml_vulkan: 0 = AMD Radeon 8060S Graphics (RADV STRIX_HALO) | uma: 1 | matrix cores: KHR_coopmat
# ggml_cuda_init: found 1 ROCm devices: Device 0: AMD Radeon 8060S Graphics, gfx1151, VRAM: 98304 MiB

All values in tokens per second.

Context depth	Vulkan pp512	Vulkan tg128	ROCm pp512	ROCm tg128
0	365.24 ± 0.47	12.33 ± 0.00	378.24 ± 3.27	12.06 ± 0.01
512	358.86 ± 4.26	12.30 ± 0.00	370.14 ± 3.10	12.04 ± 0.01
10,000	299.01 ± 1.49	11.84 ± 0.00	311.05 ± 2.19	11.66 ± 0.01
100,000	41.57 ± 0.16	9.00 ± 0.00	121.85 ± 0.33	9.05 ± 0.00

At short context the choice of backend barely matters. ROCm reads just under 4% faster, Vulkan writes a good 2% faster, and both differences sit in the range where the decision is not worth making.

At 100,000 tokens that flips. Prefill under ROCm holds 121.85 t/s, while under Vulkan it collapses to 41.57 t/s, a factor of 2.9. Decode at the same depth is practically identical, 9.05 against 9.00 t/s. That is the real lesson from the comparison: the backend does not decide how fast the machine writes, only how fast it reads a long prompt. Working on Strix Halo with short prompts, either will do. Handing the model whole repositories, you want ROCm.

Why decode does not move is something you can check with arithmetic. Every token has to pull the entire weight file through memory once, and at 12.06 to 12.33 t/s that works out to something north of 200 GB/s. The machine delivers around 215 GB/s in practice. Decode here is running at the limit of memory bandwidth, not at the limit of the driver. No backend changes that, only a different memory system does.

Compared with the RTX PRO 6000 the same effect shows up from the other side. Decode sits at roughly a factor of six in favour of the dedicated card across every depth we measured, drifting only from 6.1 to 6.3, and a ratio that barely moves with depth is exactly the signature of a bandwidth limit. Prefill starts at a factor of eleven and widens to sixteen by 100K.

In practice: at 12 t/s the Framework Desktop writes faster than a single person reads. What it costs you is the wait before the first token. Extrapolating from the curve, a cold prompt of over 100,000 tokens lands in the minutes on this machine, and under a minute on the RTX PRO 6000. For chat and individual files you will not notice. For "read this repository" you will.

What is still missing here is a working session with the MTP head on this machine. The numbers in the next section all come from the RTX PRO 6000, and the factor of roughly 1.8 that speculative decoding bought there has not been measured on Strix Halo.

DGX Spark, GB10​

./llama-bench -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -d 0,512,10000,100000,250000
# ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124546 MiB):
#   Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes, VRAM: 124546 MiB

Context depth	Prefill pp512 (t/s)	Decode tg128 (t/s)
0	837.56 ± 8.91	12.56 ± 0.00
512	834.95 ± 7.32	12.54 ± 0.01
10,000	783.19 ± 3.75	12.07 ± 0.01
100,000	502.12 ± 1.31	9.06 ± 0.01
250,000	316.77 ± 1.05	6.43 ± 0.01

The decode column is the surprise. At every depth the GB10 writes within two percent of the Framework Desktop: 12.56 against 12.33 t/s on an empty context, 9.06 against 9.00 at 100K. Two machines from completely different price brackets, the same answer, because both pull their weights out of LPDDR5X and neither is limited by anything else.

Prefill is where the Spark earns its keep. It reads 2.2 times faster than the Framework at short context, and the gap widens to 4.1 times at 100K where the Framework's Vulkan path falls apart. Our run also went out to 250K, which we had not measured on the Framework.

Against the RTX PRO 6000 it comes in at roughly a factor of five on prefill and six on decode. Same bandwidth story, now from a third angle.

In real coding work​

Benchmarks measure throughput. They do not tell you whether the model is worth using.

On the coding suites in Qwen's own model card (reference), Qwen3.8-27B lands roughly at the level of Opus 4.6.

Trading blows with a frontier hosted model is a strange thing to be able to write about 27B parameters sitting on a machine under your desk. Whether that holds up outside a benchmark harness is the part we actually care about, so we left the server running through a normal working session and read the numbers off it afterwards.

Ninety-six requests, context growing to just over 101,000 tokens by the end, the largest single prefill 89,741 tokens.

	Median	Range
Generation	134.9 t/s	86 to 194
Prefill, prompts over 1,000 tokens	2,368 t/s	1,549 to 3,662
Draft acceptance	0.67	0.39 to 0.94

The generation figure is the one to sit with. llama-bench measured 74.7 t/s decode on an empty context with no speculative decoding; in real use, with the MTP head drafting and a context that grew past 100K tokens, the median was 134.9 t/s. Speculative decoding more than paid for the context.

The spread is the other half of the story, and it is almost entirely explained by the draft acceptance rate. Requests that accepted 85% or more of their drafted tokens ran at a median of 169 t/s and peaked at 194. Requests below 45% acceptance ran at a median of 94 t/s. Across all 96 requests, acceptance and throughput track each other closely (r = 0.88).

That is the honest shape of speculative decoding: it is not a flat speed multiplier, it is a bet that pays out when the next few tokens are guessable and costs you when they are not. Same model, same flags, a factor of two between the good case and the bad one.

Key Takeaways​

A 27B dense model under Apache 2.0, with a context window this large, moves self-hosted AI from a compliance workaround to a genuine option. llama.cpp plus a good dynamic quant is what makes it practical, and the Unsloth builds do most of that work for you. Speculative decoding via the built-in MTP head is what turns 75 t/s into a median of 135 in real use, so budget for it when you size hardware. Whether you need a dedicated GPU or whether shared memory is enough is decided less by writing speed than by the wait before the first token, and by how many people are waiting on an answer at the same time. The Framework Desktop writes fast enough for a single person, but reads a 100K prompt sixteen times slower than the RTX PRO 6000.

References​

1. Qwen. (2026). Qwen3.8-27B model card. Retrieved 19.08.2026 from Qwen/Qwen3.8-27B