Running Qwen3.8-Flash-Next Locally: A Hands-On Setup Guide

Qwen3.8-Flash-Next landed on Hugging Face and ModelScope on 26 August 2026, and it is the most interesting local model release in a while — not because of the benchmark chart, but because of the shape of it. 125B total parameters, 6B active per token. That ratio is the whole story: you need a lot of memory, but you do not need a lot of compute, which is exactly the trade a workstation or a maxed-out Mac is good at.

This is a setup guide, not a review. Below is what it takes to get it answering on your own hardware, which build to pick for the RAM you actually have, and the flags that will waste an afternoon if you miss them. The last section is a full set of measured numbers from one AMD Strix Halo box, so you have a real reference point rather than a vendor chart.

What you are installing

The 51B n-gram table is the detail that changes your planning. It is not weights you need resident in fast memory — it is a lookup table that can happily live on SSD and be paged in. That is why a 125B model fits on a 64 GB machine at all.

On the numbers Qwen published: 62.5 on SWE-bench Pro, 58.7 on DeepSWE 1.1, 91.9 on LiveCodeBench, 91.7 on GPQA Diamond. Treat those as a reason to try it rather than a promise. The practical claim worth testing yourself is that it runs roughly 3× faster than the dense Qwen3.8-27B on Apple Silicon, because only 6B parameters fire per token.

Pick your build before you download 90 GB

Quantised GGUF builds land in three useful tiers. “In memory” is what has to stay resident; the rest pages from SSD.

The rule of thumb: take Q4_K_M if you have the headroom. It costs 8.7 GB more resident than IQ4_XS and picks the wrong token roughly half as often — the KL gap between those two is far bigger than the gap between Q4_K_M and Q5_K_M. Q5 buys you almost nothing for another 18 GB.

Then add context on top. Budget ~0.2 GB at 8K, ~0.8 GB at 32K, and ~6.5 GB if you actually run the full 262K window. Most agentic work sits fine at 32K, and you will feel the difference in prompt-processing latency if you leave it wide open for no reason.

So: 64 GB machine → IQ4_XS with the table paged. 80–96 GB → Q4_K_M. 128 GB+ → Q5_K_M fully resident, or Q4_K_M with a huge context.

Path 1: Ollama (fastest to a prompt, Apple Silicon only)

If you are on an M-series Mac with 128 GB or more, this is a two-minute job.

ollama run qwen3.8-flash-next

Specific tags, if you want to pin one:

ollama pull qwen3.8-flash-next:125b-a6b-nvfp4      # 113 GB
ollama pull qwen3.8-flash-next:125b-a6b-mlx-bf16   # 360 GB, full precision
ollama list

The catch, and it is a big one: every tag Ollama currently ships for this model is MLX format. Apple Silicon only. There is no GGUF tag, so if you are on Linux or Windows with Nvidia cards, Ollama is a dead end for now — skip to path 2 or 3. The 113 GB default tag also means 128 GB unified memory is a floor, not a target, once you add context and activations.

Thinking mode is on by default. You can turn it off or dial it down through the API — but read the correction near the end of this post before you do, because on this model that trade is worse than it looks:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-flash-next",
    "messages": [{"role": "user", "content": "Refactor this function."}],
    "think": false,
    "reasoning_effort": "medium"
  }'

reasoning_effort accepts low, medium and xhigh.

Path 2: llama.cpp with GGUF (the portable route)

This is where you go for Linux, Windows, or a Mac under 128 GB. Build current llama.cpp — this architecture is new enough that a stale binary will simply refuse the file.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-server

On a Mac, raise the wired-memory limit first so the GPU can hold the resident set (57344 MB ≈ 56 GB — adjust to your machine):

sudo sysctl iogpu.wired_limit_mb=57344

./build/bin/llama-cli \
  -m Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
  -ngl 99 -c 32768 --jinja -fit off

Serve it on an OpenAI-compatible endpoint so your editor and agents can reach it:

./build/bin/llama-server \
  -m Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
  --alias qwen3.8-flash-next \
  -ngl 99 -c 32768 --jinja -fit off \
  --host 127.0.0.1 --port 8080

Three flags decide whether this works:

• -fit off — skips the parameter-fitting pass that trips over this architecture. Leave it on and you get an error that looks like a corrupt download but is not.
• --jinja — applies the model’s chat template. Without it, thinking-mode output leaks into your responses as raw text.
• Leave mmap enabled. Do not pass --load-mode none. Paging the n-gram table off SSD is the whole reason this fits; disabling mmap tries to pull all 92.9 GB into RAM.

For vision input, use the multimodal CLI with the projector file:

./build/bin/llama-mtmd-cli \
  -m Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf \
  --mmproj mmproj-Qwen3.8-Flash-Next-F16.gguf \
  --image screenshot.png \
  -ngl 99 -c 32768 -b 512 --image-min-tokens 1024 --jinja -fit off \
  -p "What is failing in this stack trace?"

For reference, a 64 GB MacBook Pro M5 Max on the Q4_K_M build with the table paged from SSD reported ~517 tok/s prompt processing and ~36 tok/s generation. That is genuinely usable for agentic loops. (Numbers from a different class of machine are further down.)

Path 3: vLLM or SGLang (if you have server GPUs)

Full-precision weights on a multi-GPU box, straight from the Hub:

vllm serve "Qwen/Qwen3.8-Flash-Next"

python3 -m sglang.launch_server --model-path "Qwen/Qwen3.8-Flash-Next"

Both expose an OpenAI-compatible API. Same rule as llama.cpp: pull a recent release, because the Gated DeltaNet + sparse-attention hybrid needs current kernels.

Sampling settings that matter

Qwen ships different recommended parameters per mode, and the defaults in most clients are wrong for both. Note this top-k is the token sampler’s, unrelated to how many experts the MoE routes to.

The presence penalty of 1.5 in instruct mode is unusual and deliberate — it is what stops the model looping on repetitive tool output. If you copy your Qwen3 config across unchanged, this is the line to fix first.

Wiring it into your tooling

Every path above gives you an OpenAI-compatible base URL: http://localhost:11434/v1 for Ollama, http://localhost:8080/v1 for llama-server, http://localhost:8000/v1 for vLLM. Point your editor extension or agent runner at it with any non-empty API key. Start at 32K context, confirm you get clean tool calls, then widen the window — not the other way round.

Measured on AMD Strix Halo (Ryzen AI MAX+ 395)

Everything above is the general setup path. This section is one machine’s numbers: a Ryzen AI MAX+ 395 / Radeon 8060S with 122 GB unified memory, running the UD-IQ4_XS build (93.68 GB across three shards, plus a 0.90 GB F16 projector) on the Nathanw1014/strix-halo-llamacpp v0.7.1 fork.

The comparison is against DeepSeek V4 Flash IQ2_XXS on the same box at identical geometry (-c 524288 -np 4 -b 2048 -ub 2048 -ctk/-ctv q8_0):

Load takes 62s warm and 157s cold. GTT settles at 74–80 GiB. Vision works: a colour-identification probe on a test image returned correctly at 20.10 t/s.

The Strix-Halo-specific flags that matter here, beyond the generic ones above:

-ngl 99 --n-cpu-moe 0 -fa on \
--load-mode mmap --no-host --no-repack --fit off \
-c 524288 -np 4 -cb -b 2048 -ub 2048 -ctk q8_0 -ctv q8_0

--n-cpu-moe 0 keeps the experts in GTT, which is what makes decode fast. And --load-mode mmap is not optional: with --load-mode auto the server silently allocates a second ~95 GiB copy of the weights. You can detect it by watching RssAnon — correct behaviour keeps it in the hundreds of MB, not tens of GB.

Multi-token prediction inverts under concurrency

The 4B MTP head is genuinely fast for a single user and actively harmful for a server. Same box, same model, MTP the only variable:

So MTP is the right call for an interactive single-user session and the wrong call for anything serving concurrent agents. Speculative decoding spends compute on draft tokens that may be rejected; with one stream there is idle capacity to burn, and with four there is not.

Correction to the thinking-mode advice above

Earlier in this post I showed how to disable thinking or drop reasoning_effort. On this model that is a bad trade, and here are the numbers. A 16-task suite at temperature 0, mechanically checked, run twice with thinking as the only variable:

Nine times faster and half as accurate. It does not fail gracefully on the hard items only — it fails the basics:

arith        5638   (correct 7192)   multi-step arithmetic
code_out       10   (correct 9)
logic         Ben   (correct Ana)    five-runner ordering puzzle
wordcount   "Databases store data efficiently."   4 words, asked for 5

Those first four come from a suite that scores 10/10 with thinking enabled. Disabling it does not trim padding; it removes the model’s working memory, and answers arrive in half a second confidently wrong. The latency of a reasoning model is the price of it being correct. If you need faster, use a smaller model — do not lobotomise this one.

Two tuning mistakes worth copying the fix for

q4_0 KV cache was a regression, not an optimisation. Halving KV precision looked like free cache capacity. Identical test, idle box:

Accuracy held at 16/16 either way, which is why this nearly went unnoticed — the predicted risk was verified and the unpredicted one was not re-measured. Smaller KV is not automatically faster: dequantisation costs work on every token. It also did not shrink the context checkpoints at all, which stayed at 112.571 MiB each, so the memory saving was smaller than projected too.

Prompt cache sizing, wrong in both directions. An inherited --cache-ram 4096 was too small — 15 evictions in 10.5 hours, with individual entries up to 2117 MiB, because KV costs roughly 27 KiB per token here and a full 131072-token slot is about 3.4 GiB. Raising it to 16384 overshot: with five agent profiles live the cache filled and pushed the box into swap. 8192 holds four or five live conversations without overcommitting.

The diagnostic lesson from that second mistake is the more useful one. Swap at 99.8% got read as memory pressure and the cache got blamed. It was not the cache: per-process accounting showed the largest swap consumer was Chrome, at 2.7 GB, while MemAvailable sat at 37 GB. Full swap is not the same as memory pressure — pages evicted earlier stay in swap until something touches them, and Linux never drains it proactively. On a box like this, MemAvailable is the pressure signal, not swap percentage.

Running five agents against one server

Five agent profiles now share four slots at 131072 tokens each. It holds up better than expected: over two representative days the server was idle 70–99% of the time, with two-or-more slots active for only 5–14%, and four slots simultaneously for 0.4%. No deferred requests. The binding constraint on this hardware is generation speed, not slot capacity.

One number to size your own context with: the agent profiles here assemble a ~40 KB system prompt and run 20–58k token conversations, so a 131072-token slot is adequate rather than generous. Note that the per-slot context, -c divided by -np, must match whatever your agent framework expects, or requests fail outright.

Is it worth the disk?

If you have 128 GB of unified memory or a workstation with the RAM to spare, yes — a 262K-context multimodal model with agentic coding scores in this range, running entirely on your own hardware under a licence that permits commercial use, is a genuinely different proposition to renting an API. If you are on 32 GB, this one is not for you and no quantisation trick will change that. Wait for the smaller siblings.

The honest caveat: this is a preview architecture that Qwen has openly described as the foundation for Qwen4. Tooling support is moving weekly. Pin your versions, and expect the flags above to change.