- Published on
I capped Qwen3.6's reasoning tokens. 8192 matched free thinking.
- Authors
- Name
- Will
Qwen3.6-27B is a strong coding model, but it will happily burn 18,000 hidden reasoning tokens on a single problem before it writes the answer. That might not cost much money when you are self-hosting. The real cost is the time spent waiting and the context window it consumes.
On a 50-task LiveCodeBench slice, an 8192-token vLLM reasoning cap matched free thinking at 86% pass@1 while cutting mean generation time 55% from 289 seconds to 131 seconds.
Qwen overthinks
I knew the shape of the problem from running Qwen 3.5 and 3.6 locally. They are good models, but the reasoning traces get enormous.
That changes the practical meaning of "fast." Tokens per second is not the whole story. A model that emits tokens slowly can still answer faster if it needs far fewer tokens to get there.
Source: Benjamin Marie's Kaitchup analysis of Qwopus vs. Qwen3.5.
That is why Qwopus caught my attention. Qwopus3.5-27B-v3 is built on Qwen3.5-27B and post-trained toward a different reasoning style. Benjamin Marie's analysis framed it as a trade: a small accuracy cost in exchange for much shorter generations.
Source: Benjamin Marie's Kaitchup article, via the Qwopus model card.
Qwopus made the question feel plausible: could you keep most of Qwen's quality while making it reason less?
But Qwopus uses training. I wanted to know how far I could get at inference time.
The trigger was grammar-constrained reasoning
What made me stop theorizing and run experiments was Kaya Omer's Structured CoT post.
The claim was surprising enough that I did not understand how it could work: constrain only the model's <think> block with a tiny grammar, leave the answer channel open, and get much shorter reasoning without losing accuracy.
I cloned the repo because I wanted to understand the mechanism by working with it myself. But grammar was not the lever I wanted to lean on.
Kaya's post describes the constrained grammar as guided generation applied to the private scratchpad. The grammar makes the thinking block look like a compact plan instead of an unbounded monologue. Historically, output restrictions often hurt model performance, so the claim was worth checking rather than accepting.
The repo I was working in already tested that grammar path. I wanted to isolate a simpler lever:
What if I let Qwen reason freely, but put a hard cap on how long the reasoning block is allowed to run?
vLLM already had the control I needed
My vLLM server was already running a recently landed feature that applies a hard limit to Qwen's reasoning tokens, plus an optional string to smooth the hard stop. I had set a fairly large reasoning budget, around 32,000 tokens. That was a safety rail, not an optimized setting.
My server also passes a custom end string that vLLM injects when the reasoning budget runs out:
{
reasoning_start_str: "<think>",
reasoning_end_str:
"My reasoning budget is exhausted, but I have enough information to answer directly now.</think>",
}The important part is the injected sentence: "My reasoning budget is exhausted, but I have enough information to answer directly now." This is the text the model sees at the cliff edge between hidden reasoning and the final answer.
That set the shape of the experiment:
- grammar-constrained thinking versus free thinking
- no thinking versus explicit reasoning caps
- the same task list every time
- not just accuracy, but hidden tokens, visible tokens, cap hits, extraction failures, and generation time
The benchmark slice was LiveCodeBench release_v6, LeetCode only, contest_date >= 2025-01-01, 50 tasks, public functional tests only.
That setup makes the modes comparable to each other but it is obviously not an official leaderboard score.
Small caps underperformed thinking off
I started with thinking off plus small caps: 256, 512, and 1024 reasoning tokens.
| Mode | Pass@1 | Mean think tokens | Mean total tokens |
|---|---|---|---|
| Thinking off | 40/50, 80% | 0 | 1,943 |
| Budget 256 | 34/50, 68% | 251 | 2,135 |
| Budget 512 | 36/50, 72% | 512 | 2,176 |
| Budget 1024 | 39/50, 78% | 1,022 | 2,598 |
The miss was useful. The tiny caps underperformed thinking off.
But they also showed a clear trend: as I allowed more reasoning tokens, accuracy recovered.
Thinking off also taught the first measurement lesson: turning hidden reasoning off does not mean the model stops doing work. A lot of that work moves into the visible answer: comments, scaffolding, repeated explanation, or code-adjacent scratchpad.
That same pattern showed up in the grammar-constrained runs. The <think> block got shorter, but the model sometimes compensated by pushing deliberation into comments or final-answer text.
If you only count hidden reasoning tokens, you can fool yourself.
Free thinking set the target, but it was expensive
After pinning the task IDs to ensure repeatability, I ran free thinking, grammar mode, and a 2048 reasoning token budget. All runs used --max-tokens 65536 as a hard cap on total output.
| Mode | Pass@1 | Mean think tokens | Mean total tokens | Mean generation time |
|---|---|---|---|---|
| Free thinking | 43/50, 86% | 18,080 | 18,464 | 289s |
| Grammar FSM | 34/50, 68% | 158 | 5,026 | 78s |
| Budget 2048 | 40/50, 80% | 1,957 | 7,011 | 110s |
Free thinking was the quality target, but it hit the 65,536-token completion cap seven times. Five of those cap hits failed because the model never produced extractable code. That is a weak argument for letting the model write forever.
The grammar result was instructive for the wrong reason. It crushed hidden reasoning tokens, down to 158 on average, but only passed 34/50. It also averaged almost 4,900 post-think tokens, which is exactly the answer-channel compensation problem.
The 2048 cap was more promising: still below free thinking, but close enough to justify pushing higher.
An 8192-token cap matched free thinking
The final run tested 4096, 8192, and 12288 reasoning-token caps on the same pinned 50 tasks, with --max-tokens 65536.
Source: my local run against cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4 served through vLLM. Dataset slice: LiveCodeBench release_v6, LeetCode, contest_date >= 2025-01-01, 50 public-test tasks.
| Mode | Pass@1 | Mean think tokens | Mean total tokens | Mean generation time | Completion cap hits |
|---|---|---|---|---|---|
| Budget 4096 | 41/50, 82% | 3,537 | 7,430 | 115s | 2 |
| Budget 8192 | 43/50, 86% | 6,046 | 8,534 | 131s | 1 |
| Budget 12288 | 44/50, 88% | 8,006 | 11,022 | 169s | 2 |
| Free thinking | 43/50, 86% | 18,080 | 18,464 | 289s | 7 |
To me, the headline result is 8192.
It matched free thinking at 43/50 while using about one-third of the reasoning tokens, less than half the total tokens, and less than half the wall-clock generation time.
12288 scored one task higher. I would call that the current quality ceiling, not a proven better default. On 50 tasks, one task is two percentage points, and the task-level changes were not monotonic.
Compared with 8192, the 12288 cap gained tasks 3688, 3714, 3789, and 3793, but lost 3562, 3674, and 3781.
More reasoning changed the path. It did not simply preserve every lower-budget success and add more.
Time made the trade-off clearer than tokens alone
Reasoning-token caps are useful, but generation time is closer to the real goal. Fewer hidden tokens matter only if they get me a good answer faster.
Source: my local benchmark logs and JSON summaries. Generation timing was added to the harness during the work, so the later runs record it directly.
The timing plot is why 8192 looks like the obvious winner to me. It sits on the same accuracy line as free thinking, but far to the left.
12288 is still much cheaper than free thinking, but it moves meaningfully right for one extra public-test pass.
Source: same pinned benchmark runs. Percentages compare each cap's mean reasoning tokens, total tokens, and generation time against free thinking.
The cost picture is blunt:
| Mode | Pass delta vs free | Reasoning tokens vs free | Total tokens vs free | Generation time vs free |
|---|---|---|---|---|
| Budget 4096 | -2 | 20% | 40% | 40% |
| Budget 8192 | 0 | 33% | 46% | 45% |
| Budget 12288 | +1 | 44% | 60% | 59% |
If I had to choose a setting from this run, I would use 8192 as the default and keep 12288 as a retry or quality mode.
The cutoff string is an unexplored optimization surface
The current end string is:
My reasoning budget is exhausted, but I have enough information to answer directly now.
It works, but I have no reason to believe it is optimal.
When vLLM injects that phrase, the model has to recover from an interrupted reasoning trace and produce final code. A better phrase might make that transition cleaner. A worse phrase might cause apology text, answer-channel reasoning, bloated comments, or incomplete code.
This is where I want to try GEPA or a similar optimizer. The search target is not the user prompt. It is the server-side reasoning_end_str.
The evaluator should reward:
- pass rate at lower caps, especially 4096 and 8192
- fewer total tokens
- lower generation time
- fewer completion cap hits
- fewer extraction failures
- less comment bloat and answer-channel reasoning
The awkward part is operational. reasoning_end_str is a vLLM server config, not a normal per-request parameter. Testing many candidates probably means restarting vLLM for each candidate or serving several variants in parallel.
Still, this is the obvious next lever. If a better cutoff phrase makes 4096 behave more like 8192, the payoff is large.
What I would rerun before making stronger claims
This was a useful experiment, not a final benchmark.
First, these are public LiveCodeBench functional tests, not private official scores. They are good enough for comparing modes on the same pinned task list, but they are not leaderboard numbers.
Second, I did not retune sampling. Qwen's recommended precise-coding settings use temperature=0.6, top_p=0.95, and top_k=20. The optimal cap may move under those settings. This experiment used temperature=0.0 for repeatability across runs.
Third, the low-budget run should be repeated with the corrected tokenizer and the same 65,536 completion cap used later.
Fourth, 50 tasks is small. The shape of the curve is consistent enough to be useful, but the exact 8192-versus-12288 trade-off needs repeats.
The takeaway
It's possible to match the accuracy of unconstrained thinking with a hard reasoning budget limit. For this particular set of tasks, that limit was about 8000 tokens.
There may be opportunities to refine or smooth this behavior further.