SpecLane: Serving Small Hybrid-Attention Models Under Agent Traffic
How should an inference engine be configured to serve small language models to AI agents? I built a serving configuration for two open-weight models, Qwen3.8-27B-FP8 and Qwen3.5-9B, on the vLLM engine. I start from the default setup, change one setting at a time, and measure speed, responsiveness and answer quality to verify each change.
Speed and latency figures are generated from the raw result files by paper/make_results.py. The accuracy check was run on live endpoints with scripts/gsm8k_endpoint.py.
In short
- What this is. A serving configuration built from features that already exist in vLLM. The benchmarks verify the configuration; no new method is proposed.
- Main finding. Speculative decoding, using the multi-token-prediction (MTP) head that ships with each model, made text generation 50–124% faster on the 27B model and up to 77% faster on the 9B model. I measured no loss in answer quality.
- A setting I do not recommend. Storing the attention cache in 8-bit floating point (FP8) made generation slower and the first response later at this model scale, so I do not recommend it.
- A startup problem. With default settings, vLLM 0.33.0 cannot start the 27B model on a single H100 GPU, because its default limit on concurrent sequences is higher than the memory for per-sequence state can support.
- Agent harness pilot. I also tried a prompt-based agent harness on two agent benchmarks. At the sample sizes run, it showed no measurable accuracy difference, so I report it for completeness and draw no conclusion from it.
Key terms
Throughput
How many output tokens (word pieces) the server generates per second, summed over all requests. Higher is better.
Time to first token (TTFT)
How long a request waits before the first token appears. It governs how responsive the system feels. Lower is better.
p99
The 99th percentile: the value that all but the slowest 1% of requests stay under. With 24 requests per measurement, it is simply the slowest request.
Concurrency (c)
How many requests are in flight at the same time. c = 1 measures a single user; c = 16 measures a loaded server.
Multi-token prediction (MTP)
Alongside its normal output layer, each model carries a small extra head trained to predict several upcoming tokens at once. In speculative decoding that head proposes the next few tokens cheaply, and the full model verifies all of them in a single pass, keeping every correct proposal. The number after MTP in the detailed tables is how many tokens are proposed per step.
KV cache
Memory that stores the attention keys and values of earlier tokens so they are not recomputed. “FP8 KV cache” stores them in 8-bit precision to save memory.
Results
MTP is speculative decoding with the model’s multi-token-prediction head. Output tokens per second on the agent_sessions workload (multi-turn conversations that resend their history), one run per measurement. The change from the default (“baseline”) setup is in parentheses. c is concurrency. The last column is the p99 time to first token, in seconds, at c = 16.
| model | profile | c = 1 | c = 16 | p99 TTFT, c = 16 (s) |
|---|---|---|---|---|
| Qwen3.8-27B-FP8 | baseline | 78 | 800 | 1.09 |
| Qwen3.8-27B-FP8 | MTP, 3 proposed tokens | 175 (+124%) | 1,199 (+50%) | 0.94 |
| Qwen3.8-27B-FP8 | MTP (3) + FP8 KV cache | 163 (+110%) | 1,007 (+26%) | 1.20 |
| Qwen3.8-27B-FP8 | FP8 KV cache | 76 (−2%) | 684 (−14%) | 1.46 |
| Qwen3.5-9B | baseline | 43 | 475 | 0.95 |
| Qwen3.5-9B | MTP, 2 proposed tokens | 75 (+77%) | 753 (+59%) | 0.74 |
| Qwen3.5-9B | FP8 KV cache | 43 (+0%) | 467 (−2%) | 1.42 |
shared_prefix workload), MTP-3 raised throughput by 75% but also raised the p99 time to first token from 0.53 s to 0.79 s. The FP8 KV cache raised it to 1.47 s while also lowering throughput, so it was worse on both measures.Detailed evaluation
Throughput
Output tokens per second, with the change from baseline in parentheses. shared_prefix sends many requests that start with the same long prompt; agent_sessions sends multi-turn conversations.
| Qwen3.8-27B-FP8 (H100) | shared_prefix c=1 | shared_prefix c=16 | agent_sessions c=1 | agent_sessions c=16 |
|---|---|---|---|---|
| baseline | 79 | 743 | 78 | 800 |
| kv-fp8 | 76 (−3%) | 612 (−18%) | 76 (−2%) | 684 (−14%) |
| mtp-3 | 176 (+123%) | 1,297 (+75%) | 175 (+124%) | 1,199 (+50%) |
| mtp-3 + kv-fp8 | 167 (+111%) | 1,018 (+37%) | 163 (+110%) | 1,007 (+26%) |
| Qwen3.5-9B (L40S) | shared_prefix c=1 | shared_prefix c=16 | agent_sessions c=1 | agent_sessions c=16 |
|---|---|---|---|---|
| baseline | 43 | 420 | 43 | 475 |
| kv-fp8 | 43 (−0%) | 399 (−5%) | 43 (+0%) | 467 (−2%) |
| mtp-1 | 61 (+41%) | 499 (+19%) | 61 (+44%) | 617 (+30%) |
| mtp-2 | 71 (+65%) | 644 (+53%) | 75 (+77%) | 753 (+59%) |
Tail latency
p99 time to first token, in seconds. With 24 requests per measurement, the p99 equals the slowest request.
| Qwen3.8-27B-FP8 | shared_prefix c=1 | shared_prefix c=16 | agent_sessions c=1 | agent_sessions c=16 |
|---|---|---|---|---|
| baseline | 0.35 | 0.53 | 0.27 | 1.09 |
| kv-fp8 | 0.36 | 1.47 | 0.27 | 1.46 |
| mtp-3 | 0.36 | 0.79 | 0.28 | 0.94 |
| mtp-3 + kv-fp8 | 0.37 | 1.51 | 0.29 | 1.20 |
| Qwen3.5-9B | shared_prefix c=1 | shared_prefix c=16 | agent_sessions c=1 | agent_sessions c=16 |
|---|---|---|---|---|
| baseline | 0.53 | 0.69 | 0.41 | 0.95 |
| kv-fp8 | 0.58 | 1.39 | 0.43 | 1.42 |
| mtp-1 | 0.54 | 1.54 | 0.46 | 0.95 |
| mtp-2 | 0.57 | 0.84 | 0.45 | 0.74 |
Answer quality
In principle, speculative decoding does not change what the model outputs, because every proposed token is verified by the full model. An implementation can still differ from the theory, so I checked. I deployed the default and the MTP setup of each model as live servers and asked both the same 200 grade-school math questions (a fixed random sample of GSM8K), with the sampling temperature set to 0 and the model's extended “thinking” mode turned off. An answer counts as correct only if its final number exactly matches the reference.
| model | baseline | MTP | difference |
|---|---|---|---|
| Qwen3.8-27B-FP8 (MTP-3) | 0.940 | 0.935 | −1 question |
| Qwen3.5-9B (MTP-2) | 0.900 | 0.910 | +2 questions |
The differences are one and two questions out of 200, which is within normal sampling noise. I did not run this check for the FP8 KV cache.
Method
Models and hardware
Qwen3.8-27B-FP8 (a 27-billion-parameter model, 31 GB, Apache-2.0 licence) on one H100 GPU with 80 GB of memory, and Qwen3.5-9B (9 billion parameters, 19 GB, Apache-2.0) on one L40S GPU with 48 GB. Each model runs on its own GPU in a separate container. Software: vLLM 0.33.0, PyTorch 2.14, CUDA 13.0.
The baseline
The engine's default settings, plus only what is needed to start the model and a fixed context length of 16,384 tokens. The 9B model needs --trust-remote-code. The 27B model needs --max-num-seqs 512 (explained below).
Settings tested
Each test changes exactly one setting from the baseline: speculative decoding with k proposed tokens, the FP8 KV cache, scheduler limits, asynchronous scheduling, or turning prefix caching off. I then combine the best ones.
Workloads
shared_prefix: a 3,000-token system prompt shared by all requests, a 150-token question, and up to 256 output tokens. agent_sessions: six-turn conversations with a 2,000-token shared system prompt, where each turn resends the full history. Prompts are random words, so no request is answered from cache by accident.
The load generator runs inside the same container as the server, so no network delay is included. Every measurement is preceded by warm-up requests drawn with a different random seed. I test 1 and 16 concurrent requests. shared_prefix sends 24 requests per measurement. agent_sessions sends 24 requests at c = 1 and 96 at c = 16. Throughput is the number of output tokens divided by the elapsed time.
A startup failure on a hybrid-attention model
Qwen3.8-27B uses a cheaper “linear” form of attention in 48 of its 64 layers. Those layers keep a fixed-size state for every active sequence, so the number of sequences an engine can hold is limited by that state as well as by the KV cache. With vLLM 0.33.0’s default settings on one H100, the 27B model failed to start. The engine stopped while preparing its optimised execution graphs, with this error:
ValueError: max_num_seqs (1024) exceeds available Mamba cache blocks (745).
Each decode sequence requires one Mamba cache block, so CUDA graph capture
cannot proceed. Please lower max_num_seqs to at most 745 or increase
gpu_memory_utilization.
The default limit of 1,024 concurrent sequences is higher than the 745 state slots that fit in memory next to the model. Setting --max-num-seqs 512 fixes it, so I include it in the baseline. In this report, “baseline” therefore means the defaults plus whatever is needed to start.
More findings
The FP8 KV cache: not recommended at this scale
Storing the KV cache in 8-bit precision lowered throughput (by 14% on the 27B model with 16 requests at once) and raised the p99 time to first token from 0.53 s to 1.47 s on shared_prefix. Its purpose is to free memory, but at these model sizes memory is not the limiting factor, and the extra conversion work shows up as lost speed. It is not part of the recommended setup.
Agent harness pilot
I also asked whether a carefully engineered prompt for the coding agent (the “harness”), combined with the tuned engine, makes agents more accurate. I compared it with the default agent on the default engine, using Terminal-Bench 2.1 (command-line tasks, Qwen3.8-27B) and τ²-bench (customer-service conversations, Qwen3.5-9B). At the sample sizes I could afford, there was no reliable difference.
| task set (same tasks in both arms; valid outcomes only) | tasks | default | engineered | note |
|---|---|---|---|---|
| τ²-bench airline, test | 19 | 0.84 | 0.84 | tie; uncertainty about ±0.21 |
| τ²-bench retail, test | 5 | 1.00 | 0.80 | one task worse |
| Terminal-Bench, first 10 test tasks | 10 | 0.30 | 0.40 | one task better, none worse |
| Terminal-Bench, final round | 8 | 2 solved | 2 solved | tie; the rest hit sandbox errors |
| Terminal-Bench, dev tasks | 5 | 0.20 | 0.80 | not used for tuning; not confirmed on the test tasks |
Evaluation pipeline fixes
- Missing tool-calling settings. My first Terminal-Bench pilot scored 0 on every task because the server rejected the agent’s tool definitions with an HTTP 400 error, so no tokens were ever generated. I withdrew that pilot and enabled tool calling on every server.
- τ²-bench judge defaults to OpenAI. Some τ²-bench tasks are graded by a language model. By default it calls an OpenAI model, which failed without a key and scored the task 0 in both arms. The grader now uses my default 27B server in every run.
- Sandbox image builds. The older container-image builder failed on plain Ubuntu task images (
python: not found); selecting its newer version fixed it. Later sandbox conflict errors came from starting 30 sandboxes at once. Those trials are excluded and are not counted as model failures.
Reproduce
Everything is driven by a small, engine-agnostic configuration layer: a model registry, a set of named serving profiles (the baseline and one-change variants), a workload generator, a load-testing client and a quality check. Any machine with a supported GPU and vLLM can run it. The repository README lists the exact commands for each step:
- Stage the model weights and verify that every candidate setting is accepted by the installed vLLM version.
- Sweep the profiles and workloads at the chosen concurrency levels, with the accuracy check enabled.
- Serve the baseline or the tuned (MTP) profile behind an authenticated, OpenAI-compatible endpoint.
- Rebuild the tables, figure and paper directly from the raw result files with
paper/make_results.py.
A deployed endpoint keeps its GPU allocated until it is stopped, so shut it down when a run is finished.
Acknowledgements
This work builds on vLLM and the Qwen model family. The agent-harness pilot used Terminal-Bench 2.1 (run with Harbor) and τ²-bench.
Appendix: how multi-token prediction works
1. The head proposes, the model verifies
Plain decoding runs the full model once per generated token. Speculative decoding adds a cheap proposer that guesses the next k tokens; the full model then checks all k guesses in a single forward pass. With MTP, the proposer is a small head attached to the model. In the general MTP design (published for DeepSeek-V3; I have not inspected the exact Qwen layer layout), the head for the j-th future token combines the model’s hidden state with the embedding of the token just proposed, passes them through one light transformer block, and reuses the model’s own output layer:
qj = softmax( LMhead( norm(h′j) ) ), x̂j = argmax qjproposal j = 1…k; E and LMhead are shared with the main model, so the head is small
The model authors train the head with an extra loss for each future position, added to the usual next-token loss, where λ weights the extra terms:
2. Verification keeps the output identical
Let p be the full model’s distribution at a position and q the proposer’s. A proposed token x is accepted with probability min(1, p(x)/q(x)). At the first rejection, the token is replaced by a sample from the leftover distribution, and the remaining proposals are discarded:
The emitted token then follows exactly p, the full model’s own distribution:
So speculative decoding changes the speed, not the distribution of outputs. If all k proposals are accepted, the same verification pass also yields one extra token for free, so a step produces between 1 and k+1 tokens.
3. Expected tokens per step and the speedup
Let α be the probability that a proposal is accepted, assumed equal at every position. A step produces j tokens when the first j−1 proposals are accepted and the next is not, so the expected number of tokens per step is a geometric sum. Let c be the cost of one proposal step relative to one full-model step. Because verifying k+1 positions costs about one full-model step when decoding is limited by memory bandwidth, the speedup is:
Speedup S(k) = E[tokens per step] / (1 + c·k)upper limit as k grows: 1 / (1 − α)
Two forces set the best k. Each extra proposal adds a term αk to the numerator, which shrinks geometrically, while it adds c to the denominator. The speedup therefore rises, peaks and then falls. For illustration, with α = 0.6 and c = 0.1, S(1) = 1.46, S(2) = 1.63, S(3) = 1.67, S(4) = 1.65 and S(6) = 1.52, so the best value is 3. This is a worked example of the shape, not a measurement. It is why each model has a best k and why I compared 1 and 2 on the 9B.
4. Checking the theory against my measurements
My measured speedups bound the acceptance rate. Since S = E / (1 + c·k) ≤ E, setting E equal to the measured speedup and solving for α gives a lower bound on the acceptance rate. This assumes equal acceptance at every position, which is an approximation.
| model, k | requests at once | measured speedup | max possible (k+1) | implied acceptance rate α ≥ |
|---|---|---|---|---|
| Qwen3.8-27B, k = 3 | 1 | 2.24× | 4 | 0.62 |
| Qwen3.8-27B, k = 3 | 16 | 1.50× | 4 | 0.34 |
| Qwen3.5-9B, k = 1 | 1 | 1.44× | 2 | 0.44 |
| Qwen3.5-9B, k = 2 | 1 | 1.77× | 3 | 0.51 |
| Qwen3.5-9B, k = 2 | 16 | 1.59× | 3 | 0.42 |
Measured speedups come from the agent_sessions workload. Two observations are consistent with the model. First, the speedup is below the cap of k+1 in every row, as it must be. Second, the speedup is smaller with 16 requests at once than with one. Batching makes the full-model step more compute-bound, so verifying extra tokens costs more and the effective c rises. I did not measure α or c directly, so this explanation is a consistent reading of the numbers and not a tested claim.