Constrained Decoding Models: What CDMs Are and What They Are Not

Before anything else, a terminology warning, because searching for CDM right now returns two unrelated things and conflating them will cost you a bad architecture decision.

Constrained decoding is a technique that has been in production since roughly 2023. At every decoding step, you compute which tokens could possibly come next without violating a grammar, mask the logits of everything else to negative infinity, and sample from what remains. The model physically cannot emit output that breaks your schema. This is mature, measurable, and shipping inside vLLM, SGLang, TensorRT-LLM, and llama.cpp today.

Constrained Decoding Models, the product category, arrived on October 13, 2026, when a startup called ScaleDown launched a family of task specific models under that name and marketed them as "a new AI model architecture" that skips autoregressive generation entirely, claiming 200 times faster and 467 times cheaper than frontier models. Within three days there was a detailed rebuttal arguing that this is packaging rather than architecture.

Both of those are worth understanding. One is a technique you should probably already be using. The other is a category with a real use case and an overstated origin story. This post covers the mechanism first, because that is the durable knowledge, then the engines and the actual flags, then the research finding that matters more than any vendor benchmark, then an honest read on the product category.

How Constrained Decoding Actually Works

A language model's final layer produces a logit for every token in the vocabulary. Normally you sample from that distribution. Constrained decoding inserts one step before sampling: given everything generated so far, a grammar engine computes a bitmask of tokens that would keep the output valid, and everything else is zeroed out.

Two consequences fall straight out of this design, and both matter.

The guarantee is absolute, not statistical. This is not prompting the model to please return JSON. A token that would break the schema has probability zero. No retry loop, no parse failure handling, no "the model returned markdown fences again" bug.

The guarantee covers syntax only. The mask knows what shape is legal. It has no idea what is true. A schema requiring {"sentiment": "positive" | "negative"} guarantees you get one of those two words. It does not guarantee the right one. Hold that thought, because the most important research result in this area is about exactly that gap.

There is a third, subtler mechanism worth knowing: fast forwarding. When the grammar permits exactly one continuation, for instance the ":" after a field name in JSON, the engine can emit those tokens directly without a forward pass. On schema heavy output that is a real speedup, not just a correctness feature, because you skip model calls entirely for the deterministic parts.

The Hard Part: Tokens Do Not Match Grammars

The reason this is an engineering problem rather than a weekend project is that grammars are defined over characters and models emit subword tokens. A single token might be "age": including the quotes and colon, or it might be ag plus e. The engine has to answer, for every token in a 128,000 token vocabulary, at every step, whether that specific byte sequence keeps the parse valid.

Doing that naively is far too slow. The engines differ almost entirely in how they make it fast.

A few things to pull out of that table.

Outlines trades compile time for runtime speed. Once the transition table exists, lookups are O(1). But that table costs seconds of CPU to build and tens to hundreds of megabytes to hold. That is fine if you reuse a handful of schemas forever and cache them. It is bad in a multi tenant service where callers send arbitrary schemas, because you pay compilation on the TTFT of the first request for every new schema.

XGrammar is the default in most serving stacks for the reason the table shows: low compile cost and masking that overlaps with GPU work, so the overhead largely disappears into time the GPU was going to spend anyway.

The bitmask representation is a real optimization, not a detail. For a Llama 3 vocabulary at 128 concurrent requests and 50 steps per second, naive float masks move around 6.55 GB/s of mask data. Bitsets cut that to roughly 204.8 MB/s, a 32 times reduction. On a bandwidth bound decode phase, that difference is the difference between usable and not.

Using It: The Real Flags

vLLM exposes five constraint types through a structured_outputs field. Note the API changed: the old guided_json, guided_regex, guided_choice, guided_grammar, and guided_decoding_backend fields were removed in v0.12.0, so code written against tutorials from last year will break.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
model = client.models.list().data[0].id
 
# 1. choice: output is exactly one of these
client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}],
    extra_body={"structured_outputs": {"choice": ["positive", "negative"]}},
)
 
# 2. regex: output matches the pattern
client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Generate an email for Alan Turing at Enigma."}],
    extra_body={"structured_outputs": {"regex": r"\w+@\w+\.com\n"}, "stop": ["\n"]},
)

For JSON, the cleanest path is a Pydantic model into response_format, which is also the most portable across providers:

from pydantic import BaseModel
from enum import Enum
 
class CarType(str, Enum):
    sedan = "sedan"
    suv = "SUV"
    truck = "Truck"
 
class CarDescription(BaseModel):
    brand: str
    model: str
    car_type: CarType
 
client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Describe the most iconic car of the 90s."}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "car-description", "schema": CarDescription.model_json_schema()},
    },
)

And for anything genuinely structured, such as a restricted SQL dialect, an EBNF grammar:

simplified_sql_grammar = """
    root ::= select_statement
    select_statement ::= "SELECT " column " from " table " where " condition
    column ::= "col_1 " | "col_2 "
    table ::= "table_1 " | "table_2 "
    condition ::= column "= " number
    number ::= "1 " | "2 "
"""
 
client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Show username and email from users."}],
    extra_body={"structured_outputs": {"grammar": simplified_sql_grammar}},
)

Backend selection is a server flag, defaulting to auto, which picks per request based on what the request needs:

vllm serve <model> --structured-outputs-config.backend auto
# or pin it: xgrammar, guidance, outlines, lm-format-enforcer

Two gotchas that will cost you a day each

Reasoning models need explicit enabling. Structured outputs work alongside reasoning, but with some models the constraint gets silently disabled if reasoning content is not parsed into a separate field. The vLLM docs call this out specifically for Qwen3 Coder with reasoning enabled from v0.11.2 onward, and the fix is a flag:

vllm serve <model> --reasoning-parser deepseek_r1 \
  --structured-outputs-config.enable_in_reasoning=True

Custom grammars and parser boundary tokens do not mix naively. Some reasoning and tool call parsers identify protocol boundaries by token ID, not by decoded text. A text only grammar can emit ordinary vocabulary tokens that spell </think> without being the dedicated </think> token, and the parser will not recognize the boundary. The symptom is maddening: a tool invocation stays stuck in the reasoning field and never produces tool_calls. With XGrammar you fix it by using token level structural tag elements:

{"type": "const_string", "value": "</think>"}

becomes

{"type": "token", "token": "</think>"}

Both decode to the same text, but only the second requires the dedicated token ID. Resolve token strings against the same tokenizer your model uses, and never copy numeric token IDs from another model.

The Finding That Matters Most

Here is where most content on this topic stops, having established that constrained decoding gives you valid JSON. A study published in September 2026 asked the obvious follow up question, and the answer should change how you use this.

The setup: five dense models across three families, Qwen3-0.6B, Llama-3.2-1B, Llama-3.2-3B, Phi-4-mini, and Qwen3-4B, across 14 structured output tasks, under three conditions each, native decoding, Outlines, and XGrammar. That is 210 evaluations, scored on two separate axes: schema validity and content accuracy.

On structure, constrained decoding is a total win. Native schema validity ranged from 78.6 percent for the Llama models up to 92.9 percent for Qwen-0.6B and Phi-4-mini, with only Qwen-4B already at 100 percent. Both constrained decoding frameworks took every model to 100 percent schema validity. No retraining, no prompt engineering.

The overhead numbers from the same study are also worth having: XGrammar cost 4 to 8 ms of compilation and reduced throughput by 1.6 to 3.7 percent. Outlines averaged 2.0 to 4.9 seconds of compilation per model, peaking at 19.5 seconds, and ran 9 to 13 percent slower on four of five models. That is a strong empirical case for XGrammar as the default.

On content, the picture is more interesting. Content accuracy by model and condition:

Constrained decoding improves content accuracy substantially, which is not obvious a priori. The mechanism is that many apparent content errors were actually type errors. On a receipt extraction task, content accuracy went from 0.000 to 1.000 under constraint, because natively the models emitted prices as strings or with currency prefixes. The data was right, the type was wrong, and the mask fixed it.

But look at the scaling question the paper was really asking. Constrained Llama-1B scores 0.986, which beats native Llama-3B at 0.777. So constraint on a small model beats no constraint on a model three times larger. Tempting conclusion: constrain a tiny model and save the money. Except constrained Llama-3B reaches 0.991. The gap narrows and does not close. Qwen-0.6B stays the weakest model even under XGrammar.

And then there is the failure mode that should worry you. The paper documents two cases that no schema can catch:

Instruction semantic failure. On a task requiring two function calls, Qwen-0.6B produced only one under every decoder. The schema permitted that, so constrained decoding delivered valid, well formed, incomplete output.

Hollow rescue. Llama-1B under XGrammar scored a perfect 1.000 on a parameter presence metric for an email task, while the actual email body was the string "1. 2. 3. 4. 5.". Every required field was present and correctly typed. The content was meaningless.

The paper's one line summary is the thing to remember: constrained decoding's reach ends exactly where schema conformance ends. Its practical recommendation is XGrammar with a 3B to 4B model as the cheapest reliable configuration for structured output work.

What this means operationally: constrained decoding eliminates a class of bugs you can detect and replaces none of the bugs you cannot. A parse failure is loud. A valid JSON object containing "1. 2. 3. 4. 5." passes every check in your pipeline and ships. If you adopt constrained decoding and do not add semantic validation, you have made your failures quieter rather than rarer.

Now, the Product Category

With the mechanism understood, the October 2026 launch is easier to assess fairly.

ScaleDown shipped four single task models, each accepting up to 1 million input tokens, sold as metered endpoints at roughly $0.04 to $0.05 per million input tokens:

• Classify, which assigns a label from a caller defined list and adds a one sentence reason
• Extract, which pulls structured fields as spans of the input
• Summarize, which condenses content, output capped at 4,000 tokens on OpenRouter
• Compress, which shrinks content such as tool results

Extract, Summarize, and Classify also accept images and scanned PDFs. The headline claim was 200 times faster and 467 times cheaper than frontier models on these narrow tasks, with per task figures of 41 times cheaper and 2.7 times faster for Summarize, and 138 times cheaper and 1.8 times faster for Compress.

The rebuttal, published October 16, makes several points that hold up on inspection:

The "new architecture" claim does not survive contact with their own docs. Per ScaleDown's documentation, Classify is a scoring head over fixed labels and Extract is span finding with character offsets. Those are well established approaches, closer to classical classification and named entity recognition than to anything new. And Summarize is abstractive, meaning it does generate text, so "skips autoregressive generation entirely" is not true for the whole family.

Open libraries already deliver the structural guarantee. The September 2026 study above took five open weight models from 78.6 to 100 percent schema validity using Outlines and XGrammar, with no retraining and no new model. If the value proposition is "well formed output," that is available for free on models you already run.

Vendor benchmarks deserve the usual scrutiny. The benchmark datasets named, CLINC OOS, CUAD, QMSum, and FinanceBench, are public and that transparency is above average. But baselines and sample sizes are not fully reproducible from the docs. Against the open weight GLiNER2.5-multi-v1, released under Apache 2.0 in August 2026, ScaleDown's own page claims a 15.5 point lead on classification and a 0.2 point lead on extraction. A 0.2 point difference is a tie, and no configuration is published for the comparison run. The rebuttal also notes the headline "+10.5 percent" average accuracy figure does not reconcile with the per task numbers, where the mean of +15.5 and +0.2 is 7.85.

One caveat on the rebuttal itself: it was published by a vendor selling self hosted model agnostic infrastructure, so it has a commercial interest in the "own your stack" conclusion. Read both with that in mind.

What is genuinely new here

Strip out the architecture claim and something real remains, which is worth separating from the marketing.

A category of task specific decision models is forming, and it has a shared API. OpenRouter now runs a Decisions API at POST /api/alpha/decisions that is a different shape from chat completions entirely. You send a state, which can be a string, object, or array, plus a questions object where each question has a type and instructions. Four types are supported: noul for true or false, choice for one option from a set, score against a rubric, and box for image bounding boxes. Responses come back keyed by question name with the typed answer plus optional confidence and probabilities.

TypeSafe's Jev was the first model on that endpoint, and ScaleDown's Classify-1 is served there too and is request compatible with it. That compatibility is the interesting signal: two independent vendors converging on one interface for "ask a model a typed question and get a typed answer with calibrated confidence." An interface that multiple vendors implement is a more durable development than any individual model.

The economics on narrow tasks are real even if the architecture claim is not. If 80 percent of your agent's token spend is compressing tool results and classifying inputs, and a purpose built endpoint does that at $0.04 per million input tokens, that is a genuine cost lever regardless of what the model is called internally. The langchain-scaledown middleware for Deep Agents, shipped the same day under MIT, includes history summarization, tool result compression, and structured extraction, with a documented example compressing a docs page trace from 18,402 to 3,110 characters. Notably it fails open: if the call fails, history is left untouched and the agent keeps running, which is the right default.

Choosing Between the Options

The decision rule I would apply: use constrained decoding always, and task specific models when the task is genuinely narrow and the volume justifies evaluating one.

Constrained decoding is nearly free and removes a bug class. There is no good reason to parse free text JSON in 2026.

Task specific models are an evaluation exercise, not a purchase decision. The thing to measure is not the vendor's benchmark, it is your documents against three options: a frontier model with a schema constraint, a small open weight specialist such as a GLiNER class extractor that you host, and the hosted endpoint. On narrow tasks at high volume, the specialist often wins on cost by a wide margin, and sometimes the open weight one wins outright. The 0.2 point extraction gap in that vendor comparison is the clue: on some tasks the open model is already level.

Metrics to Track

If you deploy constrained decoding, these are the numbers that tell you whether it is working.

The pairing that matters is the first two. Schema validity at 100 percent with no semantic metric is a dashboard that will tell you everything is fine while shipping nonsense.

Pitfalls

• Treating valid as correct. The central error. A schema is a type check, not a truth check.
• Forcing the schema at token zero when the model needs to reason. Give it a free form reasoning field first, or a prefix outside the constraint, then constrain the answer.
• Unbounded string fields. An open ended string in a schema invites degenerate loops. Bound it with a length limit or a regex.
• Recompiling grammars per request. With Outlines especially, this lands seconds of CPU on your TTFT. Cache aggressively.
• Using Outlines for arbitrary caller supplied schemas. Its compile cost model assumes reuse.
• Pretty printed JSON. You are paying for whitespace tokens at output prices. Minify.
• Copying token IDs between models. Structural tag token constraints must be resolved against the tokenizer actually in use.
• Assuming code written before vLLM v0.12.0 still works. The guided_* fields were removed, not deprecated with a shim.
• Buying a task specific model on the vendor's benchmark. Evaluate on your documents against an open weight specialist and a constrained frontier model before committing.
• Expecting constraint to substitute for model capability. The scaling gap narrows but does not close. A 0.6B model under perfect constraint is still a 0.6B model.

Conclusion

The technique is unambiguously worth adopting. Constrained decoding takes schema validity to 100 percent across models that natively sit between 78 and 93 percent, costs a few percent of throughput with XGrammar, and deletes an entire category of parse failure and retry logic from your codebase. If you are still regex extracting JSON out of markdown fences, that is the highest return change available to you this week.

The product category is worth evaluating and not worth believing uncritically. Task specific endpoints for classification, extraction, and compression can be dramatically cheaper than frontier models on narrow high volume work, and the convergence of multiple vendors on a shared Decisions API is a genuinely useful development. But "new architecture" is doing a lot of work in that marketing, a scoring head over fixed labels is not new, and the open weight specialists are close enough on some tasks that you owe yourself a bake off before you sign anything.

The one thing to carry out of this post, though, is the research finding. Constrained decoding changes loud failures into quiet ones. It will give you perfectly formed, correctly typed, schema valid output containing "1. 2. 3. 4. 5." and every automated check in your pipeline will pass it. Adopt the technique, and then go build the semantic validation that the technique cannot give you, because that is now the only thing standing between your users and confidently structured nonsense.

References

1. Chavan: Constrained Decoding Eliminates Structural Failures in Small LLMs but Reveals a Scale-Dependent Semantic Gap, September 2026
2. vLLM docs: Structured Outputs
3. XGrammar: Documentation and structural tag token level formats
4. llguidance: guidance-ai/llguidance
5. Outlines: dottxt-ai/outlines
6. LLMs Blog: Constrained decoding and structured output engines in production, comparing XGrammar, llguidance, Outlines and llama.cpp GBNF
7. ibl.ai: Constrained Decoding Models Are Not a New Architecture, October 16, 2026
8. AICoder: ScaleDown ships Constrained Decoding Models with LangChain Deep Agents middleware, October 13, 2026
9. OpenRouter: Submit a Decisions request
10. OpenRouter: Jev documentation, TypeSafe decision model