Insights / 2026
Engineering notes for an agentic world.
Notes on agents, evaluation, MCP, security, inference and production architecture, from the engineers who build them.
Articles
- LLM inference19 min read
Constrained Decoding Models: What CDMs Are and What They Are Not
CDM now means two different things. One is constrained decoding, a mature technique that masks the model's output tokens so the result always matches your schema. The other is a product category launched in October 2026 selling task specific models under the same acronym, with a contested claim to being a new architecture. This guide separates them, explains how token masking actually works, covers the engines and the real flags, and walks through the research finding that matters most: constrained decoding guarantees form, not correctness.
Read - LLM inference33 min read
DeepInfer Deep Dive: Inside DeepSeek's Production Inference Engine
On October 20, 2026 DeepSeek open sourced the engine that serves its own API, and it looks nothing like vLLM. There is no tensor parallelism, the scheduler is C++, an n-gram memory layer gets its own parallel dimension, speculative decoding picks its draft length per request, and Huawei Ascend is supported from day one. Here is how it is built, what each feature does and which flag controls it, what the benchmarks do and do not show, and how it compares with vLLM, SGLang, and TensorRT-LLM.
Read - Ai agents18 min read
Spec Driven Development in the Agentic Era: What Works and What Does Not
Specs went from documentation nobody read to the primary input for coding agents, and a whole category of tooling appeared to support them: GitHub Spec Kit, AWS Kiro, steering files, AGENTS.md. But the honest reporting from teams who tried it is mixed, with real findings of ten times slower runs and documents that drift out of sync. This is a look at what spec driven development actually buys you, when the overhead pays back, when plan mode beats it, and how to tell the difference before you commit a team to a methodology.
Read - LLM inference24 min read
How to Optimize KV Cache
KV cache is the real constraint on almost every LLM deployment. It decides how many users you can serve at once, how long your context can get, and how much each token costs. This guide works through the memory math, then goes layer by layer through every lever you actually have: paging and sizing, prefix caching, FP8 quantization, CPU and NVMe offloading, and cross instance cache pools, with the real flags and the metrics that tell you which layer is your bottleneck.
Read - LLM inference22 min read
Token Economics: What Really Drives Your Cost Per Token
The same model on the same GPU can cost you 36 cents or 12 dollars per million tokens. The hardware is identical. What changes is throughput, utilization, workload shape, and how much work you avoid repeating. This guide breaks down every factor that moves your cost per token, with the formula, worked numbers, and the full set of metrics you should be tracking to see where the money actually goes.
Read - MCP13 min read
Hiding a Tool From an AI Agent Doesn't Secure It: Notes on Zero-Trust MCP
A PayPal paper tested four frontier models against 60 prompt-injection payloads on an MCP server. Filtering the tool list stopped every attempt, but models still named the hidden tool in up to 94% of runs, and a scripted client could call it directly. What the paper built, what the numbers show, and what to change on your own MCP servers.
Read - AI infrastructure15 min read
AX by Google: The Agent Executor That Helps You Scale AI Agents
Agents are neither stateless microservices nor batch jobs, which is exactly why Kubernetes struggles with them. AX is Google's open source, declarative control plane for running agent workloads at scale: sandboxed tasks, pre-wired workspaces, centrally managed models, and suspend and resume that checkpoints an idle agent instead of paying for it. This guide covers what it is, how the architecture fits together, and a full worked example from prerequisites to a running, suspended, and resumed task.
Read - AI security11 min read
Plugin4Shell: What We Found When We Checked Our Own Stack
A zero-click flaw broke SHA pinning in four major AI coding agents at once. We ran the actual check against our own environment. Here's how it works and what we found.
Read - What is8 min read
TypeSafe Jev Explained: Fast, Type Safe AI Decisions for Automation
LLMs are great at chat, but software doesn't need prose, it needs decisions it can branch on. This piece breaks down TypeSafe AI's System One models and Jev: what problem they solve, how typed Choice, Score, and Noul questions replace fragile JSON parsing, why calibrated confidence matters for automation, and how to build a practical daily triage harness in Python on top of it.
Read - AI infrastructure24 min read
Understanding llm-d and HAMi: The Kubernetes Stack for Private AI Infrastructure
Running your own LLM is easy until you have multiple models, GPUs, workloads, and users fighting for the same infrastructure. This piece breaks down what llm-d and HAMi actually solve, how their architectures work, what changes when you introduce them into Kubernetes, and where they fit in a modern private AI stack.
Read - DevSecOps57 min read
DevSecOps for AI Agents on Kubernetes and Cloud
A compromised web app might expose a database. A compromised AI agent can reach cloud APIs, internal services, secrets, documents and deployment systems in the same session. This is the long-form guide to DevSecOps for AI agents: what traditional DevSecOps gets right, what changes when the workload decides what to do next, and how to design identity, authorization, isolation, supply chain, runtime security, observability and multi-tenancy on Kubernetes and cloud so the blast radius stays small.
Read - AI safety7 min read
What Is Hugging Face's Open Alignment Initiative?
When an unreleased OpenAI model breached Hugging Face's own infrastructure this summer, a debate that used to live inside a handful of frontier labs suddenly became everyone's problem. This piece breaks down what Hugging Face's new Open Alignment Initiative actually is, why cofounder Thomas Wolf is leading it, how it connects to Dario Amodei's push for embedded evaluators, and what it could mean for anyone building on open models.
Read - Document AI12 min read
What is Doclang?
PDF was built for print. DOCX was built for editors. Markdown was built for readers. None of them were built for models, which is exactly why your AI pipeline keeps hallucinating facts that were sitting right there in the source document. This piece breaks down why document format is the real bottleneck behind bad AI accuracy, and how DocLang, an open, AI-native document standard backed by IBM, NVIDIA, Red Hat, ABBYY, and the Linux Foundation, fixes it at the source.
Read - RAG13 min read
Enterprise RAG in Production: Architecture, Multi-Agent Systems, and What Actually Works in 2026
There's a gap between a RAG prototype that answers questions about your docs and a RAG system thousands of employees rely on daily, one that respects access controls, doesn't hallucinate into a compliance incident, and that agents can actually reason with. This is what closes that gap: hybrid retrieval, permission enforcement, multi-agent orchestration, and the failure modes that show up once real users hit the system.
Read - MCP29 min read
How to Build a Good MCP Tool: Lessons From Running 50+ Servers in Production
Most MCP servers converge on the same shape within minutes of writing one. What separates the good ones isn't the shape, it's what happens inside it: schemas that actually teach a model how to call them, an auth pattern matched to what's being protected, a real internal/external error boundary, and SSRF and SQL-injection guards that don't rely on a blocklist alone. This is the field guide drawn from running and reviewing a fleet of fifty-plus production MCP servers, including the two real bugs that taught more than any tutorial could.
Read - Agent memory14 min read
Agent Memory in Production
AI agents are only as useful as their ability to remember. In production, agent memory goes beyond storing chat history—it requires the right architecture for retaining, retrieving, and managing context at scale. This guide explains how AI agent memory works, the different types of memory, and how to build a reliable, scalable memory system for production AI applications.
Read - Document processing8 min read
Success Without Correctness: Why Your Document Pipeline Needs to Distrust Its Own Libraries
The most dangerous bug in a document pipeline isn't the one that crashes, it's the one that returns successfully, looks fine in the logs, and quietly hands you garbage. A real chain of three silent failures in a Docling and Tesseract OCR pipeline, stacked one on top of the other, each one hiding behind the last, and what it teaches about trusting a library's happy path.
Read - AI agents7 min read
AI Agents Don't Just Need Tools. They Need Knowledge They Can Trust
Open Knowledge Format v0.2 is Google's attempt to make organizational knowledge portable, machine-readable, and trustworthy for AI agents.
Read - LLM9 min read
Six Billion Requests Later: What a Full Year of LLM Serving Actually Looks Like
A new preprint analyzes a full year of unsampled production LLM traffic, 6.12 billion requests across 9,174 models, and the findings reshape how you should think about caching, routing, and capacity planning.
Read