Developer Tools
Sandboxing Model-Generated Code: Limits on Resources, Network, and Filesystem Before You Execute Anything
If your agent can run code it wrote itself, the sandbox is the real security boundary. Here is what to constrain, and why a prompt instruction is not a control.
Record and Replay LLM Calls: Offline Tests That Don't Burn Tokens or Flake
A recorded-response fixture layer lets most of your test suite run without a network, a bill, or nondeterminism. The trick is knowing what to record, how to key it, and when to throw recordings away.
Write Tool Errors for the Model: Error Messages an Agent Can Actually Recover From
A tool that returns 'Error: failed' leaves the agent guessing. A good error message names the problem, the cause, and the next valid move.
Schema Design for LLM Output: Field Order, Enums, and Honest Escape Values
Your JSON schema is part of the prompt. How you order fields, define enums, and mark optionality changes both validity and content quality.
Canary Queries for LLM Features: Cheap Always-On Probes That Catch Quality Drift
A small fixed set of mechanically checkable requests, run continuously against the live path, tells you the feature is still good, not just still up.
Golden-File Tests for Rendered Prompts: Catching Template Changes Before the Model Does
Your prompt is assembled from code, and the code can change it without anyone noticing. Snapshotting the rendered text makes those changes visible in review.
Versioning Stored LLM Output: Schema Stamps, Upcasters, and Deliberate Backfills
Generated records outlive the prompts that wrote them. Stamp versions at write time, upcast on read, and backfill only where it pays.
'Not Stated' Is an Answer: Designing Extraction Prompts That Don't Invent Missing Fields
Models asked to fill a schema will fill it. Give them a legitimate way to say the value is absent, and design the downstream code to treat that as data.
Build a Minimal Eval Harness in an Afternoon: Cases, a Runner, and Diffable Results
A useful eval harness is three small parts: JSONL cases, a boring runner and diffable results. Built on Anthropic's principles for developing tests.
Tokenization Surprises: Why Character Counts, Whitespace, and Numbers Break Your Assumptions
String length is a poor proxy for tokens, and tokenizers change between models. Here is how to count correctly and avoid silent budget overruns.
Dynamic Few-Shot Selection: Choosing Examples Per Request Instead of Per Prompt
A fixed set of examples is a guess about every future input. Selecting examples at request time lets the prompt adapt, but it brings its own failure modes.
Write a Prompt Linter: Cheap Static Checks Before Any Model Call
A handful of deterministic checks on placeholders, size, and format promises can catch real prompt defects in CI before they reach users.
Evaluate RAG in Two Halves: Measuring Retrieval Recall Before You Judge the Answers
An end-to-end score says something is wrong; a split evaluation says which half to fix.
Feature-Flagging Prompts: Rolling Back a Bad System Prompt in Seconds, Not a Deploy
A one-line prompt edit can change model behavior as much as a real code change, and usually ships with a fraction of the review — and none of the rollback speed.
Building a Human Review Queue for AI Outputs: Triage, Escalation, and Closing the Feedback Loop
A review queue that just lists outputs for a human to approve or reject is a bottleneck waiting to happen. The queues that hold up under volume are designed around triage, escalation, and turning every decision back into signal.
Building a Shared Prompt Library Without It Turning Into Tribal Knowledge
Once more than one person is writing prompts for the same product, copy-paste drift sets in fast. A few concrete conventions keep a prompt library usable instead of becoming a graveyard of near-duplicate files.
Shadow-Mode Deployments for LLM Features: Validating a New Pipeline Without Showing It to Anyone
Before a new prompt, model, or retrieval pipeline ever reaches a real user, it can run in parallel with production and get judged against real traffic. Here's how to build that harness without it turning into a second production system you have to maintain forever.
Hybrid Search Beats Pure Vector Search More Often Than Vendors Admit
Embeddings miss exact matches by design. Here is how keyword plus vector search with rank fusion and reranking fills the gap, and how to tell if you need it.
A Practical Walkthrough: Adding Semantic Caching to an LLM Endpoint
Exact-match caching barely helps LLM traffic since real users rarely repeat a query verbatim. Here is how to build a semantic cache that saves money without silently serving the wrong answer.
Tool Overload: When More Functions Make an Agent Worse, Not Better
Every tool you add competes for the same attention budget the model uses to pick the right one — past a certain roster size, that competition starts costing you accuracy.
Concurrency Limits for Agent Tool Calls: What Happens When an Agent Fans Out Faster Than Your Downstream Systems Can Take
When an agent decides on its own to call five tools at once, something downstream has to absorb that burst. Most teams don't design for it until it's an outage.
Designing a Multi-Provider Model Router: Fallback Chains, Not Just a Single Switch
Routing between model providers for cost or redundancy sounds like a load-balancer problem. The failure modes say otherwise.
Structured Output, Compared: JSON Mode, Function Calling, and Constrained Decoding
Three ways to force an LLM into a schema look similar from the outside. They fail differently, and that difference should drive which one you pick.
The Hidden Cost of Retries: Why "Just Retry the LLM Call" Isn't Free
Retries add load, multiply across layers, and re-bill tokens. Here is a retry policy built on AWS guidance and adapted to LLM calls with side effects.
How Prompt Caching Actually Works, and Why Your Hit Rate Is Lower Than You Think
Prompt caching only pays off if your prompt is built for it. Here is how it works, what breaks it, and how to measure your real hit rate.
Chunk Size Isn't the Whole Story: A Practical Guide to Splitting Documents for RAG
Chunk size is the least important chunking choice. Split on structure, give each chunk its context, and test retrieval recall before tuning the knob.
Batch APIs and Async Processing: When Not to Call the Model in Real Time
Much LLM work has no human waiting. Here is how batch APIs cut cost by half, what limits and outcomes to plan for, and how to audit your own call sites.
Embedding Drift: What Actually Breaks When You Upgrade Your Embedding Model
Embeddings from different models cannot be compared, so upgrading in place quietly breaks search. Here is how to migrate with a versioned, parallel index.
Parsing Partial JSON From a Streaming LLM Response Without Guessing
Streaming tokens as they arrive breaks the standard JSON parser — here's how to validate a structured response incrementally instead of waiting for the whole thing or writing a fragile regex patch.
Mocking LLM Calls Without Lying to Your Test Suite
Unit tests that stub an LLM response usually end up testing the stub, not the system — here is how to keep them honest.
Designing Tool Error Messages an Agent Can Actually Act On
A tool error is the next input to the model. Here is what Anthropic recommends for actionable errors, plus a small schema that makes failures decidable.
Multi-Agent Systems: When Splitting Work Across Agents Actually Helps
Splitting work across agents can win big, at several times the token cost. Here's what Anthropic's own multi-agent system shows about when it pays off.
Redact at the Edge: Keeping Raw PII Out of LLM Prompts and Logs Without Losing the Ability to Debug
Blanket redaction makes a feature undebuggable; no redaction makes it a compliance liability. The fix is redacting consistently at one enforced boundary, not as an opt-in step engineers remember on a good day.
What to Look for in an LLM Observability Tool Before You Buy One
Pick LLM observability tooling by the failures you will actually have: trace completeness, per-step cost, stop reasons, version linkage, replay and privacy.
Model Context Protocol (MCP): A Universal Adapter for AI Tool Integration
MCP is an open standard for connecting AI applications to tools and data. Here are the roles, the features, and the security rules in the current specification.
Agentic Coding Assistants: What They Are and Why Developers Are Adopting Them
Agentic coding assistants plan, edit, run and revise in a loop. Here is why that works, where it fails, and how to get reliable results.