Software Engineering
Reasoning Budgets: When Extra Thinking Tokens Pay Off and When They Only Add Latency
Letting a model think longer before answering can help on some tasks and do nothing on others. Match the budget to the task, and verify it with your own eval.
Where You Put Things in a Long Prompt: Placement of Documents, Instructions, and the Question
With large inputs, the order of the pieces is a prompt-design decision you can test, not a formatting detail.
Embeddings Are Versioned Data: Planning for the Day You Change the Embedding Model
A vector index is only meaningful relative to the model that produced it. Treat the model, dimensions, and preprocessing as part of the schema.
Temperature Zero Is Not Determinism: What You Can and Can't Reproduce in LLM Calls
Setting temperature to zero narrows output variation but doesn't guarantee identical responses. Here is where the remaining variation comes from and how to build around it.
Chunk Retrieval Documents by Structure First, Then Tune the Size With a Measurement
Fixed 500-token windows are a fine baseline and a poor ending point. Splitting along the document's own structure and checking the result against real questions usually beats any amount of overlap tuning.
Record and Replay LLM Calls: Offline Tests That Don't Burn Tokens or Flake
A recorded-response fixture layer lets most of your test suite run without a network, a bill, or nondeterminism. The trick is knowing what to record, how to key it, and when to throw recordings away.
Agent Memory Needs a Retention Policy: Deciding What to Persist, What to Expire, and What to Never Store
Giving an agent memory across sessions is easy. Keeping that memory correct, scoped, and safe over months is the real engineering problem.
Order Your Prompt for the Cache: Stable Prefix First, Volatile Content Last
Provider-side prompt caching only pays off if the front of your prompt is byte-identical across requests. A few layout rules decide whether you get the discount or silently never hit it.
A Pinned Model Will Be Retired: Treat Model Deprecation as a Migration Project
Every hosted model version has an end date. Teams that plan for it early spend an afternoon on it; teams that don't spend a quarter.
Delimiters Are Cheap Insurance: Separating Instructions, Data, and Examples in a Prompt
A prompt that mixes your instructions with the text it should process invites misreadings. Consistent section markers fix most of them for almost no cost.
Schema Design for LLM Output: Field Order, Enums, and Honest Escape Values
Your JSON schema is part of the prompt. How you order fields, define enums, and mark optionality changes both validity and content quality.
Calibrate Your LLM Judge Against Human Labels Before You Trust Its Scores
A judge's score is a measurement, so validate it: build a small human-labeled anchor set, compare error types, and report agreement alongside every number.
Majority Voting for LLM Answers: When Sampling Several Times Beats Rewriting the Prompt
Self-consistency reduces random errors, and the vote split is a useful routing signal, but it cannot fix a prompt that is wrong the same way every time.
Send the Model Less: Data Minimization and Reversible Redaction for LLM Prompts
Every field you put in a prompt is a field you must secure, log, and explain. Pseudonymize before the call, restore after it, and keep the real values out of the blast radius.
Bisecting a Failing Prompt: Shrink It Until the Bug Has Nowhere to Hide
When a 2,000-token prompt misbehaves, reading it harder rarely helps. Delta debugging does: remove pieces until you find the smallest prompt that still fails.
When Your Prompt Gives Two Instructions That Collide: Write the Precedence Rule
Long prompts accumulate rules that conflict in edge cases. If you don't state which one wins, the model picks, and it won't pick consistently.
Slice Your Eval Results: An Average Score Hides the Users Who Got Worse
A single pass rate can rise while a whole class of inputs quietly regresses. Tag every eval case and report by slice.
Evaluating Summaries by Claims, Not Vibes: Separating Faithfulness From Coverage
A single quality score hides whether a summary is wrong or merely incomplete. Decompose the check into claims and key points instead.
Golden-File Tests for Rendered Prompts: Catching Template Changes Before the Model Does
Your prompt is assembled from code, and the code can change it without anyone noticing. Snapshotting the rendered text makes those changes visible in review.
Versioning Stored LLM Output: Schema Stamps, Upcasters, and Deliberate Backfills
Generated records outlive the prompts that wrote them. Stamp versions at write time, upcast on read, and backfill only where it pays.
Pairwise Judging Needs Order Swaps: Controlling Position Bias in Prompt Comparisons
Pairwise judging is more sensitive than absolute scores, but only if you control for position bias. Run each pair in both orders and report the flip rate.
Trajectory Tests for Agents: Assert Invariants, Not Exact Steps
Test outcomes first, then a few invariants about the path. Anthropic warns that scripting exact tool-call sequences makes agent tests brittle.
Idempotency Keys for Agent Tool Calls: Making Retries and Replays Safe
Agents retry, resume and repeat themselves. Here is how idempotency keys, modeled on payment APIs, keep side-effecting tool calls safe.
Your Few-Shot Examples Are Probably Leaking Into Your Eval Set
When the examples in a prompt and the cases in a test suite come from the same pool, your score measures memorization, not capability.
Build a Minimal Eval Harness in an Afternoon: Cases, a Runner, and Diffable Results
A useful eval harness is three small parts: JSONL cases, a boring runner and diffable results. Built on Anthropic's principles for developing tests.
Deadline Propagation: Budgeting Time Across Every Stage of a Multi-Call LLM Pipeline
A per-call timeout tells you nothing about whether the whole request will finish in time. Pass one deadline down the pipeline and let each stage spend only what is left.
Checking Citations Mechanically: Verifying That a RAG Answer's Quotes Really Exist in the Source
If your model quotes its sources, you can verify the quotes with ordinary string matching, which catches a whole class of fabrication for almost no cost.
Client-Side Rate Limiting for LLM APIs: Token Buckets, Backpressure, and Fair Sharing
Waiting for a 429 and retrying is a recovery strategy, not a capacity plan. Here is how to stay under provider limits on purpose.
Measuring Over-Refusal: Testing the Requests Your Model Declines That It Shouldn't
Most safety evals check that the model refuses bad requests. Fewer check that it still helps with legitimate ones that merely look risky.
Tokenization Surprises: Why Character Counts, Whitespace, and Numbers Break Your Assumptions
String length is a poor proxy for tokens, and tokenizers change between models. Here is how to count correctly and avoid silent budget overruns.
Tool Results Are a View, Not a Dump: Truncation and Pagination for Agent Tools
Large tool outputs bury the details an agent needs. Design the result shape, cap it in code, and make every omission visible to the model.
Evaluate RAG in Two Halves: Measuring Retrieval Recall Before You Judge the Answers
An end-to-end score says something is wrong; a split evaluation says which half to fix.
Streaming LLM Responses Is a UX Problem: Designing Around Time-to-First-Token and Mid-Stream Failure
Turning on streaming makes a feature feel faster, but it also creates new states and failure modes that need deliberate design.
Log Enough to Replay a Bad LLM Response, Not Just to Know It Happened
Most LLM logging records that a call occurred. Debugging requires recording enough to reproduce it. Here is the minimum replay record and how to keep it safe.
Building a Human Review Queue for AI Outputs: Triage, Escalation, and Closing the Feedback Loop
A review queue that just lists outputs for a human to approve or reject is a bottleneck waiting to happen. The queues that hold up under volume are designed around triage, escalation, and turning every decision back into signal.
Cost Attribution for LLM Features: Tracking Spend by Team, Feature, and User Without Guesswork
A monthly bill that's triple the estimate is a governance failure, not a pricing surprise — the fix is tagging every call with who asked for it before the invoice arrives, not after.
Building a Shared Prompt Library Without It Turning Into Tribal Knowledge
Once more than one person is writing prompts for the same product, copy-paste drift sets in fast. A few concrete conventions keep a prompt library usable instead of becoming a graveyard of near-duplicate files.
Shadow-Mode Deployments for LLM Features: Validating a New Pipeline Without Showing It to Anyone
Before a new prompt, model, or retrieval pipeline ever reaches a real user, it can run in parallel with production and get judged against real traffic. Here's how to build that harness without it turning into a second production system you have to maintain forever.
Context Budgeting: A Practical Framework for Deciding What Actually Fits in the Window
A bigger context window just moves the problem. Here is a concrete way to allocate tokens across system instructions, retrieved evidence, conversation history, and tool output before you run out of room.
Turning Thumbs-Up/Thumbs-Down Signals Into Better Prompts and Evals, Without Chasing Noise
A feedback button is easy to ship and easy to misuse. Here is how to turn raw clicks into labeled data that actually improves the feature, instead of a vanity metric.
Hybrid Search Beats Pure Vector Search More Often Than Vendors Admit
Embeddings miss exact matches by design. Here is how keyword plus vector search with rank fusion and reranking fills the gap, and how to tell if you need it.
A Practical Walkthrough: Adding Semantic Caching to an LLM Endpoint
Exact-match caching barely helps LLM traffic since real users rarely repeat a query verbatim. Here is how to build a semantic cache that saves money without silently serving the wrong answer.
Your LLM-as-Judge Eval Is Only as Good as Its Rubric
A vague "rate this response 1 to 10" prompt turns your judge model into a coin flip with extra steps. Here is how to build a rubric that actually holds up.
Debugging Nondeterminism in LLM Outputs: Temperature Is Rarely the Only Culprit
Setting temperature to zero doesn't buy you a deterministic system — it buys you one less variable in a pipeline that still has several.
Tool Overload: When More Functions Make an Agent Worse, Not Better
Every tool you add competes for the same attention budget the model uses to pick the right one — past a certain roster size, that competition starts costing you accuracy.
Prompt Templates Have an Injection Problem Too: Treat User Data Like Untrusted Input, Not Just Text
Most prompt injection advice focuses on malicious instructions hidden in retrieved content. There's a quieter, more common version of the same bug sitting in your own template code.
Structured Output, Compared: JSON Mode, Function Calling, and Constrained Decoding
Three ways to force an LLM into a schema look similar from the outside. They fail differently, and that difference should drive which one you pick.
The Hidden Cost of Retries: Why "Just Retry the LLM Call" Isn't Free
Retries add load, multiply across layers, and re-bill tokens. Here is a retry policy built on AWS guidance and adapted to LLM calls with side effects.
When to Use a State Machine Instead of a Prompt for Agent Control Flow
Deterministic transitions belong in code, not in prose. Here is how Anthropic's workflow patterns help you decide what the model should and shouldn't control.
How Prompt Caching Actually Works, and Why Your Hit Rate Is Lower Than You Think
Prompt caching only pays off if your prompt is built for it. Here is how it works, what breaks it, and how to measure your real hit rate.
Chunk Size Isn't the Whole Story: A Practical Guide to Splitting Documents for RAG
Chunk size is the least important chunking choice. Split on structure, give each chunk its context, and test retrieval recall before tuning the knob.
Handling PDFs, Images, and Other Multimodal Inputs in an LLM Pipeline
Piping raw files straight into a vision-capable model works in a demo and falls apart in production. Separating extraction from understanding — and caching the extraction — is what makes multimodal features debuggable.
Batch APIs and Async Processing: When Not to Call the Model in Real Time
Much LLM work has no human waiting. Here is how batch APIs cut cost by half, what limits and outcomes to plan for, and how to audit your own call sites.
Embedding Drift: What Actually Breaks When You Upgrade Your Embedding Model
Embeddings from different models cannot be compared, so upgrading in place quietly breaks search. Here is how to migrate with a versioned, parallel index.
Lost in the Middle, Found in Testing: Why Context Document Order Changes Your RAG Answers
Where you put a retrieved passage in the prompt can change the answer. Here is what the research found and a cheap test to run on your own pipeline.
Designing the Handoff Contract Between Agents: What Actually Has to Transfer When Work Changes Hands
Splitting work across agents is easy to diagram and hard to make reliable — most of the reliability problems live in the handoff, not the split.
Parsing Partial JSON From a Streaming LLM Response Without Guessing
Streaming tokens as they arrive breaks the standard JSON parser — here's how to validate a structured response incrementally instead of waiting for the whole thing or writing a fragile regex patch.
Designing Agent Memory: What to Persist, What to Forget, and How to Retrieve It Later
Most agent memory systems fail not because retrieval is weak, but because nobody decided what belongs in memory in the first place.
Mocking LLM Calls Without Lying to Your Test Suite
Unit tests that stub an LLM response usually end up testing the stub, not the system — here is how to keep them honest.
Where You Put an Instruction Matters As Much As What It Says
Long-context prompts don't treat every line equally — models attend more to the start and end of a context window than the middle, and that asymmetry has real engineering consequences.
Silent Model Upgrades Are a Production Risk — Here's How to Guard Against Them
Models have a lifecycle that ends in retirement, and replacements change behavior. Here is how to pin, log, test and canary your model versions.
Designing Tool Error Messages an Agent Can Actually Act On
A tool error is the next input to the model. Here is what Anthropic recommends for actionable errors, plus a small schema that makes failures decidable.
Multi-Agent Systems: When Splitting Work Across Agents Actually Helps
Splitting work across agents can win big, at several times the token cost. Here's what Anthropic's own multi-agent system shows about when it pays off.
Redact at the Edge: Keeping Raw PII Out of LLM Prompts and Logs Without Losing the Ability to Debug
Blanket redaction makes a feature undebuggable; no redaction makes it a compliance liability. The fix is redacting consistently at one enforced boundary, not as an opt-in step engineers remember on a good day.
Treat Agent Tool Access Like IAM, Not Like a Feature Flag
Most teams gate agent tools with a boolean and a system-prompt instruction. The teams that avoid expensive mistakes scope permissions the way they would for any service account: narrowly, explicitly, and revocably.
The Case for Synthetic Adversarial Personas in Your Eval Suite
Most eval sets are built from happy-path examples and yesterday's bugs. Neither one tells you how the system fails against a user trying to break it.
A Framework for Deciding Where an LLM Call Belongs in Your Request Path
Sync, streaming, or background job — the placement decision shapes your product more than the prompt does.
What to Cache — and What Never to Cache — in an LLM-Backed Feature
Caching feels like a free win for any expensive model call, but applied carelessly it turns a probabilistic system into a source of silent, confidently wrong answers.
Agentic Coding Assistants: What They Are and Why Developers Are Adopting Them
Agentic coding assistants plan, edit, run and revise in a loop. Here is why that works, where it fails, and how to get reliable results.