Large Language Models
Chunk Retrieval Documents by Structure First, Then Tune the Size With a Measurement
Fixed 500-token windows are a fine baseline and a poor ending point. Splitting along the document's own structure and checking the result against real questions usually beats any amount of overlap tuning.
Record and Replay LLM Calls: Offline Tests That Don't Burn Tokens or Flake
A recorded-response fixture layer lets most of your test suite run without a network, a bill, or nondeterminism. The trick is knowing what to record, how to key it, and when to throw recordings away.
Order Your Prompt for the Cache: Stable Prefix First, Volatile Content Last
Provider-side prompt caching only pays off if the front of your prompt is byte-identical across requests. A few layout rules decide whether you get the discount or silently never hit it.
A Pinned Model Will Be Retired: Treat Model Deprecation as a Migration Project
Every hosted model version has an end date. Teams that plan for it early spend an afternoon on it; teams that don't spend a quarter.
Delimiters Are Cheap Insurance: Separating Instructions, Data, and Examples in a Prompt
A prompt that mixes your instructions with the text it should process invites misreadings. Consistent section markers fix most of them for almost no cost.
Schema Design for LLM Output: Field Order, Enums, and Honest Escape Values
Your JSON schema is part of the prompt. How you order fields, define enums, and mark optionality changes both validity and content quality.
Calibrate Your LLM Judge Against Human Labels Before You Trust Its Scores
A judge's score is a measurement, so validate it: build a small human-labeled anchor set, compare error types, and report agreement alongside every number.
Majority Voting for LLM Answers: When Sampling Several Times Beats Rewriting the Prompt
Self-consistency reduces random errors, and the vote split is a useful routing signal, but it cannot fix a prompt that is wrong the same way every time.
Canary Queries for LLM Features: Cheap Always-On Probes That Catch Quality Drift
A small fixed set of mechanically checkable requests, run continuously against the live path, tells you the feature is still good, not just still up.
Bisecting a Failing Prompt: Shrink It Until the Bug Has Nowhere to Hide
When a 2,000-token prompt misbehaves, reading it harder rarely helps. Delta debugging does: remove pieces until you find the smallest prompt that still fails.
When Your Prompt Gives Two Instructions That Collide: Write the Precedence Rule
Long prompts accumulate rules that conflict in edge cases. If you don't state which one wins, the model picks, and it won't pick consistently.
Slice Your Eval Results: An Average Score Hides the Users Who Got Worse
A single pass rate can rise while a whole class of inputs quietly regresses. Tag every eval case and report by slice.
Evaluating Summaries by Claims, Not Vibes: Separating Faithfulness From Coverage
A single quality score hides whether a summary is wrong or merely incomplete. Decompose the check into claims and key points instead.
Pairwise Judging Needs Order Swaps: Controlling Position Bias in Prompt Comparisons
Pairwise judging is more sensitive than absolute scores, but only if you control for position bias. Run each pair in both orders and report the flip rate.
'Not Stated' Is an Answer: Designing Extraction Prompts That Don't Invent Missing Fields
Models asked to fill a schema will fill it. Give them a legitimate way to say the value is absent, and design the downstream code to treat that as data.
Your Few-Shot Examples Are Probably Leaking Into Your Eval Set
When the examples in a prompt and the cases in a test suite come from the same pool, your score measures memorization, not capability.
Checking Citations Mechanically: Verifying That a RAG Answer's Quotes Really Exist in the Source
If your model quotes its sources, you can verify the quotes with ordinary string matching, which catches a whole class of fabrication for almost no cost.
Client-Side Rate Limiting for LLM APIs: Token Buckets, Backpressure, and Fair Sharing
Waiting for a 429 and retrying is a recovery strategy, not a capacity plan. Here is how to stay under provider limits on purpose.
Dynamic Few-Shot Selection: Choosing Examples Per Request Instead of Per Prompt
A fixed set of examples is a guess about every future input. Selecting examples at request time lets the prompt adapt, but it brings its own failure modes.
Truncated Output Is a Silent Failure: Handling Stop Reasons and Length Limits
A response cut off at the token limit still returns HTTP 200. Here is how to read stop reasons, handle truncation, and track your truncation rate.
Write a Prompt Linter: Cheap Static Checks Before Any Model Call
A handful of deterministic checks on placeholders, size, and format promises can catch real prompt defects in CI before they reach users.
Evaluate RAG in Two Halves: Measuring Retrieval Recall Before You Judge the Answers
An end-to-end score says something is wrong; a split evaluation says which half to fix.
Streaming LLM Responses Is a UX Problem: Designing Around Time-to-First-Token and Mid-Stream Failure
Turning on streaming makes a feature feel faster, but it also creates new states and failure modes that need deliberate design.
Negative Examples in Prompts: When Showing the Model What Not to Do Backfires
Telling a model what to avoid can anchor it on exactly that behavior. When negative examples help, when they hurt, and what to write instead.
Log Enough to Replay a Bad LLM Response, Not Just to Know It Happened
Most LLM logging records that a call occurred. Debugging requires recording enough to reproduce it. Here is the minimum replay record and how to keep it safe.
Feature-Flagging Prompts: Rolling Back a Bad System Prompt in Seconds, Not a Deploy
A one-line prompt edit can change model behavior as much as a real code change, and usually ships with a fraction of the review — and none of the rollback speed.
Cost Attribution for LLM Features: Tracking Spend by Team, Feature, and User Without Guesswork
A monthly bill that's triple the estimate is a governance failure, not a pricing surprise — the fix is tagging every call with who asked for it before the invoice arrives, not after.
Context Budgeting: A Practical Framework for Deciding What Actually Fits in the Window
A bigger context window just moves the problem. Here is a concrete way to allocate tokens across system instructions, retrieved evidence, conversation history, and tool output before you run out of room.
Turning Thumbs-Up/Thumbs-Down Signals Into Better Prompts and Evals, Without Chasing Noise
A feedback button is easy to ship and easy to misuse. Here is how to turn raw clicks into labeled data that actually improves the feature, instead of a vanity metric.
Hybrid Search Beats Pure Vector Search More Often Than Vendors Admit
Embeddings miss exact matches by design. Here is how keyword plus vector search with rank fusion and reranking fills the gap, and how to tell if you need it.
A Practical Walkthrough: Adding Semantic Caching to an LLM Endpoint
Exact-match caching barely helps LLM traffic since real users rarely repeat a query verbatim. Here is how to build a semantic cache that saves money without silently serving the wrong answer.
Your LLM-as-Judge Eval Is Only as Good as Its Rubric
A vague "rate this response 1 to 10" prompt turns your judge model into a coin flip with extra steps. Here is how to build a rubric that actually holds up.
Debugging Nondeterminism in LLM Outputs: Temperature Is Rarely the Only Culprit
Setting temperature to zero doesn't buy you a deterministic system — it buys you one less variable in a pipeline that still has several.
Prompt Templates Have an Injection Problem Too: Treat User Data Like Untrusted Input, Not Just Text
Most prompt injection advice focuses on malicious instructions hidden in retrieved content. There's a quieter, more common version of the same bug sitting in your own template code.
Designing a Multi-Provider Model Router: Fallback Chains, Not Just a Single Switch
Routing between model providers for cost or redundancy sounds like a load-balancer problem. The failure modes say otherwise.
Structured Output, Compared: JSON Mode, Function Calling, and Constrained Decoding
Three ways to force an LLM into a schema look similar from the outside. They fail differently, and that difference should drive which one you pick.
How Prompt Caching Actually Works, and Why Your Hit Rate Is Lower Than You Think
Prompt caching only pays off if your prompt is built for it. Here is how it works, what breaks it, and how to measure your real hit rate.
Chunk Size Isn't the Whole Story: A Practical Guide to Splitting Documents for RAG
Chunk size is the least important chunking choice. Split on structure, give each chunk its context, and test retrieval recall before tuning the knob.
Handling PDFs, Images, and Other Multimodal Inputs in an LLM Pipeline
Piping raw files straight into a vision-capable model works in a demo and falls apart in production. Separating extraction from understanding — and caching the extraction — is what makes multimodal features debuggable.
Batch APIs and Async Processing: When Not to Call the Model in Real Time
Much LLM work has no human waiting. Here is how batch APIs cut cost by half, what limits and outcomes to plan for, and how to audit your own call sites.
Why Prompts That Work in English Quietly Fail in Other Languages
A prompt validated only on English test cases carries assumptions about tokenization, formatting, and register that don't automatically travel — and the failures rarely look like failures.
Embedding Drift: What Actually Breaks When You Upgrade Your Embedding Model
Embeddings from different models cannot be compared, so upgrading in place quietly breaks search. Here is how to migrate with a versioned, parallel index.
Lost in the Middle, Found in Testing: Why Context Document Order Changes Your RAG Answers
Where you put a retrieved passage in the prompt can change the answer. Here is what the research found and a cheap test to run on your own pipeline.
Parsing Partial JSON From a Streaming LLM Response Without Guessing
Streaming tokens as they arrive breaks the standard JSON parser — here's how to validate a structured response incrementally instead of waiting for the whole thing or writing a fragile regex patch.
Mocking LLM Calls Without Lying to Your Test Suite
Unit tests that stub an LLM response usually end up testing the stub, not the system — here is how to keep them honest.
Where You Put an Instruction Matters As Much As What It Says
Long-context prompts don't treat every line equally — models attend more to the start and end of a context window than the middle, and that asymmetry has real engineering consequences.
Silent Model Upgrades Are a Production Risk — Here's How to Guard Against Them
Models have a lifecycle that ends in retirement, and replacements change behavior. Here is how to pin, log, test and canary your model versions.
Multi-Turn State: What Breaks When Your AI Feature Moves From Single-Shot to Conversational
The API remembers nothing, so your code owns the conversation. Here is what accumulates, what breaks, and what to decide on purpose.
Prompt Injection Isn't Solved by a Better System Prompt: Defense Patterns for Tool-Using Agents
A system prompt is a suggestion, not a security boundary. Here is what OWASP recommends and how to design agents so an injection is survivable.
The Case for Synthetic Adversarial Personas in Your Eval Suite
Most eval sets are built from happy-path examples and yesterday's bugs. Neither one tells you how the system fails against a user trying to break it.
A Practical Rubric for Choosing Between Few-Shot Prompting and Fine-Tuning
Prompting, retrieval and fine-tuning fix different problems. Here is OpenAI's diagnostic framework plus rules of thumb for choosing between few-shot and fine-tuning.
What to Cache — and What Never to Cache — in an LLM-Backed Feature
Caching feels like a free win for any expensive model call, but applied carelessly it turns a probabilistic system into a source of silent, confidently wrong answers.
Google's Flash Sprint Can't Mask a Flagship Stuck in the Shop
Google shipped Gemini 3.7 Flash just weeks after its predecessor while Gemini 3.5 Pro's release drags on, underscoring a brutal, price-driven AI arms race.