AI Engineering Practice
5 in the last 7 days (+25% vs. prior week)Articles
32A Reranker Is a Second Opinion on Retrieval: When the Extra Pass Is Worth Its Latency
First-stage retrieval is built for recall at speed. A reranking pass trades milliseconds for precision, and whether that trade pays off is something you can measure.
Embeddings Are Versioned Data: Planning for the Day You Change the Embedding Model
A vector index is only meaningful relative to the model that produced it. Treat the model, dimensions, and preprocessing as part of the schema.
Rewrite the Question Before You Retrieve: Query Rewriting for RAG That Doesn't Change What the User Meant
Users ask follow-ups, use pronouns, and write in fragments. A small rewriting step before retrieval can fix recall, if you constrain it and test it on its own.
Parallel Tool Calls Need a Failure Policy: What to Do When Three of Five Calls Succeed
Models can request several tools at once. Your orchestrator has to decide, ahead of time, what a half-successful batch means.
Handoffs Between Agents Are Contracts: What to Pass, What to Leave Behind
When one agent delegates to another, the handoff message is the only context the receiver has. Treat it as an interface, not a chat message.
Agent Memory Needs a Retention Policy: Deciding What to Persist, What to Expire, and What to Never Store
Giving an agent memory across sessions is easy. Keeping that memory correct, scoped, and safe over months is the real engineering problem.
A Pinned Model Will Be Retired: Treat Model Deprecation as a Migration Project
Every hosted model version has an end date. Teams that plan for it early spend an afternoon on it; teams that don't spend a quarter.
Calibrate Your LLM Judge Against Human Labels Before You Trust Its Scores
A judge's score is a measurement, so validate it: build a small human-labeled anchor set, compare error types, and report agreement alongside every number.
Canary Queries for LLM Features: Cheap Always-On Probes That Catch Quality Drift
A small fixed set of mechanically checkable requests, run continuously against the live path, tells you the feature is still good, not just still up.
Slice Your Eval Results: An Average Score Hides the Users Who Got Worse
A single pass rate can rise while a whole class of inputs quietly regresses. Tag every eval case and report by slice.
Trajectory Tests for Agents: Assert Invariants, Not Exact Steps
Test outcomes first, then a few invariants about the path. Anthropic warns that scripting exact tool-call sequences makes agent tests brittle.
Your Few-Shot Examples Are Probably Leaking Into Your Eval Set
When the examples in a prompt and the cases in a test suite come from the same pool, your score measures memorization, not capability.
Deadline Propagation: Budgeting Time Across Every Stage of a Multi-Call LLM Pipeline
A per-call timeout tells you nothing about whether the whole request will finish in time. Pass one deadline down the pipeline and let each stage spend only what is left.
Client-Side Rate Limiting for LLM APIs: Token Buckets, Backpressure, and Fair Sharing
Waiting for a 429 and retrying is a recovery strategy, not a capacity plan. Here is how to stay under provider limits on purpose.
Measuring Over-Refusal: Testing the Requests Your Model Declines That It Shouldn't
Most safety evals check that the model refuses bad requests. Fewer check that it still helps with legitimate ones that merely look risky.
Truncated Output Is a Silent Failure: Handling Stop Reasons and Length Limits
A response cut off at the token limit still returns HTTP 200. Here is how to read stop reasons, handle truncation, and track your truncation rate.
Tool Results Are a View, Not a Dump: Truncation and Pagination for Agent Tools
Large tool outputs bury the details an agent needs. Design the result shape, cap it in code, and make every omission visible to the model.
Streaming LLM Responses Is a UX Problem: Designing Around Time-to-First-Token and Mid-Stream Failure
Turning on streaming makes a feature feel faster, but it also creates new states and failure modes that need deliberate design.
Log Enough to Replay a Bad LLM Response, Not Just to Know It Happened
Most LLM logging records that a call occurred. Debugging requires recording enough to reproduce it. Here is the minimum replay record and how to keep it safe.
Feature-Flagging Prompts: Rolling Back a Bad System Prompt in Seconds, Not a Deploy
A one-line prompt edit can change model behavior as much as a real code change, and usually ships with a fraction of the review — and none of the rollback speed.
Signal
No signals yet
Signals related to this topic will appear here as new intelligence is published.