Design a Failure-Injection Experiment for a Service Dependency Before You Run It in Production

0 upvotes

Help me design a controlled failure-injection experiment for one dependency of a service. I want a plan a team can review and run safely, not a generic chaos-engineering essay.

<system>
Service under test: [name and what it does]
Dependency to disrupt: [e.g. the payments API, a cache, a message broker, an LLM provider]
How the service currently handles that dependency failing: [timeouts, retries, fallbacks, circuit breakers, or "unknown"]
Environment available: [staging, a canary slice of production, etc.]
Observability available: [metrics, logs, traces, alerts]
</system>

Produce a plan with these parts:
1. Hypothesis: one sentence in the form "When X happens, the service will Y, and users will experience Z." If the current behavior is unknown, say so and state the hypothesis as a prediction to confirm or refute.
2. Failure modes to test, ranked by how likely they are in real life: for example added latency, timeouts, error responses, partial responses, and complete unavailability. For each, say how to inject it with the least invasive method.
3. Steady-state signals: the specific metrics that show the service is healthy, with thresholds for "abort now".
4. Blast radius controls: how to limit the experiment (traffic percentage, duration, scope), who needs to be told in advance, and how to stop it immediately.
5. What to look for beyond pass/fail: retry storms, queue buildup, thread or connection exhaustion, misleading error messages, and slow recovery after the dependency returns.
6. Follow-up: how to record findings and turn them into fixes or runbook updates.

Flag any assumption you had to make because my input was incomplete. If the proposed environment makes the experiment unsafe, say so plainly and suggest a safer starting point.