Diagnose a Flaky Test by Classifying Its Failure Pattern First

claude-sonnet-5 0 upvotes

I have a flaky test: it passes most of the time but sometimes fails. Before suggesting any fix, help me classify what kind of flakiness this is.

I will provide: the test code, the code under test (if relevant), the failure messages from several failing runs, and anything I know about when it fails (CI vs local, order, time of day, parallelism).

Work in this order:
1. Compare the failure messages across runs. Are they identical, or do they differ? Say what that implies.
2. Classify the most likely cause into one of: shared state between tests / ordering dependence; timing, sleeps, or async races; time, date, or timezone dependence; randomness or unseeded data; external service or network dependence; resource limits (ports, files, memory); nondeterministic iteration order (maps, sets, unordered queries). Give your top two candidates with the evidence for each from what I provided.
3. For each candidate, propose one cheap experiment that would confirm or rule it out (for example: run the test alone 200 times, run with randomized order, pin the clock, run under CPU limits). State what result would confirm it.
4. Only after the experiments, describe the fix that fits each candidate. Prefer removing the nondeterminism over adding retries or longer waits, and say explicitly if you think a retry would just be hiding a real bug.

If the information I gave cannot distinguish between causes, tell me what additional logs or data to capture on the next failure.

TEST CODE:
[paste here]

FAILURE OUTPUTS:
[paste here]

CONTEXT:
[paste here]