Flaky Test
A Flaky Test can pass or fail without a code change. Timing, order, shared state, randomness, or outside systems often cause it.
Another example
A browser test fails in parallel because two workers share one account. Unique fixtures and event waiting remove the race.
Main parts
- 01Code or behavior under test
- 02Observable evidence
- 03Coverage, failure, and recovery
Use it when
Record and repeat the failure. If briefly blocked, name an owner and fix the changing cause.
Do not use it when
Avoid blind retries, permanent quarantine, unsupported blame, and treating one green rerun as a fix.
Name used in code
repetition + seed + timing/order isolation Before you ship
Check the environment and prepared test data (fixture). Define a result you can observe, failure and edge cases, isolation, stable inputs, useful error details, and cleanup. Set a time budget, owner, and flaky-test rule.
Outcome: Use Flaky Test to gather repeatable evidence about this behavior or quality risk. User context: A browser test fails in parallel because two workers share one account. Unique fixtures and event waiting remove the race. Validation approach: Flaky Test. Why it fits: Record and repeat the failure. If briefly blocked, name an owner and fix the changing cause. Do not use it when: Avoid blind retries, permanent quarantine, unsupported blame, and treating one green rerun as a fix. Test requirements: Name the system and test environment. Define prepared test data, other inputs, and the result you can observe. Include failure and edge cases, allowed error, isolation, and cleanup. Test operations: Make failures easy to diagnose. Control random or changing inputs. Name an owner. Keep the test fast and simple enough for its release gate. Acceptance criteria: The test fails for the known bug and passes after the fix. It repeats in the target environment, gives useful evidence, and does not expose or change unrelated data. Evidence and limits (evidence boundary): No single official standard controls the whole term. Its meaning may change across teams, platforms, or frameworks. Unknowns to confirm: Release gate, environment, prepared test data, allowed error, time budget, owner, and flaky-test rule.
Check this request
Review the current use of Flaky Test. Definition: A Flaky Test can pass or fail without a code change. Timing, order, shared state, randomness, or outside systems often cause it. Release checks: Check the environment and prepared test data (fixture). Define a result you can observe, failure and edge cases, isolation, stable inputs, useful error details, and cleanup. Set a time budget, owner, and flaky-test rule. Before changing code, report the evidence you found, gaps, severity, and the smallest safe fix.