Find the First Bad Change in a Software Release

A release ships on Tuesday. On Wednesday a customer reports that invoices crossing midnight sometimes appear twice. Reverting the release may contain the damage, but it does not establish which change caused it. First turn “sometimes” into a repeatable observation.

Make the failure small enough to run twice

Capture the input, timezone, relevant configuration and expected result. Use synthetic data to build a failing case. In this example, process one invoice at a timezone boundary and count the resulting records. Run the same case on a known good revision as well as the current one. If it fails on both, your assumed good boundary is wrong.

Preserve dependency versions, database fixtures and the clock where possible. A test that depends on today's date or a remote service may change its answer as you move between commits.

Bisect only what the test can distinguish

In a disposable checkout, start with git bisect start, mark the failing revision bad and a verified good revision good. Git proposes a midpoint. Run the reproduction and mark that revision good or bad. If a revision cannot be built or the result cannot be interpreted, skip it rather than inventing an answer. Finish with git bisect reset.

Binary search is efficient when the tested property has a usable good-to-bad boundary. A flaky test, a revert followed by a reintroduction or incompatible schema versions can complicate the result. Keep the bisect log so a colleague can reproduce the sequence.

Test the explanation

The first bad commit may expose a pre-existing bug rather than contain the underlying defect. Read the diff and design a narrower test: does changing the timezone conversion alone alter the duplicate count? Add a regression test covering both sides of midnight. Verify it fails for the relevant reason before the fix and passes after it. That evidence remains useful long after the release has been forgotten.

Further reading

Git: bisect

Related guide: Use Git Worktrees to Review a Patch Without Losing Your Place.