Skip to main content

Test triage, a ShirePath lab study

Is this test worth keeping?

16 tests from an audit of 17 codebases, each rewritten with new names so it keeps its shape and loses its owner. Answer the five questions, pick a verdict, then see what the audit said.

These verdicts come from the LLM judge. I planned to hand-audit 100 of them and stopped after a few rows: most of these tests were written by AI, and at that volume I could not tell their value apart any better than the judge could. So read them as the judge's calls, with no accuracy figure behind them.

From a league-management SaaS

it("rejects a cutoff date before the stored close date", async () => {
  getSeason.mockResolvedValue({ id: "s1", closesOn: "2026-08-31", cutoffOn: null });

  const result = await updateSeason("s1", { cutoffOn: "2026-08-15" });

  expect(result).toEqual({ ok: false, error: "Cutoff cannot be before registration closes" });
  expect(saveSeason).not.toHaveBeenCalled();
});
The five questions

1Does it protect a behavior a user or caller relies on?

2Is there a credible change that would turn it red?

3Does another test already catch that failure?

4Does it need production code that exists only for tests?

5Did bugs ship in this repo in code no test reaches?

Your verdict

Protects a contract, or kills a mutant nothing else kills.

The retention bar

Some tests stay whatever their smells: auth and authorization, money, security, storage and migrations, API contracts, and data isolation. A test there gets merged or fixed, never just deleted.