Tests with teeth

01Why this page exists

An agent with no testing guidance writes tests that assert the shape of the code it just wrote, watches them pass on the first run, and reports success. The suite is green and proves nothing. Feeding the agent a corpus of testing wisdom (best-practice sources plus your own taste) fixes this, and this page is my distillation: the rules my agents load whenever they write production code.

Two audiences. Your agent: paste this URL or the text into its context (a skill file, CLAUDE.md, a rules doc) before it writes a feature, a function, or an app. You, the operator: §15 is the list of constraints worth promoting from “instructions the agent reads” into “gates the agent cannot bypass,” and the judgment calls only you can make.

Everything here was either distilled from named sources (cited inline) or paid for with a real incident in my shop, marked measured. Nothing is speculative.

02Start here if you are the agent

Precedence. Your operator’s explicit instructions win, then the repo’s existing conventions and test setup, then this page. This page fills gaps; it never overrides a repo that has already decided how it tests.

The loop, every time you are about to write production code

1. Is this production code? Throwaway spike, generated code, or pure config: announce the skip out loud and move on. Anything else: continue.
2. Find how this repo runs its tests (the package.json test script, Makefile target, CI config) before writing or running anything. §10.
3. Find the requirement the change serves. If you cannot state it, you are about to test an implementation detail you invented. §03.
4. Stop and ask before the first test when the spec is ambiguous, when the work touches auth / payments / tokens / crypto, or when one feature looks like it needs more than ~50 tests (the unit is too big; propose a split). The time to ask is before the first test, not after 30 wrong ones.
5. Write one failing test. Watch it fail for the right reason. Minimum code to green. Full suite green. Refactor on green. Repeat.
6. Before claiming done, run the checklist in §14.

Existing codebases. These rules govern new tests and new code. Do not rewrite a working suite to match this page, and do not refuse to work in a repo that violates it; state the gap and follow the repo. One exception cuts the other way: getting a first test around untested legacy code justifies moves banned for fresh code (private-access tooling, wide characterization tests). The bar for new code stays where this page puts it.

Vocabulary, since this page uses it precisely: a double is any stand-in object; a stub answers queries with canned values; an expectation (mock) asserts a call was made. “Integration test” here means a test proving your code can reach and talk to a real external thing, not “any big test.”

03The iron law, and the cycle

No production code without a failing test in front of you first. Code written before its test gets deleted and rewritten, not adapted (“adapted” is tests-after with extra steps). Skip TDD only for throwaway spikes, generated code, or pure config, and say out loud that you are skipping. A silent skip is a rationalization.

Why this is aimed at agents specifically: the default LLM move is implementation-first with tests as a stamp afterward. That inverts the point. Tests are a design tool. “TDD is a design process, not a testing process.” (Steve Sanderson)

One vertical slice at a time. RED (one failing assertion), verify it fails for the right reason, GREEN (minimum code), full suite green, refactor on green. Repeat. Never all-tests-then-all-code: tests written in bulk test imagined behavior, assert on shape instead of behavior, and go stale before the code exists.

• A test that passes on its first run tested nothing. Delete it and write a sharper one. Watching it fail is the only validation the test itself will ever get; there are no tests for tests.
• The trigger for a new test is a requirement, never “I am about to add a method.” Nobody handed you a spec saying foo() should multiply two numbers; that is an implementation detail you invented, and testing at that altitude produces the suite that breaks on every refactor. (Ian Cooper, “TDD, Where Did It All Go Wrong,” DevTernity 2017)

Proportion. Slice count scales to risk. A one-function pure fix gets 2 to 4 slices and no ceremony. Reserve the full treatment (many slices, an executed mutation, §07) for auth, payments, money, tokens, crypto, concurrency, or shared code with more than one caller. Do not gold-plate a trivial change; do not wave through a dangerous one.

04A bug fix targets the class, not the instance

A bug report is one sample of a defect class. Before editing:

1. Name the exact unit that is wrong.
2. State that unit’s whole job in one sentence.
3. Enumerate every input dimension of that job. For text: case, accents, internal whitespace, leading/trailing whitespace, punctuation, unicode lookalikes, empty. For numbers: sign, zero, rounding, overflow, locale. For collections: zero, one, many, lots (where paging and truncation live), and oops (it throws). (J.B. Rainsberger)
4. Write one probe test per dimension before the fix, and make each pass.

Exclude a dimension only on positive evidence from the spec or the data, never on your own speculation that it probably does not matter. measured The hard no-opt-out wording catches a cheap model’s symptom-patch 3 out of 3 tries; the softer “rule it out with a written reason” version scored 0 to 1 out of 3, because the model enumerates the dimension and then excuses itself from exactly the one it should fix.

And at least one probe must exercise behavior you did not change. A green suite on only the reported case is a stamp, not a proof.

05Where to point the test, and what to assert

Put a boundary you own between your logic and someone else’s framework, database, or network, then test through it. Four speakers reach this from four vocabularies: Cooper’s ports and adapters, Rainsberger’s “ring,” Gary Bernhardt’s “functional core, imperative shell,” Michael Feathers’ separation of concerns.

“The functional core makes all the decisions, and the shell has all the dependencies.” Gary Bernhardt, “Boundaries,” RubyConf 2012

So spend tests accordingly:

• Decision-heavy, dependency-light code (parsers, scorers, rules, calculations): many cheap isolated tests.
• Dependency-heavy, path-light code (adapters, transport, CLI glue): a handful of integration tests. They prove the adapter is reachable and configured. Never test the adapter itself; it is third-party code. (Cooper)

Getting this backwards is the single most common cause of an expensive, slow, brittle suite. And partial adoption is most of the benefit: “80% functional is a heck of a lot better than 0%.” Do not stall trying to purify everything.

What to assert once you are there

Sandi Metz’s grid (“The Magic Tricks of Testing,” RailsConf 2013). Three message origins times two message types, and most of the grid is “do nothing”:

Assert values only on the receiver of an incoming message. If you assert somewhere else, you have duplicated a test that already lives on the receiver, and one implementation change now breaks it in two places.

Never test private methods. A correct test of the public method already exercises the private one. A test that pins a private message, as Metz puts it, “does not make it safe to refactor, it makes it impossible.”

For an outgoing command, assert the send itself, not what it eventually causes. “Reaching across a bunch of intermediate objects and testing a distant side effect is an integration test. It’s just hiding in your unit test directory.” (Metz)

06Doubles, fixtures, and the contract they owe

A test double is permitted only where the collaborator crosses a real boundary: network, filesystem, clock, randomness, subprocess, third-party SDK, another process. Everywhere else use the real object. Expected shape of a correct suite: roughly one double per external boundary touched, near zero elsewhere.

Two rules keep the fakes honest:

A fixture that crosses a boundary comes from the producer, never from reading your own parser. If the data is written in real life by something else (a wire format, an API response, another tool’s output), copy that producer’s real output verbatim. A fixture written to match your parser confirms the parser to itself. measured Twice in one night: a fake message bus in a test invented an envelope field the real bus never emits; nine tests were green while the code under test read attacker-controllable text instead of the true origin, inverting the script’s own security claim.

Every stub owes a test on the other side. For every expectation set on a double, at least one test below performs that exact call against the real implementation; for every stubbed answer, a test below asserts the real thing produces that answer, including the throws. Otherwise the suite is green over an agreement nobody checked.

“You have a test that says what happens if the answer is 23, and you don’t have a test that says what happens if the answer is 22. And that, my friends, is a contract problem.” J.B. Rainsberger, “Integrated Tests Are a Scam”

07Teeth: never trust a green you have not watched go red

A green suite looks identical whether the fix works or your change was inert and no test ever exercised it. You cannot tell those apart from the green bar. measured A one-line gate edit was reported done with “80/80 green” as proof; the edit was inert (a new parameter no caller passed), so the suite was green precisely because behavior did not change.

So for high-stakes code (§03’s list: a security gate, a guard, anything where the green suite is your primary evidence the fix is real), run an executed mutation:

1. Baseline: run the suite, confirm green.
2. Introduce one tracked mutation that should break exactly the tests for the behavior you changed (force the new branch off, flip the comparison).
3. Run. Confirm exactly the right tests go red and nothing else. Nothing red means the test has no teeth. Red somewhere unexpected means false coupling.
4. Revert cleanly, confirm green returns.

Mutation tooling automates this per language (Pitest for Java, Stryker for JS/TS, Infection for PHP, Mutant for Ruby, mull for C/C++; verify names before use, they rot). Scope it to the code you just touched, run it as you develop, not overnight. The best single example of what it finds: Pitest deleted the line that throws the assertion error inside Google Truth, an assertion library, and no test failed; the covering test was missing a fail() call and passed by falling through. (Henry Coles, “Making Mutants Work for You,” GOTO 2019)

08No self-validation

Assert the spec’s words, traced to a source, never a value you authored. A string you wrote and then asserted is assert(true) with better formatting. This applies to user-facing copy exactly as it applies to code: a test asserting marketing words the agent invented, against a view the agent wrote, goes green while shipping slop. measured An agent working from a screenshot and a summary (instead of the actual spec it had been handed) invented copy, asserted it, watched it pass, and reported “aligned” repeatedly.

• The proxy is not the spec. A screenshot, a handoff summary, or your memory of the spec are lossy. Open the actual spec.
• When delegating to a subagent, pass the spec file path, never your own summary of it. A summary is a fresh proxy, and two agents validating against the same summary is the tautology reborn.
• A spec can itself be wrong: one published-standard audit found over 100 defects in the standard itself, including a requirement contradicted by its own next paragraph. When the spec contradicts itself, stop and say so; do not silently pick the reading that is easier to implement. (John Hughes, “Testing the Hard Stuff and Staying Sane,” Clojure/West 2014)

09An intermittent failure is a finding, never “flaky”

A failure that disappears on re-run is unexplained, not resolved. The next step is capture (logs, artifacts, a recorded run), not another attempt. The measured case that settles it: a team re-ran 5 failing groups out of 50 until green and shipped; customers then hit a caching bug affecting about 10% of sessions. Five out of fifty is also 10%. The failure rate was a direct measurement of the blast radius. (Alister Scott, “Your Tests Aren’t Flaky,” GTAC 2015)

Retries in production code are fine. Treating a retry that succeeded as an explanation is not. And never harden a test against an application defect (a sleep, a tolerance for a random popup): that absorbs the defect into the suite, where it becomes permanent and invisible.

10Run the suite with its own runner

Before debugging a suite, read how the project actually runs it: the package.json test script, the Makefile target, the config file. A project written in a toolchain that ships a test runner is not thereby a user of that runner. measured Two nights lost to running bun test on a bun project whose suite was vitest; under the wrong runner the suite did not fail, it hung, and a whole suite that cannot complete is a statement about the runner, not the code.

Two siblings of the same failure:

• An unregistered test is absent, not failing, and nothing in the toolchain says a word. measured Three test files sat outside the directories the test config registers and had never executed anywhere; each passed when run directly.
• In a hand-rolled suite (bash, no framework), confirm the cases run at all. Defining test_foo() and then reading $? reads the exit status of the function definition, which is always 0, so every case prints ok having asserted nothing. measured 5/5 green against a script that was literally exit 0.

11What a test says

• Test names are propositions, not method names. They are never called by your code, so language naming conventions do not govern; use underscores and write the rule: a_year_divisible_by_4_but_not_by_100_is_a_leap_year beats test_leap_year_works. Writing the rule out is itself a check; it surfaces imprecision in your own understanding. When a propositional name fails, the failure carries information (the opposite proposition is true) instead of just sounding an alarm. (Kevlin Henney, “Structure and Interpretation of Test Cases,” GOTO 2022)
• One behavior per test, as many assertions as that behavior needs. Do not mistake this for one assertion per test: “I keep seeing people making this recommendation. Please don’t do that.” (Henney) Delete any assertion that could fail while the named behavior still holds; it belongs to a different test.
• Prefer asserting what is there over what is absent. A negative assertion can become permanently true for an unrelated reason and then can never fail again (assert a word is absent from a page, then rebuild the page to load its data async: absent forever, green forever). (Bernhardt, “Fast Test, Slow Test,” PyCon 2012)
• Watch for invented behavior. Developers (and agents, worse) routinely add limits and validations nobody asked for, with no test. Every behavior in the diff traces to a requirement or gets called out.

12When testing hurts, the design is talking

Michael Feathers, in “The Deep Synergy Between Testability and Good Design”: “Every time you encounter a testability problem, there is an underlying design problem.” Read the implication one way only. Better design brings testability. Testability does not bring better design.

So when a private method goes public, the obstacle is gone and nothing is fixed. Tell an agent to make the code testable and it will loosen visibility, thread in a container, or reach for a private-access tool, then report success. Check the claim against the asymmetry.

13Measuring the suite: coverage and properties

Know which coverage number you are reading

“Coverage” unqualified is a useless word. Function, statement, branch, and path coverage are four different numbers that disagree; a single FizzBuzz test scores 100 / 100 / 50 / 25 across them. Worse, statement coverage partly measures how the code is written: an imperative version that accumulates a string reports 100% where an expression-based version of the same logic reports 62.5%, on the same tests. The number rewards the less honest implementation. (Henney)

Coverage tells you what has definitely not been tested; mutation testing tells you what definitely has. (Coles) They are opposite instruments. Measure both if you like; target neither. “The minute we set a target, the nature of the numbers changes.”

Property-based testing, for the cases you cannot enumerate

Bugs that span two features need a quadratic number of hand-written cases to cover pairwise interactions; races worse. Write a small model of your system: for each call, its precondition, the state transition, and the postcondition. Generate call sequences against the model, run them, and let the tool shrink any failure to a minimal reproduction. Scale evidence: a model of under 100 lines against a 6,000-line implementation. At one company, a race that corrupted data bimonthly for a year, its evidence self-repairing away, was reproduced with one record and five calls. (Hughes)

Two lessons hold even without the tooling. Minimize any reproduction before you debug it, however you found it. And do not trust your own judgment about which steps are unnecessary. Hughes deleted an open/close pair he thought obviously irrelevant; the bug vanished for tens of thousands of runs. The pair was essential.

14Before claiming done

• Every new behavior has a test that was observed failing before the implementation existed. Each failure came from the missing behavior, not from a defective test.
• The implementation is the smallest that passes. Nothing beyond what the requirements asked for.
• Full suite green under the repo’s own runner, no new warnings.
• For each plausible mutation, you can name the test that catches it: wrong constant, swapped branch, missing side effect, empty return, missing validation. Cannot name one? Write it.
• For high-stakes changes, an executed mutation failed exactly the right tests and was then reverted cleanly. See §07.
• Every asserted user-facing value traces back to a spec source you did not write. See §08.
• No test was made to pass by simply re-running it. See §09.
• Report what was tested and what was not. Distinguish “It should work” from “I watched it fail then pass”. Only write the second.

15Discuss with your operator, then make it physics

A rule the agent merely reads is a posted speed limit. A hook it cannot route around is a speed bump. Jay Springett put it this way: “One can be ignored; the other cannot.”

So treat every ALWAYS or NEVER in your agent instructions as a candidate for promotion from prose into a gate: a pre-tool-use hook, a pre-commit script, a CI check, a schema. The mechanics have their own field note: Physics, not pleas, with a worked example and a prompt you can reuse.

The test for whether your gate is real: ownership of the switch. If the agent can disable the constraint through its own environment (an environment variable, a flag, an editable config line), it is advice, no matter how loudly it logs. The bypass has to live somewhere the agent cannot reach. Otherwise admit it is a suggestion and stop calling it a gate.

Candidates, roughly in order of value per effort:

1. Runner check. Block or warn when test commands do not match the project's declared test script. The cost of missing this is two nights (§10).
2. Registration audit. Fail when a test file lives outside the directories registered in the test config. A test that is never collected is silent by default; this makes the absence visible.
3. Copy and value provenance. Block commits when an assertion about a user-facing string has no citation to a real spec line. This closed my worst incident (§08). Pair it with a delegation guard: subagents get spec file paths, not summaries.
4. Executed-mutation requirement for high-stakes changes. Auth, payments, guards, gates: require proof the suite failed under a tracked mutation before the change counts as complete. Or run real mutation tooling limited to the diff.
5. No re-run-to-green in CI. Auto-retry off by default. If a pass arrives on retry, file an anomaly with captured artifacts instead of letting it vanish.
6. Announce-the-skip. When tooling spots production-code edits with no accompanying test edits, force the agent to declare the skip out loud rather than let the omission pass silently.
7. Test-only affordances, config-tested. Test-only endpoints and state injection are fine for hard-to-reach states, but the production config must provably exclude them, and that config needs its own unit test. Without the second half it is a security hole (GTAC 2015).

Three judgment calls that stay with the operator

Reasonable practitioners disagree on these, and your agent should not resolve them silently:

16The corpus

If you want your agent to get better at this, feed it the sources rather than only the summary. Everything above distills from:

Talks (all findable by title): Ian Cooper, “TDD, Where Did It All Go Wrong” (DevTernity 2017) · Sandi Metz, “The Magic Tricks of Testing” (RailsConf 2013) · J.B. Rainsberger, “Integrated Tests Are a Scam” · Gary Bernhardt, “Boundaries” (RubyConf 2012) and “Fast Test, Slow Test” (PyCon 2012) · Michael Feathers, “The Deep Synergy Between Testability and Good Design” · John Hughes, “Testing the Hard Stuff and Staying Sane” (Clojure/West 2014) · Henry Coles, “Making Mutants Work for You” (GOTO 2019) · Alister Scott, “Your Tests Aren’t Flaky” (GTAC 2015) · Kevlin Henney, “Structure and Interpretation of Test Cases” (GOTO 2022)

Skill repos and writing: obra/superpowers (test-driven-development), quii/learn-go-with-tests (anti-patterns.md), Steve Sanderson’s “Writing Great Unit Tests,” Jay Springett’s “Hard Worlds for Little Guys” (the physics-over-prose frame)

Named as the obvious next reading, not yet absorbed: Kent Beck, Test-Driven Development: By Example · Freeman and Pryce, Growing Object-Oriented Software, Guided by Tests · Feathers, Working Effectively with Legacy Code

The pipeline, if you want to replicate it: transcribe the talks, extract per-talk notes with timestamps, distill into a skill file your agent loads whenever it writes production code, and record every incident where the agent got burned anyway as a new rule with the word “measured” on it. The incidents are worth more than the talks.