← Back to Blog

Coverage floors without theater

·6 min read

Coverage numbers are the most gamed metric in software. A floor set by aspiration gets met by tests that execute lines and assert nothing. pyRPC just added a coverage gate anyway, because the failure mode of no measurement is worse: untested code ships indistinguishable from tested code. The design question was how to add the gate without inviting the theater.

Measure first, then pick a number

pytest-cov ran against all five Python packages with no threshold. Result: 80 percent across 1475 statements. The interesting rows were not the total:

pyrpc_codegen/ts_codegen.py   66%   (the string-grammar type parser)
pyrpc_core/cli.py             73%   (wizard, watchers, console)
pyrpc_core/bundlers.py        81%
pyrpc_core/config.py          88%
core/procedure.py, models.py  ~100% (protocol heart)

The shape tells a story: protocol internals are near-perfectly covered because they are pure and easy to test; interactive surfaces lag because they need scaffolding. That is exactly where you want visibility before deciding policy.

The floor is 78, not 80

Setting fail-under equal to the current number guarantees an immediate red PR for someone whose change shifts a percentage point by accident. Two points of slack absorb that noise while still catching real regressions like a new module landing at 30 percent. The number came from measurement, not ambition, and CONTRIBUTING.md states the social contract explicitly: raise it when adding code; never lower it to make a PR pass.

What line coverage will never tell you

The ts_codegen parser row deserves honesty: its uncovered 42 statements are mostly deep union and generic fallback branches. Line coverage counts them equally with critical paths even though a bug in dict-type rendering matters far more than one in an exotic Set[Tuple] corner. Coverage is a smoke detector, not a fire marshal. It cannot prove the suite asserts the right things; it can only prove which lines nothing touches at all.

Mechanics worth copying

  • Coverage runs as its own CI job pinned to one Python version; matrix legs stay fast and coverage stays deterministic.
  • Report uses term-missing so a red job shows exactly which lines are dark, in the log, no artifact hunting.
  • The local command in CONTRIBUTING.md is character-identical to CI. Gates you cannot reproduce locally are gates other people run for you.

Next step when it earns it: per-module floors so cli.py cannot hide behind procedure.py. Floors, like types, are most useful when they are specific.