Audit Baseline Ratchet
homeboy.json carries a permanent suppression list for homeboy audit at
baselines.audit.known_fingerprints. This document explains what is in it, why
it may only shrink, and how to change the ceiling that enforces that.
The ratchet is enforced by tests/audit_baseline_ratchet_test.rs.
What the baseline is
When homeboy audit --update-baseline runs, every finding of the current run is
reduced to a fingerprint string and stored. On later runs,
homeboy_engine_primitives::baseline::compare treats any finding whose
fingerprint is already in that list as not new. Only findings absent from the
list set drift_increased and fail the audit.
The list is therefore not a record of history. It is live configuration that decides what the audit is allowed to report.
A row is not one finding
This is the part worth internalizing before adding a row.
AuditFinding::fingerprint (in crates/homeboy-code-audit/src/baseline.rs) builds:
convention::file::KindThe description is deliberately excluded, because structural findings embed
volatile values — the test fingerprint_ignores_description pins that a
GodFile finding keeps one identity as the file grows from 2,484 to 2,645
lines. No fingerprint contains a line number.
Matching is then plain set membership on that string.
The consequence: baselining is per file + kind, not per instance. One row
suppresses every finding of that kind in that file — the ones that existed when
the row was written, and every one added afterward. A file baselined for
IntraMethodDuplicate is silent on that detector forever, no matter how many
new duplicated blocks land in it.
CoreBoundaryLeak is the only exception. It appends a description whose line
numbers are normalized (... at line <line>), giving:
convention::file::description::Kindso it is per file + kind + message. Still not per instance.
Of the current rows, 821 are the three-segment per-file+kind form (covering 821
distinct file+kind pairs across 582 files) and 312 are the four-segment
CoreBoundaryLeak form.
Two ways the list grows on its own
homeboy audit --update-baselinere-saves whatever the current run found.- A conflicted
homeboy.jsonis resolved bybaseline_mergeto the union of both sides’ fingerprints, on the reasoning that each side accepted some debt.
Neither path asks anyone to justify a new row, and nothing ages rows out. That is why the size needs an external check.
The rule
baselines.audit.known_fingerprints may only shrink.
tests/audit_baseline_ratchet_test.rs pins AUDIT_BASELINE_CEILING to the
exact current count and fails if the list exceeds it. A second assertion fails
if the list is below the ceiling, which forces the ceiling down as debt is
paid — a ceiling with slack in it is room for silent re-growth.
Adding a suppression
In order of preference:
- Fix the finding. This is the default and usually the cheaper option.
- Retire an existing suppression in the same PR, so the total does not rise.
- Raise
AUDIT_BASELINE_CEILINGin the same PR, and justify it in the PR body. Expect to be asked why the finding cannot be fixed.
Lowering the ceiling
When suppressions are retired, set AUDIT_BASELINE_CEILING to the new exact
count in the same PR. Get the number with:
jq '.baselines.audit.known_fingerprints | length' homeboy.jsonThe audit_baseline_ceiling_leaves_no_slack test will tell you the number if
you forget.
Current debt, by finding kind
1,133 rows as of the commit that introduced this ratchet. This table exists so
the debt is visible rather than hidden inside a 158 KB JSON blob — that array is
93% of homeboy.json by byte count.
| Kind | Entries |
|---|---|
CoreBoundaryLeak | 312 |
SkeletonDuplicate | 147 |
HighItemCount | 133 |
ConstantBypassLiteral | 128 |
IntraMethodDuplicate | 114 |
GodFile | 94 |
UnreferencedExport | 47 |
CommandWrapperBypass | 37 |
NearDuplicate | 24 |
DirectAggregateConstruction | 15 |
ParallelImplementation | 15 |
RemoteExecutionPreflight | 15 |
MissingMethod | 8 |
DuplicateFunction | 7 |
StaleDocReference | 5 |
UnboundedOutputCapture | 5 |
ThinCommandAdapterViolation | 4 |
VacuousTest | 4 |
DeadCodeMarker | 3 |
RepeatedEnumDispatchContract | 3 |
UnusedParameter | 3 |
DirectorySprawl | 2 |
GlobalEnvMutationGuard | 2 |
LayerOwnershipViolation | 2 |
BrokenDocReference | 1 |
MissingInterface | 1 |
NamingMismatch | 1 |
ParallelRunnerSetup | 1 |
| Total | 1133 |
Roughly: 307 duplication findings, 227 size findings, 47 dead public exports.
Regenerate with:
jq -r '.baselines.audit.known_fingerprints[] | split("::") | .[-1]' homeboy.json
| sort | uniq -c | sort -rnBurning this list down is separate work from the ratchet. The ratchet only stops it from getting worse.
Known wrinkles
metadata.item_countis stale. It reads 1137 while the array holds 1133.normalize_loaded_baselineonly recomputesitem_countwhen a policy section enumerates fingerprints, and the sections now use thefingerprint_prefixform with empty enumerations, so that branch never runs.compareusesitem_countfor its reporteddelta, so that number is off by four. It does not affectdrift_increased, which is computed from the fingerprint set, so suppression behavior is correct. The ratchet measures the array length, notitem_count.- Rows are not sorted.
savesorts, but 363 of the 1,133 rows are out of sorted position, so order is not an invariant and no test pins it. - The baseline is not in a separate file. Splitting it out would make config
diffs readable, but
BaselineConfig::json_path()hardcodeshomeboy.json, and so do the git-ref reader (git show <ref>:homeboy.json) and the wholebaseline_mergeconflict driver. There is no external-baseline-path option today, so the split is a real change rather than a config tweak.