Validation
How we prove each waste check
A detector that passes its unit tests can still be wrong about real AWS. So every waste check sits on a ladder, L0 to L4, and the ladder says how far it has been proven. We publish where each one stands, including the ones that are not proven yet.
The ladder
Counts are the highest level each of the 187 waste checks has reached today, read from the same data as the catalog.
- L0Unit-tested against synthetic provider data.0
- L1Desk-reviewed against current AWS documentation.94
- L2Fired on its fixture with a plausible saving, and did not fire on its control.64
- L3Its finding rendered correctly in the UI (workspace, drawer, dashboard totals).29
- L4Remediation planned and executed or rolled back against its fixture.0
93 of 187 have been fired on a real resource and held back on its control (L2 or above). The other 94 are at L0 or L1: desk-reviewed against AWS documentation, not yet proven on a live resource.
What a fixture is
A fixture is a real AWS resource built on purpose, in a dedicated fixture account, never in a customer's: an idle NAT gateway, an unattached volume, a log group with no retention policy. The waste check has to fire on it.
Next to each fixture sits a control: the same kind of resource, set up so it is not waste. The check has to stay quiet on the control. Firing on the fixture alone proves nothing, because a check that fires on everything would pass.
L3 and L4
L3 checks that the finding reaches the screen intact: workspace row, detail drawer, dashboard totals. L4 goes further for checks that can be fixed: a remediation is planned and either executed or rolled back against the fixture, so we know the fix does what it says.
A level only goes up with new evidence. It never goes up because time passed.
Advisory checks
44 of the 187 waste checks are Advisory. They have no path to L2 within our fixture budget: the smallest resource that triggers them costs more to run for the check's window than we allow, or they need a Reserved Instance or Savings Plan commitment that cannot be torn down, or AWS no longer lets you create the configuration they look for.
So we label them. An Advisory finding says it is estimated from your configuration and has not been validated against live waste, and its savings are kept out of every total we show you.
Open and closed
91 waste checks are open: they run in a local scan, and their catalog pages show the detection criterion that is already in our public docs. The other 96 run in the partner build and the paid hosted tiers; their pages say what they find and why it costs money. Both kinds are held to the same ladder.