How We Prove an AWS Waste Check Works: Fire on a Real Cluster, Stay Silent on Its Twin
A unit test tells you your code does what you think AWS does. It can't tell you what AWS actually does.
For a tool that reads CloudWatch metrics and billing data to decide whether a resource is wasted, that gap is the whole product. So we test each check the same way: build the wasted thing for real, and make the check prove itself on it.
The rule is two halves:
- Fixture: a real resource that should trigger the check. The check has to fire on it.
- Control: the healthy version of the same thing. The check has to stay silent on it.
Fire without silence and you've built an alarm that always rings. Silence without fire and you've built nothing. You need both, measured against live AWS, before you trust the check.
Here is what that looked like for one check: idle DocumentDB.
The check, and the real cluster we pointed it at
The rule: flag a DocumentDB cluster with zero connections and combined read plus write IOPS under 10 per second, averaged over 7 days.
We stood up a real db.t3.medium cluster in a separate test AWS account and left it alone. It costs $1.87 a day and it's a short-lived test resource, not something we keep. Measured on 2026-09-19:
| Metric (7-day window) | Value | Gate |
|---|---|---|
| DatabaseConnections | 0.0000 | must be 0 |
| Combined read + write IOPS | 5.46 / s | under 10 / s |
That is an idle cluster by any reasonable reading. The check should fire.
What the real cluster told us that the unit tests couldn't
It did not fire. Not on the first try, and not for one reason. Three separate bugs were stacked in front of it, and every one of them lived in the gap between our code and AWS's behavior.
| Layer | What was wrong | How it hid |
|---|---|---|
| Wiring | The detector called a metrics method that didn't exist on the live data provider, so it always got nothing back. Fixed 2026-07-09. | A missing answer reads as "no data", which reads as "nothing wrong here". |
| Units | CloudWatch reports DocumentDB IOPS as a rate (Count per Second). We read it with Statistics=Sum over 7 days, which adds a rate up as if it were a running total. A steady ~5.15 per second over 10,080 one-minute datapoints sums to roughly 52,000, against a gate expecting a small number. | Our regression tests pinned the same Sum-shaped mock, so they enshrined the bug. |
| Naming | Cost data calls the product "Amazon DocumentDB (with MongoDB compatibility)". Our service-to-detector map only knew "Amazon DocumentDB" and "AmazonDocDB", and the lookup is exact, so on every scheduled scan the DocumentDB detector was skipped, even on accounts whose costs listed DocumentDB. | A skipped detector reports nothing, which looks exactly like an account with no waste. |
The second row is the one worth sitting with. The tests were not sloppy. They were written against a mock that matched the code's assumption, so they could only ever confirm the assumption. Only a real cluster reporting a real rate could disagree, and it did.
The fix for the second bug reads the average of the rate instead of the sum, and gates on combined IOPS averaging under 10 per second with zero connections.
The first real firing, and why we almost missed it
With all three fixed, the 2026-09-19 06:01 UTC scan ran the DocumentDB detector on that cluster for the first time and it fired: idle cluster, list price $56.94 a month for one db.t3.medium.
You would not have seen it in the findings list, though. Our scan reconciles every finding against the account's actual billing before showing it, and the test account was on DocumentDB's free trial, so the finding reconciled to $0.00 and was dropped from the customer-facing list on purpose. It only existed in the pre-floor debug output we keep for exactly this kind of question.
That's the system working as designed, and it's also why "look at the findings page" is not a test. We had to read the layer beneath it.
The control: the same cluster, connected
A healthy twin usually means a second resource. For this check the healthy state is simply "someone is using it", and the two states are mutually exclusive by construction: the check requires zero connections, so any connection at all rules idle out.
So the control was the same cluster with one variable changed. On 2026-09-19 we opened a handful of connections from a short-lived Lambda inside the VPC and deleted it the same day.
| Fixture (idle) | Control (connected) | |
|---|---|---|
| DatabaseConnections | 0 | 0 to 6 per minute while connected, 7-day average 0.0169 |
| Combined IOPS | 5.46 / s | 5.44 / s |
| Scan on 2026-09-20 | fired the day before | detector ran, 0 findings |
Nothing moved except the connections. IOPS stayed under the gate, and one afternoon of traffic was enough to pull the 7-day average off zero and silence the check. Fired on the fixture, silent on the control.
One honest caveat, because it would be easy to leave out: those two observations are separated in time, not simultaneous. It's one cluster measured in two states, and the code guarantees the states can't both be true. That's a weaker claim than "two clusters side by side", and we say so in the write-up.
Controls need checking too
The same sweep caught the opposite failure. A Glue job we built as a healthy control set off the "missing timeout" check. We had left the timeout out when creating it, and Glue quietly defaults to 2,880 minutes, which is exactly the number the check looks for. The control was never healthy.
A control that fires isn't a control. We now set the timeout explicitly, and we verify every control is silent across all checks, not only its own.
What we took from it
- A passing unit test is a statement about your assumptions. Real resources are the only thing that can contradict them.
- Test the whole path, not the function. All three bugs lived outside the detector's decision logic: in how metrics were fetched and read, and in how the check got selected to run.
- Prove silence as hard as you prove firing. A check nobody has watched stay quiet is a guess.
- It's cheap. One small cluster for roughly a week costs about what a lunch does, and it found bugs our mock-based tests couldn't.
That's the method we use when we validate a check against real AWS. CloudWise runs 189 waste checks across 40+ AWS services, and if you want to see what a read-only pass finds in your own account, it takes about five minutes.
Free, read-only AWS waste scan: cloudcostwise.io — five minutes, no card.
Stop wasting money on AWS
CloudWise monitors 45 AWS services and finds waste automatically. Free forever.
Start Free Scan →