CloudWatch Logs Never Expire by Default: The Storage Bill That Keeps Growing

You know, when we look at infrastructure costs, we usually expect like the massive EC2 instances or unoptimized databases to be the main culprits. But today, we're pulling from some AWS billing white papers and engineering postmortems to look at something much, much sneakier. Oh, absolutely. It's kind of like leaving a garden hose running in your backyard, except instead of just flooding the lawn, it's filling up this infinitely expanding swimming pool. Right.

And you are paying for every single drop of that water basically forever. Exactly. So our mission for this deep dive is to unpack exactly how this specific AWS billing mechanic works, why it survives undetected for so long, and how you can fix it right now. Yeah. So the reason that pool expands infinitely really just comes down to a tiny default setting in Amazon CloudWatch log groups.

I mean, if we look at the unit math in US East-1, standard log ingestion runs about 50 cents per gigabyte. And then storing those logs costs about 3 cents per gigabyte per month. Okay. 50 cents to get it in, 3 cents a month to keep it. I mean, in isolation, 3 cents is nothing.

Yeah. It sounds like practically nothing. But if an application generates, say, 100 gigabytes of logs a day, that 3 cent storage fee is compounding on a constantly growing pile of data. So by month 12, you aren't just paying $90 a month, you're paying over $1,000 a month for data you probably haven't even looked at in a year. Exactly.

That's the trap. It's the default configuration. When a new log group is created, the retention policy defaults to never expire. So literally every single log line your application writes is stored and billed endlessly. Wait.

AWS just keeps them forever by default? How does a confident engineering team not notice this creeping up on the bill? Well, you have to think about the alternative. Deleting user data by default is a massive liability for a cloud provider. AWS prioritizes data safety over your wallet, which makes perfect sense.

So cost optimization is entirely your responsibility. Right. They aren't going to just throw your data away for you. Exactly. And this waste accumulates silently because honestly, nobody really owns it.

It's just a byproduct of deploying a service. Things function normally, no operational alarms go off. And on a standard per service invoice, it's just rolled up under a generic CloudWatch line item. So it's hiding in plain sight. If I want to find this leak, I guess looking at the retention column in the AWS console is fine for a quick check.

But if I've got, you know, 500 log groups across different regions, clicking through the UI isn't going to scale at all. No, not at all. So how do we actually automate hunting this down? You use the command line. You can run the API call describe dash log dash groups and just parse the JSON output.

What you're specifically looking for are any log groups that are missing the retention in days attribute entirely. If that attribute isn't there, AWS is keeping that data forever. Okay. So we find the groups missing the attribute. The fix seems straightforward.

We just add a retention policy. But wait, if I run an API command to set that policy to say 30 days, AWS is going to instantly purge everything older than that, right? That sounds like a one-way door. Yeah, you hit the core risk exactly. The remediation is running the put dash retention dash policy API call with a timeframe that matches your debugging or compliance needs.

But applying that policy triggers background garbage collection in AWS. Oh, wow. So it just wipes it out? Yes, logs older than the new retention period are deleted permanently. And lengthening the retention period later will not bring those deleted logs back.

They're totally gone. Which means we can't just slap a 30-day policy on everything just to save a buck. No, absolutely not. And this brings up a fundamental safety rule for AWS. You should never assume a resource is provably or guaranteed unused.

Even if a log group looks stale to you, downstream processes, compliance audits, or security tools might still depend on referencing it. Right, like some auditor script that runs once a year. Exactly. So before you shorten a retention policy on critical infrastructure, you absolutely must export anything you might need to keep into an S3 bucket. Right, export to S3 first, then cap the retention.

Now, we just talked about manually querying the API to find these missing attributes. But if you don't want to script that out yourself, there's actually a free scanner mentioned in our notes. Yeah, if you run pip install cloudcostwise, you get a local scanner that covers 20 AWS services. It runs read-only directly on your own machine and flags this exact missing retention pattern for you. And it doesn't send data anywhere, right?

Right, it sends absolutely nothing back to CloudWise. It's simply a fast way to get visibility without writing your own shell scripts to check for the missing attribute. Perfect. So as you look at your own infrastructure today, consider this. If a simple never expire default can silently fill an infinite costly pool in your AWS account, well, what other set-and-forget default configurations are quietly accumulating charges right this very second?

Something to think about.