Your Container Tag Is Lying to You: A Mutable ECR Tag Put the Wrong Build in Production

On 2026-08-21, a routine docs-only PR merged to main. CI built the frontend image, pushed it, and called start-deployment on the App Runner service — and got rejected. The pipeline went red. "Deploy to Production" failed.

That part is normal; pipelines go red. What isn't normal is what CloudTrail and ECR showed once we went looking: production wasn't broken by the red step. It was broken by the deploy that succeeded 3 seconds earlier — from a laptop, not CI — and the pipeline's red error message never once mentioned it.

The timeline, from CloudTrail and ECR, not from memory

Time (EDT)	Event
10:52:25	StartDeployment on cloudwise-production-frontend, called by an IAM user from a local machine (aws-cli/2.34.2, macOS, arm64) — not CI. At this instant the production tag points at the previous build.
10:52:43	CI pushes the new frontend image and moves the production tag to point at it.
10:52:44	CI calls StartDeployment as its own deploy role → InvalidRequestException: Can't start a deployment … because it isn't in RUNNING state → exit 254 → pipeline red.
10:55:49	The manual deployment from 10:52:25 succeeds — having resolved the production tag back at 10:52:25, before CI moved it. It deploys the old image.

Net state: the production tag pointed at the image CI had just built. That image was never deployed. App Runner was RUNNING on the previous build, and the pipeline was red for a reason that named none of this.

18 seconds is the whole story. That's the window between the manual deploy resolving the tag and CI moving it. Move a mutable tag inside that window and the registry and the running container permanently disagree about which image "production" means.

Why a mutable tag is the actual defect

ECR_TAG=production for the frontend repository always points at the newest push. App Runner doesn't pin a digest — it resolves whatever production points to at the moment a deployment starts, not at the moment the container swap completes. Those are two different instants, and normally nothing happens in between.

Two deployments racing to start closes that gap. One of them read the tag old; the other wrote it new. Whichever deployment actually executes the swap runs whatever it resolved, and there's no signal anywhere in App Runner's own state that says "the tag has moved since I started."

This had already been flagged as a hazard in the abstract — two CI runs landing close together can serve an older image, all green, no laptop involved. This incident reached the same failure by a different route: the second actor wasn't CI at all, so none of the existing concurrency guards between pipeline runs applied.

The fix: verify the digest, don't trust the tag (PR #1181)

Two independent changes, both in .github/workflows/deploy-environment.yml:

1. Bounded retry on start-deployment. A RUNNING-state rejection now retries for up to 20 minutes instead of failing on the first attempt — a wait can only delay a deploy, never corrupt one, so it's a safe response to "something else is deploying right now." But a retry alone doesn't fix the race; it just makes CI patient. The actual fix is next.

2. Re-read the tag's digest on every retry attempt, and refuse to deploy if it moved. Before each start-deployment call, CI re-reads what production currently points to and compares it against the digest of the image this run pushed. If they no longer match — because something else moved the tag in the interim — CI stops and errors loudly instead of deploying an image it can no longer identify:

"The production tag no longer points at the image this run built. Built <digest>, tag now <digest>. Something pushed to this tag outside this pipeline; deploying now would ship an unknown image."

That single check closes the 18-second window: CI will now only ever call start-deployment when it can prove the tag still means what it thinks it means.

3. A post-swap check — warn-only, on purpose. After the deployment completes, CI reads the digest App Runner actually deployed and compares it against what CI pushed. If they don't match, it logs a warning and keeps going rather than failing the pipeline.

That's a deliberate asymmetry, and it's the design decision worth explaining: the pre-swap digest check is a hard gate, because it's checkable before anything ships and a mismatch there means CI is about to deploy the wrong thing. The post-swap check can't be a hard gate, because by the time it runs, the running container already is what it is — the thing that ran ahead was the registry tag, not production. Failing the pipeline red at that point would block a legitimate deploy for a discrepancy that already happened and that the running service had no part in causing. Warn, don't block, on a fact you can no longer change.

The part that makes it worth writing about

The merge that triggered this was docs-only. The image that never got deployed differed from the one that did by markdown and release notes — nothing user-visible. Production was healthy and serving the whole time.

That's luck, not design, and it's the actual point: the same race on a real frontend change ships a red pipeline and production quietly serving the old build, and the red error points at App Runner deployment state — not at "prod is stale." You'd fix the wrong problem. InvalidRequestException: isn't in RUNNING state is a true statement about App Runner. It was never a true statement about what mattered, which was that the tag we were about to deploy had already been consumed by someone else. A pipeline that goes red for the right reason but tells you the wrong story is worse than one that goes red honestly — you close the ticket on the error you can see instead of the one that actually happened.

Boring fixes are the ones worth shipping: retry with a bound, verify a digest instead of trusting a tag, and only fail hard on the check that happens before anything ships.

---

We hit this on our own production App Runner service, running the app you'd be scanning your AWS bill with. Curious what a read-only pass over your own account finds? Free, read-only AWS waste scan: cloudcostwise.io — five minutes, no card.