Woke up to a billing alarm email. Thought I had leaked my AWS keys accidentally and somehow run up 437 billion dollars of charges. Joke's on them though, I don't have 437 billion dollars
Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS
I'm not a betting person but I am looking forward to the postmortem, whether or not AI was involved, and what their code to production verification stuff looks like now. This kind of thing should have been caught by automated tests.
There is an enormous amount of money invested in trying to prove the opposite, for this reason not only they will never admit it, they will actively negate AI was used. It happened before:
My Budget is 10$.
The month forecasted cost associated with this budget is $182,278,249,263.06.
Even though I new they could not collect the whole amount, I wondered whether I was hacked.
I closed the account, it was an old testing account anyways.
Woke up to a 100 billion dollars in S3, which is above the USD 4 alarm threshold I had set (I pay $0.55 monthly) . Some AI decided to prune the most impressionable of us.
My estimate was only $21M (vs ~$0.01 average bill). Wish I had checked status sooner and saved myself the panic!
My process went: verify email is not phishing (it was), login to console and check dashboard (same amount), attempt to understand cost (cost management kept contradicting itself), try to log support ticket and only on that part did I notice the status notification. At least I can breathe again now!
A friend of mine went through the same - he got an alert for bill over budget, he logged in and boom 107M$.
I was on the phone with him and we checked that he didn’t leak any APIs keys but no traffic at all. I even thought of a breach at a vendor he uses for some s3 stuff before I found this thread on HN.
same process, same near heart attack - especially some days ago I had malicious activity on my platform and thought that now they actually found something.
Apparently you can trigger an Action (e.g. prevent uploads) when the billing alert triggers, but then my platform wouldn't work anymore, just because AWS had an issue. Also insane that Amazon still hasn't send an email to clarify.
Whatever you do, AWS, don't post a related service health alert site wide on the console. Heck, don't even post one in the billing module. We wouldn't want to overdo the alerts, especially when we already have one being displayed to market the new FinOps Agent in Public Preview.
Several comments here talk about "nearly" having a heart attack. But I wonder: since it's happened to so many people, chances are someone had a heart attack for real. Can they legally be made responsible for that?
When someone says "I nearly had a heart attack," it's _highly_ unlikely that they actually nearly had a heart attack. I don't think the chances are good that anyone actually had a heart attack.
Scared me even though it was obviously a bug once I stopped to consider the magnitude ($bn). Very unfriendly that they don't allow for hard spend caps; closed my mostly dormant personal account as a result.
Thank you so much! I just woke up, and saw budget alert email for a dormant account to use $434,896.90. I haven't gotten so awake so fast in such a long time.
Ditto on the heart attack. My cost estimate for the month is currently $223,509,270,216.17. My girlfriend suggested contacting Elon for help. Glad I found this thread. Maybe I should create new keys anyway, this stuff freaks me out.
If I can expect to be penalized for not paying my legitimate bills, companies should also be penalized for failing to implement common-sense reasonable safeguards that prevent them from slapping their consumers with such absurdities.
At some point my role was to reduce our startup’s AWS bill. I managed to keep 7 figures on our books instead of handing it to AWS. But a message like that would have given me a heart attack in those days.
Long story short: it saved the company from irrelevance. “Well-architected” is for the hyperscalers’ balance sheet, not yours.
Jul 17 3:03 AM PDT We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more
information becomes available.
Jul 17 2:07 AM PDT Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.
Jul 17 1:33 AM PDT We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.
I've had a mysterious Neptune cluster appear on my billing. Never used it, no API key access (or IAM instance profile access, OIDC etc), nothing in my console shows I've ever had one in any region. Raised a case with support, they ignored it.
I almost had a heart attack because of this. I was like, did I mess up my API management? Why didn't I just use Lightsail? Those were the thoughts running through my head.
My personal website is on Lightsail, but those alerts started popping up from some test services I had set up while I was studying AI. I swear my heart nearly stopped and I cried. I really think AWS should have spending limits in place.
My bank account barely has enough for next month's rent.
Three billion dollars sounds about right for a free week of Kiro in the default Agentic mode. We usually see slightly higher numbers, so I wouldn’t be too concerned.
Zimbabwe redenominated its currency three times in 2006, 2008, and 2009, dropping a total of 25 trailing zeros in the process. the 4th Zimbabwean dollar in 2009 was worth 10^25 of the first zimbabwean dollar.
You don't need hours to recalculate billing. You need to go back to basic algebra.
Anyone using Amazon and dealing with this should be moving away from their services because something this basic going wrong means the correct people are not at the helm of the ship.
Anyways I didn't need coffee. That produced an adrenaline release unlike any I've experienced before. Thanks AWS
If you owe AWS 437B bucks, that’s a big problem for AWS
It's entirely irresponsible of Amazon to even display such values to the user.
https://www.bbc.com/news/world-us-canada-55990461
https://edition.cnn.com/2021/07/01/business/robinhood-lawsui...
>The second path involves rolling back a recent change to the billing computation subsystem.
Want to bet AI code was involved?
https://www.theregister.com/software/2026/02/24/aws-would-ra...
AI: "No problem, let me change how we bill and fix the tests for the new increase in value"
Even though I new they could not collect the whole amount, I wondered whether I was hacked. I closed the account, it was an old testing account anyways.
My process went: verify email is not phishing (it was), login to console and check dashboard (same amount), attempt to understand cost (cost management kept contradicting itself), try to log support ticket and only on that part did I notice the status notification. At least I can breathe again now!
I was on the phone with him and we checked that he didn’t leak any APIs keys but no traffic at all. I even thought of a breach at a vendor he uses for some s3 stuff before I found this thread on HN.
Apparently you can trigger an Action (e.g. prevent uploads) when the billing alert triggers, but then my platform wouldn't work anymore, just because AWS had an issue. Also insane that Amazon still hasn't send an email to clarify.
https://health.aws.amazon.com/health/status
Of course, you provided heart stress failure tests for free.
Thanks AWS, no caffeine needed this morning!
Love to see how hyperscalers make your life easier and less worrying.
https://news.ycombinator.com/item?id=48945233
OT: $286 million for me, almost had a heart attack
Long story short: it saved the company from irrelevance. “Well-architected” is for the hyperscalers’ balance sheet, not yours.
AWS on their support data is reporting this:
Inaccurate Estimated Billing Data
Jul 17 3:03 AM PDT We continue to work to resolve the issue affecting estimated cost and usage data displayed in the Billing and Cost Management Console. We have identified the root cause as an issue with unit pricing within the estimated billing computation subsystem and we are working on a mitigation. The displayed billing estimates do not reflect actual usage and charges. There are no customer actions required at this time. Once the issue has been mitigated, we expect full resolution to take multiple hours as we work through recomputing the estimated billing data. We will provide another update by 4:00 AM PDT or sooner if more information becomes available.
Jul 17 2:07 AM PDT Beginning on July 16 7:38 PM PDT, we began displaying incorrect estimated billing data in the Billing and Cost Management Console. Our engineering teams are engaged and investigating root cause. We will provide another update by 3:00 AM PDT or sooner if more information becomes available.
Jul 17 1:33 AM PDT We are investigating issues with Cost Explorer reflecting inaccurate estimated billing data.
My personal website is on Lightsail, but those alerts started popping up from some test services I had set up while I was studying AI. I swear my heart nearly stopped and I cried. I really think AWS should have spending limits in place.
My bank account barely has enough for next month's rent.
Zimbabwe redenominated its currency three times in 2006, 2008, and 2009, dropping a total of 25 trailing zeros in the process. the 4th Zimbabwean dollar in 2009 was worth 10^25 of the first zimbabwean dollar.
But so is imprisoning or exterminating all humans for their own good, as most AI dystopias end up as.
$151 billion the number for me.
You don't need hours to recalculate billing. You need to go back to basic algebra.
Anyone using Amazon and dealing with this should be moving away from their services because something this basic going wrong means the correct people are not at the helm of the ship.
LLMs are notoriously bad at it...