70 lines
6.6 KiB
Markdown
70 lines
6.6 KiB
Markdown
|
+++
|
||
|
|
date = '2026-07-06T09:00:31-05:00'
|
||
|
|
draft = false
|
||
|
|
title = 'Well this sucks'
|
||
|
|
summary = 'GCP billing, AI hacking and no sleep'
|
||
|
|
+++
|
||
|
|
|
||
|
|
I cannot believe what a roller coaster of emotions I had yesterday, July 5th.
|
||
|
|
|
||
|
|
I got a page from PagerDuty shortly after noon saying my cloud budget was reaching 50%
|
||
|
|
|
||
|
|
[](/images/2026-07-gcpissues-01.png)
|
||
|
|
|
||
|
|
I started to explore AWS to start - I didn't even think it could be Google GCP. I had about $800 in credits queued in that account and shouldn't see spend for a few months.
|
||
|
|
|
||
|
|
But indeed it was Google. Over the next hour I was madly trying to find the culprit. I knew the project (myAnthosProject2), my oldest GCP project and the one I use in blogs the most. I could see from billing it was coming from Vertex AI but I hadn't done any recent posts.
|
||
|
|
|
||
|
|
None of my systems would account for that kind of massive spend. I had not written any recent Vertex AI articles.
|
||
|
|
|
||
|
|
As I rotated keys and disabled things, I still saw the spend continuing. I finally (around 1:15p or so) just deleted the project altogether.
|
||
|
|
|
||
|
|
I saw at this point, the prior night late in the evening Google detected abnormal usage but had done nothing to stop it...
|
||
|
|
|
||
|
|
[](/images/2026-07-gcpissues-02.png)
|
||
|
|
|
||
|
|
Meanwhile, it charged my credit card $100 and kept going.. the spend just kept climbing - against a project that was now completely deleted
|
||
|
|
|
||
|
|
[](/images/2026-07-gcpissues-03.png)
|
||
|
|
|
||
|
|
I couldn't get a person , the only billing support was an automated Chat bot - no person to find. The chat bot just kept telling me "well, you did all the things - account disabled and removed.. billing can lag 2-5 hours".
|
||
|
|
|
||
|
|
**2 to 5 hours** - that is a firehose of costs for a regular person. It crawled all the way up to $210 by about 6:30p when it stopped growing.
|
||
|
|
|
||
|
|
Around 4:25p that night, in sheer desperation I started to post to my GDE channel
|
||
|
|
|
||
|
|
> Okay - such torture today. the long story/version. I keep watching the monies go up and rollover as it dings my credit card another $100.. I'm hopping it should slow down soon but im so stuck. In short, an API key somehow was found out (tho searching, i never exposed it in writing, but whose to say they didnt get into my GCP account last week, view it, then go away). Starting July 4th evening someone in southeast asia (by the anomaly logs of trying to use AI keys in that region) started to hammer an unrestricted Vertex AI key.. it burned through about 850$ or so in credits i still had before it started to auto-charge my credit card. The failures in the system (imho) were i got an anomaly email at 11:47p at night saying 'hey, you spent 515.85 in your project on Vertex AI when we expected you to spend 0.10" but it didnt stop the account, or charges or shut things off.. it let it run thru all my credits and into my card. Only when my own GCP Budget alert hit pagerduty after church and rang my phone did i come down to my office to see why i had any charges in GCP. by that point it had billed my card 100, and started fresh again. and i cannot disable credit cards (must have 1 primary), deleting the project all together after expiring the account and unlinking it still shows growing charges.. i chatted with the billing ai bot (there is just no way to reach a person) tells me i did all the things i can do and charges can queue for 2-5 hours.. so i just sit here watching it grow and grow with nothing i can do and no one i can talk to. I thought maybe i could apply for cloud credits as a GDE - i have before and i do have a GDG event to prep for, but that form auto declined saying the credit program is paused. so im SOL.
|
||
|
|
|
||
|
|
The week prior my project had been tagged briefly for "hijacking" but as the compliance team would not say where or how, I just had to review my billing and keys and they all seemed fine so I assumed it was because i was using old GCA/Cloud Code APIs
|
||
|
|
|
||
|
|
[](/images/2026-07-gcpissues-04.png)
|
||
|
|
|
||
|
|
## All Developers should be Wary
|
||
|
|
|
||
|
|
The end result here, is we (GDEs) all perked up.
|
||
|
|
|
||
|
|
One GDE shared an article they wrote, ["Automated GCP Killswitch: How to Avoid an Unexpected Google Cloud Bill"](https://medium.com/google-cloud/how-to-avoid-a-massive-cloud-bill-41a76251caba)
|
||
|
|
|
||
|
|
In that article they cover massive bills from students - $55k, 11k pounds, 50k Euros...
|
||
|
|
|
||
|
|
Another GDE linked to [this TheRegister article which is pretty much my exact scenario including the issues getting details about 'hijacked resources'](https://www.theregister.com/cyber-crime/2026/07/03/dev-says-google-warned-him-about-account-hijack-then-charged-him-11000-anyway/5266234). That person racked up $11k and wasn't helped by billing (according to the article).
|
||
|
|
|
||
|
|
There is also a new [Spending Caps feature in preview](https://cloud.google.com/blog/topics/cost-management/introducing-spend-caps-ai-cost-visibility-next26) that might help, but its a request for private access and there are no guarantees it will be granted.
|
||
|
|
|
||
|
|
## Where we sit
|
||
|
|
|
||
|
|
I didn't sleep well last night.
|
||
|
|
|
||
|
|
I kept getting up and going back to my office in the basement to check my Cloud Budget to see that spend was done. I was still sitting on $210. This sucks, but this isn't thousands of dollars so i need to keep perspective.
|
||
|
|
|
||
|
|
I've submitted some forms, and reached out to Googlers I know to see if they can put me in touch with billing, but I do not expect much.
|
||
|
|
|
||
|
|
The hardest part for me is I have proudly been a Google Cloud Expert since I transitioned from the Cloud Innovators Champions program. I've been less thrilled with some of the AI things lately, but I was a huge supporter and involved with Gemini CLI and Project IDX to Firebase.
|
||
|
|
|
||
|
|
However, I cannot do a program for funsies that might cost me the same as a new Buick. And if the lag in billing really is 6 hours (as I personally experienced), I just cannot trust using the AI tools unless I use a key, do a demo, then delete that key right away.
|
||
|
|
|
||
|
|
I never wanted to get paid, or make a profit talking about Cloud systems, including Google - but I don't want it to impact my families finances. yesterday i was to the point of wanting to throw up thinking I just jeopardized our family summer vacation because a bad actor was using a hacked key.
|
||
|
|
|
||
|
|
And since I cannot know _how_ they got in, I now have some very serious trust issues. This could mean quitting GCP and GDE all together if push comes to shove. I can't be put into positions of doing free evangelism and then getting a massive bill.
|
||
|
|
|
||
|
|
I'll have to share how things end up. As it stands I'm feeling like trash and loading up on coffee...
|