I had my GCP quota algorithmically set to 0 after spending 6 months working with them to launch a startup.
I went through a ton of hoops to get approval for our quota. We sent them system diagrams, code samples, financial reports, growth predictions, etc. It was months of back and forth. I'll also add that it was very annoying because they auto-reject your quota request if you don't respond to their emails within 48 hours but their responses take 1-3 weeks. In any case, after 6 months, they eventually approved us for our quota, we launched, and they shut us down to 0 quota across all services the instant our production app got traffic.
We contacted them again asking for help. We never got any human response. We got a boiler plate template a few times, but that was it.
I will never ever ever again use a cloud service where I can't guarantee that I can get good customer service. Unfortunately for a small business that means no big clouds like AWS, GCP, etc.
Has AWS support gone downhill in the last two years? I've worked with them in the past - as both an individual and a couple startups - I always reached a human. Issues weren't always resolved as quickly as I'd like but response times were short.
Working in small and medium businesses I've observed the same thing, and I've been quite satisfied with it. So I don't think it's really gone downhill, so GP's comment doesn't really resonate for me, but that isn't to negate their experience. Otoh I keep hearing horror stories about GCP and now I'm reluctant to try it.
Shitting on GCP is just popular on HN and always gets upvoted. AWS and Azure have royally fucked thousands of customers if you care to search for those writeups. My wild ass guess, considering posts like these have zero background details, is that they were careless with service account keys and their account got suspended for mining crypto or something. They also probably weren’t actually paying for support of any kind and that’s why no one is responding to them.
Can't say for GCP and Azure. But I run two small projects on AWS, with monthly billings of $8k and $150 respectively. I have always received very good support to any requests that I have made on both projects.
Nope. We had been testing in our development and staging environments for months. We were deploying to production the exact same stack and we got our quota revoked within about an hour. We must have tripped some random thing. We have absolutely no idea what I could have been though.
I have the same question. I always got human response after 24-48hours or after one round of messages (with an automated human or machine, not sure). But so far, across 3 accounts and a dozens of correspondence, I always got a human.
In the 2010s I always got an AWS support team to help.
Now I get handed off to an external partner of AWS certified contractors.
They are often terrible. They have no backend systems access and just run through the AWS equivalent of "reboot it", "defrag your disk". Basically trying to find an issue in my pipeline. Which they never do because it's the same TF scripts used for years.
Only once we waste time going through the motions do I get passed up to someone who can actually correct the backend issue in the AWS stack itself.
Tbf though I rarely ever have to contact AWS support at this point. The few times I have in the last 2-3 was due to issues after they rolled out an update or with a newer service we wanted to use.
Never have issues with stable services like S3, ECS, EKS, or RDS.
My last experience was with someone giving me answers that where clearly in part LLM generated: It contained payloads/configs that did not even match the actual API
> They have no backend systems access and just run through the AWS equivalent of "reboot it", "defrag your disk"
To be fair I would bet money that the overwhelmingly vast majority of support tickets are exactly those kind of issues, and ones that refer to actual bugs on their end are, comparatively, extremely rare, and should have to be escalated through normal procedures to weed out common problems.
if you want best support (while staying with big cloud) then Microsoft is the best .
Azure has its flaws but Microsoft puts a lot of people and effort behind it . We are not that large but there are so many instances where Microsoft reps will come in call with our customers or their people working with common customers will help out etc.
AWS has a done a decent job of taking enterprise business seriously last 10 years. you can get human support but generally they will charge you , I.e if better support you want you have to pay for premium support plans .
They are constrained unlike MS they don’t have non-cloud large enterprise business relationships for decades M365 or AD etc that helps with building the enterprise DNA.
In all three clouds it works best if you don’t buy directly, buy through a partner reseller , who both have the relationships to the CSP and have the people to work with you .
MS is the same network were even their lead engineers answer "well, uhh create a new account and hope you're not banned", when it comes to fixing a illegitimate ban issue.
None of the biggies are good. None of them.
You're better off building your own data enter. Can't believe I'm saying that, but I am. And it doesnt have to be acres and MW and water cooled. It can be a 42U rack.
Hell, I'm a homeowner and have 27U rack with 10U full, battery backup, solar, fiber and a backup internet connection, and stuff.
A small business could easy do this and own the hardware and software to their enterprise. In fact, they probably should. Helps prevent rug pulls!
Now I am curious what is the realistic price a business would expect to put down for a full rack. Say UPS, switch, 4-8U storage, and the rest CPU compute. Without entertaining GPUs, I bet you can get very respectably speced 1-2U servers for $5k a pop. So few hundred thousand probably gets you just an unbelievable amount of horsepower.
You can get a petabyte for like $14k now. Make a massive SAN. Even 10G networking is reasonable.
Compute? You can run stuff like Proxmox in commercial mode and get tons of features for really reasonable price.
Ram is now the big nasty, but I'm thinking with the 500 billion USD dropped on this circlejerk economy, ram will come back down.
You can also get a few graphics cards for AI stuff, but I'd constrain it to actual dedicated reasons, rather than some "AI everywhere".
Backups can be tricky. Run tape, but also run encrypted S3 backups remote, for 'holyshiteverythingsgone' reasons.
I used to be syseng for a small dev company. They had 3 racks by a local MSP, and was grossly mismanaged. Could have did everything in 1.5 racks. I have pictures, and you'd be aghast.
Quota for what? In my experience the GCP service quotas are pretty sensible and if you’re running up against them you’re either dealing with unusual levels of traffic or (more often) you’re just using that service incorrectly.
The quota we needed increased far beyond the usual was the YouTube API. The startup was a media editing and publishing tool, with a feature to upload videos to YouTube on your behalf. Uploading a video requires a ton of quota, which they gave us.
Regardless, dropping all quotas to 0 effectively killed our GCP account.
How many services have meaningful SLAs for extreme downtime?
Github and (parts of) AWS will give you a small discount at 0.1% downtime, a bigger discount at 1% downtime, and AWS will refund the whole month for 5% downtime. But beyond that they don't care. If a particular customer gets no service at all then their entire $0 gets refunded and that's it.
If you werent willing to pay for an SLA, and they clearly werent going to offer one to you… why is it surprising if literally no promises were made in writing?
Why would they intentionally lose money on your private commercial activity without even that?
They denied my request for a service account quota increase even though my use case[1] was literally straight from their documentation. They only increased it after I complained on Twitter and got retweeted by Corey Quinn.
> In my experience the GCP service quotas are pretty sensible and if you’re running up against them you’re either dealing with unusual levels of traffic or (more often) you’re just using that service incorrectly.
Well 0 is not sensible, and who cares if it's weird if they got detailed approval and they're paying for it.
Sure, but the comment is so vague I’m skeptical the OP knew what they were doing in the first place, or it happened exactly as they wrote. Maybe a service quota was reset to the default? But just set to zero? Doesn’t pass the sniff test.
"... and they're paying for it..." - that might be the exact issue. Google has no way to ensure that these small shops and startups will pay their bill, so quotas are used to prevent the company from running up a large bill they won't be able to pay.
I see a bunch of threads on reddit about startups accidentally going way over budget and then asking for credits back.
This doesn't at all mean the startups have bad intent, but things happen and Google doesn't want to deal with a huge collection issue.
If someone rolled up to your gas station and wanted to pump 10,000 gallons of gas but only pay you next month - would you allow it?
Well that is kind of a problem of their own making. The clouds refuse to entertain the prospect of pre-paying for services/having some sort of hard spending limits because they know that over-allocation is probably driving a decent amount of revenue.
That’s not how quotas work in GCP. Google sets quotas for certain APIs for interacting with GCP itself, like how many VMs you can create per second. They’re not billable. Sometimes these quotas can be be increased if you need them to be. But the way op described it makes no sense.
> they auto-reject your quota request if you don't respond to their emails within 48 hours but their responses take 1-3 weeks
It boggles my mind anyone would base their business on their good will. By now it should be obvious that companies with a huge number of customers don't care about individual cases that much for obvious reasons. That's why they cut on customer support. You get much better support with smaller companies where you (as an individual or business) are much more important to them.
What did your account manager say about this. Getting this interaction right is the core of their job, enabling your business on the platform so you spend more money. With this bad an interaction I'd have asked for a new account manager.
A colleague had a similar quota issue. 4 times quota restoration request was rejected. Upon the final request he put “women owned startup helping underprivileged kids” and it was approved.
So it isn't I take it from your answer? If that's the case, not only did your colleague abuse protections for the vulnerable population, and possibly break the law by misrepresenting the legal aspects of your startup (IANAL), but you also didn't see anything wrong with potentially fucking over actual women owned businesses helping underprivileged kids (which I reckon is infinitely more valuable to society than whatever startup your colleague has) and even go as far as publicizing it on this high traffic website as a tip. This kind of behaviour is exactly the reason these protections exist, but big tech never gave enough of a fuck to do anything except virtue signal and democrats are too impotent to actually make them do it.
What Google account? Is it personal Gmail? Or your academic account? Are you using this for personal reasons or professional or commercial reasons? What kind of payment method is attached? What was your level of usage? Any idea why you were suspended initially?
Because it could be that Google is reviewing your appeal and simply shadow-denying it, and you haven't provided the right information to make it look legit. E.g. if they think you're a spammer or mining crypto or they think you're creating additional free accounts to use free credits, they're obviously not going to tell you what makes them think that.
But if this is for university-related work, and your university purchases IT+cloud services from Google (as they probably do), talk to your IT department so they can get you in touch with their institution-level support. Obviously, for the attached Google sales rep, the last thing they want is a CS researcher losing access to GCP.
> I am a CS researcher at UC Berkeley. This has seriously impacted my work.
I would try to get help from your department. Somewhere within CS and CS-adjacent departments at Berkeley there’s likely to be someone with an official or unofficial connection to Google that can get you in touch with a human to at least clarify the situation.
In my last company, we _did_ pay for Google Cloud support and when BigQuery jobs started to fail randomly, causing huge trouble producing critical reports, the response was essentially "we are investigating", "we have identified the issue", and "please wait for it to be fixed". Hardly what I would call support. They couldn't care less.
Not sure why this is getting downvoted. This may be snark, but this is 100% needed in the world we live in today. It is a fact of today's world that individuals have no leverage over these companies. I can understand why big companies, who have leverage, buy their services. But I don't understand why individuals, who have no leverage, buy their services and build their profession and livelihood around them. Any day, they can cut you off from their services. You are being irresponsible to yourself if you put all your eggs in these big tech baskets.
We seriously do need this kind of research and compelling articles that argue why relying on these big tech cloud services is harmful for individuals.
The only reliable way to get things fixed with these large companies, is to have a direct point of contact to them. I have one. You can find my email in my profile, ping me and I will try to get a human in the loop for you.
It's gone. No human will ever respond to you. That's how these companies operate. From here, you realistically have two options.
1. Forget the account and move on. You could create a new one, but nobody can tell how long it would take before that gets suspended as well.
2. If the suspension has a tangible negative impact on your profession, hire a lawyer and get proper legal advice.
Most important of all, let this be a lesson for you and your colleagues. It is a terrible idea to let any critical part of your life depend on unregulated industries that can wipe out someone's livelihood at the whim of machine learning systems. Learn this lesson and pass it on to everyone you know.
As an individual, you are nobody to Google and you have no leverage. It is reckless to build your livelihood or profession around their platforms. If you were a company, your team could speak to an account manager and negotiate. As an individual, your only real leverage is legal action.
Stories like this appear every month. I don't know how many more it will take before it becomes best practice not to depend on these utterly abominable rackets for anything critical.
> let this be a lesson for you and your colleagues.
Nah, big tech infiltrates everything, it’s 100% their fault. Why did everyone switch to webmail? Why did we gravitate to web apps? Big tech persuaded us all to do it.
With big promises comes great responsibility, and the stuff in the fine print doesn’t count. It’s not ethical to invite dependency and randomly kneecap people; it shouldn’t be legal either.
100% agree with you. The big techs are definitely 100% at fault. But you know, fool me once, shame on me. Fool me twice...
I mean, we get these stories every month. Yes, 100% it is not ethical to randomly kneecap people. But let's be honest. Nobody is working on making these big tech companies accountable for the potentially devastating, algorithm-driven decisions they take. How many more times do they have to fool us before we all realize that it's time to move away from them?
All I ask from you, myself and all the tech folks here is to learn from these lessons and pass them on to everyone around you. With how things are today, it is reckless to depend on these big tech cloud services for your livelihood and profession. If you're working for a company where the company has leverage, all good. But as an individual, you should stay away from these big tech companies, because they can screw up your life any day, without warning and without recourse.
With email in particular, it's not like chocolate. If you self-host email, or use any other cloud host besides Microsoft or Google, then Microsoft and Google will randomly fail to deliver emails you send to their users, even though you have SPF, DKIM, DMARC, etc. set up exactly right.
Seriously #2 is your only recourse. Download the terms of service / your service contract , highlight their violations and send them a certified letter about breach of contract and that you intend legal recourse.
Cloud lock-in is real and requires a lot of forethought to avoid or at least mitigate. on the LLM side I have pushed my last two companies to always have at least 3 vendors for hosting and a system to fail over instantly or even load balance based on different criteria. It has paid massive dividends. I wish that philosophy was easier to implement at the cloud level for all services.
UC Berkeley gets much of its IT infrastructure from Google in a big and expensive contract. Perhaps some of the campus staff could try to negotiate on your behalf?
I was wondering exactly how that it related. We as a German university are actually still shaping our cloud strategy and such stories are kind of worst case scenario to protect our research. However, I would imagine that stuff bought via our official contracts (actual there is even a European wide framework contract for all unis) protects a bit. Still we are in an even more complex situation because individual researchers could be hit by US sanctions, so it difficult to also understand the reverse risk. I think StackIT (Schwartz/LIDL) will run Google PaaS/SaaS for further protections. In the end, however, building your business (in OP's/our case is research) on cloud seens to remain a gamble without a concrete exit strategy.
This is an asinine question. Even if you build agnostic solutions (like a docker image), you have storage resources, networks, configs, ACLs, snapshots and more all trapped inside GCP. we’re human — we forget to backup things, or push important commits . And we know cloud solutions quickly develop lock-in – even a simple cloud DB instance locks you into the vendors config .
So there are at least a dozen perfectly good reasons this guy is panicking that his account was suddenly revoked without warning.
This is such a repetitious issue that I wonder why there has been no class action suits so far?
I think documenting these cases somewhere, and targeting not just Alphabet but all the other "we're too big to support little people like you" companies would be a good idea. I don't think the pay out would be significant, but the punitive impact might change things.
OP is not clear , but it looks like GCP suspension not Google one (I.e email android etc)
All clouds reject a lot of businesses for their services for variety of reasons and there are alternatives in the market unlike say a Google account suspension .
I don’t think class action is feasible for cloud computing suspension (unless of course they are discriminating against a protected class etc)
i was thinking more in terms of tort law or contract law. They probably have a disclaimer and Tos that addresses all that, but given enough plaintiffs and their market dominance, it might amount to possible deliberate/calculated financial harm. It might be enough to not get thrown out of court at least. They can reject business for any reason, but once someone relies on their services for their business, there is always a certain expectation of continued service, and in the event of service termination, they may not need to explain themselves, but they must accommodate reasonable requests to transfer data, customers,etc.. elsewhere. Otherwise it sounds like tortiuous interference.
> Tortious interference, also known as intentional interference with contractual relations, in the common law of torts, occurs when one person intentionally damages someone else's contractual or business relationships with a third party, causing economic harm.
In this case, people who use GCP have customers and other contractual relationships. Google's termination of service interfered with that. Google also doing this as a matter of standard business practice indicates that they are aware that their action will interfere with people's contractual obligations (well common sense should tell them that anyways).
You can't force someone to sign a contract with you that says "if I interfere with your future contracts with arbitrary third parties on purpose, you can't sue me". The deliberate part is crucial from what I understand. If their decision making couldn't have accounted for the interference, and the interference wasn't calculated as an acceptable risk, there is no issue. But the plaintiffs can claim that repeated social media posts and acknowledgements of said interference by Google over the years means it's enough grounds for a suit. and a suit will mean discovery, google will have go hand over internal documents, depose employees,etc...
In the end, this might be more costly to companies like Google than just giving customers a grace period to move elsewhere before termination.
Obligatory: IANAL, I'm just a guy using big words I barely understand.
Call me cynical but I have little to no hope that even class actions would solve anything. These companies have become so big that they can take one class action after another for years to come without making a dent in their financials and without bringing any change to their operating procedures.
I'm just expecting them to change their calculus. Right now it costs them nothing to randomly shut down accounts. If it had some cost, perhaps some minor notice, accommodation,etc.. however automated might be worth just the man hours spent on lawsuits.
Of all the posts like this I’ve seen the customers are always 1) extremely scant on details about what they were using GCP for or why they were suspended, and more importantly 2) never actually paying for support.
Having worked with a fair few academics, I’m guessing they lost track of their service account keys and the account got suspended for crypto mining.
There have been plenty of posts where the reason was apparent. One i recall was caused by a guy having malware on his phone, and he happened to use a work email on his phone, so the entire GWS organization was banned, shutting down the company's operations.
I had a bad Cloudflare experience. So, my card on file got no balance one day (my bad, I forgot to update to a new card), and they just turned off the services.
They somehow managed to charge partial amount (like 80% of the bill), but decided to turn off everything anyway, even the services that could be covered by those 80%. They turned off what they offer for free, and we were unable to change the setting, like instead of their CDN point traffic to an S3 bucket, etc.
When they do that they basically freeze your account. I mean you cannot provide a new card to pay the outstanding bill, or do anything at all actually. You're not welcomed here anymore. Locked out. That's is a terrible way to react to a payment failure after being a paying customer for a few years.
It was hard to reach the support, and it took multiple days until I found someone on Reddit who looked at our ticket and it eventually helped.
PS I had much worse experience with GCP after being a loyal customer of them for like 15 years, so Clouflare is good.
Yeah, we were looking for image CDN services (with resizing etc). Asked CloudFlare and they said $200 a month, everyone else was saying $3-5k per month.
Had a sales call with CloudFlare, they said yes they do flat rate billing and it's only $200 a month for all we can eat image hosting.
We of course called bullshit and third time around (talking to human sales reps) we said, just to get it in writing, we can do X bandwidth/Y images for $200 a month?
...oh errr, no, that would be more like $7k.
Thankfully we smelled bullshit and didn't take sales word for it. We'd have built an integration and started paying only to be bitten a month or two later when they readjusted our pricing. They basically refuse to talk about real pricing until you're already paying $200/m and locked in.
We ended up hosting our own on GKE for $500-$1k/m.
Is HN full of amateurs, AI slop, or people who don't use GCP at all? You need to spend a minimum of $29 for the Standard support which gives 4-8 hour response depending on the case type.
Xbox & Discord are the only 2 services I’ve seen handle bans with adequate transparency (yes there is still room to improve). Both offer a ban status tab ranging from hand-slap to giga-banned , allowing you to have some level of warning before being booted.
Given how dependent we all are on these services: we run our businesses and our lives, it’s despicable that more due process and transparency is not offered for shadow and proper bans like this.
I had my GCP quota algorithmically set to 0 after spending 6 months working with them to launch a startup.
I went through a ton of hoops to get approval for our quota. We sent them system diagrams, code samples, financial reports, growth predictions, etc. It was months of back and forth. I'll also add that it was very annoying because they auto-reject your quota request if you don't respond to their emails within 48 hours but their responses take 1-3 weeks. In any case, after 6 months, they eventually approved us for our quota, we launched, and they shut us down to 0 quota across all services the instant our production app got traffic.
We contacted them again asking for help. We never got any human response. We got a boiler plate template a few times, but that was it.
I will never ever ever again use a cloud service where I can't guarantee that I can get good customer service. Unfortunately for a small business that means no big clouds like AWS, GCP, etc.
Yes, I am bitter.
Has AWS support gone downhill in the last two years? I've worked with them in the past - as both an individual and a couple startups - I always reached a human. Issues weren't always resolved as quickly as I'd like but response times were short.
Working in small and medium businesses I've observed the same thing, and I've been quite satisfied with it. So I don't think it's really gone downhill, so GP's comment doesn't really resonate for me, but that isn't to negate their experience. Otoh I keep hearing horror stories about GCP and now I'm reluctant to try it.
Shitting on GCP is just popular on HN and always gets upvoted. AWS and Azure have royally fucked thousands of customers if you care to search for those writeups. My wild ass guess, considering posts like these have zero background details, is that they were careless with service account keys and their account got suspended for mining crypto or something. They also probably weren’t actually paying for support of any kind and that’s why no one is responding to them.
Can't say for GCP and Azure. But I run two small projects on AWS, with monthly billings of $8k and $150 respectively. I have always received very good support to any requests that I have made on both projects.
Nope. We had been testing in our development and staging environments for months. We were deploying to production the exact same stack and we got our quota revoked within about an hour. We must have tripped some random thing. We have absolutely no idea what I could have been though.
Last time I checked you had to pay something like $400/mo extra for legit, timely human support from AWS.
I have the same question. I always got human response after 24-48hours or after one round of messages (with an automated human or machine, not sure). But so far, across 3 accounts and a dozens of correspondence, I always got a human.
It has!
In the 2010s I always got an AWS support team to help.
Now I get handed off to an external partner of AWS certified contractors.
They are often terrible. They have no backend systems access and just run through the AWS equivalent of "reboot it", "defrag your disk". Basically trying to find an issue in my pipeline. Which they never do because it's the same TF scripts used for years.
Only once we waste time going through the motions do I get passed up to someone who can actually correct the backend issue in the AWS stack itself.
Tbf though I rarely ever have to contact AWS support at this point. The few times I have in the last 2-3 was due to issues after they rolled out an update or with a newer service we wanted to use.
Never have issues with stable services like S3, ECS, EKS, or RDS.
My last experience was with someone giving me answers that where clearly in part LLM generated: It contained payloads/configs that did not even match the actual API
> They have no backend systems access and just run through the AWS equivalent of "reboot it", "defrag your disk"
To be fair I would bet money that the overwhelmingly vast majority of support tickets are exactly those kind of issues, and ones that refer to actual bugs on their end are, comparatively, extremely rare, and should have to be escalated through normal procedures to weed out common problems.
if you want best support (while staying with big cloud) then Microsoft is the best .
Azure has its flaws but Microsoft puts a lot of people and effort behind it . We are not that large but there are so many instances where Microsoft reps will come in call with our customers or their people working with common customers will help out etc.
AWS has a done a decent job of taking enterprise business seriously last 10 years. you can get human support but generally they will charge you , I.e if better support you want you have to pay for premium support plans .
They are constrained unlike MS they don’t have non-cloud large enterprise business relationships for decades M365 or AD etc that helps with building the enterprise DNA.
In all three clouds it works best if you don’t buy directly, buy through a partner reseller , who both have the relationships to the CSP and have the people to work with you .
Sorry, no.
MS is the same network were even their lead engineers answer "well, uhh create a new account and hope you're not banned", when it comes to fixing a illegitimate ban issue.
None of the biggies are good. None of them.
You're better off building your own data enter. Can't believe I'm saying that, but I am. And it doesnt have to be acres and MW and water cooled. It can be a 42U rack.
Hell, I'm a homeowner and have 27U rack with 10U full, battery backup, solar, fiber and a backup internet connection, and stuff.
A small business could easy do this and own the hardware and software to their enterprise. In fact, they probably should. Helps prevent rug pulls!
Now I am curious what is the realistic price a business would expect to put down for a full rack. Say UPS, switch, 4-8U storage, and the rest CPU compute. Without entertaining GPUs, I bet you can get very respectably speced 1-2U servers for $5k a pop. So few hundred thousand probably gets you just an unbelievable amount of horsepower.
You can get a petabyte for like $14k now. Make a massive SAN. Even 10G networking is reasonable.
Compute? You can run stuff like Proxmox in commercial mode and get tons of features for really reasonable price.
Ram is now the big nasty, but I'm thinking with the 500 billion USD dropped on this circlejerk economy, ram will come back down.
You can also get a few graphics cards for AI stuff, but I'd constrain it to actual dedicated reasons, rather than some "AI everywhere".
Backups can be tricky. Run tape, but also run encrypted S3 backups remote, for 'holyshiteverythingsgone' reasons.
I used to be syseng for a small dev company. They had 3 racks by a local MSP, and was grossly mismanaged. Could have did everything in 1.5 racks. I have pictures, and you'd be aghast.
@dang why was my response autokilled?
Having run a small business on one of the big clouds for almost 10 years now building your own data center is insane advice.
>easy
Hell no
Quota for what? In my experience the GCP service quotas are pretty sensible and if you’re running up against them you’re either dealing with unusual levels of traffic or (more often) you’re just using that service incorrectly.
The quota we needed increased far beyond the usual was the YouTube API. The startup was a media editing and publishing tool, with a feature to upload videos to YouTube on your behalf. Uploading a video requires a ton of quota, which they gave us.
Regardless, dropping all quotas to 0 effectively killed our GCP account.
Interesting. I guess we’ve learned an important lesson in not building businesses around APIs that don’t have an SLA…
How many services have meaningful SLAs for extreme downtime?
Github and (parts of) AWS will give you a small discount at 0.1% downtime, a bigger discount at 1% downtime, and AWS will refund the whole month for 5% downtime. But beyond that they don't care. If a particular customer gets no service at all then their entire $0 gets refunded and that's it.
That was just a feature of the product to be helpful. Not a core function at all.
If you werent willing to pay for an SLA, and they clearly werent going to offer one to you… why is it surprising if literally no promises were made in writing?
Why would they intentionally lose money on your private commercial activity without even that?
They denied my request for a service account quota increase even though my use case[1] was literally straight from their documentation. They only increased it after I complained on Twitter and got retweeted by Corey Quinn.
[1] https://www.agwa.name/blog/post/accessing_your_customers_goo...
> Quota for what?
Sure, I'm interested too.
> In my experience the GCP service quotas are pretty sensible and if you’re running up against them you’re either dealing with unusual levels of traffic or (more often) you’re just using that service incorrectly.
Well 0 is not sensible, and who cares if it's weird if they got detailed approval and they're paying for it.
Sure, but the comment is so vague I’m skeptical the OP knew what they were doing in the first place, or it happened exactly as they wrote. Maybe a service quota was reset to the default? But just set to zero? Doesn’t pass the sniff test.
"... and they're paying for it..." - that might be the exact issue. Google has no way to ensure that these small shops and startups will pay their bill, so quotas are used to prevent the company from running up a large bill they won't be able to pay.
I see a bunch of threads on reddit about startups accidentally going way over budget and then asking for credits back.
This doesn't at all mean the startups have bad intent, but things happen and Google doesn't want to deal with a huge collection issue.
If someone rolled up to your gas station and wanted to pump 10,000 gallons of gas but only pay you next month - would you allow it?
Well that is kind of a problem of their own making. The clouds refuse to entertain the prospect of pre-paying for services/having some sort of hard spending limits because they know that over-allocation is probably driving a decent amount of revenue.
I dont really understand ops problem as I've been able to set monthly limits on expenditure. Seems trivial to setup.
Are you sure you're not thinking of billing alerts? AFAIK there is still no way to set spending limits at GCP.
I'll admit it's a bit obscure But there is info here.
https://docs.cloud.google.com/apis/docs/capping-api-usage
Or you can do it programmatically.
https://docs.cloud.google.com/billing/docs/how-to/disable-bi...
That’s not how quotas work in GCP. Google sets quotas for certain APIs for interacting with GCP itself, like how many VMs you can create per second. They’re not billable. Sometimes these quotas can be be increased if you need them to be. But the way op described it makes no sense.
> they auto-reject your quota request if you don't respond to their emails within 48 hours but their responses take 1-3 weeks
It boggles my mind anyone would base their business on their good will. By now it should be obvious that companies with a huge number of customers don't care about individual cases that much for obvious reasons. That's why they cut on customer support. You get much better support with smaller companies where you (as an individual or business) are much more important to them.
What did your account manager say about this. Getting this interaction right is the core of their job, enabling your business on the platform so you spend more money. With this bad an interaction I'd have asked for a new account manager.
As a petty person I would at least attempt to push this through small claims court.
Add up the amount you lost moving to another platform, value your time appropriately, and submit your claim.
A judge will decide if it’s fair, and you can be awarded ~10k in most places.
If they don’t pay for some reason you can put a lien on their property.
A colleague had a similar quota issue. 4 times quota restoration request was rejected. Upon the final request he put “women owned startup helping underprivileged kids” and it was approved.
It can’t hurt.
Was the startup actually women owned and helping underprivileged kids?
have you met a vegan that doesn’t tell you they are vegan every day?
This is a stereotype, and I'd say a bit harmful. There are way more people complaining about such vegans.
I guess stereotypes are a conspiracy ? Besides, you get the illustration, obviously .
So it isn't I take it from your answer? If that's the case, not only did your colleague abuse protections for the vulnerable population, and possibly break the law by misrepresenting the legal aspects of your startup (IANAL), but you also didn't see anything wrong with potentially fucking over actual women owned businesses helping underprivileged kids (which I reckon is infinitely more valuable to society than whatever startup your colleague has) and even go as far as publicizing it on this high traffic website as a tip. This kind of behaviour is exactly the reason these protections exist, but big tech never gave enough of a fuck to do anything except virtue signal and democrats are too impotent to actually make them do it.
Call it prompt engineering
So much information is missing from this.
What Google account? Is it personal Gmail? Or your academic account? Are you using this for personal reasons or professional or commercial reasons? What kind of payment method is attached? What was your level of usage? Any idea why you were suspended initially?
Because it could be that Google is reviewing your appeal and simply shadow-denying it, and you haven't provided the right information to make it look legit. E.g. if they think you're a spammer or mining crypto or they think you're creating additional free accounts to use free credits, they're obviously not going to tell you what makes them think that.
But if this is for university-related work, and your university purchases IT+cloud services from Google (as they probably do), talk to your IT department so they can get you in touch with their institution-level support. Obviously, for the attached Google sales rep, the last thing they want is a CS researcher losing access to GCP.
Also, why wait 2 years to take it public?
> I am a CS researcher at UC Berkeley. This has seriously impacted my work.
I would try to get help from your department. Somewhere within CS and CS-adjacent departments at Berkeley there’s likely to be someone with an official or unofficial connection to Google that can get you in touch with a human to at least clarify the situation.
> I am a CS researcher at UC Berkeley. This has seriously impacted my work.
Can I suggest a topic for your next research? "Cloud exascalers and their negative impact on the society"
Or “Why I should have been paying for support if my entire career depended on it.”
In my last company, we _did_ pay for Google Cloud support and when BigQuery jobs started to fail randomly, causing huge trouble producing critical reports, the response was essentially "we are investigating", "we have identified the issue", and "please wait for it to be fixed". Hardly what I would call support. They couldn't care less.
There are so many things with this statement I don't even know where to start. I hope you're being sarcastic.
Not sure why this is getting downvoted. This may be snark, but this is 100% needed in the world we live in today. It is a fact of today's world that individuals have no leverage over these companies. I can understand why big companies, who have leverage, buy their services. But I don't understand why individuals, who have no leverage, buy their services and build their profession and livelihood around them. Any day, they can cut you off from their services. You are being irresponsible to yourself if you put all your eggs in these big tech baskets.
We seriously do need this kind of research and compelling articles that argue why relying on these big tech cloud services is harmful for individuals.
Unless you know someone at Google with a little swagger to get things done you’re shit out of luck.
Our GCP account manager was always pretty good at solving these sorts of problems for us at my last company.
The role of GCP is to help enterprises negotiate better deals out of Azure/AWS. Why would anyone actually use it is beyond me...
The only reliable way to get things fixed with these large companies, is to have a direct point of contact to them. I have one. You can find my email in my profile, ping me and I will try to get a human in the loop for you.
It's gone. No human will ever respond to you. That's how these companies operate. From here, you realistically have two options.
1. Forget the account and move on. You could create a new one, but nobody can tell how long it would take before that gets suspended as well.
2. If the suspension has a tangible negative impact on your profession, hire a lawyer and get proper legal advice.
Most important of all, let this be a lesson for you and your colleagues. It is a terrible idea to let any critical part of your life depend on unregulated industries that can wipe out someone's livelihood at the whim of machine learning systems. Learn this lesson and pass it on to everyone you know.
As an individual, you are nobody to Google and you have no leverage. It is reckless to build your livelihood or profession around their platforms. If you were a company, your team could speak to an account manager and negotiate. As an individual, your only real leverage is legal action.
Stories like this appear every month. I don't know how many more it will take before it becomes best practice not to depend on these utterly abominable rackets for anything critical.
> let this be a lesson for you and your colleagues.
Nah, big tech infiltrates everything, it’s 100% their fault. Why did everyone switch to webmail? Why did we gravitate to web apps? Big tech persuaded us all to do it.
With big promises comes great responsibility, and the stuff in the fine print doesn’t count. It’s not ethical to invite dependency and randomly kneecap people; it shouldn’t be legal either.
100% agree with you. The big techs are definitely 100% at fault. But you know, fool me once, shame on me. Fool me twice...
I mean, we get these stories every month. Yes, 100% it is not ethical to randomly kneecap people. But let's be honest. Nobody is working on making these big tech companies accountable for the potentially devastating, algorithm-driven decisions they take. How many more times do they have to fool us before we all realize that it's time to move away from them?
All I ask from you, myself and all the tech folks here is to learn from these lessons and pass them on to everyone around you. With how things are today, it is reckless to depend on these big tech cloud services for your livelihood and profession. If you're working for a company where the company has leverage, all good. But as an individual, you should stay away from these big tech companies, because they can screw up your life any day, without warning and without recourse.
Mars persuaded me to buy their chocolate. I couldn't help myself.
With email in particular, it's not like chocolate. If you self-host email, or use any other cloud host besides Microsoft or Google, then Microsoft and Google will randomly fail to deliver emails you send to their users, even though you have SPF, DKIM, DMARC, etc. set up exactly right.
Seriously #2 is your only recourse. Download the terms of service / your service contract , highlight their violations and send them a certified letter about breach of contract and that you intend legal recourse.
Did you tweet them - https://x.com/googlecloud?
If it's so important, maybe talk with a lawyer.
Cloud lock-in is real and requires a lot of forethought to avoid or at least mitigate. on the LLM side I have pushed my last two companies to always have at least 3 vendors for hosting and a system to fail over instantly or even load balance based on different criteria. It has paid massive dividends. I wish that philosophy was easier to implement at the cloud level for all services.
UC Berkeley gets much of its IT infrastructure from Google in a big and expensive contract. Perhaps some of the campus staff could try to negotiate on your behalf?
I was wondering exactly how that it related. We as a German university are actually still shaping our cloud strategy and such stories are kind of worst case scenario to protect our research. However, I would imagine that stuff bought via our official contracts (actual there is even a European wide framework contract for all unis) protects a bit. Still we are in an even more complex situation because individual researchers could be hit by US sanctions, so it difficult to also understand the reverse risk. I think StackIT (Schwartz/LIDL) will run Google PaaS/SaaS for further protections. In the end, however, building your business (in OP's/our case is research) on cloud seens to remain a gamble without a concrete exit strategy.
How come your business or research are so tied to GCP? What about other providers?
This is an asinine question. Even if you build agnostic solutions (like a docker image), you have storage resources, networks, configs, ACLs, snapshots and more all trapped inside GCP. we’re human — we forget to backup things, or push important commits . And we know cloud solutions quickly develop lock-in – even a simple cloud DB instance locks you into the vendors config .
So there are at least a dozen perfectly good reasons this guy is panicking that his account was suddenly revoked without warning.
It's not asinine. This suspension happened two years ago. I don't see any sudden panicking.
Appeals take time. And it’s not an uncommon case . It doesn’t make his desire to recover the resources any less valid .
GCP fired my neighbor just before they were about to receive a promotion by attrition fill, but instead was replaced by offshore, lower tier support.
This is such a repetitious issue that I wonder why there has been no class action suits so far?
I think documenting these cases somewhere, and targeting not just Alphabet but all the other "we're too big to support little people like you" companies would be a good idea. I don't think the pay out would be significant, but the punitive impact might change things.
OP is not clear , but it looks like GCP suspension not Google one (I.e email android etc)
All clouds reject a lot of businesses for their services for variety of reasons and there are alternatives in the market unlike say a Google account suspension .
I don’t think class action is feasible for cloud computing suspension (unless of course they are discriminating against a protected class etc)
i was thinking more in terms of tort law or contract law. They probably have a disclaimer and Tos that addresses all that, but given enough plaintiffs and their market dominance, it might amount to possible deliberate/calculated financial harm. It might be enough to not get thrown out of court at least. They can reject business for any reason, but once someone relies on their services for their business, there is always a certain expectation of continued service, and in the event of service termination, they may not need to explain themselves, but they must accommodate reasonable requests to transfer data, customers,etc.. elsewhere. Otherwise it sounds like tortiuous interference.
https://en.wikipedia.org/wiki/Tortious_interference
> Tortious interference, also known as intentional interference with contractual relations, in the common law of torts, occurs when one person intentionally damages someone else's contractual or business relationships with a third party, causing economic harm.
In this case, people who use GCP have customers and other contractual relationships. Google's termination of service interfered with that. Google also doing this as a matter of standard business practice indicates that they are aware that their action will interfere with people's contractual obligations (well common sense should tell them that anyways).
You can't force someone to sign a contract with you that says "if I interfere with your future contracts with arbitrary third parties on purpose, you can't sue me". The deliberate part is crucial from what I understand. If their decision making couldn't have accounted for the interference, and the interference wasn't calculated as an acceptable risk, there is no issue. But the plaintiffs can claim that repeated social media posts and acknowledgements of said interference by Google over the years means it's enough grounds for a suit. and a suit will mean discovery, google will have go hand over internal documents, depose employees,etc...
In the end, this might be more costly to companies like Google than just giving customers a grace period to move elsewhere before termination.
Obligatory: IANAL, I'm just a guy using big words I barely understand.
> but the punitive impact might change things
Call me cynical but I have little to no hope that even class actions would solve anything. These companies have become so big that they can take one class action after another for years to come without making a dent in their financials and without bringing any change to their operating procedures.
I'm just expecting them to change their calculus. Right now it costs them nothing to randomly shut down accounts. If it had some cost, perhaps some minor notice, accommodation,etc.. however automated might be worth just the man hours spent on lawsuits.
Of all the posts like this I’ve seen the customers are always 1) extremely scant on details about what they were using GCP for or why they were suspended, and more importantly 2) never actually paying for support.
Having worked with a fair few academics, I’m guessing they lost track of their service account keys and the account got suspended for crypto mining.
There have been plenty of posts where the reason was apparent. One i recall was caused by a guy having malware on his phone, and he happened to use a work email on his phone, so the entire GWS organization was banned, shutting down the company's operations.
I have yet to see someone say why they were suspended.
I’ve always wondered why, this makes sense.
anything that has to do with google avoid at major cost
or have an alternative ready
for serious work -- don't use google & don't use google devices either
Plenty of GCP, AWS and Azure experiences in this thread, but any Cloudflare experiences?
I had a bad Cloudflare experience. So, my card on file got no balance one day (my bad, I forgot to update to a new card), and they just turned off the services.
They somehow managed to charge partial amount (like 80% of the bill), but decided to turn off everything anyway, even the services that could be covered by those 80%. They turned off what they offer for free, and we were unable to change the setting, like instead of their CDN point traffic to an S3 bucket, etc.
When they do that they basically freeze your account. I mean you cannot provide a new card to pay the outstanding bill, or do anything at all actually. You're not welcomed here anymore. Locked out. That's is a terrible way to react to a payment failure after being a paying customer for a few years.
It was hard to reach the support, and it took multiple days until I found someone on Reddit who looked at our ticket and it eventually helped.
PS I had much worse experience with GCP after being a loyal customer of them for like 15 years, so Clouflare is good.
I am going through a very similar issue with Cloudflare right now, and billing support is almost of no help.
Yeah, we were looking for image CDN services (with resizing etc). Asked CloudFlare and they said $200 a month, everyone else was saying $3-5k per month.
Had a sales call with CloudFlare, they said yes they do flat rate billing and it's only $200 a month for all we can eat image hosting.
We of course called bullshit and third time around (talking to human sales reps) we said, just to get it in writing, we can do X bandwidth/Y images for $200 a month?
...oh errr, no, that would be more like $7k.
Thankfully we smelled bullshit and didn't take sales word for it. We'd have built an integration and started paying only to be bitten a month or two later when they readjusted our pricing. They basically refuse to talk about real pricing until you're already paying $200/m and locked in.
We ended up hosting our own on GKE for $500-$1k/m.
The Lord giveth, and the Lord taketh away. Blessed be the name of the Lord. - Job 1:21
One of the many reasons I continue to degoogle and remove that garbage from my life wherever I can. So many cases like this.
Is HN full of amateurs, AI slop, or people who don't use GCP at all? You need to spend a minimum of $29 for the Standard support which gives 4-8 hour response depending on the case type.
Xbox & Discord are the only 2 services I’ve seen handle bans with adequate transparency (yes there is still room to improve). Both offer a ban status tab ranging from hand-slap to giga-banned , allowing you to have some level of warning before being booted.
Given how dependent we all are on these services: we run our businesses and our lives, it’s despicable that more due process and transparency is not offered for shadow and proper bans like this.