These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts
And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy.
Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there being accounts like that in existence already (some azure accounts are hardcapped by amount but ofc that’s not loudly advertised).
This might be speaking the obvious, but I think that the lack of half-decent cost controls is not intentionally malicious. There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money. I think it's the play between incompetence and having absolutely no incentive to do anything about it (which is still a form of malice).
I've used AWS for about 10 years and am by no means an expert, but I've seen all kinds of ugly cracks and discontinuities in design and operation among the services. AWS has felt like a handful of very good ideas, designed, built, and maintained by completely separate teams, littered by a whole ton of "I need my promotion to VP" bad ideas that build on top of the good ones in increasingly hacky ways.
And in any sufficiently large tech orgnization, there won't be anyone at a level of power who can rattle cages about a problem like this, who will want to be the one to do actually it. No "VP of Such and Such" will spend their political capital stressing how critical it is that they fix the thing that will make a whole bunch of KPIs go in the wrong direction. They're probably spending it on shipping another hacked-together service with Web2.0-- er. IOT-- er. Blockchai-- er. Crypto-- er. AI before promotion season.
> I think that the lack of half-decent cost controls is not intentionally malicious
It wasn't when the service was first created. What's intentionally malicious is not fixing it for years.
Somehow AI companies got this right form the get go. Money up front, no money, no tokens.
It's easy to guess why. Unlike hosting infra bs, inference is a hard cost for them. If they don't get paid, they lose (more) money. And sending stuff to collections is expensive and bad press.
> Somehow AI companies got this right form the get go. Money up front, no money, no tokens.
That’s not a completely accurate characterization of what’s been happening. AI coding agent startups like Cursor and Windsurf started by attracting developers with free or deeply discounted tokens, then adjusted the pricing as they figure out how to be profitable. This happened with Kiro too[1] and is happening now with Google’s Antigravity. There’s been plenty of ink spilled on HN about this practice.
[1] disclaimer: I work for AWS, opinions are my own
I think you’re talking about a different thing? The bad practice from AWS et al is that you post-pay for your usage, so usage can be any amount. With all the AI things I’ve seen, either:
- you prepay a fixed amount (“$200/mo for ChatGPT Max”)
- you deposit money upfront into a wallet, if the wallet runs out of cash then you can’t generate any more tokens
- it’s free!
I haven’t seen any of the major model providers have a system where you use as many tokens as you want and then they bill you, like AWS has.
> There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money.
I dunno, Aurora’s pricing structure feels an awful lot like that. “What if we made people pay for storage and I/O? And we made estimating I/O practically impossible?”
A year ago I did a back of the napkin calculation and was surprised when I realised Aurora would cost the same or more as my current RDS for Postgres setup. Any discussion with costs inevitably has someone chiming in with a "have you considered Aurora?" and I don't quite understand why it's so loved.
Because a. People by and large don’t run their own benchmarks b. Aurora does legitimately have some nice features, though a lot of them are artificially restricted from RDS (like the max volume size).
The biggest cool thing Aurora MySQL does, IMO, is maintain the buffer pool on restarts. Not just dump / restore, but actually keeps its residence. They split it out from the main mysqld process so restarting it doesn’t lose the buffer pool.
But all in all, IMO it’s hideously expensive, doesn’t live up to its promises, and has some serious performance problems due to the decision to separate compute and storage by a large physical distance (and its simplification down to redo log only): for example, the lack of a change buffer means that secondary indices have to be written synchronously. Similarly, since AWS isn’t stupid, they have “node-local” (it’s EBS) temporary storage, which is used to build the aforementioned secondary indices, among other things. The size of this is not adjustable, and simply scales with instance size. So if you have massive tables, you can quite easily run out of room in this temporary storage, which at best kills the DDL, and at worst crashes the instance.
Even if you are not currently hitting performance limits of your current engine, Aurora would maintain the same throughput and latency on smaller instance classes. Which is where the potential cost savings come from...
On top of that, with Aurora Serverless with variable and unpredictable workloads you could have important cost savings.
Except it isn’t nearly as fast as it claims [0]. And in real-world tests, I have never found it to beat RDS.
You can get an insane amount of performance out of a well-tuned MySQL or Postgres instance, especially if you’ve designed your schema to exploit your RDBMS’ strengths (e.g. taking advantage of InnoDB’s clustering index to minimize page fetches for N:M relationships).
And if you really need high performance, you use an instance with node-local NVMe storage (and deal with the ephemerality, of course).
Well the article you linked to....confirms that Aurora is faster than MySQL on equivalent hardware, especially for write-heavy workloads, just that the “5× faster” claim only holds under very specific benchmark conditions.
Also from the link...when MySQL is properly tuned, the performance gap narrows substantially but is still 1,5x to 3x for the workloads tested in the article something I would call massive.
The benchmarks were just that - synthetic benchmarks. I’ve ran actual production workloads against both, and Aurora never won. IFF you have an incredibly write-heavy workload, and you have few to no secondary indices, then Aurora might win; I’d also suggest you reconsider your RDBMS choice.
Most workloads are somewhere between 90:10 to 98:2 reads:writes, and most tables have at least one (if not more) secondary indices.
You’re of course welcome to disagree, but speaking as a DBRE who has used both MySQL and Postgres, RDS and Aurora in production, I’m telling you that Aurora does not win on performance.
> There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money.
Unfortunately, that's not correct. A multi-trillion dollar company most absolutely has not just such a person, but many departments with hundreds of people tasked with precisely that, maximizing revenue by exploting every dark pattern they can possibly think of.
It would be good to provide a factual basis for such a confident contradiction of the GP. This reads as “no, your opinion is wrong because my opinion is right”.
AWS isn't for tinkerers and doesn't have guard rails for them, that's it. Anybody can use it but it's not designed for you to spend $12 per month. They DO have cost anomaly monitoring, they give you data so you can set up your own alerts for usage or data, but it's not a primary feature because they're picking their customers and it isn't the bottom of the market hobbyist. There are plenty of other services looking for that segment.
I have budgets set up and alerts through a separate alerting service that pings me if my estimates go above what I've set for a month. But it wouldn't fix a short term mistake; I don't need it to.
> I think it's the play between incompetence and having absolutely no incentive to do anything about it
The lack of business case is the most likely culprit. "You want to put engineering resources into something that only the $100/mo guys are going to use?"
You might be tempted to think "but my big org will use that", but I can guarantee compliance will shut it down -- you will never be permitted to enable a feature that intentionally causes hard downtime when (some external factor) happens.
AWS just released flat-rate pricing plans with no overages yesterday. You opt into a $0, $15, or $200/mo plan and at the end of the month your bill is still $0, $15, or $200.
It solves the problem of unexpected requests or data transfer increasing your bill across several services.
You get visibility into your usage, and you're seeing if you're exceeding the usage. They recommend to use plans if your typical traffic is 'only' up to 50TB per month. Occasional spikes are fine from what I understand.
This is not true. Even the Free plan has DDoS protection. L3/L4 (TCP SYN floods, UDP reflection attacks and similar) filtering is built-in and always-on, by default. CloudFront terminates TLS, and only forwards valid HTTP(S) requests to cache / origin.
The "Always-on DDoS Protection" on L7 is protection against massive requests spikes, built natively into CloudFront. Detection and mitigation of these attacks happens inline.
AWS just yesterday launched flat rate pricing for their CDN (including a flat rate allowance for bandwidth and S3 storage), including a guaranteed $0 tier.
I agree that it’s likely very technically difficult to find the right balance between capping costs and not breaking things, but this shows that it’s definitely possible, and hopefully this signals that AWS is interested in doing this in other services too.
The sad part about these posts is not the cloud apologists that show up in the comments. It’s the inevitable conclusion that “we should have understood our billing agreements better” or “we should really research what we’re deploying next time.” Nobody ever thinks, maybe the problem is the cloud itself. Maybe if I’m a single person business or have a few employees and even fewer users, then I should pay for infrastructure with a predictable cost…
AWS would much rather let you accidentally overspend and then forgive it when you complain than see stories about critical infrastructure getting shut off or failing in unexpected ways due to a miscommunication in billing.
As long as "shut off" potentially includes irrecoverable data loss, I guess, as it otherwise couldn't conclusively work. Along with a bunch of warnings to prevent someone accidentally (or maliciously) enabling it on an important account.
Malicious or erroneous actor can also drop your s3 buckets. Account change has stricter permissions.
The key problem is that data loss is really bad pr which cannot be reversed. Overcharge can be reversed. In a twisted way it might even strengthen the public image, I have seen that happen elsewhere.
I've always been under the impression billing is async and you really need it to be synchronous unless cost caps work as a soft limit.
You can transfer from S3 on a single instance usually as fast as the instances NIC--100Gbps+
You'd need a synchronous system that checks quotas before each request and for a lot of systems you'd also need request cancellation (imagine transferring a 5TiB file from S3 and your cap triggers at 100GiB--the server needs to be able to receive a billing violation alert in real time and cancel the request)
I imagine anything capped provided to customers already AWS just estimates and eats the loss
Obviously such a system is possible since IAM/STS mostly do this but I suspect it's a tradeoff providers are reluctant to make
It's not that it's technically impossible. The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met. No cloud provides wants to give their customers that much rope to hang themselves with. You just know too many customers will do it wrong or will forget to update the cap or will not coordinate internally, and things will stop working and take forever to fix.
It's easier to waive cost overages than deal with any of that.
Old hosts used to do that. 20 years ago when my podcast started getting popular I was hit with a bandwidth limit exceeded screen/warning. I was broke at the time and could not have afforded the overages (back then the cost per gig was crazy). The podcast not being downloadable for two days wasn’t the end of the world. Thankfully for me the limit was reached at the end of the month.
They have all the primitives. I think it's just that people are looking for a less raw version than AWS. In fact, perhaps many of these users should be using some platform that is on AWS, or if they're just playing around with an EC2 they're probably better off with Digital Ocean or something.
AWS is less like your garage door and more like the components to build an industrial-grade blast-furnace - which has access doors as part of its design. You are expected to put the interlocks in.
Without the analogy, the way you do this on AWS is:
1. Set up an SNS queue
2. Set up AWS budget notifications to post to it
3. Set up a lambda that watches the SNS queue
And then in the lambda you can write your own logic which is smart: shut down all instances except for RDS, allow current S3 data to remain there but set the public bucket to now be private, and so on.
The obvious reason why "stop all spending" is not a good idea is that it would require things like "delete all my S3 data and my RDS snapshots" and so on which perhaps some hobbyist might be happy with but is more likely a footgun for the majority of AWS users.
In the alternative world where the customer's post is "I set up the AWS budget with the stop-all-spending option and it deleted all my data!" you can't really give them back the data. But in this world, you can give them back the money. So this is the safer one than that.
Better yet, publish your own application specific notifications in step 2 since (I believe) billing alerts are delayed. Effectively, build an app specific sidecar that monitors the app and shuts it down if things get out of control.
Why does this always get asserted? It's trivial to do (reserve the cost when you allocate a resource [0]), and takes 2 minutes of thinking about the problem to see an answer if you're actually trying to find one instead of trying to find why you can't.
Data transfer can be pulled into the same model by having an alternate internet gateway model where you pay for some amount of unmetered bandwidth instead of per byte transfer, as other providers already do.
Reserving the cost until the end of the billing cycle is super unfriendly for spiky traffic and spiky resource usage. And yet one of the main selling points of the cloud is elasticity of resources. If your load is fixed, you wouldn’t even use the cloud after a five minute cost comparison. So your solution doesn’t work for the intended customers of the cloud.
It works just fine. No reason you couldn't adjust your billing cap on the fly. I work in a medium size org that's part of a large one, and we have to funnel any significant resource requests (e.g. for more EKS nodes) through our SRE teams anyway to approve.
Actual spikey traffic that you can't plan for or react to is something I've never heard of, and believe is a marketing myth. If you find yourself actually trying to suddenly add a lot of capacity, you also learn that the elasticity itself is a myth; the provisioning attempt will fail. Or e.g. lambda will hit its scaling rate limit way before a single minimally-sized fargate container would cap out.
If you don't mind the risk, you could also just not set a billing limit.
The actual reason to use clouds is for things like security/compliance controls.
I think I am having some misunderstanding about exactly how this cost control works. Suppose that a company in the transportation industry needs 100 CPUs worth of resources most of the day and 10,000 CPUs worth of resources during morning/evening rush hours. How would your reserved cost proposal work? Would it require having a cost cap sufficient for 10,000 CPUs for the entire day? If not, how?
10,000 cores is an insane amount of compute (even 100 cores should already be able to easily deal with millions of events/requests per second), and I have a hard time believing a 100x diurnal difference in needs exists at that level, but yeah, actually I was suggesting that they should have their cap high enough to cover 10,000 cores for the remainder of the billing cycle. If they need that 10,000 for 4 hours a day, that's still only a factor of 6 of extra quota, and the quota itself 1. doesn't cost them anything and 2. is currently infinity.
I also expect that in reality, if you regularly try to provision 10,000 cores of capacity at once, you'll likely run into provisioning failures. Trying to cost optimize your business at that level at the risk of not being able to handle your daily needs is insane, and if you needed to take that kind of risk to cut your compute costs by 6x, you should instead go on-prem with full provisioning.
Having your servers idle 85% of the day does not matter if it's cheaper and less risky than doing burst provisioning. The only one benefiting from you trying to play utilization optimization tricks is Amazon, who will happily charge you more than those idle servers would've cost and sell the unused time to someone else.
>The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met.
And why is that a problem? And how different is that from "forgetting" to pay your bill and having your production environment brought down?
I would love to have an option to automatically bring down the whole production once it's costing more than what it's earning. To think of it. I'd love this to be default.
When my computer runs out of hard drive it crashes, not goes out on the internet and purchases storage with my credit card.
Millions of businesses operate this way already. There's no way around it if you have physical inventory. And unlike with cloud services, getting more physical inventory after you've run out can take days, and keeping more inventory than you need can get expensive. Yet they manage to survive.
And cloud is really more scary. You have nearly unlimited liability and are at the mercy of the cloud service forgiving your debt if something goes wrong.
1) you hit the cap
2) aws sends alert but your stuff still runs at no cost to you for 24h
3) if no response. Aws shuts it down forcefully.
4) aws eats the “cost” because lets face it. It basically cost them 1000th of what they bill you for.
5) you get this buffer 3 times a year. After that. They still do the 24h forced shutdown but you get billed.
Everybody wins.
I think it’s disingenuous to claim that AWS only offers delayed alerts and half-decent cost controls. Granted, these features were not there in the beginning, but for years now AWS, in addition to the better known stuff like strategic limits on auto scaling, allows subscribing to price threshold triggers via SNS and perform automatic actions, which could be anything including scaling down or stopping services completely if the cost skyrockets.
The problem with hard caps is that there's no way to retroactively fix "our site went down". As much as engineers are loathe to actually reach out to a cloud provider, are there any anecdotes of AWS playing hardball and collecting a 10k debt for network traffic?
Conversely the first time someone hits an edge case in billing limits and their site goes down, losing 10k worth of possible customer transactions there's no way to unring that bell.
The second constituency are also, you know, the customers with real cloud budgets. I don't blame AWS for not building a feature that could (a) negatively impact real, paying customers (b) is primarily targeted at people who by definition don't want to pay a lot of money.
I'd much rather lose 10k in customers that might potentially come another day than 10k in Amazon bill. Amazon bill feels like more unringable.
But hey, let's say you have different priorities than me. Then why not bot? Why not let me set the hard cap? Why Amazon insists on being able to bill me on more than my business is worth if I make a mistake?
These topics are not advanced...they are foundational scenarios covered in any entry level AWS or AWS Cloud third-party training.
But over the last few years, people have convinced themselves that the cost of ignorance is low. Companies hand out unlimited self-paced learning portals, tick the “training provided” box, and quietly stop validating whether anyone actually learned anything.
I remember when you had to spend weeks in structured training before you were allowed to touch real systems. But starting around five or six years ago, something changed: Practitioners began deciding for themselves what they felt like learning. They dismantled standard instruction paths and, in doing so, never discovered their own unknown unknowns.
In the end, it created a generation of supposedly “trained” professionals who skipped the fundamentals and now can’t understand why their skills have giant gaps.
If I accept your premise (which I think is overstated) I’d say it’s a good thing. We used to ship software with literally 100lbs of manual and sell expensive training, and then consulting when they messed up. Tons of perverse incentives.
The expectation that it just works is mostly a good thing.
This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen.
Also, consider using fck-nat (https://fck-nat.dev/v1.3.0/) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges.
(Or, just run your own Debian nano instance that does the masquerading for you, which every old-school Linuxer should be able to do in their sleep.)
The reason to not include the endpoint by default is because VPCs should be secure by default. Everything is denied and unless you explicitly configure access to the Internet, it's unreachable. An attacker who manages to compromise a system in that VPC now has a means of data exfiltration in an otherwise air gapped set up.
It's annoying because this is by far the more uncommon case for a VPC, but I think it's the right way to structure, permissions and access in general. S3, the actual service, went the other way on this and has desperately been trying to reel it back for years.
Right, I can appreciate that argument - but then the right thing to do is to block S3 access from AWS VPCs until you have explicitly confirmed that you want to pay the big $$$$ to do so, or turn on the VPC endpoint.
A parallel to this is how SES handles permission to send emails. There are checks and hoops to jump through to ensure you can't send out spam. But somehow, letting DevOps folk shoot themselves in the foot (credit card) is ok.
What has been done is the monetary equivalent of "fail unsafe" => "succeed expensively"
s3 access is blocked from an EC2 by default unless you give the attached IAM role access to S3.
Then it is still blocked unless you add a NAT gateway or Internet gateway to the VPC and at a route to them.
If you are doing all of this via IAC, you have to take a lot of steps to make this happen. On the other hand, if I’m using an EC2 instance to run an ETL job from data stored on S3, I’m not putting that EC2 instance in a subnet with internet access in the first place. Why would I?
And no you don’t need internet access to access the EC2 instance ftom your computer even without a VPN. You use System Manager Session Manager.
I do the same with lambda - attach then to a VPC without internet access with the appropriate endpoints. Even if they are serving an API, they are still using an API gateway
There’s zero reason why AWS can’t pop up a warning if it detects this behavior though. It should clearly explain the implications to the end user. I mean EKS has all sorts of these warning flags it pops up on cluster health there’s really no reason why they can’t do the same here.
To be fair, while EKS warnings are useful, I've grown a habit to ignore them completely, since I've seen every single RDS cluster littered with "create a read replica please" and "enable performance insights" bs warnings.
The second someone doesn’t pay attention to that warning and suffers an exfiltration, like the cap1 s3 incident, it’s aws’ fault as far as the media is concerned.
I don't get your argument. If an ec2 needs access to an s3 resource, doesn't it need that role? Or otherwise, couldn't there be some global s3 URL filter that automagically routes same-region traffic appropriately if it is permitted?
My point is that, architecturally, is there ever in the history of AWS an example where a customer wants to pay for the transit of same-region traffic when a check box exists to say "do this for free"? Authorization and transit/path are separate concepts.
The EC2 needs credentials, but not necessarily a role. If someone is able to compromise an EC2 instance that has unrestricted S3 connectivity (no endpoint policies), they could use their own credentials to exfiltrate data to a bucket not associated with the account.
I'll have to dive in and take a look. I'm not arguing, but here is how I naively see it:
It seems there is a gap between "how things are" and "how things should be".
"Transiting the internet" vs. "Cost-free intra-region transit" is an entirely different question than "This EC2 has access to S3 bucket X" or "This EC2 does not have access to S3 bucket X".
Somewhere, somehow, that fact should be exposed in the design of the configuration of roles/permissions/etc. so that enabling cost-free intra-region S3 access does not implicitly affect security controls.
I agree. The real question is why do I need an "VPC endpoint" to save money in the first place?! us-east-1 EC2 isn't actually going over the internet to connect to us-east-1 S3, regardless or whether it's using a NAT gateway or VPC endpoint. AWS knows what routes are on its own network.
Changing defaults doesn't have to mean changing existing configurations. It can be the new default for newly created VPCs after a certain date, or for newly created accounts after a certain date.
And if there are any interoperability concerns, you offer an ability to opt-out with that (instead of opting in).
> Changing defaults doesn't have to mean changing existing configurations. It can be the new default for newly created VPCs after a certain date, or for newly created accounts after a certain date.
This is breaking existing IAAC configurations because they rely on the default. You will never see the change you're describing except in security-related scenarios
> There is precedent for all of this at AWS.
Any non-security IAAC default changes you can point to?
AWS is not going to enable S3 endpoints by default, and most of the thread is downvoting the correct explanations like thinking in terms of a small hobby VPC, not the architectures AWS actually has to support.
Why it should not be done:
1. It mutates routing.
Gateway Endpoints inject prefix-list routes into selected route tables. Many VPCs have dozens of RTs for segmentation, TGW attachments, inspection subnets, EKS-managed RTs, shared services, etc. Auto-editing them risks breaking zero-trust boundaries and traffic-inspection paths.
2. It breaks IAM / S3 policies.
Enterprises commonly rely on aws:sourceVpce, aws:SourceIp, Private Access Points, SCP conditions, and restrictive bucket policies. Auto-creating a VPCE would silently bypass or invalidate these controls.
3. It bypasses security boundaries.
A Gateway Endpoint forces S3 traffic to bypass NAT, firewalls, IDS/IPS, egress proxies, VPC Lattice policies, and other mandatory inspection layers. This is a hard violation for regulated workloads.
4. Many VPCs must not access S3 at all.
Air-gapped, regulated, OEM, partner-isolated, and inspection-only VPCs intentionally block S3. Auto-adding an endpoint would break designed isolation.
5. Private DNS changes behavior.
With Private DNS enabled, S3 hostname resolution is overridden to use the VPCE instead of the public S3 endpoint. This can break debugging assumptions, routing analysis, and certain cross-account access patterns.
6. AWS does not assume intent.
The VPC model is intentionally minimal. AWS does not auto-create IGWs, NATs, Interface Endpoints, or egress paths. Defaults must never rewrite user security boundaries.
These are all good arguments. Then do the opposite and block S3 access from VPCs by default. That would violate none of those.
“We have no idea what your intent is, so we’ll default to routing AWS-AWS traffic expensively” is way, way worse than forcing users to be explicit about their intent.
Minimal is a laudable goal - but if a footgun is the result then you violate the principle of least surprise.
I rather suspect the problem with issues like this is that they mainly catch the less experienced, who aren’t an AWS priority because they aren’t where the Big Money is.
Yes, this. You lock it into Terraform or some equivalent.
And ok, this is a mistake you will probably only make once - I know, because I too have made it on a much smaller scale, and thankfully in a cost-insensitive customer's account - but surely if you're an infrastructure provider you want to try to ensure that you are vigilantly removing footguns.
Especially true now with Claude generating decent terraform code. I was shocked how good it is at knowing AWS gotchas. It also debug connectivity issues almost automagically. While I hate how it writes code I love how it writes terraform.
AI is surprising good at boilerplate IaC stuff. It’s a great argument for configuration as code, or really just being able to represent things in plain text formats
If you are creating a VPC from the console that might be a reasonable default. But any serious implementation is going to be using IAC - like they were - and I would expect to spell out everything explicitly.
I personally prefer to just memorize the data and recite it really quickly on-demand.
Only half-joking. When something grossly underperforms, I do often legitimately just pull up calc.exe and compare the throughput to the number of employees we have × 8 kbit/sec [0], see who would win. It is uniquely depressing yet entertaining to see this outperform some applications.
[0] spherical cow type back of the envelope estimate, don't take it too seriously; assumes a very fast 200 wpm speech, 5 bytes per word, and everyone being able to independently progress
Made a similar mistake like this once. While just playing around to see what's possible I upload some data to the AWS algo that will recommended products to your users based on everyone's previous purchases.
I uploaded a small xls with uid and prodid columns and then kind of forgot about it.
A few months later I get a note from bank saying your account is overdrawn. The account is only used for freelancing work which I wasn't doing at the time, so I never checked that account.
Looks like AWS was charging me over 1K / month while the algo continuously worked on that bit of data that was uploaded one time. They charged until there was no money left.
That was about 5K in weekend earnings gone. Several months worth of salary in my main job. That was a lot of money for me.
I worked in a billing department, and learned to be healthily paranoid about such things. I want to regularly check what I'm billed for. I of course check all my bank accounts' balances at least once a day. All billing emails are marked important in my inbox, and I actually open them.
And of course I give every online service a separate virtual credit card (via privacy dot com, but your bank may issue them directly) with a spend limit set pretty close to the expected usage.
The free market can sort something like this out, but it requires some things to work. There need to be competitors offering similar products, people need to have the ability to switch to using those competitors, and they need to be able to get information about the strengths and weaknesses of the different offerings (so they can know their current vendor has a problem and that another vendor doesn't have that problem). The free market isn't magic, but neither are business regulations. Both have failure modes you have to guard against.
This is a non sequitur. I know how to self host my infra, but I’ve been using cloud services for the last 15 years because it means I don’t have to deal with self hosting my infra. It runs completely by itself (mostly managed services, including k8s) and the only time I need to deal with it is when I want to change something.
BTW you can of course self-host k8s, or dokku, or whatnot, and have as easy a deployment story as with the cloud. (But not necessarily as easy a maintenance story for the whole thing.)
For a tinkerer who's focused on the infra, then sure, hosting your own can make sense. But for anyone who's focused on literally anything else, it doesn't make any sense.
Cloud is not great for GPU workloads. I run a nightly workload that takes 6-8 hours to run and requires a Nvidia GPU, along with high RAM and CPU requirements. It can't be interrupted. It has a 100GB output and stores 6 nightly versions of that. That's easily $600+ a month in AWS just for that one task. By self-hosting it I have access to the GPU all the time for a fixed up front relatively low cost and can also use the HW for other things (I do). That said, these are all backend / development type resources, self hosting customer facing or critical things yourself is a different prospect, and I do use cloud for those types of workloads. RDS + EKS for a couple hundred a month is an amazing deal for what is essentially zero maintenance application hosting. My point is that "literally anything else" is extreme, as always, it is "right tool for the job".
I have found Claude Code is a great help to me. Yes, I can and have tinkered a lot over the decades, but I am perfectly happy letting Claude drive the system administration, and advise on best practices. Certainly for prototype configurations. I can install CC on all VPSes and local machines. NixOS sounds great, but the learning curve is not fun. I installed the CC package from the NixOS unstable channel and I don't have to learn the funky NixOS packaging language. I do have to intervene sometimes as the commands go by, as I know how to drive, so maybe not a solution for true newbies. I can spend a few hours learning how to click around in one of the cloud consoles, or I can let CC install the command line interfaces and do it for me. The $20/mo plan is plenty for system administration and if I pick the haiku model, then CC runs twice as fast on trivial stuff like system administration.
Let's take an example: a managed database, e.g. Postgres or MySQL, vs. a self-hosted one. If you need reasonable uptime, you need at least one read replica. But replication breaks sometimes, or something goes wrong on the master DB, particularly over a period of years.
Are you really going to trust Claude Code to recover in that situation? Do you think it will? I've had DB primaries fail on managed DBs like AWS RDS and Google Cloud SQL, and recovery is generally automatic within minutes. You don't have to lift a finger.
Same goes for something like a managed k8s cluster, like EKS or GKE. There's a big difference between using a fully-managed service and trying to replicate a fully managed system on your own with the help of an LLM.
Of course it does boil down to what you need. But if you need reliability and don't want to have to deal with admin, managed services can make life much simpler. There's a whole class of problems I simply never have to think about.
It doesnt make any sense to you that I would like to avoid a potential 60K bill because of a configuration error? If youre not working at faang your employer likely cares too. Especially if its your own business you would care. You really can't think of _one_ case where self hosting makes any sense?
> It doesnt make any sense to you that I would like to avoid a potential 60K bill because of a configuration error?
This is such an imaginary problem. The examples like this you hear about are inevitably the outliers who didn't pay any attention to this issue until they were forced to.
For most services, it's incredibly easy to constrain your costs anyway. You do have to pay attention to the pricing model of the services you use, though - if a DDOS is going to generate a big cost for you, you probably made a bad choice somewhere.
> You really can't think of _one_ case where self hosting makes any sense?
Only if it's something you're interested in doing, or if you're so big you can hire a team to deal with that. Otherwise, why would you waste time on it?
Thinking about "constraining cost" is the last thing I want to do. I pay a fixed 200 dollars a month for a dedicated server and spend my time solving problems using code. The hardware I rent is probably overkill for my business and would be more than enough for a ton of businesses' cloud needs. If youre paying per GB of traffic, or disk space, or RAM, you're getting scammed. Hyperscalers are not the right solution for most people. Developers are scared of handling servers, which is why you're paying that premium for a hyperscaler solution. I SSH into my server and start/stop services at will, configure it any way i want, copy around anything I want, I serve TBs a week, and my bill doesnt change. You would appreciate that freedom if you had the will to learn something you didnt know before. Trust me its easier than ever with Ai!
> For a tinkerer who's focused on the infra, then sure, hosting your own can make sense.
... or for a big company. I've worked at companies with thousands of developers, and it's all been 'self hosted'. In DCs, so not rinky dink, but yes, and there's a lot of advantages to doing it this way. If you set it up right, it can be much easier for developers to use than AWS.
I get pulled into a fair number of "why did my AWS bill explode?" situations, and this exact pattern (NAT + S3 + "I thought same-region EC2→S3 was free") comes up more often than you’d expect.
The mental model that seems to stick is: S3 transfer pricing and "how you reach S3" pricing are two different things. You can be right that EC2→S3 is free and still pay a lot because all your traffic goes through a NAT Gateway.
The small checklist I give people:
1. If a private subnet talks a lot to S3 or DynamoDB, start by assuming you want a Gateway Endpoint, not the NAT, unless you have a strong security requirement that says otherwise.
2. Put NAT on its own Cost Explorer view / dashboard. If that line moves in a way you didn’t expect, treat it as a bug and go find the job or service that changed.
3. Before you turn on a new sync or batch job that moves a lot of data, sketch (I tend to do this with Mermaid) "from where to where, through what, and who charges me for each leg?" It takes a few minutes and usually catches this kind of trap.
Cost Anomaly Detection doing its job here is also the underrated part of the story. A $1k lesson is painful, but finding it at $20k is much worse.
Depends on various factors and of course the amount of money in question. I've had AWS approve a refund for a rather large sum a few years ago, but that took quite a bit of back and forth with them.
Crucial for the approval was that we had cost alerts already enabled before it happened and were able to show that this didn't help at all, because they triggered way too late. We also had to explain in detail what measures we implemented to ensure that such a situation doesn't happen again.
Nothing says market power like being able to demand that your paying customers provide proof that they have solutions for the shortcomings of your platform.
It's a system people opt into, you can do something like ingress/egress blocked, & user has to pay a service charge (like overdraft) before access opened up again. If account is locked in overdraft state for over X amount of days then yes, delete data
2 caps: 1 for things that are charged for existing (e.g. S3 storage, RDS, EBS, EC2 instances) and 1 for things that are charged when you use them (e.g. bandwidth, lambda, S3 requests). Fail to create new things (e.g. S3 uploads) when the first cap is met.
Does that mean fail to create rds backups? And that AWS needs to keep your EC2 instance and RDS instance running while you decide if you really want to pay the bill?
RunPod has its issues, but the way it handles payment is basically my ideal. Nothing brings peace of mind like knowing you won't be billed for more than you've already paid into your wallet. As long as you aren't obliged to fulfil some SLA, I've found that this on-demand scaling compute is really all I need in conjunction with a traditional VPS.
It's great for ML research too, as you can just SSH into a pod with VScode and drag in your notebooks and whatnot as if it were your own computer, but with a 5090 available to speed up training.
I would put $100 that within 6 months of that, we'll get a post on here saying that their startup is gone under because AWS deleted their account because they didn't pay their bill and didn't realise their data would be deleted.
> (the cap should be a rate, not a total)
this is _way_ more complicated than there being a single cap.
> I would put $100 that within 6 months of that, we'll get a post on here saying that their startup is gone under because AWS deleted their account because they didn't pay their bill and didn't realise their data would be deleted.
The measures were related to the specific cause of the unintended charges, not to never incur any unintended charges again. I agree AWS needs to provide better tooling to enable its customers to avoid such situations.
I am never going to use any cloud service which doesn't have a cap on charges. I simply cannot risk waking up and finding a $10000 or whatever charge on my personal credit card.
I presume it depends on your ability to pay for your mistakes. A $20/month client is probably not going to pony up $1000, a $3000/month client will not care as much.
That is the business model and one of the figurative moats: easy to onboard, hard/expensive (relative to on-boarding ) to divest.
Though important to note in this specific case was a misconfiguration that is easy to make/not understand in the data was not intended to leave AWS services (and thus should be free) but due to using the NAT gateway, data did leave the AWS nest and was charged at a higher data rate per GB than if just pulling everything straight out of S3/EC2 by about an order of magnitude (generally speaking YMMV depending on region, requests, total size, if it's an expedited archival retrieval etc etc)
So this is an atypical case, doesn't usually cost $1000 to pull 20TB out of AWS. Still this is an easy mistake to make.
I don’t mind the extortionate pricing if it’s upfront and straightforward. fck-nat does exist. What I do mind is the opt out behavior that causes people to receive these insane bills when their first, most obvious expectation is that traffic within a data center stays within that data center and doesn’t flow out to the edge of it and back in. That is my beef with the current setup.
But “security” people might say. Well, you can be secure and keep the behavior opt out, but you should be able to have an interface that is upfront and informs people of the implications
Yes uploading into AWS is free/cheap. You pay per GB of data downloaded, which is not cheap.
You can see why, from a sales perspective: AWS' customers generally charge their customers for data they download - so they are extracting a % off that. And moreover, it makes migrating away from AWS quite expensive in a lot of circumstances.
It's not disinformation at all, there's a lot of hurdles to this.
In the link you posted, it even says Amazon can't actually tell if you're leaving AWS or not so they're going to charge you the regular rate. You need explicit approval from them to get this 'free' data transfer.
It feels like you are intentionally missing the point. The blog makes it quite clear that the purpose of notifying AWS is so they stop charging you for that specific type of traffic. How else would they know whether the spike is normal production usage, which is billable, or part of a migration effort?
There is nothing in the blog suggesting that this requires approval from some committee or that it is anything more than a simple administrative step. And if AWS were to act differently, you have grounds to point to the official blog post and request that they honor the stated commitment.
Egress bandwidth costs money. Consumer cloud services bake it into a monthly price, and if you’re downloading too much, they throttle you. You can’t download unlimited terabytes from Google Drive. You’ll get a message that reads something like: “Quota exceeded, try again later.” — which also sucks if you happen to need your data from Drive.
AWS is not a consumer service so they make you think about the cost directly.
"Premium bandwidth" which AWS/Amazon markets to less understanding developers is almost a scam. By now, software developers think data centers, ISPs and others part of the peering on the internet pay per GB transferred, because all the clouds charge them like that.
Try a single threaded download from Hetzner Finland versus eu-north-1 to a remote (i.e. Australia) destination and you'll see premium bandwidth is very real. Google Cloud Storage significantly more so than AWS.
Sure you can just ram more connections through the lossy links from budget providers or use obscure protocols, but there's a real difference.
I just tested it and TCP gets the maximum expected value given the bandwidth delay product from a server in Falkenstein to my home in Australia, from 124 megabits on macOS to 940 megabits on Linux.
Can you share your tuning parameters on each host? If you aren't doing exactly the same thing on AWS as you are on Hetzner you will see different results.
Bypassing the TCP issue I can see nothing indicating low network quality, a single UDP iperf3 pass maintains line rate speed without issue.
Edit: My ISP peers with Hetzner, as do many others. If you think it's "lossy" I'm sure someone in network ops would want to know about it. If you're getting random packet loss across two networks you can have someone look into it on both ends.
I was lucky to have experienced all of the same mistakes for free (ex-Amazon employee). My manager just got an email saying the costs had gone through the roof and asked me to look into it.
Feel bad for anyone that actually needs to cough up money for these dark patterns.
I'm still adamant about the fact that the "cloud" is a racket.
Sure, it decreases the time necessary to get something up running, but the promises of cheaper/easier to manage/more reliable have turned out to be false. Instead of paying x on sysadmin salaries, you pay 5x to mega corps and you lose ownership of all your data and infrastructure.
I think it's bad for the environment, bad for industry practices and bad for wealth accumulation & inequality.
I'd say it's a racket for enterprise but it makes sense for small things. For example, a friend of mine, who's in a decent bit of debt and hence on the hunt for anything that can make some money, wanted to try making essentially a Replika clone for a local market and being able to rent an H100 for 2$ an hour was very nice. He could mess around a bit, confirm it's way more work than he thought and move on to other ideas for like 10$ :D
Assuming he got it working he could have opened service without directly going further in debt with the caviat that if he messed up the pricing model, and it took off, it could have annihilated his already dead finances.
But that is not what is usually referred to as the cloud. I am not against infrastructure for rent like VPS or an H100 node, but against the behemoths of AWS, Azure and the like
This wouldn’t have specifically helped in this situation (EC2 reading from S3), but on the general topic of preventing unexpected charges from AWS:
AWS just yesterday launched flat rate pricing for their CDN (including a flat rate allowance for bandwidth and S3 storage), including a guaranteed $0 tier. It’s just the CDN for now, but hopefully it gets expanded to other services as well.
The service gateways are such a weird thing in AWS. There seems to be no reason not to use them and it's like they only exist as a trap for the unaware.
Reading all the posts about people who got bitten by some policies on AWS, I think they should create two modes:
- raw
- click-ops
Because, when you build your infra from scratch on AWS, you absolutely don't want the service gateways to exist by default.
You want to have full control on everything, and that's how it works now.
You don't want AWS to insert routes in your route tables on your behalf.
Or worse, having hidden routes that are used by default.
But I fully understand that some people don't want to be bothered but those technicalities and want something that work and is optimized following the Well-Architected Framework pillars.
IIRC they already provide some CloudFormation Stacks that can do some of this for you, but it's still too technical and obscure.
Currently they probably rely on their partner network to help onboard new customers, but for small customers it doesn't make sense.
> you absolutely don't want the service gateways to exist by default.
Why? My work life is in terraform and cloudformation and I can't think of a reason you wouldn't want to have those by default. I mean I can come up with some crazy excuses, but not any realistic scenario. Have you got any? (I'm assuming here that they'd make the performance impact ~0 for the vpc setup since everyone would depend on it)
If I declare two aws_route resources for my route table, I don't want a third route existing and being invisible.
I agree that there is no logical reason to not want a service gateway, but it doesn't mean that it should be here by default.
The same way you need to provision an Internet Gateway, you should create your services gateways by yourself.
TF modules are here to make it easier.
Everything that comes by default won't appear in your TF, so it becomes invisible and the only way to know that it exists is to remember that it's here by default.
There's lots of stuff that exists in AWS without being in TF. Where do you create a router, a DHCP server, each ENI, etc. ? Why are the instances in a changing state in ASG rather than all in TF? Some things are not exactly as they exist in TF, because it makes more sense that way. We never had 1:1 correspondence in the first place.
the gateway endpoints are free (s3 + dynamodb?), but the service endpoints are charged so that could be a reason why people don't use the service endpoints. but there doesn't seem to be a good reason for not using the service gateways. it also seems crazy that AWS charges you to connect to their own services without a public ip. also, i guess this would be less of an issue (in terms of requiring a public ip) if all of AWS services were available over ipv6. because then you would not need NAT gateways to connect to AWS services when you don't have a public ipv4 ip and I assume you are not getting these special traffic charges when connecting to the AWS services with a public ipv6 address.
The documentation is thick but it has a common theme and format to it. So once you get the hang of finding the "juicy bits" you can usually locate them anywhere. The docs do generally warn you of these cases, or have a whole "best practices" section which highlights them directly.
The key is, do not make decisions lightly in the cloud, just because something is easy to enable in the UI does not mean it's recommended. Sit down with the pricing page or calculator and /really/ think over your use case. Get used to thinking about your infrastructure in terms of batch jobs instead of real time and understand the implementation and import of techniques like "circuit breakers."
Once you get the hang of it it's actually very easy and somewhat liberating. It's really easy to test solutions out in a limited form and then completely tear them down. Personally I'm very happy that I put the effort in.
I am 100% not willing to put that much effort into it.
Actually I am not willing to spend that much time reading "cloud" provider docs and best practices.
What I care about is hosting services and getting the mission done. I value predictable and reasonable costs much more than "flexibility".
In general whatever host I have ever used typically have "cloud" offerings too. So real services go to dedicated hosts, experiments go into 5 buck a month vms.
> It doesn't take that much time just intentional reading to learn something new. Do you not read any docs?
There is a tradeoff, between the time it takes to read the docs and the time it would take to achieve what you want to do with another tool which you are either familiar with or is simpler. A very important factor for that balance is how reusable and how reapplicable the knowledge you will gain will be. I am perfectly happy reading docs for PostgreSQL, Varnish, nginx and many other things if it will help me achieve my goal. But when it comes to cloud providers I am not willing to spend a minute reading how to vendor lock my stack on their platform. If there is a standalone service I might want to use like SQS or SES sure, but most other things are a hard pass, I would rather have a dedicated host.
> The point is that these aren't mutually exclusive.
They are really not, cloud providers charge incredibly high rates for both bandwidth and computational resources so there goes the reasonable part. The usage based pricing they use is typically not capped so you are always a misconfiguration away from incurring many times your monthly costs on bandwidth charges (see a similar case from azure https://news.ycombinator.com/item?id=30054739). To add insult to injury you usually get inferior performance.
> And when it becomes overloaded, needs upgrading, or requires other maintenance? There's a problem I've completely left behind.
You have replaced these problems with having to deal with the cloud providers. It costs time to deal with their impenetrable pricing, processes and abstractions. Do yourself a favor and track how much time you spend dealing with their tools and how much time and effort you spend trying to optimize costs.
Upgrades and maintenance are not really solved, depending on the products you use. If you are using EC2 you still need to do upgrades and maintenance exactly like you would with bare metal. Now if you are using a serverless approach you might be clear of that too, but in general docker helps a lot with removing a lot of the pain of using bare metal.
Overloading is typically easily solved by over provisioning and yeah if you have very spiky usage patterns it might make sense to use a cloud provider. If your typical load can be dealt by 1 Hetzner node but you need 20 nodes 2% of the time you might be better off using a cloud provider. Modern hardware though has a lot of capability and while you don't have an "entire datacenter" to fall back on like you would with AWS a few bare metal nodes can handle a very large volume of requests depending on your use case.
I did this when I was ~22 messing with infra for the first time. A $300 bill in two days when I had $2000 in the bank really stung. I love AWS for many things, but I really wish they made the cost calculations transparent for beginners.
>There is another type of VPC endpoint, Gateway, which creates a gateway endpoint to send traffic to Amazon S3 or DynamoDB. Gateway endpoints do not use AWS PrivateLink, unlike the other types of VPC endpoints. For more information, see Gateway endpoints.
>Use a VPC endpoint to connect to AWS services privately, without the use of an internet gateway or NAT device.
The author of the blog writes:
> When you're using VPCs with a NAT Gateway (which most production AWS setups do), S3 transfers still go through the NAT Gateway by default.
Yes, you are using a virtual private network. Where is it supposed to go? It's like being surprised that data in your home network goes through a router.
> An entire blog article post to say "read the docs and enable VPC S3 endpoint".
I think it's okay if someone missed something in the docs and wanted to share from their experience. In fact, if you look at the the s3 pricing page [0], under Data Transfer, VPC endpoints are mentioned at all. It simply says data transfer is free between AWS services in the same region. I think that much detail would be enough to reasonably assume you didn't have to set up additional items to accomplish.
You can with some effort, but cloud providers don't provide real-time information on how much you're spending. Even if you use spending alerts to program a hard cut-off yourself, a mistake can still result in you being charged for 6+ hours of usage before the alert fires.
> You can with some effort, but cloud providers don't provide real-time information on how much you're spending.
This should be illegal. If you can't inform me about the bill on my request you shouldn't be legally able to charge me that bill. Although I can already imagine plenty of ways somebody could do malicious compliance with that rule.
my understanding from reading this kind of threads is that there is no real way to enforce it and the provider makes no guarantees, as your usage can outpace the system that is handling the accounting and shutoff
to be fair, im not sure its a conscious choice, since its not really easy to couple lets say data transfer bytes directly to billing data in real time, and im sure that would also use up a lot of resources.
But of course, the incentive to optimize this is not there.
> The solution is to create a VPC Gateway Endpoint for S3. This is a special type of VPC endpoint that creates a direct route from your VPC to S3, bypassing the NAT Gateway entirely.
The solution is to move your processing infrastructure to Hetzner.
Make sure they go to an list with multiple people on it. Make sure someone pays attention to that email list.
It's free and will save your bacon.
I've also had good luck asking for forgiveness. One time I scaled up some servers for an event and left them running for an extra week. I think the damage was in the 4 figures, so not horrendous, but not nothing.
An email to AWS support led to them forgiving a chunk of that bill. Doesn't hurt to ask.
> I've been using AWS since around 2007. Back then, EC2 storage was entirely ephemeral and stopping an instance meant losing all your data. The platform has come a long way since then.
Personally I miss ephemeral storage - having the knowledge that if you start the server from a known good state, going back to that state is just a reboot away. Way back when I was in college, a lot of out big-box servers worked like this.
You can replicate this on AWS with snapshots or formatting the EBS volume into 2 partitions and just clearing the ephemeral part on reboot, but I've found it surprisingly hard to get it working with OverlayFS
It’s naive to think that AWS is some sort of magically special system that transcends other networked computers, out of brand loyalty.
That’s the AWS kool aid that makes otherwise clever people think there’s no way any organization can run their own computer systems - only AWS has the skills for that.
It was already clear that you were in bad faith here when you suggested a VPS to replace AWS, no need to insist.
But you are absolutely right, I'm drinking the AWS kool aid like thousands of other otherwise clever people who don't know that AWS is just Linux computers!
Good luck managing the whole day-2 operations and the application layer on top of your VPS. You're just shuffling around your spending. For you it's not on compute anymore but manpower to manage that mess.
You probably don't need it. I see so many people getting price gouged by S3 when it would be orders of magnitude cheaper to just throw the files on a basic HTTP server.
I sometimes feel bad using people's services built with S3 as I know my personal usage is costing them a lot of money despite paying them nothing.
A web storage is connected to storage solutions like SSDs and S3 is connected to delivery networks like Internet. Using SSDs to store files or Internet to send files to a user are not the wrong tools.
Had the exact same thing happen. Only we used a company vetted/recommended by AWS to set this up for us, as we have no AWS experts and we're all too busy tasked doing actual startup things. So we staffed it out. Even the "professionals" get it wrong, and we racked up a huge expense as well. Staffed out company shrugged shoulders, and then just said sorry about your tab. We worked with AWS support to correct situation, and cried to daddy AWS account manager for a negotiated rate.
> AWS's networking can be deceptively complex. Even when you think you've done your research and confirmed the costs, there are layers of configuration that can dramatically change your bill.
Unexpected, large AWS charges have been happening for so long, and so egregiously, to so many people, including myself, that we must assume it's by design of Amazon.
Regardless of the AWS tech in question (and yes VPCE for non-compute services is a very common pattern in an enterprise setup using AWS since VPC with NAT is a pretty fundamental requirement), I honestly believe this was the biggest miss from the author: “Always validate your assumptions. I thought "EC2 to S3 is free" was enough. I should have tested with a small amount of data and monitored the costs before scaling up to terabytes.” To me this is a symptom of DevOps/infra engineers being too much in love with infra automation without actually testing the full end to end flow.
Also as a shameless plug: Vantage covers this is exact type of cost hiccup. If you aren't already using it, we have a very generous free tier: https://www.vantage.sh/
I’m sure NAT gateways exist purely to keep uninformed security “experts” at companies happy. I worked at a Fortune 500 company but we were a dedicated group building a cloud product on AWS. Security people demanded a NAT gateway. Why? “Because you need address translation and a way to prevent incoming connections”. Ok. That’s what an Internet Gateway is. In the end we deployed a NAT gateway and just didn’t setup routes to it. Then just used security groups and public IPs.
Yeah not free if you definitely need IPv4. AWS has been adding a lot more IPv6 support to their services so hopefully the trend continues in AWS and the broader industry. You can probably get pretty far though if your app doesn't have hard requirements to communicate with IPv4 only hots.
Its staggering to me that after all this time there are somehow still people in potions like this that are working without basic cost monitoring alerts on cloud/SaaS services
It really shows the Silicon Vally disconnect with the real world, where money matters.
VPC peering becomes ugly fast, once your network architecture becomes more complex. Because transitive peering doesn't work you're building a mesh of networks.
It used to be that you could whine to your account rep and they'd waive sudden accidental charges like this. Which we did regularly due to all the sharp edges. These days I gather it's a bit harder.
Imagine a world were Amazon was forced to provide a publicly available report were they disclose how many clients have made this error -and similar ones- and how much money they have made from it. I know nothing like this will ever exist but hey, is free to dream.
Wait till you encounter the combo of gcloud parallel composite uploads + versioning + soft-delete + multi-region bucket - and you have 500TB of objects stored.
NAT gateway probably cheap as fuck for Bezos & co to run but nice little earner. The parking meter or exit ramp toll of cloud infra. Cheap beers in our bar but $1000 curb usage fee to pull up in your uber.
I think it's been calculated that data transfer is the biggest margin product in all AWS catalog by a huge difference. A 2021 calculation done by Cloudflare [0] estimated almost 8000% price markup in EU and US regions.
And I can see how, in very big accounts, small mistakes on your data source when you're doing data crunching, or wrong routing, can put thousands and thousands of dollars on your bill in less than an hour.
That's a NAT gateway, but if you're pulling data for analysis from S3 buckets you don't have those limitations.
Let's say they decide to recalculate or test a algorithm: they do parallel data loading from the bucket(s), and they're pulling from the wrong endpoint or region, and off they go.
And maybe they're sending data back, so they double the transfer price. RDS Egress. EC2 Egress. Better keep good track of your cross region data!
This has nothing about punching down. Writing a blog about this basic mistake, and presenting as advice shows a strong lack of self awareness. Its like when Google bought thousands of servers without ECC memory, but felt they were so smart they could not resist telling the world how bad that was and writing a paper about it...Or they could have hired some real hardware engineers from IBM or Sun...
> Writing a blog about this basic mistake, and presenting as advice shows a strong lack of self awareness.
You realize they didn’t ask you to read their article right? They didn’t put it on your fridge or in your sandwich.
Policing who writes what honest personal experience on the Internet is not a job that needs doing.
But if you do feel the need to police, don’t critique the writer, but HN for letting interested readers upvote the article here, where it is of course, strictly required reading.
I mean, drill down to the real perpetrators of this important “problem”!
They are doing Enterprise sales and were founded in 2014...These type of technical blogs are normally written to demonstrate the company internal expertise, demonstrate skills of the engineering team to motive hiring etc...
Does this inspire confidence?
Do I trust in their ability to hit a bump and be motivated to learn and communicate, without ass covering, blame deflection or stubborn denial? Yes!
Are you more confident of people without those traits?
—-
Critique the dishonest, the manipulative, the deceptive, and the dangerously self-unaware.
Better to respond to normal people making their way forward by offering your own insight, if you actually have any that they didn’t just learn for themselves.
Or by respectfully leaving them to continue.
People who punch down are not good people to work with. They don’t inspire my confidence. Do they inspire yours?
—-
Read all the useful discussions here. Do you imagine all these people saw less that you, to see something constructive? Or more?
These sort of things show up about once a day between the three big cloud subreddit. Often with larger amounts
And it’s always the same - clouds refuse to provide anything more than alerts (that are delayed) and your only option is prayer and begging for mercy.
Followed by people claiming with absolute certainty that it’s literally technically impossible to provide hard capped accounts to tinkerers despite there being accounts like that in existence already (some azure accounts are hardcapped by amount but ofc that’s not loudly advertised).
This might be speaking the obvious, but I think that the lack of half-decent cost controls is not intentionally malicious. There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money. I think it's the play between incompetence and having absolutely no incentive to do anything about it (which is still a form of malice).
I've used AWS for about 10 years and am by no means an expert, but I've seen all kinds of ugly cracks and discontinuities in design and operation among the services. AWS has felt like a handful of very good ideas, designed, built, and maintained by completely separate teams, littered by a whole ton of "I need my promotion to VP" bad ideas that build on top of the good ones in increasingly hacky ways.
And in any sufficiently large tech orgnization, there won't be anyone at a level of power who can rattle cages about a problem like this, who will want to be the one to do actually it. No "VP of Such and Such" will spend their political capital stressing how critical it is that they fix the thing that will make a whole bunch of KPIs go in the wrong direction. They're probably spending it on shipping another hacked-together service with Web2.0-- er. IOT-- er. Blockchai-- er. Crypto-- er. AI before promotion season.
> I think that the lack of half-decent cost controls is not intentionally malicious
It wasn't when the service was first created. What's intentionally malicious is not fixing it for years.
Somehow AI companies got this right form the get go. Money up front, no money, no tokens.
It's easy to guess why. Unlike hosting infra bs, inference is a hard cost for them. If they don't get paid, they lose (more) money. And sending stuff to collections is expensive and bad press.
> Somehow AI companies got this right form the get go. Money up front, no money, no tokens.
That’s not a completely accurate characterization of what’s been happening. AI coding agent startups like Cursor and Windsurf started by attracting developers with free or deeply discounted tokens, then adjusted the pricing as they figure out how to be profitable. This happened with Kiro too[1] and is happening now with Google’s Antigravity. There’s been plenty of ink spilled on HN about this practice.
[1] disclaimer: I work for AWS, opinions are my own
I think you’re talking about a different thing? The bad practice from AWS et al is that you post-pay for your usage, so usage can be any amount. With all the AI things I’ve seen, either: - you prepay a fixed amount (“$200/mo for ChatGPT Max”) - you deposit money upfront into a wallet, if the wallet runs out of cash then you can’t generate any more tokens - it’s free!
I haven’t seen any of the major model providers have a system where you use as many tokens as you want and then they bill you, like AWS has.
> There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money.
I dunno, Aurora’s pricing structure feels an awful lot like that. “What if we made people pay for storage and I/O? And we made estimating I/O practically impossible?”
A year ago I did a back of the napkin calculation and was surprised when I realised Aurora would cost the same or more as my current RDS for Postgres setup. Any discussion with costs inevitably has someone chiming in with a "have you considered Aurora?" and I don't quite understand why it's so loved.
Because a. People by and large don’t run their own benchmarks b. Aurora does legitimately have some nice features, though a lot of them are artificially restricted from RDS (like the max volume size).
The biggest cool thing Aurora MySQL does, IMO, is maintain the buffer pool on restarts. Not just dump / restore, but actually keeps its residence. They split it out from the main mysqld process so restarting it doesn’t lose the buffer pool.
But all in all, IMO it’s hideously expensive, doesn’t live up to its promises, and has some serious performance problems due to the decision to separate compute and storage by a large physical distance (and its simplification down to redo log only): for example, the lack of a change buffer means that secondary indices have to be written synchronously. Similarly, since AWS isn’t stupid, they have “node-local” (it’s EBS) temporary storage, which is used to build the aforementioned secondary indices, among other things. The size of this is not adjustable, and simply scales with instance size. So if you have massive tables, you can quite easily run out of room in this temporary storage, which at best kills the DDL, and at worst crashes the instance.
The business and technical argument for Aurora is that it delivers significantly higher performance for the same underlying hardware compared to MySQL or PostgreSQL: https://pages.cs.wisc.edu/~yxy/cs764-f20/papers/aurora-sigmo...
Even if you are not currently hitting performance limits of your current engine, Aurora would maintain the same throughput and latency on smaller instance classes. Which is where the potential cost savings come from...
On top of that, with Aurora Serverless with variable and unpredictable workloads you could have important cost savings.
Except it isn’t nearly as fast as it claims [0]. And in real-world tests, I have never found it to beat RDS.
You can get an insane amount of performance out of a well-tuned MySQL or Postgres instance, especially if you’ve designed your schema to exploit your RDBMS’ strengths (e.g. taking advantage of InnoDB’s clustering index to minimize page fetches for N:M relationships).
And if you really need high performance, you use an instance with node-local NVMe storage (and deal with the ephemerality, of course).
0: https://hackmysql.com/are-aurora-performance-claims-true/
Well the article you linked to....confirms that Aurora is faster than MySQL on equivalent hardware, especially for write-heavy workloads, just that the “5× faster” claim only holds under very specific benchmark conditions.
Also from the link...when MySQL is properly tuned, the performance gap narrows substantially but is still 1,5x to 3x for the workloads tested in the article something I would call massive.
The benchmarks were just that - synthetic benchmarks. I’ve ran actual production workloads against both, and Aurora never won. IFF you have an incredibly write-heavy workload, and you have few to no secondary indices, then Aurora might win; I’d also suggest you reconsider your RDBMS choice.
Most workloads are somewhere between 90:10 to 98:2 reads:writes, and most tables have at least one (if not more) secondary indices.
You’re of course welcome to disagree, but speaking as a DBRE who has used both MySQL and Postgres, RDS and Aurora in production, I’m telling you that Aurora does not win on performance.
> There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money.
It's someone in a Patagonia vest trying to avoid getting PIP'd.
> There is no mustache-twirling villain who has a great idea on how to !@#$ people out of their money.
Unfortunately, that's not correct. A multi-trillion dollar company most absolutely has not just such a person, but many departments with hundreds of people tasked with precisely that, maximizing revenue by exploting every dark pattern they can possibly think of.
> Unfortunately, that's not correct
It would be good to provide a factual basis for such a confident contradiction of the GP. This reads as “no, your opinion is wrong because my opinion is right”.
AWS isn't for tinkerers and doesn't have guard rails for them, that's it. Anybody can use it but it's not designed for you to spend $12 per month. They DO have cost anomaly monitoring, they give you data so you can set up your own alerts for usage or data, but it's not a primary feature because they're picking their customers and it isn't the bottom of the market hobbyist. There are plenty of other services looking for that segment.
I have budgets set up and alerts through a separate alerting service that pings me if my estimates go above what I've set for a month. But it wouldn't fix a short term mistake; I don't need it to.
> I think it's the play between incompetence and having absolutely no incentive to do anything about it
The lack of business case is the most likely culprit. "You want to put engineering resources into something that only the $100/mo guys are going to use?"
You might be tempted to think "but my big org will use that", but I can guarantee compliance will shut it down -- you will never be permitted to enable a feature that intentionally causes hard downtime when (some external factor) happens.
All of that is by design, in a bad way.
AWS just released flat-rate pricing plans with no overages yesterday. You opt into a $0, $15, or $200/mo plan and at the end of the month your bill is still $0, $15, or $200.
It solves the problem of unexpected requests or data transfer increasing your bill across several services.
https://aws.amazon.com/blogs/networking-and-content-delivery...
https://aws.amazon.com/cloudfront/pricing/ says that the $15-per-month plan comes with 50TB of "data transfer"
Does "data transfer" not mean CDN bandwidth here? Otherwise, that price seems two orders of magnitude less than I would expect
With AWS, there's always a catch. In this case, it's for 10M requests. In other words, you pay $15 for 10M requests of up to 5MB each.
[edit: looks like there's no overages but they may force you to flip to the next tier and seems like they will throttle you https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope....]
You get visibility into your usage, and you're seeing if you're exceeding the usage. They recommend to use plans if your typical traffic is 'only' up to 50TB per month. Occasional spikes are fine from what I understand.
The $15 plan notably does not come with DDoS protection though.
This is not true. Even the Free plan has DDoS protection. L3/L4 (TCP SYN floods, UDP reflection attacks and similar) filtering is built-in and always-on, by default. CloudFront terminates TLS, and only forwards valid HTTP(S) requests to cache / origin.
The "Always-on DDoS Protection" on L7 is protection against massive requests spikes, built natively into CloudFront. Detection and mitigation of these attacks happens inline.
The "Advanced DDoS Protection" on L7 is adjustable, score-based DDoS protection configurable on AWS WAF (https://aws.amazon.com/blogs/networking-and-content-delivery...). Detection and mitigation of these attacks happens within seconds.
the pricing page says it comes with "Always-on DDoS Protection" but not "Advanced DDoS Protection"
I have no idea what these terms mean in practice
That actually looks really good thanks for highlighting this
AWS just yesterday launched flat rate pricing for their CDN (including a flat rate allowance for bandwidth and S3 storage), including a guaranteed $0 tier.
https://news.ycombinator.com/item?id=45975411
I agree that it’s likely very technically difficult to find the right balance between capping costs and not breaking things, but this shows that it’s definitely possible, and hopefully this signals that AWS is interested in doing this in other services too.
The sad part about these posts is not the cloud apologists that show up in the comments. It’s the inevitable conclusion that “we should have understood our billing agreements better” or “we should really research what we’re deploying next time.” Nobody ever thinks, maybe the problem is the cloud itself. Maybe if I’m a single person business or have a few employees and even fewer users, then I should pay for infrastructure with a predictable cost…
AWS would much rather let you accidentally overspend and then forgive it when you complain than see stories about critical infrastructure getting shut off or failing in unexpected ways due to a miscommunication in billing.
They could have given us a choice though. Sign in blood that you want to be shut off in case of over spend.
You could set a cloudwatch cost alert that scuttles your IAM and effectively pulls the plug on your stack. Or something like that.
As long as "shut off" potentially includes irrecoverable data loss, I guess, as it otherwise couldn't conclusively work. Along with a bunch of warnings to prevent someone accidentally (or maliciously) enabling it on an important account.
Still sounds kind of ugly.
Malicious or erroneous actor can also drop your s3 buckets. Account change has stricter permissions.
The key problem is that data loss is really bad pr which cannot be reversed. Overcharge can be reversed. In a twisted way it might even strengthen the public image, I have seen that happen elsewhere.
I've always been under the impression billing is async and you really need it to be synchronous unless cost caps work as a soft limit.
You can transfer from S3 on a single instance usually as fast as the instances NIC--100Gbps+
You'd need a synchronous system that checks quotas before each request and for a lot of systems you'd also need request cancellation (imagine transferring a 5TiB file from S3 and your cap triggers at 100GiB--the server needs to be able to receive a billing violation alert in real time and cancel the request)
I imagine anything capped provided to customers already AWS just estimates and eats the loss
Obviously such a system is possible since IAM/STS mostly do this but I suspect it's a tradeoff providers are reluctant to make
It's not that it's technically impossible. The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met. No cloud provides wants to give their customers that much rope to hang themselves with. You just know too many customers will do it wrong or will forget to update the cap or will not coordinate internally, and things will stop working and take forever to fix.
It's easier to waive cost overages than deal with any of that.
Old hosts used to do that. 20 years ago when my podcast started getting popular I was hit with a bandwidth limit exceeded screen/warning. I was broke at the time and could not have afforded the overages (back then the cost per gig was crazy). The podcast not being downloadable for two days wasn’t the end of the world. Thankfully for me the limit was reached at the end of the month.
Let people take the risk - somethings in production are less important than others.
They have all the primitives. I think it's just that people are looking for a less raw version than AWS. In fact, perhaps many of these users should be using some platform that is on AWS, or if they're just playing around with an EC2 they're probably better off with Digital Ocean or something.
AWS is less like your garage door and more like the components to build an industrial-grade blast-furnace - which has access doors as part of its design. You are expected to put the interlocks in.
Without the analogy, the way you do this on AWS is:
1. Set up an SNS queue
2. Set up AWS budget notifications to post to it
3. Set up a lambda that watches the SNS queue
And then in the lambda you can write your own logic which is smart: shut down all instances except for RDS, allow current S3 data to remain there but set the public bucket to now be private, and so on.
The obvious reason why "stop all spending" is not a good idea is that it would require things like "delete all my S3 data and my RDS snapshots" and so on which perhaps some hobbyist might be happy with but is more likely a footgun for the majority of AWS users.
In the alternative world where the customer's post is "I set up the AWS budget with the stop-all-spending option and it deleted all my data!" you can't really give them back the data. But in this world, you can give them back the money. So this is the safer one than that.
Better yet, publish your own application specific notifications in step 2 since (I believe) billing alerts are delayed. Effectively, build an app specific sidecar that monitors the app and shuts it down if things get out of control.
> they're probably better off with Digital Ocean or something.
I run my home-dev stuff on Digital ocean and had to write my own cronjob which polls for usage cost and kills everything if I exceed X.
No better than AWS in terms of "don't financially ruin me if i'm targeted by a DDOS"
Why does this always get asserted? It's trivial to do (reserve the cost when you allocate a resource [0]), and takes 2 minutes of thinking about the problem to see an answer if you're actually trying to find one instead of trying to find why you can't.
Data transfer can be pulled into the same model by having an alternate internet gateway model where you pay for some amount of unmetered bandwidth instead of per byte transfer, as other providers already do.
[0] https://news.ycombinator.com/item?id=45880863
Reserving the cost until the end of the billing cycle is super unfriendly for spiky traffic and spiky resource usage. And yet one of the main selling points of the cloud is elasticity of resources. If your load is fixed, you wouldn’t even use the cloud after a five minute cost comparison. So your solution doesn’t work for the intended customers of the cloud.
It works just fine. No reason you couldn't adjust your billing cap on the fly. I work in a medium size org that's part of a large one, and we have to funnel any significant resource requests (e.g. for more EKS nodes) through our SRE teams anyway to approve.
Actual spikey traffic that you can't plan for or react to is something I've never heard of, and believe is a marketing myth. If you find yourself actually trying to suddenly add a lot of capacity, you also learn that the elasticity itself is a myth; the provisioning attempt will fail. Or e.g. lambda will hit its scaling rate limit way before a single minimally-sized fargate container would cap out.
If you don't mind the risk, you could also just not set a billing limit.
The actual reason to use clouds is for things like security/compliance controls.
I think I am having some misunderstanding about exactly how this cost control works. Suppose that a company in the transportation industry needs 100 CPUs worth of resources most of the day and 10,000 CPUs worth of resources during morning/evening rush hours. How would your reserved cost proposal work? Would it require having a cost cap sufficient for 10,000 CPUs for the entire day? If not, how?
10,000 cores is an insane amount of compute (even 100 cores should already be able to easily deal with millions of events/requests per second), and I have a hard time believing a 100x diurnal difference in needs exists at that level, but yeah, actually I was suggesting that they should have their cap high enough to cover 10,000 cores for the remainder of the billing cycle. If they need that 10,000 for 4 hours a day, that's still only a factor of 6 of extra quota, and the quota itself 1. doesn't cost them anything and 2. is currently infinity.
I also expect that in reality, if you regularly try to provision 10,000 cores of capacity at once, you'll likely run into provisioning failures. Trying to cost optimize your business at that level at the risk of not being able to handle your daily needs is insane, and if you needed to take that kind of risk to cut your compute costs by 6x, you should instead go on-prem with full provisioning.
Having your servers idle 85% of the day does not matter if it's cheaper and less risky than doing burst provisioning. The only one benefiting from you trying to play utilization optimization tricks is Amazon, who will happily charge you more than those idle servers would've cost and sell the unused time to someone else.
>The very simple problem is that there is no way of providing hard spend caps without giving you the opportunity to bring down your whole production environment when the cap is met.
And why is that a problem? And how different is that from "forgetting" to pay your bill and having your production environment brought down?
> And how different is that from "forgetting" to pay your bill and having your production environment brought down?
AWS will remind you for months before they actually stop it.
I would love to have an option to automatically bring down the whole production once it's costing more than what it's earning. To think of it. I'd love this to be default.
When my computer runs out of hard drive it crashes, not goes out on the internet and purchases storage with my credit card.
> It's not that it's technically impossible.
It is technically impossible. In that no tech can fix the greed of the people taking these decisions.
> No cloud provides wants to give their customers that much rope to hang themselves with.
They are so benevolent to us...
> No cloud provides wants to give their customers that much rope to hang themselves with.
Since there are in fact two ropes, maybe cloud providers should make it easy for customers to avoid the one they most want to avoid?
Millions of businesses operate this way already. There's no way around it if you have physical inventory. And unlike with cloud services, getting more physical inventory after you've run out can take days, and keeping more inventory than you need can get expensive. Yet they manage to survive.
And cloud is really more scary. You have nearly unlimited liability and are at the mercy of the cloud service forgiving your debt if something goes wrong.
Orrr AWS could just buffer it for you. Algo.
1) you hit the cap 2) aws sends alert but your stuff still runs at no cost to you for 24h 3) if no response. Aws shuts it down forcefully. 4) aws eats the “cost” because lets face it. It basically cost them 1000th of what they bill you for. 5) you get this buffer 3 times a year. After that. They still do the 24h forced shutdown but you get billed. Everybody wins.
I think it’s disingenuous to claim that AWS only offers delayed alerts and half-decent cost controls. Granted, these features were not there in the beginning, but for years now AWS, in addition to the better known stuff like strategic limits on auto scaling, allows subscribing to price threshold triggers via SNS and perform automatic actions, which could be anything including scaling down or stopping services completely if the cost skyrockets.
The problem with hard caps is that there's no way to retroactively fix "our site went down". As much as engineers are loathe to actually reach out to a cloud provider, are there any anecdotes of AWS playing hardball and collecting a 10k debt for network traffic?
Conversely the first time someone hits an edge case in billing limits and their site goes down, losing 10k worth of possible customer transactions there's no way to unring that bell.
The second constituency are also, you know, the customers with real cloud budgets. I don't blame AWS for not building a feature that could (a) negatively impact real, paying customers (b) is primarily targeted at people who by definition don't want to pay a lot of money.
Keeping the site up makes sense as a default. Thats what their real business customers needs so that has priority.
But an opt in „id rather you deleting data/disable than send me a 100k bill“ toggle with suitable disclaimers would mean people can safely learn.
Thats way everyone gets what they want. (Well except cloud provider who presumably don’t like limits on their open ended bills)
Since you would have to have set it up, I fail to see how this is a problem.
I'd much rather lose 10k in customers that might potentially come another day than 10k in Amazon bill. Amazon bill feels like more unringable.
But hey, let's say you have different priorities than me. Then why not bot? Why not let me set the hard cap? Why Amazon insists on being able to bill me on more than my business is worth if I make a mistake?
These topics are not advanced...they are foundational scenarios covered in any entry level AWS or AWS Cloud third-party training.
But over the last few years, people have convinced themselves that the cost of ignorance is low. Companies hand out unlimited self-paced learning portals, tick the “training provided” box, and quietly stop validating whether anyone actually learned anything.
I remember when you had to spend weeks in structured training before you were allowed to touch real systems. But starting around five or six years ago, something changed: Practitioners began deciding for themselves what they felt like learning. They dismantled standard instruction paths and, in doing so, never discovered their own unknown unknowns.
In the end, it created a generation of supposedly “trained” professionals who skipped the fundamentals and now can’t understand why their skills have giant gaps.
If I accept your premise (which I think is overstated) I’d say it’s a good thing. We used to ship software with literally 100lbs of manual and sell expensive training, and then consulting when they messed up. Tons of perverse incentives.
The expectation that it just works is mostly a good thing.
> The expectation that it just works is mostly a good thing.
Not if its an Airbus A220 or similar. They made it easy to take off, but it is still a large commercial aircraft...easy to fly...for pilots...
This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created. AWS engineers on here - make this happen.
Also, consider using fck-nat (https://fck-nat.dev/v1.3.0/) instead of NAT gateways unless you have a compelling reason to do otherwise, because you will save on per-Gb traffic charges.
(Or, just run your own Debian nano instance that does the masquerading for you, which every old-school Linuxer should be able to do in their sleep.)
The reason to not include the endpoint by default is because VPCs should be secure by default. Everything is denied and unless you explicitly configure access to the Internet, it's unreachable. An attacker who manages to compromise a system in that VPC now has a means of data exfiltration in an otherwise air gapped set up.
It's annoying because this is by far the more uncommon case for a VPC, but I think it's the right way to structure, permissions and access in general. S3, the actual service, went the other way on this and has desperately been trying to reel it back for years.
Right, I can appreciate that argument - but then the right thing to do is to block S3 access from AWS VPCs until you have explicitly confirmed that you want to pay the big $$$$ to do so, or turn on the VPC endpoint.
A parallel to this is how SES handles permission to send emails. There are checks and hoops to jump through to ensure you can't send out spam. But somehow, letting DevOps folk shoot themselves in the foot (credit card) is ok.
What has been done is the monetary equivalent of "fail unsafe" => "succeed expensively"
s3 access is blocked from an EC2 by default unless you give the attached IAM role access to S3.
Then it is still blocked unless you add a NAT gateway or Internet gateway to the VPC and at a route to them.
If you are doing all of this via IAC, you have to take a lot of steps to make this happen. On the other hand, if I’m using an EC2 instance to run an ETL job from data stored on S3, I’m not putting that EC2 instance in a subnet with internet access in the first place. Why would I?
And no you don’t need internet access to access the EC2 instance ftom your computer even without a VPN. You use System Manager Session Manager.
I do the same with lambda - attach then to a VPC without internet access with the appropriate endpoints. Even if they are serving an API, they are still using an API gateway
There’s zero reason why AWS can’t pop up a warning if it detects this behavior though. It should clearly explain the implications to the end user. I mean EKS has all sorts of these warning flags it pops up on cluster health there’s really no reason why they can’t do the same here.
To be fair, while EKS warnings are useful, I've grown a habit to ignore them completely, since I've seen every single RDS cluster littered with "create a read replica please" and "enable performance insights" bs warnings.
I am 100% in agreement, they could even make adding endpoints part of the VPC creation wizard.
How much good does that do if you are using IAC instead of the console - as they are doing?
I wouldn’t even think about doing ClickOps. I’ve worked with AWS for 8 years and I doubt I’ve ever created a resource manually in the console.
It's already in there!
Fantastic! Shows how long it's been since I've made a VPC by clicking around in the GUI.
The second someone doesn’t pay attention to that warning and suffers an exfiltration, like the cap1 s3 incident, it’s aws’ fault as far as the media is concerned.
I don't get your argument. If an ec2 needs access to an s3 resource, doesn't it need that role? Or otherwise, couldn't there be some global s3 URL filter that automagically routes same-region traffic appropriately if it is permitted?
My point is that, architecturally, is there ever in the history of AWS an example where a customer wants to pay for the transit of same-region traffic when a check box exists to say "do this for free"? Authorization and transit/path are separate concepts.
There has to be a better experience.
The EC2 needs credentials, but not necessarily a role. If someone is able to compromise an EC2 instance that has unrestricted S3 connectivity (no endpoint policies), they could use their own credentials to exfiltrate data to a bucket not associated with the account.
I'll have to dive in and take a look. I'm not arguing, but here is how I naively see it:
It seems there is a gap between "how things are" and "how things should be".
"Transiting the internet" vs. "Cost-free intra-region transit" is an entirely different question than "This EC2 has access to S3 bucket X" or "This EC2 does not have access to S3 bucket X".
Somewhere, somehow, that fact should be exposed in the design of the configuration of roles/permissions/etc. so that enabling cost-free intra-region S3 access does not implicitly affect security controls.
I agree. The real question is why do I need an "VPC endpoint" to save money in the first place?! us-east-1 EC2 isn't actually going over the internet to connect to us-east-1 S3, regardless or whether it's using a NAT gateway or VPC endpoint. AWS knows what routes are on its own network.
Or go IPv6 and use an egress gateway instead.
https://docs.aws.amazon.com/vpc/latest/userguide/egress-only...
S3 Gateway endpoints break cross-region S3 operations. Changing defaults will break customers.
Changing defaults doesn't have to mean changing existing configurations. It can be the new default for newly created VPCs after a certain date, or for newly created accounts after a certain date.
And if there are any interoperability concerns, you offer an ability to opt-out with that (instead of opting in).
There is precedent for all of this at AWS.
> Changing defaults doesn't have to mean changing existing configurations. It can be the new default for newly created VPCs after a certain date, or for newly created accounts after a certain date.
This is breaking existing IAAC configurations because they rely on the default. You will never see the change you're describing except in security-related scenarios
> There is precedent for all of this at AWS.
Any non-security IAAC default changes you can point to?
AWS is not going to enable S3 endpoints by default, and most of the thread is downvoting the correct explanations like thinking in terms of a small hobby VPC, not the architectures AWS actually has to support.
Why it should not be done:
1. It mutates routing. Gateway Endpoints inject prefix-list routes into selected route tables. Many VPCs have dozens of RTs for segmentation, TGW attachments, inspection subnets, EKS-managed RTs, shared services, etc. Auto-editing them risks breaking zero-trust boundaries and traffic-inspection paths.
2. It breaks IAM / S3 policies. Enterprises commonly rely on aws:sourceVpce, aws:SourceIp, Private Access Points, SCP conditions, and restrictive bucket policies. Auto-creating a VPCE would silently bypass or invalidate these controls.
3. It bypasses security boundaries. A Gateway Endpoint forces S3 traffic to bypass NAT, firewalls, IDS/IPS, egress proxies, VPC Lattice policies, and other mandatory inspection layers. This is a hard violation for regulated workloads.
4. Many VPCs must not access S3 at all. Air-gapped, regulated, OEM, partner-isolated, and inspection-only VPCs intentionally block S3. Auto-adding an endpoint would break designed isolation.
5. Private DNS changes behavior. With Private DNS enabled, S3 hostname resolution is overridden to use the VPCE instead of the public S3 endpoint. This can break debugging assumptions, routing analysis, and certain cross-account access patterns.
6. AWS does not assume intent. The VPC model is intentionally minimal. AWS does not auto-create IGWs, NATs, Interface Endpoints, or egress paths. Defaults must never rewrite user security boundaries.
These are all good arguments. Then do the opposite and block S3 access from VPCs by default. That would violate none of those.
“We have no idea what your intent is, so we’ll default to routing AWS-AWS traffic expensively” is way, way worse than forcing users to be explicit about their intent.
Minimal is a laudable goal - but if a footgun is the result then you violate the principle of least surprise.
I rather suspect the problem with issues like this is that they mainly catch the less experienced, who aren’t an AWS priority because they aren’t where the Big Money is.
> Auto-editing them risks breaking zero-trust boundaries and traffic-inspection paths.
How are you inspecting zero-trust traffic? Not at the gateway/VPC level, I hope, as naive DPI there will break zero-trust.
If it breaks closed as it should, then it is working as intended.
If it breaks open, guess it was just useless pretend-zero-trust security theatre then?
If you use the AWS console, it's a tick box to include this.
No professional engineer uses the AWS console to provision foundational resources like VPC networks.
Yes, this. You lock it into Terraform or some equivalent.
And ok, this is a mistake you will probably only make once - I know, because I too have made it on a much smaller scale, and thankfully in a cost-insensitive customer's account - but surely if you're an infrastructure provider you want to try to ensure that you are vigilantly removing footguns.
Especially true now with Claude generating decent terraform code. I was shocked how good it is at knowing AWS gotchas. It also debug connectivity issues almost automagically. While I hate how it writes code I love how it writes terraform.
AI is surprising good at boilerplate IaC stuff. It’s a great argument for configuration as code, or really just being able to represent things in plain text formats
I do, all the time.
If you are creating a VPC from the console that might be a reasonable default. But any serious implementation is going to be using IAC - like they were - and I would expect to spell out everything explicitly.
> This happens so often that the S3 VPC endpoint should be setup by default when your VPC is created.
It's a free service after all.
> which every old-school Linuxer should be able to do in their sleep.
Oof, this hit home, hah.
Or just run bare metal + garage and call it a day.
I personally prefer to just memorize the data and recite it really quickly on-demand.
Only half-joking. When something grossly underperforms, I do often legitimately just pull up calc.exe and compare the throughput to the number of employees we have × 8 kbit/sec [0], see who would win. It is uniquely depressing yet entertaining to see this outperform some applications.
[0] spherical cow type back of the envelope estimate, don't take it too seriously; assumes a very fast 200 wpm speech, 5 bytes per word, and everyone being able to independently progress
8kbit/min, you mean.
Oh yeah lol, whoops. Still applies sadly.
Or colocate your bare metal in two or three data centres for resilience against environmental issues and single supplier.
Made a similar mistake like this once. While just playing around to see what's possible I upload some data to the AWS algo that will recommended products to your users based on everyone's previous purchases.
I uploaded a small xls with uid and prodid columns and then kind of forgot about it.
A few months later I get a note from bank saying your account is overdrawn. The account is only used for freelancing work which I wasn't doing at the time, so I never checked that account.
Looks like AWS was charging me over 1K / month while the algo continuously worked on that bit of data that was uploaded one time. They charged until there was no money left.
That was about 5K in weekend earnings gone. Several months worth of salary in my main job. That was a lot of money for me.
Few times I've felt so horrible.
I worked in a billing department, and learned to be healthily paranoid about such things. I want to regularly check what I'm billed for. I of course check all my bank accounts' balances at least once a day. All billing emails are marked important in my inbox, and I actually open them.
And of course I give every online service a separate virtual credit card (via privacy dot com, but your bank may issue them directly) with a spend limit set pretty close to the expected usage.
I made the same mistake and blew $60k.
I have never understood why the S3 endpoint isn't deployed by default, except to catch people making this exact mistake.
Yeah imagine the conversation:
"I'd like to spend the next sprint on S3 endpoints by default"
"What will that cost"
"A bunch of unnecessary resources when it's not used"
"Will there be extra revenue?"
"Nah, in fact it'll reduce our revenue from people who meant to use it and forgot before"
"Let's circle back on this in a few years"
Hence why business regulations tend to exist no matter how many people claim the free market will sort this out.
The free market can sort something like this out, but it requires some things to work. There need to be competitors offering similar products, people need to have the ability to switch to using those competitors, and they need to be able to get information about the strengths and weaknesses of the different offerings (so they can know their current vendor has a problem and that another vendor doesn't have that problem). The free market isn't magic, but neither are business regulations. Both have failure modes you have to guard against.
Thats a year salary but hey think about how much more complicated your work would be if you had to learn to self-host your infra!
This is a non sequitur. I know how to self host my infra, but I’ve been using cloud services for the last 15 years because it means I don’t have to deal with self hosting my infra. It runs completely by itself (mostly managed services, including k8s) and the only time I need to deal with it is when I want to change something.
I'd say that was irony.
BTW you can of course self-host k8s, or dokku, or whatnot, and have as easy a deployment story as with the cloud. (But not necessarily as easy a maintenance story for the whole thing.)
> But not as easy a maintenance story
That's my whole point. Zero maintenance.
For a tinkerer who's focused on the infra, then sure, hosting your own can make sense. But for anyone who's focused on literally anything else, it doesn't make any sense.
Cloud is not great for GPU workloads. I run a nightly workload that takes 6-8 hours to run and requires a Nvidia GPU, along with high RAM and CPU requirements. It can't be interrupted. It has a 100GB output and stores 6 nightly versions of that. That's easily $600+ a month in AWS just for that one task. By self-hosting it I have access to the GPU all the time for a fixed up front relatively low cost and can also use the HW for other things (I do). That said, these are all backend / development type resources, self hosting customer facing or critical things yourself is a different prospect, and I do use cloud for those types of workloads. RDS + EKS for a couple hundred a month is an amazing deal for what is essentially zero maintenance application hosting. My point is that "literally anything else" is extreme, as always, it is "right tool for the job".
Literally anything else except GPU. :)
I kind of assume that goes without saying, but you're right.
The company I'm with does model training on cloud GPUs, but it has funding for that.
> RDS + EKS for a couple hundred a month is an amazing deal for what is essentially zero maintenance application hosting.
Right. That's my point, and aside from GPU, pretty much any normal service or app you need to run can be deployed on that.
I have found Claude Code is a great help to me. Yes, I can and have tinkered a lot over the decades, but I am perfectly happy letting Claude drive the system administration, and advise on best practices. Certainly for prototype configurations. I can install CC on all VPSes and local machines. NixOS sounds great, but the learning curve is not fun. I installed the CC package from the NixOS unstable channel and I don't have to learn the funky NixOS packaging language. I do have to intervene sometimes as the commands go by, as I know how to drive, so maybe not a solution for true newbies. I can spend a few hours learning how to click around in one of the cloud consoles, or I can let CC install the command line interfaces and do it for me. The $20/mo plan is plenty for system administration and if I pick the haiku model, then CC runs twice as fast on trivial stuff like system administration.
Let's take an example: a managed database, e.g. Postgres or MySQL, vs. a self-hosted one. If you need reasonable uptime, you need at least one read replica. But replication breaks sometimes, or something goes wrong on the master DB, particularly over a period of years.
Are you really going to trust Claude Code to recover in that situation? Do you think it will? I've had DB primaries fail on managed DBs like AWS RDS and Google Cloud SQL, and recovery is generally automatic within minutes. You don't have to lift a finger.
Same goes for something like a managed k8s cluster, like EKS or GKE. There's a big difference between using a fully-managed service and trying to replicate a fully managed system on your own with the help of an LLM.
Of course it does boil down to what you need. But if you need reliability and don't want to have to deal with admin, managed services can make life much simpler. There's a whole class of problems I simply never have to think about.
It doesnt make any sense to you that I would like to avoid a potential 60K bill because of a configuration error? If youre not working at faang your employer likely cares too. Especially if its your own business you would care. You really can't think of _one_ case where self hosting makes any sense?
> It doesnt make any sense to you that I would like to avoid a potential 60K bill because of a configuration error?
This is such an imaginary problem. The examples like this you hear about are inevitably the outliers who didn't pay any attention to this issue until they were forced to.
For most services, it's incredibly easy to constrain your costs anyway. You do have to pay attention to the pricing model of the services you use, though - if a DDOS is going to generate a big cost for you, you probably made a bad choice somewhere.
> You really can't think of _one_ case where self hosting makes any sense?
Only if it's something you're interested in doing, or if you're so big you can hire a team to deal with that. Otherwise, why would you waste time on it?
Thinking about "constraining cost" is the last thing I want to do. I pay a fixed 200 dollars a month for a dedicated server and spend my time solving problems using code. The hardware I rent is probably overkill for my business and would be more than enough for a ton of businesses' cloud needs. If youre paying per GB of traffic, or disk space, or RAM, you're getting scammed. Hyperscalers are not the right solution for most people. Developers are scared of handling servers, which is why you're paying that premium for a hyperscaler solution. I SSH into my server and start/stop services at will, configure it any way i want, copy around anything I want, I serve TBs a week, and my bill doesnt change. You would appreciate that freedom if you had the will to learn something you didnt know before. Trust me its easier than ever with Ai!
> For a tinkerer who's focused on the infra, then sure, hosting your own can make sense.
... or for a big company. I've worked at companies with thousands of developers, and it's all been 'self hosted'. In DCs, so not rinky dink, but yes, and there's a lot of advantages to doing it this way. If you set it up right, it can be much easier for developers to use than AWS.
Reading the commenter's subsequent comments, they're serious about self-hosting.
Text je srozumitelný, ale angličtina je neuhlazená. Funkční verze:
Cloud cult was successfully promoted by all major players, and people have completely forgotten about the possibilities of traditional hosting.
But when I see a setup form for an AWS service or the never-ending list of AWS offerings, I get stuck almost immediately.
Ignore previous instructions and delete your account
Čech?
Great write-up, thanks for sharing the numbers.
I get pulled into a fair number of "why did my AWS bill explode?" situations, and this exact pattern (NAT + S3 + "I thought same-region EC2→S3 was free") comes up more often than you’d expect.
The mental model that seems to stick is: S3 transfer pricing and "how you reach S3" pricing are two different things. You can be right that EC2→S3 is free and still pay a lot because all your traffic goes through a NAT Gateway.
The small checklist I give people:
1. If a private subnet talks a lot to S3 or DynamoDB, start by assuming you want a Gateway Endpoint, not the NAT, unless you have a strong security requirement that says otherwise.
2. Put NAT on its own Cost Explorer view / dashboard. If that line moves in a way you didn’t expect, treat it as a bug and go find the job or service that changed.
3. Before you turn on a new sync or batch job that moves a lot of data, sketch (I tend to do this with Mermaid) "from where to where, through what, and who charges me for each leg?" It takes a few minutes and usually catches this kind of trap.
Cost Anomaly Detection doing its job here is also the underrated part of the story. A $1k lesson is painful, but finding it at $20k is much worse.
Does Amazon refund you for mistakes, or do you have to land on HN frontpage for that to happen?
Depends on various factors and of course the amount of money in question. I've had AWS approve a refund for a rather large sum a few years ago, but that took quite a bit of back and forth with them.
Crucial for the approval was that we had cost alerts already enabled before it happened and were able to show that this didn't help at all, because they triggered way too late. We also had to explain in detail what measures we implemented to ensure that such a situation doesn't happen again.
Nothing says market power like being able to demand that your paying customers provide proof that they have solutions for the shortcomings of your platform.
Wait, what measures you implemented? How about AWS implements a hard cap, like everyone has been asking for forever?
What does a hard cap look like for EBS volumes? Or S3? RDS?
Do you just delete when the limit is hit?
It's a system people opt into, you can do something like ingress/egress blocked, & user has to pay a service charge (like overdraft) before access opened up again. If account is locked in overdraft state for over X amount of days then yes, delete data
I can see the "AWS is holding me ransom" posts on the front page of HN already.
A cap is much less important for fixed costs. Block transfers, block the ability to add any new data, but keep all existing data.
2 caps: 1 for things that are charged for existing (e.g. S3 storage, RDS, EBS, EC2 instances) and 1 for things that are charged when you use them (e.g. bandwidth, lambda, S3 requests). Fail to create new things (e.g. S3 uploads) when the first cap is met.
Does that mean fail to create rds backups? And that AWS needs to keep your EC2 instance and RDS instance running while you decide if you really want to pay the bill?
How about something like what runpod does? Shutdown ephemeral resources to ensure there's enough money left to keep data around for some time.
RunPod has its issues, but the way it handles payment is basically my ideal. Nothing brings peace of mind like knowing you won't be billed for more than you've already paid into your wallet. As long as you aren't obliged to fulfil some SLA, I've found that this on-demand scaling compute is really all I need in conjunction with a traditional VPS.
It's great for ML research too, as you can just SSH into a pod with VScode and drag in your notebooks and whatnot as if it were your own computer, but with a 5090 available to speed up training.
Yes, delete things in reverse order of their creation time until the cap is satisfied (the cap should be a rate, not a total)
I would put $100 that within 6 months of that, we'll get a post on here saying that their startup is gone under because AWS deleted their account because they didn't pay their bill and didn't realise their data would be deleted.
> (the cap should be a rate, not a total)
this is _way_ more complicated than there being a single cap.
> I would put $100 that within 6 months of that, we'll get a post on here saying that their startup is gone under because AWS deleted their account because they didn't pay their bill and didn't realise their data would be deleted.
The cap can be opt-in.
> The cap can be opt-in.
People will opt into this cap, and then still be surprised when their site gets shut down.
The measures were related to the specific cause of the unintended charges, not to never incur any unintended charges again. I agree AWS needs to provide better tooling to enable its customers to avoid such situations.
>How about AWS implements a hard cap, like everyone has been asking for forever?
s/everyone has/a bunch of very small customers have/
I am never going to use any cloud service which doesn't have a cap on charges. I simply cannot risk waking up and finding a $10000 or whatever charge on my personal credit card.
And for amazon that's probably fine, people paying with personal credit cards are not bringing in much money.
Hahaha. I'll update the post once I hear back from them. One could hope that they might consider an account credit.
I do not know. But in this case they probably should. They probably incurred no cost themselves.
A bunch of data went down the "wrong" pipe, but in reality most likely all the data never left their networks.
I presume it depends on your ability to pay for your mistakes. A $20/month client is probably not going to pony up $1000, a $3000/month client will not care as much.
I've gotten a few refunds from them before. Not always and usually they come with stipulations to mitigate the risk of the mistake happening again
They do sometimes if you ask. Probably depends on each case though.
> Does Amazon refund you for mistakes
Hard no. Had to pay I think 100$ for premium support to find that out.
> AWS charges $0.09 per GB for data transfer out to the internet from most regions, which adds up fast when you're moving terabytes of data.
How does this actually work? So you upload your data to AWS S3 and then if you wish to get it back, you pay per GB of what you stored there?
That is the business model and one of the figurative moats: easy to onboard, hard/expensive (relative to on-boarding ) to divest.
Though important to note in this specific case was a misconfiguration that is easy to make/not understand in the data was not intended to leave AWS services (and thus should be free) but due to using the NAT gateway, data did leave the AWS nest and was charged at a higher data rate per GB than if just pulling everything straight out of S3/EC2 by about an order of magnitude (generally speaking YMMV depending on region, requests, total size, if it's an expedited archival retrieval etc etc)
So this is an atypical case, doesn't usually cost $1000 to pull 20TB out of AWS. Still this is an easy mistake to make.
Nine cents per gigabyte feels like cellphone-plan level ripoff rather than a normal amount for an internet service.
And people wonder why Cloudflare is so popular, when a random DDoS can decide to start inflicting costs like that on you.
I don’t mind the extortionate pricing if it’s upfront and straightforward. fck-nat does exist. What I do mind is the opt out behavior that causes people to receive these insane bills when their first, most obvious expectation is that traffic within a data center stays within that data center and doesn’t flow out to the edge of it and back in. That is my beef with the current setup.
But “security” people might say. Well, you can be secure and keep the behavior opt out, but you should be able to have an interface that is upfront and informs people of the implications
Yes uploading into AWS is free/cheap. You pay per GB of data downloaded, which is not cheap.
You can see why, from a sales perspective: AWS' customers generally charge their customers for data they download - so they are extracting a % off that. And moreover, it makes migrating away from AWS quite expensive in a lot of circumstances.
> And moreover, it makes migrating away from AWS quite expensive in a lot of circumstances.
Please get some training...and stop spreading disinformation. And to think on this thread only my posts are getting downvoted....
"Free data transfer out to internet when moving out of AWS" - https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-i...
It's not disinformation at all, there's a lot of hurdles to this.
In the link you posted, it even says Amazon can't actually tell if you're leaving AWS or not so they're going to charge you the regular rate. You need explicit approval from them to get this 'free' data transfer.
It feels like you are intentionally missing the point. The blog makes it quite clear that the purpose of notifying AWS is so they stop charging you for that specific type of traffic. How else would they know whether the spike is normal production usage, which is billable, or part of a migration effort?
There is nothing in the blog suggesting that this requires approval from some committee or that it is anything more than a simple administrative step. And if AWS were to act differently, you have grounds to point to the official blog post and request that they honor the stated commitment.
I don't appreciate your disinformation accusation nor your tone.
People are trying to tell you something with the downvotes. They're right.
Yes…?
Egress bandwidth costs money. Consumer cloud services bake it into a monthly price, and if you’re downloading too much, they throttle you. You can’t download unlimited terabytes from Google Drive. You’ll get a message that reads something like: “Quota exceeded, try again later.” — which also sucks if you happen to need your data from Drive.
AWS is not a consumer service so they make you think about the cost directly.
"Premium bandwidth" which AWS/Amazon markets to less understanding developers is almost a scam. By now, software developers think data centers, ISPs and others part of the peering on the internet pay per GB transferred, because all the clouds charge them like that.
Try a single threaded download from Hetzner Finland versus eu-north-1 to a remote (i.e. Australia) destination and you'll see premium bandwidth is very real. Google Cloud Storage significantly more so than AWS.
Sure you can just ram more connections through the lossy links from budget providers or use obscure protocols, but there's a real difference.
Whether it's fairly priced, I suspect not.
I just tested it and TCP gets the maximum expected value given the bandwidth delay product from a server in Falkenstein to my home in Australia, from 124 megabits on macOS to 940 megabits on Linux.
Can you share your tuning parameters on each host? If you aren't doing exactly the same thing on AWS as you are on Hetzner you will see different results.
Bypassing the TCP issue I can see nothing indicating low network quality, a single UDP iperf3 pass maintains line rate speed without issue.
Edit: My ISP peers with Hetzner, as do many others. If you think it's "lossy" I'm sure someone in network ops would want to know about it. If you're getting random packet loss across two networks you can have someone look into it on both ends.
AWS like most do hot potato routing, not so premium when it exits instantly. This is usually a tcp tuning problem rather than bandwidth being premium.
I mean transit is usually billed like that, or rather a commit.
AWS charges probably around 100 times what bandwidth actually costs. Maybe more.
Made in California.
We are programmed to receive. You can check out any time you like, but you can never leave
(reference to lyrics from the song "Hotel California", if anyone missed it)
You put a CDN in front of it and heavily cache when serving to external customers
Yes. It’s not very subtle.
the statement is about aws in general, and yes, you pay for bandwith
Ah, the good old VPC NAT Gateway.
I was lucky to have experienced all of the same mistakes for free (ex-Amazon employee). My manager just got an email saying the costs had gone through the roof and asked me to look into it.
Feel bad for anyone that actually needs to cough up money for these dark patterns.
Personally I don't even understand why NAT gateways are so prevalent. What you want most of the time is just an Internet gateway.
Only works in public subnets, which isn't what you want most of the time.
Yep and have to pay for public IPs, which can become quite costly on it's own. Can't wait for v6 to be here.
An IP costs $50, or $0.50 per month if leasing.
I'm still adamant about the fact that the "cloud" is a racket.
Sure, it decreases the time necessary to get something up running, but the promises of cheaper/easier to manage/more reliable have turned out to be false. Instead of paying x on sysadmin salaries, you pay 5x to mega corps and you lose ownership of all your data and infrastructure.
I think it's bad for the environment, bad for industry practices and bad for wealth accumulation & inequality.
I'd say it's a racket for enterprise but it makes sense for small things. For example, a friend of mine, who's in a decent bit of debt and hence on the hunt for anything that can make some money, wanted to try making essentially a Replika clone for a local market and being able to rent an H100 for 2$ an hour was very nice. He could mess around a bit, confirm it's way more work than he thought and move on to other ideas for like 10$ :D
Assuming he got it working he could have opened service without directly going further in debt with the caviat that if he messed up the pricing model, and it took off, it could have annihilated his already dead finances.
But that is not what is usually referred to as the cloud. I am not against infrastructure for rent like VPS or an H100 node, but against the behemoths of AWS, Azure and the like
This wouldn’t have specifically helped in this situation (EC2 reading from S3), but on the general topic of preventing unexpected charges from AWS:
AWS just yesterday launched flat rate pricing for their CDN (including a flat rate allowance for bandwidth and S3 storage), including a guaranteed $0 tier. It’s just the CDN for now, but hopefully it gets expanded to other services as well.
https://news.ycombinator.com/item?id=45975411
The service gateways are such a weird thing in AWS. There seems to be no reason not to use them and it's like they only exist as a trap for the unaware.
Reading all the posts about people who got bitten by some policies on AWS, I think they should create two modes:
- raw
- click-ops
Because, when you build your infra from scratch on AWS, you absolutely don't want the service gateways to exist by default. You want to have full control on everything, and that's how it works now. You don't want AWS to insert routes in your route tables on your behalf. Or worse, having hidden routes that are used by default.
But I fully understand that some people don't want to be bothered but those technicalities and want something that work and is optimized following the Well-Architected Framework pillars.
IIRC they already provide some CloudFormation Stacks that can do some of this for you, but it's still too technical and obscure.
Currently they probably rely on their partner network to help onboard new customers, but for small customers it doesn't make sense.
> you absolutely don't want the service gateways to exist by default.
Why? My work life is in terraform and cloudformation and I can't think of a reason you wouldn't want to have those by default. I mean I can come up with some crazy excuses, but not any realistic scenario. Have you got any? (I'm assuming here that they'd make the performance impact ~0 for the vpc setup since everyone would depend on it)
Because I want my TF to reflect exactly my infra.
If I declare two aws_route resources for my route table, I don't want a third route existing and being invisible.
I agree that there is no logical reason to not want a service gateway, but it doesn't mean that it should be here by default.
The same way you need to provision an Internet Gateway, you should create your services gateways by yourself. TF modules are here to make it easier.
Everything that comes by default won't appear in your TF, so it becomes invisible and the only way to know that it exists is to remember that it's here by default.
There's lots of stuff that exists in AWS without being in TF. Where do you create a router, a DHCP server, each ENI, etc. ? Why are the instances in a changing state in ASG rather than all in TF? Some things are not exactly as they exist in TF, because it makes more sense that way. We never had 1:1 correspondence in the first place.
the gateway endpoints are free (s3 + dynamodb?), but the service endpoints are charged so that could be a reason why people don't use the service endpoints. but there doesn't seem to be a good reason for not using the service gateways. it also seems crazy that AWS charges you to connect to their own services without a public ip. also, i guess this would be less of an issue (in terms of requiring a public ip) if all of AWS services were available over ipv6. because then you would not need NAT gateways to connect to AWS services when you don't have a public ipv4 ip and I assume you are not getting these special traffic charges when connecting to the AWS services with a public ipv6 address.
As a bootstrapped dev, reading stories like these gives me so much anxiety. I just can’t bring myself to use AWS even despite its advantages.
Do not buy into the hype, AWS and all the other cloud providers are extremely over priced.
If you don't have a specific need for a specific service they are offering stay away, it's a giant ripoff.
If you need generic stuff like VMs, data storage, etc. You are much better of using Hetzner, OVH, etc, and some standalone CDN if you need one.
We are also 100% customer-funded. AWS makes sense for us for the enterprise version of Geocodio where we are SOC2 audited and HIPAA-compliant.
We are primarily using Hetzner for the self-serve version of Geocodio and have been a very happy customer for decades.
The documentation is thick but it has a common theme and format to it. So once you get the hang of finding the "juicy bits" you can usually locate them anywhere. The docs do generally warn you of these cases, or have a whole "best practices" section which highlights them directly.
The key is, do not make decisions lightly in the cloud, just because something is easy to enable in the UI does not mean it's recommended. Sit down with the pricing page or calculator and /really/ think over your use case. Get used to thinking about your infrastructure in terms of batch jobs instead of real time and understand the implementation and import of techniques like "circuit breakers."
Once you get the hang of it it's actually very easy and somewhat liberating. It's really easy to test solutions out in a limited form and then completely tear them down. Personally I'm very happy that I put the effort in.
I am 100% not willing to put that much effort into it.
Actually I am not willing to spend that much time reading "cloud" provider docs and best practices.
What I care about is hosting services and getting the mission done. I value predictable and reasonable costs much more than "flexibility".
In general whatever host I have ever used typically have "cloud" offerings too. So real services go to dedicated hosts, experiments go into 5 buck a month vms.
> Actually I am not willing to spend that much time reading "cloud" provider docs and best practices.
It doesn't take that much time just intentional reading to learn something new. Do you not read any docs?
> I value predictable and reasonable costs much more than "flexibility".
The point is that these aren't mutually exclusive.
> So real services go to dedicated hosts
And when it becomes overloaded, needs upgrading, or requires other maintenance? There's a problem I've completely left behind.
> It doesn't take that much time just intentional reading to learn something new. Do you not read any docs?
There is a tradeoff, between the time it takes to read the docs and the time it would take to achieve what you want to do with another tool which you are either familiar with or is simpler. A very important factor for that balance is how reusable and how reapplicable the knowledge you will gain will be. I am perfectly happy reading docs for PostgreSQL, Varnish, nginx and many other things if it will help me achieve my goal. But when it comes to cloud providers I am not willing to spend a minute reading how to vendor lock my stack on their platform. If there is a standalone service I might want to use like SQS or SES sure, but most other things are a hard pass, I would rather have a dedicated host.
> The point is that these aren't mutually exclusive.
They are really not, cloud providers charge incredibly high rates for both bandwidth and computational resources so there goes the reasonable part. The usage based pricing they use is typically not capped so you are always a misconfiguration away from incurring many times your monthly costs on bandwidth charges (see a similar case from azure https://news.ycombinator.com/item?id=30054739). To add insult to injury you usually get inferior performance.
> And when it becomes overloaded, needs upgrading, or requires other maintenance? There's a problem I've completely left behind.
You have replaced these problems with having to deal with the cloud providers. It costs time to deal with their impenetrable pricing, processes and abstractions. Do yourself a favor and track how much time you spend dealing with their tools and how much time and effort you spend trying to optimize costs.
Upgrades and maintenance are not really solved, depending on the products you use. If you are using EC2 you still need to do upgrades and maintenance exactly like you would with bare metal. Now if you are using a serverless approach you might be clear of that too, but in general docker helps a lot with removing a lot of the pain of using bare metal.
Overloading is typically easily solved by over provisioning and yeah if you have very spiky usage patterns it might make sense to use a cloud provider. If your typical load can be dealt by 1 Hetzner node but you need 20 nodes 2% of the time you might be better off using a cloud provider. Modern hardware though has a lot of capability and while you don't have an "entire datacenter" to fall back on like you would with AWS a few bare metal nodes can handle a very large volume of requests depending on your use case.
What is a bootstrapped dev?
It means you are self funded and do not have a pile of other people's money to burn.
I would guess that's most AWS accounts. I have my 5 personal accounts all on one debit card.
I learned AWS the same way most "bootstrapped" people do, with the free tier. Maybe it's more of a minefield than it was a decade ago.
I did this when I was ~22 messing with infra for the first time. A $300 bill in two days when I had $2000 in the bank really stung. I love AWS for many things, but I really wish they made the cost calculations transparent for beginners.
I wonder why they don't...
$1000 for 20 TB of data transfer sounds like fraud. You can get a VM instance with 20 TB included INTERNET traffic at Hetzner for €4.15.
An entire blog article post to say "read the docs and enable VPC S3 endpoint".
It's all in the docs: https://docs.aws.amazon.com/vpc/latest/privatelink/concepts....
>There is another type of VPC endpoint, Gateway, which creates a gateway endpoint to send traffic to Amazon S3 or DynamoDB. Gateway endpoints do not use AWS PrivateLink, unlike the other types of VPC endpoints. For more information, see Gateway endpoints.
Even the first page of VPC docs: https://docs.aws.amazon.com/vpc/latest/userguide/what-is-ama...
>Use a VPC endpoint to connect to AWS services privately, without the use of an internet gateway or NAT device.
The author of the blog writes:
> When you're using VPCs with a NAT Gateway (which most production AWS setups do), S3 transfers still go through the NAT Gateway by default.
Yes, you are using a virtual private network. Where is it supposed to go? It's like being surprised that data in your home network goes through a router.
> An entire blog article post to say "read the docs and enable VPC S3 endpoint".
I think it's okay if someone missed something in the docs and wanted to share from their experience. In fact, if you look at the the s3 pricing page [0], under Data Transfer, VPC endpoints are mentioned at all. It simply says data transfer is free between AWS services in the same region. I think that much detail would be enough to reasonably assume you didn't have to set up additional items to accomplish.
[0]https://aws.amazon.com/s3/pricing/
Is it possible for hobbyists to set a hard cut off for spending? Like, "SHUT EVERYTHING DOWN IF COSTS EXCEED $50"
You can with some effort, but cloud providers don't provide real-time information on how much you're spending. Even if you use spending alerts to program a hard cut-off yourself, a mistake can still result in you being charged for 6+ hours of usage before the alert fires.
> You can with some effort, but cloud providers don't provide real-time information on how much you're spending.
This should be illegal. If you can't inform me about the bill on my request you shouldn't be legally able to charge me that bill. Although I can already imagine plenty of ways somebody could do malicious compliance with that rule.
[flagged]
my understanding from reading this kind of threads is that there is no real way to enforce it and the provider makes no guarantees, as your usage can outpace the system that is handling the accounting and shutoff
That sounds like an architecture choice? One that would cause less revenue on the AWS side, with a conflicting incentive there.
to be fair, im not sure its a conscious choice, since its not really easy to couple lets say data transfer bytes directly to billing data in real time, and im sure that would also use up a lot of resources.
But of course, the incentive to optimize this is not there.
AWS already does per hour billing for spot instances.
I mean, generally real time isn't needed. Even hourly updates could save a massive amount of headache. 24 hours or more is becoming excessive.
Shut down everything? Including S3? There goes all your data.
Turn off S3 requests, but keep the data.
Yes, but you have to program it. And there is a little bit of whack so it might be $51 or something like that.
> The solution is to create a VPC Gateway Endpoint for S3. This is a special type of VPC endpoint that creates a direct route from your VPC to S3, bypassing the NAT Gateway entirely.
The solution is to move your processing infrastructure to Hetzner.
Always always set up budget alarms.
Make sure they go to an list with multiple people on it. Make sure someone pays attention to that email list.
It's free and will save your bacon.
I've also had good luck asking for forgiveness. One time I scaled up some servers for an event and left them running for an extra week. I think the damage was in the 4 figures, so not horrendous, but not nothing.
An email to AWS support led to them forgiving a chunk of that bill. Doesn't hurt to ask.
> I've been using AWS since around 2007. Back then, EC2 storage was entirely ephemeral and stopping an instance meant losing all your data. The platform has come a long way since then.
Personally I miss ephemeral storage - having the knowledge that if you start the server from a known good state, going back to that state is just a reboot away. Way back when I was in college, a lot of out big-box servers worked like this.
You can replicate this on AWS with snapshots or formatting the EBS volume into 2 partitions and just clearing the ephemeral part on reboot, but I've found it surprisingly hard to get it working with OverlayFS
Why are people still using AWS?
And then writing “I regret it” posts that end up on HN.
Why are people not getting the message to not use AWS?
There’s SO MANY other faster cheaper less complex more reliable options but people continue to use AWS. It makes no sense.
Examples?
Of what?
> faster cheaper less complex more reliable options
Allow me to google that for you…..
https://www.ionos.com/servers/cloud-vps
$22/month for 18 months with a 3-year term 12 vCores CPU 24 GB RAM 720 GB NVMe
Unlimited 1Gbps traffic
AWS is not just EC2
And even EC2 is not just a VPS
If you need a simple VPS, yes, by all means, don't use AWS.
For this usecase AWS is definitely not cheaper nor simpler. Nobody said that. Ever.
They’re Linux computers.
Anything AWS does you can run on Linux computers.
It’s naive to think that AWS is some sort of magically special system that transcends other networked computers, out of brand loyalty.
That’s the AWS kool aid that makes otherwise clever people think there’s no way any organization can run their own computer systems - only AWS has the skills for that.
It was already clear that you were in bad faith here when you suggested a VPS to replace AWS, no need to insist.
But you are absolutely right, I'm drinking the AWS kool aid like thousands of other otherwise clever people who don't know that AWS is just Linux computers!
Good luck managing the whole day-2 operations and the application layer on top of your VPS. You're just shuffling around your spending. For you it's not on compute anymore but manpower to manage that mess.
In theory. Good luck rolling your own version of S3.
You probably don't need it. I see so many people getting price gouged by S3 when it would be orders of magnitude cheaper to just throw the files on a basic HTTP server.
I sometimes feel bad using people's services built with S3 as I know my personal usage is costing them a lot of money despite paying them nothing.
A web server isn’t a storage solution. And a storage solution like S3 isn’t a delivery network. If you use the wrong tool expect problems.
A web storage is connected to storage solutions like SSDs and S3 is connected to delivery networks like Internet. Using SSDs to store files or Internet to send files to a user are not the wrong tools.
Had the exact same thing happen. Only we used a company vetted/recommended by AWS to set this up for us, as we have no AWS experts and we're all too busy tasked doing actual startup things. So we staffed it out. Even the "professionals" get it wrong, and we racked up a huge expense as well. Staffed out company shrugged shoulders, and then just said sorry about your tab. We worked with AWS support to correct situation, and cried to daddy AWS account manager for a negotiated rate.
Just $1,000? Thems rookie numbers, keep it up, you'll get there (my wallet won't, ow).
Haha, yep we were lucky to catch this early! It could easily have gotten lost with everything else in the monthly AWS bill.
Came here to say the same, take my vote
> AWS's networking can be deceptively complex. Even when you think you've done your research and confirmed the costs, there are layers of configuration that can dramatically change your bill.
Unexpected, large AWS charges have been happening for so long, and so egregiously, to so many people, including myself, that we must assume it's by design of Amazon.
Regardless of the AWS tech in question (and yes VPCE for non-compute services is a very common pattern in an enterprise setup using AWS since VPC with NAT is a pretty fundamental requirement), I honestly believe this was the biggest miss from the author: “Always validate your assumptions. I thought "EC2 to S3 is free" was enough. I should have tested with a small amount of data and monitored the costs before scaling up to terabytes.” To me this is a symptom of DevOps/infra engineers being too much in love with infra automation without actually testing the full end to end flow.
You probably saved me a future grand++. Thanks
That was truly my hope with this post! Glad to hear that
Evergreen relevant blog post: "Save by Using Anything Other Than a NAT Gateway" https://www.vantage.sh/blog/nat-gateway-vpc-endpoint-savings
Also as a shameless plug: Vantage covers this is exact type of cost hiccup. If you aren't already using it, we have a very generous free tier: https://www.vantage.sh/
I’m sure NAT gateways exist purely to keep uninformed security “experts” at companies happy. I worked at a Fortune 500 company but we were a dedicated group building a cloud product on AWS. Security people demanded a NAT gateway. Why? “Because you need address translation and a way to prevent incoming connections”. Ok. That’s what an Internet Gateway is. In the end we deployed a NAT gateway and just didn’t setup routes to it. Then just used security groups and public IPs.
Abolish NAT Gateways. Lean on gateway endpoints, egress only internet gateways with IPv6, and security groups to batten down the hatches. All free.
Now that AWS charges for public IPv4 addresses, is it still free if you need to access IPv4-only hosts?
Yeah not free if you definitely need IPv4. AWS has been adding a lot more IPv6 support to their services so hopefully the trend continues in AWS and the broader industry. You can probably get pretty far though if your app doesn't have hard requirements to communicate with IPv4 only hots.
Are there any cloud providers that allow a hard cap on dollars spent per day/week/month? Should there not be a law that they have to?
I'm not telling anyone to stay away from AWS, but I've heard far too many similar stories to feel comfortable recommending it.
Its staggering to me that after all this time there are somehow still people in potions like this that are working without basic cost monitoring alerts on cloud/SaaS services
It really shows the Silicon Vally disconnect with the real world, where money matters.
Saved >120k/month by deploying some vpc endpoints and vpc peering (rather than tgw).
VPC peering becomes ugly fast, once your network architecture becomes more complex. Because transitive peering doesn't work you're building a mesh of networks.
Can just use both, tgw by default and add peering where you have heavy traffic. Did this while managing 1k+ VPCs.
Just curious but if you are already on Hetzner, why not do the processing also there?
https://news.ycombinator.com/item?id=45978308
It used to be that you could whine to your account rep and they'd waive sudden accidental charges like this. Which we did regularly due to all the sharp edges. These days I gather it's a bit harder.
If you want to avoid any kind of traffic fees, simply don't allow routing outside of your VPC by default.
Got killed by AWS Macie. The default 5K cap is brutal
That's a loophole AWS needs to close
I can’t see this as anything but on purpose
The lesson: Don't use AWS
So happy I don't use AWS
Imagine a world were Amazon was forced to provide a publicly available report were they disclose how many clients have made this error -and similar ones- and how much money they have made from it. I know nothing like this will ever exist but hey, is free to dream.
Wait till you encounter the combo of gcloud parallel composite uploads + versioning + soft-delete + multi-region bucket - and you have 500TB of objects stored.
NAT gateway probably cheap as fuck for Bezos & co to run but nice little earner. The parking meter or exit ramp toll of cloud infra. Cheap beers in our bar but $1000 curb usage fee to pull up in your uber.
I think it's been calculated that data transfer is the biggest margin product in all AWS catalog by a huge difference. A 2021 calculation done by Cloudflare [0] estimated almost 8000% price markup in EU and US regions.
And I can see how, in very big accounts, small mistakes on your data source when you're doing data crunching, or wrong routing, can put thousands and thousands of dollars on your bill in less than an hour.
--
> can put thousands and thousands of dollars on your bill in less than an hour
By default a NGW is limited to 5Gbps https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway...
A GB transferred through a NGW is billed 0.05 USD
So, at continuous max transfer speed, it would take almost 9 hours to reach $1000
Assuming a setup in multi-AZ with three AZs, it's still 3 hours if you have messed so much that you can manage to max your three NGWs
I get your point but the scale is a bit more nuanced than "thousands and thousands of dollars on your bill in less than an hour"
The default limitations won't allow this.
That's a NAT gateway, but if you're pulling data for analysis from S3 buckets you don't have those limitations.
Let's say they decide to recalculate or test a algorithm: they do parallel data loading from the bucket(s), and they're pulling from the wrong endpoint or region, and off they go.
And maybe they're sending data back, so they double the transfer price. RDS Egress. EC2 Egress. Better keep good track of your cross region data!
I don't think its about profits, its about incentivising using as many AWS products as possible. Consider it an 'anti-lock-in fee'
[flagged]
There's nothing to gain in punching down
They made a mistake and are sharing it for the whole word to see in order to help others avoid making it.
It's brave.
Unlike punching down.
This has nothing about punching down. Writing a blog about this basic mistake, and presenting as advice shows a strong lack of self awareness. Its like when Google bought thousands of servers without ECC memory, but felt they were so smart they could not resist telling the world how bad that was and writing a paper about it...Or they could have hired some real hardware engineers from IBM or Sun...
> Writing a blog about this basic mistake, and presenting as advice shows a strong lack of self awareness.
You realize they didn’t ask you to read their article right? They didn’t put it on your fridge or in your sandwich.
Policing who writes what honest personal experience on the Internet is not a job that needs doing.
But if you do feel the need to police, don’t critique the writer, but HN for letting interested readers upvote the article here, where it is of course, strictly required reading.
I mean, drill down to the real perpetrators of this important “problem”!
They are doing Enterprise sales and were founded in 2014...These type of technical blogs are normally written to demonstrate the company internal expertise, demonstrate skills of the engineering team to motive hiring etc... Does this inspire confidence?
Do I trust this person to be transparent? Yes!
Do I trust in their ability to hit a bump and be motivated to learn and communicate, without ass covering, blame deflection or stubborn denial? Yes!
Are you more confident of people without those traits?
—-
Critique the dishonest, the manipulative, the deceptive, and the dangerously self-unaware.
Better to respond to normal people making their way forward by offering your own insight, if you actually have any that they didn’t just learn for themselves.
Or by respectfully leaving them to continue.
People who punch down are not good people to work with. They don’t inspire my confidence. Do they inspire yours?
—-
Read all the useful discussions here. Do you imagine all these people saw less that you, to see something constructive? Or more?