I strongly advise buyers to stick with trusted suppliers, avoid Amazon/ebay channels, and carefully vet your incoming stock with SMART tools to ensure you receive what you think you are ... especially for SSD parts.
DO NOT assume SMART is reliable. You can wipe SMART stats or write any values you want.
You have to actually examine the real bits on the drive. Resellers don't want to take the time to actually zero a drive, they usually just nuke the partition table.
You also need to physically examine the drive. Corroded fingerprints on the PCB, wear on the port contacts, scratches from mounting rails, etc.
That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.
> That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of
hours of runtime.
This sounds like it could be a big problem for Backblaze customers, and consequently for Backblaze.
Can you alert the Backblaze CEO about their insufficiently-decommissioned drives leaking out like this?
Backblaze customers also need to know, but I would give Backblaze the first shot at figuring out how to notify, whom, of what.
Backblaze erasure-codes customer data across 17 (I think) servers, so customer data is probably not accessible. Yes, it would be better if they zeroed the drive, but Google says that will take 14-30 hours for a 10TB drive.
For drives that implement an internal encryption key, it's faster (instantaneous) to reset the encryption key. It won't give you a zeroed drive, but one filled with garbage.
In many erasure coding systems, the first X sets of code are simply cleartext chunks.
This is also more efficient in the happy path since then no computation is needed to decode the data. It can be DMA'd straight from the drive to the network adapter with super low CPU utilisation even for Gbps of network traffic.
The earlier description is ambiguous (i.e., is it data of or about customers, and is that data cleartext), but it seems they believe they have a drive from Backblaze with a lot of cleartext files on it, and something involving customers.
> It contained terabytes of customer data, and a shit ton of cleartext files.
Yev from Backblaze here -> That sounds very strange. I've asked our ops team and they said if you can provide the serial number of the drive you have we can track it. Would you be able to write in a support ticket w/ the serial number of the drive you bought so I can personally flag it and investigate? Once done you can write the ticket # here and I'll follow up with the team -> https://help.backblaze.com/hc/en-us/requests/new.
> drive I bought on Amazon was actually a used Backblaze drive
Assuming this is true, I find it weird/surprising that Backblaze doesn't at least zero their drives before disposing of them? I have to do that at my work, and at least by policy I could lose my job if I skipped doing it.
I work in the refurb division of an ewaste recycling company. I'm pretty sure I would be fired if I didn't wipe a drive before selling it, too. We got certifications that we NEED to keep in order to keep doing business.
Backblaze should get ironclad guarantees that decomissioned drives have been destroyed, besides that their storage system should not be storing anything in plaintext at all.
Unless you're super, duper sure this is true ... you might want to post your evidence, rather than your conclusion. Any misinformation these days gets amplified even faster and bigger than it did just 1-3 years ago, especially via ChatGPT and Gemini.
I don't understand any scenario where a NAS drive would have a normal filesystem with plainly readable files. Not because of security (as many would expect) but because drives used in arrays are striped and intended for parallel use. They do not contain complete individual files. I know this because when you lose more than 2 drives in, e.g. a RAID 5, you lose everything.
Equally unlikely is why a storage provider would use anything OTHER than an array for any of their drives. Again, tossing security aside, it is not a viable way to store data on one filesystem. That drive could die at any time and lose all the data. So it makes no sense to store it that way.
Question for all of you more knowledgeble than I: can SMART data be tampered with? When I get, say, a refurbished Mac from Apple, I'm trusting Apple won't stoop to that. But a SSD vendor I've never heard of?
Yes. There are vendor-specific utilities that have escaped into the wild that allow bad actors to reset various SMART counters, etc.
A lot of abuse came to light during the launch and initial mining of the (ridiculous) Chiacoin[1] during which Chia miners would burn through SSDs to within a hair of their usable life, reset their SMART stats, and sell them as new on Amazon or ebay.
As can be seen in my above comment, larger distributors like "Maestro Technologies" have their stock polluted with parts like this and I find it very unlikely that they are not aware of the status of these parts they are selling as new.
Yes, it can be tampered with. Drives can even lie about the amount of storage they support. I once bought a 1TB pen drive that was only 32MB for $10. (Yes, I knew it was a scam beforehand.)
I have given up on Amazon and similar "market makers" (bol.com is a regional one here) completely. Too much fraud. Instead I use specific vertical or store as sourced outlets.
I try to stick to at least "sold by amazon" as much as possible, that or the mfg. Generally, even with comingled inventory, Amazon has been good about replacements/refunds.
I've never considered myself very paranoid about Amazon, but recently I needed a cheap router, but couldn't shake the feeling that I shouldn't get it there (went with an in person Best Buy purchase instead)
Samsung Electronics has lowered its target for NAND wafer output this year to around 4.72 million sheets, about 7% down from the previous year's 5.07 million. Kioxia also adjusted its output from 4.80 million last year to 4.69 million this year.. SK hynix and Micron are likewise keeping output conservatively constrained in a bid to benefit from higher prices. SK hynix's NAND output fell about 10%, from 2.01 million sheets last year to around 1.80 million this year. Micron's situation is similar: it is maintaining production at Fab 7 in Singapore—its largest NAND production base—in the low 300,000-sheet range, keeping a conservative supply posture.
Micron has a new US fab coming online in 2027, which should improve supply.
Yes, anywhere there is a race to the bottom. That is why you see cartels rise up, like unions are cartels. Things that are not cartels generally have ever diminishing margins.
I see the rising of cartels more just the inevitable consequence of market systems where some players just getting bigger and bigger. Then when only a few large players are left, they collude (either openly, or just by unspoken agreement). So less of a race to the bottom, but a race to the top.
This is getting out of control. Like a monster eating off all the things that are needed for normal people. Worse than nukes race, which at least did not affect common man. No one really gains anything. All gains are temporary competitive edge, that vanishes quickly while locking everyone into the vicious cycle.
Worse than nukes race? I really much prefer being slightly inconvenienced when buying hard drives to risking being obliterated along with billions of other people and having all of civilisation wiped out with hundreds or thousands of years until recovery.
> Worse than nukes race, which at least did not affect common man
At least not visibly, we were deprived of a clean source of surgical steel and many in the areas were uncompensated for sicknesses they experienced as a result of testing
I can find vague mentions of "medical" uses for low background steel, and one specific claim that it's used to shield some rooms or large devices for measuring how much radiation is coming off a person's body. That's pretty niche.
Looking into this point further, I'll agree. I shouldn't have acted on an anecdote without verifying first. The other fact though is one that I'm certain of.
Yeah, another hot take from some guy who has no clue what he's talking about.
To this day Kazakhstan pays out monthly monetary compensations to people whose health was negatively affected by the nuclear tests on the Semipalatinsk nuclear polygon. Several people I know personally receive these compensations.
That didn't even come to mind. Call me Ameri-centric but my mind was on the Native Americans who to this day are the "most nuked" and have not been compensated
I just find it so odd that the oh so environmentally conscious tech sector is now building out some of the most resource intensive facilities on the planet. It's like environmentalism was just a con.
The tech sector "environmentalism" was obviously always just performative. They went along with the movement because it cost them virtually nothing, deflected pressure from the progressive political faction, and helped them get invited to fashionable parties. Now there are real costs so it's no longer convenient.
I don't know if I agree, but it's not a very extreme take.
Pretend every country with lots of nukes instead had no more than 100 warheads at a time. Would the world be much different? Would the average person be better off in a noticeable way?
Would it make a big difference if we never had hydrogen bombs? Note that we already had them in the early 50s. Everything since then was about increasing the quantity and refining designs, no meaningful size increases.
But why? Training hardware maybe will appear, but everything that's running inference has real, paying customers on it, with current capabilities level. Even if the bubble bursts, why would that demand evaporate? People still would want to use chatgpt, claude code etc., even if they stop getting any better tomorrow.
Yes, but how much do people want to pay to use this stuff? That is the real question.
It might be that more than $30 a month is too much and people stop using it. However, I suspect it would end up a fair bit higher than many would think. If you are a 'whale' (to use the gambling term) then there is a good chance that they could charge in the hundreds of dollars.
Maybe it become a wealth divide between those that can afford it and those who cannot. In equality yet again. Then the echos of Dune, the 'Butlerian jihad' and ‘Thou shalt not make a machine in the likeness of a man’s mind' will start to come up.
I'd expect at least API pricing for all major players to be margin-positive. Positive ROI if you include model training cost? Maybe not, but in a hypothetical scenario where all capital for new model training goes away, the latest frontier model weights still continue to exist.
Subscription prices for claude code et. al? No idea.
No, that's a rumor lots of people have been taking at face value.
If you do the math, inferrence is very lucrative.
Here someone deployed a big model, the costs are $0.20/1M token
https://lmsys.org/blog/2025-05-05-large-scale-ep/
If some large companies implode local models will fill the gaps soon enough. I can see a crash in data center construction and operation once the economics get too far out of whack.
During 1998-2000, AOL was ordering so many free trial CDs that it locked up world production, and music CDs faced 8-12 week delays. It was rumored that certain weeks there were no albums getting fabricated at all, worldwide.
I wonder if history isn’t repeating itself. AOL CDs had pretty much jumped the shark by 2000.
We finally reached critical mass on seeing money as an arbitrary construct, so now we're converting it into real, physics-based heat. Entropy at its finest.
I've been saying roughly the same thing about cryptocurrencies (just a good way to waste fuckhuge amounts of resources on digital tulips to enable crime) but it never seems to stop anyone from plowing ahead on being stupid.
Water is one I really don't get, if anyone would be willing to explain it to me. Is it because of manufacturing? We already have these manufacturing plants, it's not like more are being started, that process takes years to decades. Hell intel couldn't even finish the few they were working on. Because of concrete for datacenters? I'm sure developing countries are using significantly more of that to build houses, so much so that it completely eclipses the one-to-two datacenters you might see built in the next 5 years in your area. Power generation? That water goes straight back to the atmosphere.. water cycle, even on the open loops, water becomes steam which becomes rain..
And the water is being pumped out of aquifers that aren't being refilled. That water took thousands if not millions of years to get there. Once gone, the aquifers collapse and cannot be replaced as the space once taken up by the water is now just ground.
The issue is that they need fresh water for evaporative cooling.
Much of the world experiences fresh water scarcity, so it's not the best ethics to divert this resource from people in need to tech of uncertain value.
Check out the impact xAI is having on environment and health in Memphis if you want to go further down the rabbit hole.
Maybe people shouldn't be building data centers in deserts. In the city of Toronto the Deep Lake Water Cooling System uses water taken from Lake Ontario used for drinking water to cool a number of buildings. Most notably 151 Front Street West which houses the data centers routing most of the Internet in Ontario.
For all the issues with power use, I do grant it that a lot of materials and labor in needed at the same time. It isn't JUST burning up cash to make Sora videos.
But by the same measure, the Great Egyptian pyramids would have been a huge boom for work even if the final product didn't achieve much for most people.
Thanks for the correction. The experts think the builders were not slaves. Their life in the barracks was too good for slaves, and similarly were they treated in death (which was specially important there)
I have a hypothesis that it was AI, not COVID/sanctions/etc, that was mainly responsible for the 2020-to-ongoing "chip shortage." Ignoring companies with their own fabs (Intel) and companies with pre-existing reserved-capacity contracts with fabs (Apple), everyone else is stuck waiting in line behind batch after batch of fab orders from Nvidia.
Downstream of that, AI is effectively also responsible for the current generation of game consoles never declining in price.
Because game consoles are fixed platforms that continue to be manufactured over 5+ years, normally the most expensive parts in the system (the CPU and GPU) would gradually get cheaper to manufacture [and in turn, cheaper to buy] over the course of the console's lifetime—which was often passed onto the consumer in the form of the console's MSRP gradually decreasing. Either the process node for the console's silicon design would stay fixed, and demand for this process node would gradually decrease as larger fab customers move on to newer nodes, decreasing the (effectively auction-based) pricing for fab time on the older node; or the console manufacturer['s silicon vendor] would put their silicon design through a process-shrink, and so, while still paying top dollar for use of the fab's newest node, would be getting more chips per wafer out of that, and again could charge less.
But instead, what we've seen since the start of the AI boom is that there's no longer any price-reduced timeslots to be sold to manufacture the low-BOM parts for these price-sensitive console-maker customers. Instead, both Nvidia and AMD are now getting such high value out of even the older nodes, that 1. the fabs know they can squeeze them, charging full price for those slots as well; and 2. both Nvidia and AMD, in their roles as silicon vendors to the console makers, haven't been able to justify using (very much of) the time they're paying so much for to fab low-BOM-cost parts to fulfill their pre-existing outstanding purchase orders, when they could instead be fulfilling much-higher-margin new POs from the hyperscalers.
Thus every console of the ninth generation (PS5, Xbox Series, Switch) still selling for their launch price (with no help from process shrinks); thus none of these consoles having been able to be produced in excess of demand to the point that supply-and-demand ever drove retail prices below MSRP; and thus the only tenth-gen console so far, the Switch 2, taking ~4 years longer than anticipated to release†.
---
† Nintendo were very likely waiting for Nvidia to run off enough of the Tegra T239 with a sufficiently low passed-on fab cost, for Nintendo to both 1. be able to build a backstock and non-run-dry pipeline of Switch 2s, and 2. be able to be positive-margin on charging the same price as the Switch 1 for them. They waited four years, and neither thing ever happened; so they eventually just gave up and priced the Switch 2 higher, and also built out an entirely novel D2C + online-marketplace-partner distribution pipeline so they could ration the tiny initial supply of units they had been able to build with the chips Nvidia had supplied them so far.
Though, that being said, Nintendo actually got a double whammy here. They were also waiting for fast NAND to come down in price, so that they could have physical game cards manufactured for a trivial BOM price while still enabling the "direct GPU disk-streamed assets" pipeline that games of the last generation had begun relying on. Obviously, as today's article points out, that hasn't been happening either! Thus game key cards; thus SD Express cards only beginning to trickle out, with no sizes above 128GiB available at the Switch 2's launch time; and thus those SD Express cards being ridiculously priced for their capacity compared to equivalent transfer-speed + die-size NAND (as seen in e.g. low-profile/flush-mount flash drives.)
> I have a hypothesis that it was AI, not COVID/sanctions/etc, that was mainly responsible for the 2020-to-ongoing "chip shortage."
Might be for higher-end chips like GPUs, but for smaller ones like microcontrollers it was very much COVID that threw a huge wrench in the gears.
Things are closer to normal now, but the toilet paper effect hit the electronics supply chain hard during COVID. People resorted to buying dev boards just to desolder the microcontroller on it to use in their commercial product, and similar desperate moves. With lead time measured in many months to years even when the factories are operational, sudden hoarding is not what you want.
It wasn't COVID directly, it was the car companies canceling and then uncanceling their massive chip orders that caused the shock in the semiconductor supply chain.
Something I think contributes is that TSMC seem to be so good at making chips now, it seems like the defect rate must be low enough and they can produce huge chips in volume that a decade ago would be burning money. If you can do that, why bother making less lucrative smaller chips? Also while there's still benefits to shrinking an existing chip with a revised design, I'd guess we're into diminishing returns for what it offers a console whether the chip itself gets cheaper or lets them reduce the bill of parts in other ways.
This is also the first generation where prices have gone up IIRC.
The defect rate always drops like that. But usually the big lucrative chips move on to the next node, and instead they're filling up more nodes in parallel.
The main cure is to build more fabs but that's hard too and they're not in a rush to ruin their own margins by scaling up too fast in the face of an uncertain future.
That sounds reasonable but I suspect it was a little bit of both. Initially Covid slow down followed by large demand. A lot of older Nodes were shutdown (300-400nm) with the Covid slow down and a lot of chips ended up having to be moved over to newer nodes once demand picked up again. It led to a big swing in the flow of production and would have roll on effects.
Combine that with a huge GPU boom and you have the setup for production issues.
But the basic chips like the ones that go into cars are the ones that were the most scarce--the low margin stuff was not being made. You're saying it's because of Nvidia? I'm not totally tracking here.
The game console thing is also related to monetary inflation--the price of everything just keeps jumping.
We should be making some effort to quantify the amount and cost of slop produced by both AI and simpler automated systems (spinners etc), it's a huge negative externality.
I built a computer December 2024 with a Ryzen 9900X, 64GB DDR5, 1TB + 4TB SSDs, 4060. If I were to buy the exact same parts now, nearly 1 year later, it would cost about $1,000 more for the exact same components. The memory alone went from $189 to $505. I'm not sure how much of that is from tariffs and how much is due to AI datacenter demand, but that is a massive increase.
Sometimes, I think government supply control isn't that much of a bad thing. That way, the government could force the availability of goods for the common market as well and not just for the really big dogs flush with ample VC money to burn who can pay any price.
[1] For the uninitiated: a cryptocurrency where the limiting factor wasn't CPU, RAM or GPU compute resource, but storage - in 2021, there was so much craze around it that HDD and SSD prices exploded, and after the bubble collapsed a lot of heavily abused drives flooded the markets.
Chia was/is WORM (write once , read many) so not sure how these drives were heavily abused? As a storage nerd, I bought many 18tb drives for chia and they are still spinning just fine in my storage arrays. It’s been five years.
I've been in the industry since 1985. There have always been boom and bust cycles with the basic components. It's the nature of the thing: very costly factories that take many years to bring online. Plus normal business cycles that tend to align with the US election cycle.
I think it is solidigm that has started to argue that with a 128 TB QLC drive constant writes at the maximum write rate will hit the drive’s endurance limit at about 4.6 years. The perf/TB of these drives is better than HDDs. The cost per TB when you factor in server count, switches, power, etc., is argued to favor huge QLC drives too.
Enterprise SSDs are not expensive only because they have better flash chips, but they have much more of them.
A top of the line write oriented SSD comes with 4-7x more capacity than what it says on the tin, but that extra capacity is used for cell replacement rather than capacity itself.
Mixed use comes with 2-4x overprovisioning, and read oriented is around 2x IIRC.
I find these numbers to be way outside of what I have heard of. I would be surprised if you could give an example that comes with even 1.5x capacity. (4TB capacity, 6TB actual flash on chips, for example.)
Overprovisioning is much less aggressive than this in practice. A read-oriented SSD with 15.36 TB of storage typically has 16.384 TiB of flash. The same hardware can be used to implement a 12.8 TB mixed-use SSD (3 DWPD or more).
I just got 2 16TB drives for $90 each off Craigslist. Perfect for video game storage or downloading huge AI models or whatever. Or legit Linux ISOs. Wish I got more but I was too late. Can't find anything close to that price online
$5 a TB is insanely cheap and someone just getting rid of stuff and not really caring about the price. Nothing has ever been that cheap at retail or manufacturer recertified. Lowest was like $9-10 last year and now we are sitting at more like $15.
They came uninitialized, but yeah it could be worthwhile to actually zero and fully format them. They are decently fast at 275 MB/s but even that's 16 hours to do one pass. Maybe let it run over the weekend
16 hours isn't that bad. I recently had to zero my old 2tb Time Capsule and just let it run overnight and during work, wasn't so bad. You'd also get to detect bad sectors, figure out how good the drive is in terms of what % of the drive is actually usable for large sequential writes
Soon we'll see many "AI" product that analyses network storage to identify files that could be deleted because they're the nth copy of some random report that was never published.
Depends how it goes down, if a company goes into insolvency all security policies are off the table and random hardware can get shifted into lot bidding.
HDD can be written multiple times with random data if data centers really have to protect what their former customers wrote on them. I never looked at those details in standard contracts.
All you really need to do is write one pass of zeros on them. That will prevent anyone but a very dedicated adversary with expensive equipment from recovering any data, especially on TB scale drives.
Can still take hours per drive though, which is why a lot of people skip it.
I make a random 1MB chunk, then write that all over the drive, at overlapping offsets. I've been told that really clears it. On IDE-spinning-rust disks I trusted it, not sure if I should trust these modern SSD
A drive that supports Secure Instant Erase should be encrypting all data. When the SEI function is invoked (“nvme format -s 2”, “hdparm —-security-erase”) they key is thrown away and replaced with a new one. Similar implementations exist for NVMe, SATA, and SAS drives — regardless of whether they are HDD or SSD.
This puts a fair amount of trust and in the drive’s ability to really delete the old key.
My understanding is that refurbished GPUs from e.g. crypto mining are something people are warned away from, because they've often been run into the ground. Are AI usage practices different?
> My understanding is that refurbished GPUs from e.g. crypto mining are something people are warned away from, because they've often been run into the ground.
No, this wasn't the case. While there were never comprehensive studies various tech media purchased these cards to run testing and found that, other than scammers, they all performed to expectation.
Car tire is a moving part that wears in use. Only moving parts a GPU has are its fans, which can be replaced. The rest may last really long, or fail brand new.
Thermal cycling will eventually cause problems with continuity but I'm not sure that's really predictable. We have some really old GPUs that eventually had problems under load, we attributed it to thermal degradation as we had ruled out pretty much everything else.
> And when the AI bubble bursts, "refurbished" HDDs and GPUs will flood the market.
GPUs? No way. The datacenter cards don't even have video output ports, and I think the chips destined for AI / ML training also have everything video/render related removed from the silicon, makes for more yield.
And the other way around, using (cheap) consumer GPUs in servers, I think at least NVDA tries to prevent that with driver-based DRM, so there won't be any flooding coming from there either.
> This is a long term good thing. ... By 2028 or so we'll have a 50% drop in price-per-storage for these components.
Per the op:
> and the ongoing DRAM shortage is proof of this, with memory kits costing more than double what they did just a few months ago.
> While enterprise-grade QLC SSDs would entirely power this pivot, Sandisk has already raised NAND prices by 50%, according to another DigiTimes report, after initially warning of a 10% increase two months ago.
So you're basically saying prices may return to normal in two years, and that's somehow a good thing compared to them not being inflated in the first place?
> By 2028 or so we'll have a 50% drop in price-per-storage for these components.
Do you mean relative to six months ago, or now? Because a lot of the prices have already more than doubled.
(I’m upset because the computer I’ve been planning to build, which three months ago would have come to around ₹90,000, is now up to ₹1,20,000 and climbing week by week, half due to price increases on the same part, half due to forced substitutions on RAM since the cheaper 32GB 6400MT/s DDR5 sticks are completely unavailable. And looking into laptops, for the first time ever I’m seeing manufacturet SODIMM or SSD upgrades being cheaper than aftermarket.)
Like the GPUs? How they were up since the bitcoin craze and are still up to now? Yeah I dont think these prices are ever going back down. Specially not since people are buying at these prices and will continue to pay these prices. Maybe HDDs will get phased out though, since you might as well buy SSDs if you are overpaying that much.
Depending on the future you predict 10% may be a good ROI - if AI will replace humans and traditional economy will collapse all other investments will loos value even more. In such scenario you cannot save the money you only can loose less if you will make a right investment.
I'm not sure I understand why the economy would collapse if AI replaced humans. Wouldn't the companies just make more profit because they save labor costs and stonks go up?
The economy is a macrosystem. For a company to make profit, someone needs to buy. For someone to buy a service or a product, they need to have the money needed to buy. If AI replaces humans and UBI doesn't happen, costs might drop to near zero but so will revenue as virtually no one will be able to buy anything. So the economy will certainly collapse.
It depends on how many jobs AI replaces and which. It is unlikely that LLMs will replaces plumbers any time soon for example. Many technological inventions have replaced humans in lots of jobs, the economy has only grown since then.
>Many technological inventions have replaced humans in lots of jobs, the economy has only grown since then.
Indeed it has, but many of those it replaced did not necessarily get moved to better positions if they were moved at all. The industrial revolution was great if you were young and had little to no responsibilities. Your average middle aged farmer with a large family to feed faced poverty or near poverty and those who were able to move to the cities faced worse labour conditions. People on here love talking about the winners while conveniently ignoring the loser side.
And that's not even mentioning the skilling up issue to which older workers are more vulnerable. Who will fund the nation wide skill up costs for them?
Industrial revolution was all right as long as you owned the farm, as most farmers did. The losers were those farm workers who weren't needed in as great numbers anymore, and they weren't paid well in the first place. I agree on labour conditions though, there industrialisation was a clear downgrade for many.
What worries me about AI is how it's not obvious which new jobs it may create. Younger farm workers and children of farmers could just move and work in factories, where the employers mostly took care of their training. I can't see such opportunities here. I believe whole world needs either UBI, or governments paying for jobs that previously weren't seen worth paying for. Otherwise the economy will collapse due to mass unemployment and resulting lack of demand.
Plumbers and electricians will loose too when demand for their services will plummet because people who will loose jobs will have no money to pay for their services (even if the services are still needed).
Economy is a fine tuned system with lots of interdependent parts, any sadden change can have destructive effects. If many people will loose jobs at least two things will happen: 1. they'll stop buying products and services (may be except food), companies who provide these products and services will layoff employees even if their employees are not yet replaced by AI 2. people who lost jobs will default on mortgages causing financial crisis worse than 2008 one.
UBI and wealth taxes will mitigate effects to some extent but it looks like political elites across the spectrum are controlled by wealthy people - even left of the center politicians prefer to tax middle class steering away from taxing the rich (both people and companies) more.
This is getting ridiculous. Expect this to bust with prices crashing as soon as the combination of hype driven margin inflation and unfulfilled and easily cancelled purchasing agreements hit.
Great. The Cryptocurrency hype finally died down and prices for things (except GPUs) were MOSTLY going back to normal.
Now the sector has invented a new hype machine and hardware prices are all fucked again.
Is this going to happen every few years? The industry hypes up some nonsense that skyrockets all the hardware prices and causes everything to be on backorder, just to dump it and go to the next one?
I bought 3x22TB WD Red Pros to put into a RaidZ1 vdev so I'll be good for a while. I have a clear upgrade path all the way up to 8 drives with 2x vdevs in a pool but that is something for far future me to consider.
Same here, though I did it a bit earlier. Data center SSD's I bought a while back for 50 euros each from Ebay now sell for over 200 euros. Similar increase for HDD's too.
Maybe in 3-5 years there will be a fantastic time to upgrade again, after AI bubble has burst and slightly used stuff gets dumped from data centers to online marketplaces.
I really hope the AI bubble bursts sooner rather than later. Sooner will impact the broad economy less severely (although it'll still be pretty bad) and curtail these supply chain shortages. If the bubble keeps inflating, the storage makers will have already mostly built out excess capacity and the crash will lead to even longer-term supply distortion.
I think it's still going to be a while. Go talk to anyone who isn't really tech savvy and you still hear all the same hype, now more than ever. None of them can tell you how they've actually used it to create much, if any value, but they still think it's the second coming of Christ.
I disagree, I think there's a huge bubble but I know at least 4 people using LLMs daily who are not techies, and I know what they use it for and it's valuable.
The funniest thing about this is that, with high GPU prices, rising RAM costs, and now increasing SSD prices, Apple will end up producing the most affordable PCs.
They have a magic supply chain that is unaffected by the broader world which is one of the reasons why Tim Apple was chosen by Steve Jobs as his successor.
The problem is not chip supply it's manufacturing. Apple has their own manufacturing suppply. This is not the chip crisis of the last years. Hyper scalers are switching to consumer hardware because there is nothing in storage for Prosumer anymore and the manufacturing pipelines for these are smaller and harder to scale than consumer ones.
The NAND itself seems to come from companies like SK Hynix, which are currently struggling to meet demand.
Apple doesn't make their own NAND, just like they don't make their own screens. They did write their own NAND controller and stuck it into their CPUs, but the flash memory that actually stores data doesn't come directly from their factories.
It's not a fucking nand problem. The producers reduced capacity because of the slow consumer PC market in 2024. With hyperscalers shifting to consumer products the market is strained but everyone is scaling up their production now. This is not the same as the chip shortage we had in the last few years. The production capabilities are there they are just adjusting to new market realities. We will see a whiplash for a few months but we can and will produce enough chips for everyone which we were not able to do in the 20-22 crisis. Stop getting your "insights" from influencers and stop parroting them.
I don't see any evidence of a general chip shortage in your link either. That they are down in stock because of unprecedented demand after a downturn in the last 3 years in consumer demands of consumer products is something completely different than being down because of production capacity. They reduced production in 2023 and 2024. It's exactly what I wrote in my previous comment.
"As mentioned previously, the rip-roaring demand for HBM products is reducing the available wafer capacity for other DRAM and NAND products, reducing their supply in the process. We noted in a recent post that the global average delivery times for DDR5 are now quite stretched, currently hovering between 26 and 39 weeks.
Coming back, SK Hynix announced at its earnings call that it has fully booked its DRAM and NAND capacity through 2026, with HBM4 set to ship by the end of 2025."
No, Apple has effectively promoted iCloud as the alternative to local storage as part of its product differentiation strategy in the lower price segment.
Apple will almost certainly introduce the same approach for the budget MacBook as well.
It's rough out there and has become increasingly difficult to maintain our pace of storage deployment.
Further - and most concerning - is the pollution of the supply chain with refurbished/recertified stock being sold and marketed as "new".
One example:
https://kozubik.com/items/MaestroTechnology/
I strongly advise buyers to stick with trusted suppliers, avoid Amazon/ebay channels, and carefully vet your incoming stock with SMART tools to ensure you receive what you think you are ... especially for SSD parts.
DO NOT assume SMART is reliable. You can wipe SMART stats or write any values you want.
You have to actually examine the real bits on the drive. Resellers don't want to take the time to actually zero a drive, they usually just nuke the partition table.
You also need to physically examine the drive. Corroded fingerprints on the PCB, wear on the port contacts, scratches from mounting rails, etc.
That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.
Physical examination is the only reliable method.
> That's how it found out that the last "new" drive I bought on Amazon was actually a used Backblaze drive. It contained terabytes of customer data, and a shit ton of cleartext files. SMART, of course, reported it was a brand new drive with zero hours. Cleartext logs on the drive showed many thousands of hours of runtime.
This sounds like it could be a big problem for Backblaze customers, and consequently for Backblaze.
Can you alert the Backblaze CEO about their insufficiently-decommissioned drives leaking out like this?
Backblaze customers also need to know, but I would give Backblaze the first shot at figuring out how to notify, whom, of what.
Backblaze erasure-codes customer data across 17 (I think) servers, so customer data is probably not accessible. Yes, it would be better if they zeroed the drive, but Google says that will take 14-30 hours for a 10TB drive.
For drives that implement an internal encryption key, it's faster (instantaneous) to reset the encryption key. It won't give you a zeroed drive, but one filled with garbage.
In many erasure coding systems, the first X sets of code are simply cleartext chunks.
This is also more efficient in the happy path since then no computation is needed to decode the data. It can be DMA'd straight from the drive to the network adapter with super low CPU utilisation even for Gbps of network traffic.
The earlier description is ambiguous (i.e., is it data of or about customers, and is that data cleartext), but it seems they believe they have a drive from Backblaze with a lot of cleartext files on it, and something involving customers.
> It contained terabytes of customer data, and a shit ton of cleartext files.
>Can you alert the Backblaze CEO about their insufficiently-decommissioned drives leaking out like this?
One has to wonder how many times you can deliberately fein incompetence until it's clear it's on purpose.
https://www.bleepingcomputer.com/news/security/backblaze-mis...
Yev from Backblaze here -> That sounds very strange. I've asked our ops team and they said if you can provide the serial number of the drive you have we can track it. Would you be able to write in a support ticket w/ the serial number of the drive you bought so I can personally flag it and investigate? Once done you can write the ticket # here and I'll follow up with the team -> https://help.backblaze.com/hc/en-us/requests/new.
[flagged]
> drive I bought on Amazon was actually a used Backblaze drive
Assuming this is true, I find it weird/surprising that Backblaze doesn't at least zero their drives before disposing of them? I have to do that at my work, and at least by policy I could lose my job if I skipped doing it.
I find it more weird that they don't use encrypted storage, then you don't nee to bother with zeroing drives. You only need to 'forget' the key.
I find it weird they are allowed to do anything but destroy end-of-life HDDs
If our legal team finds out about this, we are never going to be allowed to use Backblaze lol.
Seems like the only responsible thing to do is tell them and then stop using Backblaze. Seems to me they can't be trusted.
I work in the refurb division of an ewaste recycling company. I'm pretty sure I would be fired if I didn't wipe a drive before selling it, too. We got certifications that we NEED to keep in order to keep doing business.
But you don’t work at backblaze :)
Backblaze should get ironclad guarantees that decomissioned drives have been destroyed, besides that their storage system should not be storing anything in plaintext at all.
This seems like it would create a lot of e-waste, as long as the data is guaranteed unrecoverable in entirety or in part it should be okay no?
> actually a used Backblaze drive
Unless you're super, duper sure this is true ... you might want to post your evidence, rather than your conclusion. Any misinformation these days gets amplified even faster and bigger than it did just 1-3 years ago, especially via ChatGPT and Gemini.
I don't understand any scenario where a NAS drive would have a normal filesystem with plainly readable files. Not because of security (as many would expect) but because drives used in arrays are striped and intended for parallel use. They do not contain complete individual files. I know this because when you lose more than 2 drives in, e.g. a RAID 5, you lose everything.
Equally unlikely is why a storage provider would use anything OTHER than an array for any of their drives. Again, tossing security aside, it is not a viable way to store data on one filesystem. That drive could die at any time and lose all the data. So it makes no sense to store it that way.
> You have to actually examine the real bits on the drive.
What is the best utility for doing this?
Shop around digital forensics tools
Question for all of you more knowledgeble than I: can SMART data be tampered with? When I get, say, a refurbished Mac from Apple, I'm trusting Apple won't stoop to that. But a SSD vendor I've never heard of?
Yes. There are vendor-specific utilities that have escaped into the wild that allow bad actors to reset various SMART counters, etc.
A lot of abuse came to light during the launch and initial mining of the (ridiculous) Chiacoin[1] during which Chia miners would burn through SSDs to within a hair of their usable life, reset their SMART stats, and sell them as new on Amazon or ebay.
As can be seen in my above comment, larger distributors like "Maestro Technologies" have their stock polluted with parts like this and I find it very unlikely that they are not aware of the status of these parts they are selling as new.
[1] https://en.wikipedia.org/wiki/Chia_Network
Yes, it can be tampered with. Drives can even lie about the amount of storage they support. I once bought a 1TB pen drive that was only 32MB for $10. (Yes, I knew it was a scam beforehand.)
better than 10 floppies!
Trivially
I have given up on Amazon and similar "market makers" (bol.com is a regional one here) completely. Too much fraud. Instead I use specific vertical or store as sourced outlets.
Nearly any product you can buy from Amazon, even when it says shipped from Amazon, is suspect.
I wouldn't shop there at all. It's a literal scam market. Allegedly.
I try to stick to at least "sold by amazon" as much as possible, that or the mfg. Generally, even with comingled inventory, Amazon has been good about replacements/refunds.
I've never considered myself very paranoid about Amazon, but recently I needed a cheap router, but couldn't shake the feeling that I shouldn't get it there (went with an in person Best Buy purchase instead)
I'm using an N305 mini pc with OpnSense for my router, with a separate commercial AP for wireless... generally working very well/stable.
> avoid Amazon/ebay channels,
But why ? They are the good ones. TEMU and Shein bad. /s
"OMEC" (Organization of Memory Exporting Countries) NAND production quotas?
https://x.com/jukanlosreve/status/1988505115339436423
Micron has a new US fab coming online in 2027, which should improve supply.DRAM price fixing scandal: 1998-2002, 2016-2018, https://en.wikipedia.org/wiki/DRAM_price_fixing_scandal
It’s a cartel just like the OPEC oil cartel.
Is there anything that isn't functionally a cartel at this point?
Yes, anywhere there is a race to the bottom. That is why you see cartels rise up, like unions are cartels. Things that are not cartels generally have ever diminishing margins.
I see the rising of cartels more just the inevitable consequence of market systems where some players just getting bigger and bigger. Then when only a few large players are left, they collude (either openly, or just by unspoken agreement). So less of a race to the bottom, but a race to the top.
I think we agree. To risk being trampled by a race to the bottom, they choose to rise above by forming a cartel.
[dead]
This is getting out of control. Like a monster eating off all the things that are needed for normal people. Worse than nukes race, which at least did not affect common man. No one really gains anything. All gains are temporary competitive edge, that vanishes quickly while locking everyone into the vicious cycle.
Worse than nukes race? I really much prefer being slightly inconvenienced when buying hard drives to risking being obliterated along with billions of other people and having all of civilisation wiped out with hundreds or thousands of years until recovery.
At least nuclear holocaust is easily quantifiable
Maybe the real paperclip maximizer was the irrational exuberance we made along the way :')
> Worse than nukes race, which at least did not affect common man At least not visibly, we were deprived of a clean source of surgical steel and many in the areas were uncompensated for sicknesses they experienced as a result of testing
Why would it matter for surgical steel?
I can find vague mentions of "medical" uses for low background steel, and one specific claim that it's used to shield some rooms or large devices for measuring how much radiation is coming off a person's body. That's pretty niche.
Looking into this point further, I'll agree. I shouldn't have acted on an anecdote without verifying first. The other fact though is one that I'm certain of.
Yeah, another hot take from some guy who has no clue what he's talking about.
To this day Kazakhstan pays out monthly monetary compensations to people whose health was negatively affected by the nuclear tests on the Semipalatinsk nuclear polygon. Several people I know personally receive these compensations.
https://egov.kz/cms/en/news/nuclear_tests
https://egov.kz/cms/en/services/family_support/pass035_mtszn
That didn't even come to mind. Call me Ameri-centric but my mind was on the Native Americans who to this day are the "most nuked" and have not been compensated
https://www.aljazeera.com/opinions/2020/8/29/a-message-from-...
https://reimaginerpe.org/node/165
I just find it so odd that the oh so environmentally conscious tech sector is now building out some of the most resource intensive facilities on the planet. It's like environmentalism was just a con.
The tech sector "environmentalism" was obviously always just performative. They went along with the movement because it cost them virtually nothing, deflected pressure from the progressive political faction, and helped them get invited to fashionable parties. Now there are real costs so it's no longer convenient.
> No one really gains anything.
That's not true I've seen loads of amusing videos on Instagram.
>Worse than nukes race
That certainly is a take.
I don't know if I agree, but it's not a very extreme take.
Pretend every country with lots of nukes instead had no more than 100 warheads at a time. Would the world be much different? Would the average person be better off in a noticeable way?
Would it make a big difference if we never had hydrogen bombs? Note that we already had them in the early 50s. Everything since then was about increasing the quantity and refining designs, no meaningful size increases.
Maybe when the bubble bursts there will be lots of good used hardware at massive discounts?
But why? Training hardware maybe will appear, but everything that's running inference has real, paying customers on it, with current capabilities level. Even if the bubble bursts, why would that demand evaporate? People still would want to use chatgpt, claude code etc., even if they stop getting any better tomorrow.
Yes, but how much do people want to pay to use this stuff? That is the real question.
It might be that more than $30 a month is too much and people stop using it. However, I suspect it would end up a fair bit higher than many would think. If you are a 'whale' (to use the gambling term) then there is a good chance that they could charge in the hundreds of dollars.
Maybe it become a wealth divide between those that can afford it and those who cannot. In equality yet again. Then the echos of Dune, the 'Butlerian jihad' and ‘Thou shalt not make a machine in the likeness of a man’s mind' will start to come up.
They're probably buying this equipment anticipating that the current demand will continue. If it stops, all the surplus will need to be sold.
But maybe not for the price it actually takes to run these models. Afaik atm they are all subsidized.
I'd expect at least API pricing for all major players to be margin-positive. Positive ROI if you include model training cost? Maybe not, but in a hypothetical scenario where all capital for new model training goes away, the latest frontier model weights still continue to exist.
Subscription prices for claude code et. al? No idea.
No, that's a rumor lots of people have been taking at face value. If you do the math, inferrence is very lucrative. Here someone deployed a big model, the costs are $0.20/1M token https://lmsys.org/blog/2025-05-05-large-scale-ep/
If some large companies implode local models will fill the gaps soon enough. I can see a crash in data center construction and operation once the economics get too far out of whack.
Not if the paying customers aren't sustainable. Many of us have seen this before--2001.
Don’t worry, OpenAI is about to implode.
During 1998-2000, AOL was ordering so many free trial CDs that it locked up world production, and music CDs faced 8-12 week delays. It was rumored that certain weeks there were no albums getting fabricated at all, worldwide.
I wonder if history isn’t repeating itself. AOL CDs had pretty much jumped the shark by 2000.
All that manufacturing demand led to enough free machinery to get those free Ubunutu CD's! ;)
Man, I loved those. A Kubuntu CD felt like such a treat to 10 year old me.
That's somehow unsurprising given how many AOL CDs I saw at checkout of every grocery store
Stacks of them at a local post office back in the day. Covering every horizontal surface.
So far, due to the AI boom, we're lacking:
* HDDs
* SSDs
* DRAM
* GPUs, obviously
* Power to hook up to our datacenters
Anything I'm missing? What a crazy world we're living in.
Intelligence. We're lacking intelligence.
Water? Clean Air? ROI?
Nearly nothing left unexploited in the pursuit of profits.
And there is no profit, too.
Its very depressing that we’re almost quite literally converting money into BTUs via GPUs and not even making a buck, meanwhile the ocean gets hotter
We finally reached critical mass on seeing money as an arbitrary construct, so now we're converting it into real, physics-based heat. Entropy at its finest.
I've been saying roughly the same thing about cryptocurrencies (just a good way to waste fuckhuge amounts of resources on digital tulips to enable crime) but it never seems to stop anyone from plowing ahead on being stupid.
This is precisely what soured me on crypto. It's a ridiculous waste, a circle jerk.
Water is one I really don't get, if anyone would be willing to explain it to me. Is it because of manufacturing? We already have these manufacturing plants, it's not like more are being started, that process takes years to decades. Hell intel couldn't even finish the few they were working on. Because of concrete for datacenters? I'm sure developing countries are using significantly more of that to build houses, so much so that it completely eclipses the one-to-two datacenters you might see built in the next 5 years in your area. Power generation? That water goes straight back to the atmosphere.. water cycle, even on the open loops, water becomes steam which becomes rain..
Cooling (data center and power generation)
Yes, the water isn't destroyed, instead joining the water cycle. But drinkable water is scarce in a lot of places already.
And the water is being pumped out of aquifers that aren't being refilled. That water took thousands if not millions of years to get there. Once gone, the aquifers collapse and cannot be replaced as the space once taken up by the water is now just ground.
The issue is that they need fresh water for evaporative cooling.
Much of the world experiences fresh water scarcity, so it's not the best ethics to divert this resource from people in need to tech of uncertain value.
Check out the impact xAI is having on environment and health in Memphis if you want to go further down the rabbit hole.
Maybe people shouldn't be building data centers in deserts. In the city of Toronto the Deep Lake Water Cooling System uses water taken from Lake Ontario used for drinking water to cool a number of buildings. Most notably 151 Front Street West which houses the data centers routing most of the Internet in Ontario.
Don't bring ROI into this, that's loser talk
Radio on internet?
Return on Investment
HODL!
* True innovation
Every tech startup circa 2023 doesn't do anything without smearing AI all over their front page.
Makes me wonder about the things that go into datacenters and power plants:
- Air conditioners
- generators
- Racks/Cages
- fire supression
- Concrete, Steel, power lines
etc
For all the issues with power use, I do grant it that a lot of materials and labor in needed at the same time. It isn't JUST burning up cash to make Sora videos.
But by the same measure, the Great Egyptian pyramids would have been a huge boom for work even if the final product didn't achieve much for most people.
It was a huge boom of Slave work. I wouldn't consider that a positive, either.
Isn't it pretty much known at this point that pyramids(at least the Egyptian ones) were not built by slaves?
Thanks for the correction. The experts think the builders were not slaves. Their life in the barracks was too good for slaves, and similarly were they treated in death (which was specially important there)
https://en.wikipedia.org/wiki/Slavery_in_ancient_Egypt
All those have multi-year backlogs.
A coherent regulatory framework? An honest conversation about whether we need all this?
Profit too. We lack profit.
I have a hypothesis that it was AI, not COVID/sanctions/etc, that was mainly responsible for the 2020-to-ongoing "chip shortage." Ignoring companies with their own fabs (Intel) and companies with pre-existing reserved-capacity contracts with fabs (Apple), everyone else is stuck waiting in line behind batch after batch of fab orders from Nvidia.
Downstream of that, AI is effectively also responsible for the current generation of game consoles never declining in price.
Because game consoles are fixed platforms that continue to be manufactured over 5+ years, normally the most expensive parts in the system (the CPU and GPU) would gradually get cheaper to manufacture [and in turn, cheaper to buy] over the course of the console's lifetime—which was often passed onto the consumer in the form of the console's MSRP gradually decreasing. Either the process node for the console's silicon design would stay fixed, and demand for this process node would gradually decrease as larger fab customers move on to newer nodes, decreasing the (effectively auction-based) pricing for fab time on the older node; or the console manufacturer['s silicon vendor] would put their silicon design through a process-shrink, and so, while still paying top dollar for use of the fab's newest node, would be getting more chips per wafer out of that, and again could charge less.
But instead, what we've seen since the start of the AI boom is that there's no longer any price-reduced timeslots to be sold to manufacture the low-BOM parts for these price-sensitive console-maker customers. Instead, both Nvidia and AMD are now getting such high value out of even the older nodes, that 1. the fabs know they can squeeze them, charging full price for those slots as well; and 2. both Nvidia and AMD, in their roles as silicon vendors to the console makers, haven't been able to justify using (very much of) the time they're paying so much for to fab low-BOM-cost parts to fulfill their pre-existing outstanding purchase orders, when they could instead be fulfilling much-higher-margin new POs from the hyperscalers.
Thus every console of the ninth generation (PS5, Xbox Series, Switch) still selling for their launch price (with no help from process shrinks); thus none of these consoles having been able to be produced in excess of demand to the point that supply-and-demand ever drove retail prices below MSRP; and thus the only tenth-gen console so far, the Switch 2, taking ~4 years longer than anticipated to release†.
---
† Nintendo were very likely waiting for Nvidia to run off enough of the Tegra T239 with a sufficiently low passed-on fab cost, for Nintendo to both 1. be able to build a backstock and non-run-dry pipeline of Switch 2s, and 2. be able to be positive-margin on charging the same price as the Switch 1 for them. They waited four years, and neither thing ever happened; so they eventually just gave up and priced the Switch 2 higher, and also built out an entirely novel D2C + online-marketplace-partner distribution pipeline so they could ration the tiny initial supply of units they had been able to build with the chips Nvidia had supplied them so far.
Though, that being said, Nintendo actually got a double whammy here. They were also waiting for fast NAND to come down in price, so that they could have physical game cards manufactured for a trivial BOM price while still enabling the "direct GPU disk-streamed assets" pipeline that games of the last generation had begun relying on. Obviously, as today's article points out, that hasn't been happening either! Thus game key cards; thus SD Express cards only beginning to trickle out, with no sizes above 128GiB available at the Switch 2's launch time; and thus those SD Express cards being ridiculously priced for their capacity compared to equivalent transfer-speed + die-size NAND (as seen in e.g. low-profile/flush-mount flash drives.)
> I have a hypothesis that it was AI, not COVID/sanctions/etc, that was mainly responsible for the 2020-to-ongoing "chip shortage."
Might be for higher-end chips like GPUs, but for smaller ones like microcontrollers it was very much COVID that threw a huge wrench in the gears.
Things are closer to normal now, but the toilet paper effect hit the electronics supply chain hard during COVID. People resorted to buying dev boards just to desolder the microcontroller on it to use in their commercial product, and similar desperate moves. With lead time measured in many months to years even when the factories are operational, sudden hoarding is not what you want.
It wasn't COVID directly, it was the car companies canceling and then uncanceling their massive chip orders that caused the shock in the semiconductor supply chain.
Something I think contributes is that TSMC seem to be so good at making chips now, it seems like the defect rate must be low enough and they can produce huge chips in volume that a decade ago would be burning money. If you can do that, why bother making less lucrative smaller chips? Also while there's still benefits to shrinking an existing chip with a revised design, I'd guess we're into diminishing returns for what it offers a console whether the chip itself gets cheaper or lets them reduce the bill of parts in other ways.
This is also the first generation where prices have gone up IIRC.
The defect rate always drops like that. But usually the big lucrative chips move on to the next node, and instead they're filling up more nodes in parallel.
The main cure is to build more fabs but that's hard too and they're not in a rush to ruin their own margins by scaling up too fast in the face of an uncertain future.
That sounds reasonable but I suspect it was a little bit of both. Initially Covid slow down followed by large demand. A lot of older Nodes were shutdown (300-400nm) with the Covid slow down and a lot of chips ended up having to be moved over to newer nodes once demand picked up again. It led to a big swing in the flow of production and would have roll on effects.
Combine that with a huge GPU boom and you have the setup for production issues.
But the basic chips like the ones that go into cars are the ones that were the most scarce--the low margin stuff was not being made. You're saying it's because of Nvidia? I'm not totally tracking here.
The game console thing is also related to monetary inflation--the price of everything just keeps jumping.
* Time to wade through all the AI slop
* Trust that what we are watching or reading is human-sourced and real
* Time for real creators' unique voices to remain unique
* Hope for society's future
We should be making some effort to quantify the amount and cost of slop produced by both AI and simpler automated systems (spinners etc), it's a huge negative externality.
Well, at least the paperclip maximizer is not running out of water and carbon.
Definitely lacking profit. OpenAI is on track to lose $60B this year.
Don't forget rising energy prices.
jobs...
I built a computer December 2024 with a Ryzen 9900X, 64GB DDR5, 1TB + 4TB SSDs, 4060. If I were to buy the exact same parts now, nearly 1 year later, it would cost about $1,000 more for the exact same components. The memory alone went from $189 to $505. I'm not sure how much of that is from tariffs and how much is due to AI datacenter demand, but that is a massive increase.
Yup. Walked into a microcenter a few weeks ago and was surprised and saddened by how expensive everything has gotten.
So, it's Chia all over again? [1]
Sometimes, I think government supply control isn't that much of a bad thing. That way, the government could force the availability of goods for the common market as well and not just for the really big dogs flush with ample VC money to burn who can pay any price.
[1] For the uninitiated: a cryptocurrency where the limiting factor wasn't CPU, RAM or GPU compute resource, but storage - in 2021, there was so much craze around it that HDD and SSD prices exploded, and after the bubble collapsed a lot of heavily abused drives flooded the markets.
Chia was/is WORM (write once , read many) so not sure how these drives were heavily abused? As a storage nerd, I bought many 18tb drives for chia and they are still spinning just fine in my storage arrays. It’s been five years.
Honestly no idea, all I 'member is reporting from back in ye olde days [1]. Damn, time really flies.
[1] https://www.techpowerup.com/281979/chia-farming-already-caus...
The future of AI is everything we liked getting ruined by AI.
I've been in the industry since 1985. There have always been boom and bust cycles with the basic components. It's the nature of the thing: very costly factories that take many years to bring online. Plus normal business cycles that tend to align with the US election cycle.
>Picking QLC over TLC allows them to maintain costs while achieving sufficient endurance for cold storage.
How does that work, doesn't QLC have less write endurance?
Yes, but QLC has much higher density.
I think it's the higher density that makes it better for cold storage, which generally has infrequent access, and more reads than writes.
Hence the QLC's endurance being "sufficient for cold storage".
Cold storage normally doesn't have frequent writes or frequent reads.
I think it is solidigm that has started to argue that with a 128 TB QLC drive constant writes at the maximum write rate will hit the drive’s endurance limit at about 4.6 years. The perf/TB of these drives is better than HDDs. The cost per TB when you factor in server count, switches, power, etc., is argued to favor huge QLC drives too.
Ssd for cold storage seems like an odd choice in itself. If that’s genuinely done due to availability then we really are in for a wild ride
I believe "cold storage" in this parlance is more like "read-oriented" rather than being accessed once in three years.
In short: Aggressive overprovisioning.
Enterprise SSDs are not expensive only because they have better flash chips, but they have much more of them.
A top of the line write oriented SSD comes with 4-7x more capacity than what it says on the tin, but that extra capacity is used for cell replacement rather than capacity itself.
Mixed use comes with 2-4x overprovisioning, and read oriented is around 2x IIRC.
I find these numbers to be way outside of what I have heard of. I would be surprised if you could give an example that comes with even 1.5x capacity. (4TB capacity, 6TB actual flash on chips, for example.)
Overprovisioning is much less aggressive than this in practice. A read-oriented SSD with 15.36 TB of storage typically has 16.384 TiB of flash. The same hardware can be used to implement a 12.8 TB mixed-use SSD (3 DWPD or more).
Huh, so that's why openai has been asking to use my local disk.
I just got 2 16TB drives for $90 each off Craigslist. Perfect for video game storage or downloading huge AI models or whatever. Or legit Linux ISOs. Wish I got more but I was too late. Can't find anything close to that price online
$5 a TB is insanely cheap and someone just getting rid of stuff and not really caring about the price. Nothing has ever been that cheap at retail or manufacturer recertified. Lowest was like $9-10 last year and now we are sitting at more like $15.
Yeah it was interesting because the guy knew what he was doing unlike other great deals I've had. I guess he just really didn't want them
... might wanna zero those drives just in case
They came uninitialized, but yeah it could be worthwhile to actually zero and fully format them. They are decently fast at 275 MB/s but even that's 16 hours to do one pass. Maybe let it run over the weekend
16 hours isn't that bad. I recently had to zero my old 2tb Time Capsule and just let it run overnight and during work, wasn't so bad. You'd also get to detect bad sectors, figure out how good the drive is in terms of what % of the drive is actually usable for large sequential writes
Hunger sharpens ingenuity.
Soon we'll see many "AI" product that analyses network storage to identify files that could be deleted because they're the nth copy of some random report that was never published.
Cloud storage and on-prem storage vendors have long provided deterministic block-level dedupe via hashing, without expensive LLM content analysis.
And when the AI bubble bursts, "refurbished" HDDs and GPUs will flood the market. Save your money now and be prepared.
GPUs yes, but there'll be no HDDs making it alive, they'll get destroyed to protect whatever rubbish they had on.
Depends how it goes down, if a company goes into insolvency all security policies are off the table and random hardware can get shifted into lot bidding.
HDD can be written multiple times with random data if data centers really have to protect what their former customers wrote on them. I never looked at those details in standard contracts.
There is also encryption at rest.
All you really need to do is write one pass of zeros on them. That will prevent anyone but a very dedicated adversary with expensive equipment from recovering any data, especially on TB scale drives.
Can still take hours per drive though, which is why a lot of people skip it.
Or encrypt it and just trash the encryption header.
I make a random 1MB chunk, then write that all over the drive, at overlapping offsets. I've been told that really clears it. On IDE-spinning-rust disks I trusted it, not sure if I should trust these modern SSD
Why rewrite the same 1MB chunk, instead of making new random chunks?
Redundant data at least opens the possibility that the drive could deduplicate.
Cause making new random was taking too much time.
Pretty sure most modern computers can generate random data faster than they can write to a disc.
All the large datacenter/cloud companies do not let hard drives leave the building.
I wrote hddrand to write random data and optionally read it back to verify integrity. https://github.com/mqudsi/hddrand
When I used to do computer refurbishment, 'Boot and Nuke' was great for this. Load it up at boot, and write over the with random junk a few times.
They have enough investor money they don't need to recoup it selling used drives. Straight to the shredder.
> HDD can be written multiple times with random data
Which costs more in compute than simply throwing the drive in a shredder
What's the disposal method for shredded drives?
Mount Doom
Not really, if we give the HDD some resale value. There's a market for used but functional hard drives.
Nah, the liquidators aren't going to care about that. Those hard drives are going to be shipped out with all your wildest porn chat bot fantasies.
"Shredded onsite" means by the next user when they format the drive and write contents to it /s
I believe many enterprise drives have instant-erase functionality (presumably deleting an encryption key).
If they were encrypted to begin with, yes. Many are not.
A drive that supports Secure Instant Erase should be encrypting all data. When the SEI function is invoked (“nvme format -s 2”, “hdparm —-security-erase”) they key is thrown away and replaced with a new one. Similar implementations exist for NVMe, SATA, and SAS drives — regardless of whether they are HDD or SSD.
This puts a fair amount of trust and in the drive’s ability to really delete the old key.
I was under the impression that these drives may be transparently encrypted by default. (Rollable encryption key in hardware, invisible to end-user.)
new HDD have a SHA key built in that can be erased without zeroing out the drive. it just creates a new key and boom, free disk
My understanding is that refurbished GPUs from e.g. crypto mining are something people are warned away from, because they've often been run into the ground. Are AI usage practices different?
> My understanding is that refurbished GPUs from e.g. crypto mining are something people are warned away from, because they've often been run into the ground.
No, this wasn't the case. While there were never comprehensive studies various tech media purchased these cards to run testing and found that, other than scammers, they all performed to expectation.
You can buy used car tires with 1% of thread left and they'll perform amazingly during your one time test too
Car tire is a moving part that wears in use. Only moving parts a GPU has are its fans, which can be replaced. The rest may last really long, or fail brand new.
Believe it or not but the "really long time" you're mentioning is much less "long" when you run the card 24/7 at 100% in enclosed spaces
Electricity is a moving part btw, electronic components do wear and tear
Thermal cycling will eventually cause problems with continuity but I'm not sure that's really predictable. We have some really old GPUs that eventually had problems under load, we attributed it to thermal degradation as we had ruled out pretty much everything else.
It was considered a fear but I don't know if there is much truth to it. The fans and capacitors would give out long before the silicon.
Even if it say, halved the life span of the chips, that is still far longer than what most people would ever use them for.
> And when the AI bubble bursts, "refurbished" HDDs and GPUs will flood the market.
GPUs? No way. The datacenter cards don't even have video output ports, and I think the chips destined for AI / ML training also have everything video/render related removed from the silicon, makes for more yield.
And the other way around, using (cheap) consumer GPUs in servers, I think at least NVDA tries to prevent that with driver-based DRM, so there won't be any flooding coming from there either.
If this bubble pops you might need that money for food when bananas go from $1.50 to $150.00
This is a long term good thing.
It sucks right now and will probably suck through 2027.
By 2028 or so we'll have a 50% drop in price-per-storage for these components.
> This is a long term good thing. ... By 2028 or so we'll have a 50% drop in price-per-storage for these components.
Per the op:
> and the ongoing DRAM shortage is proof of this, with memory kits costing more than double what they did just a few months ago.
> While enterprise-grade QLC SSDs would entirely power this pivot, Sandisk has already raised NAND prices by 50%, according to another DigiTimes report, after initially warning of a 10% increase two months ago.
So you're basically saying prices may return to normal in two years, and that's somehow a good thing compared to them not being inflated in the first place?
> By 2028 or so we'll have a 50% drop in price-per-storage for these components.
Do you mean relative to six months ago, or now? Because a lot of the prices have already more than doubled.
(I’m upset because the computer I’ve been planning to build, which three months ago would have come to around ₹90,000, is now up to ₹1,20,000 and climbing week by week, half due to price increases on the same part, half due to forced substitutions on RAM since the cheaper 32GB 6400MT/s DDR5 sticks are completely unavailable. And looking into laptops, for the first time ever I’m seeing manufacturet SODIMM or SSD upgrades being cheaper than aftermarket.)
> manufacturer SODIMM or SSD upgrades being cheaper than aftermarket.
Temporarily thanks to old stock.
Like the GPUs? How they were up since the bitcoin craze and are still up to now? Yeah I dont think these prices are ever going back down. Specially not since people are buying at these prices and will continue to pay these prices. Maybe HDDs will get phased out though, since you might as well buy SSDs if you are overpaying that much.
Will the AI bubble last until 2028? I’m still unclear how AI will return even 10% of this investment in profit.
Depending on the future you predict 10% may be a good ROI - if AI will replace humans and traditional economy will collapse all other investments will loos value even more. In such scenario you cannot save the money you only can loose less if you will make a right investment.
I'm not sure I understand why the economy would collapse if AI replaced humans. Wouldn't the companies just make more profit because they save labor costs and stonks go up?
The economy is a macrosystem. For a company to make profit, someone needs to buy. For someone to buy a service or a product, they need to have the money needed to buy. If AI replaces humans and UBI doesn't happen, costs might drop to near zero but so will revenue as virtually no one will be able to buy anything. So the economy will certainly collapse.
It depends on how many jobs AI replaces and which. It is unlikely that LLMs will replaces plumbers any time soon for example. Many technological inventions have replaced humans in lots of jobs, the economy has only grown since then.
>Many technological inventions have replaced humans in lots of jobs, the economy has only grown since then.
Indeed it has, but many of those it replaced did not necessarily get moved to better positions if they were moved at all. The industrial revolution was great if you were young and had little to no responsibilities. Your average middle aged farmer with a large family to feed faced poverty or near poverty and those who were able to move to the cities faced worse labour conditions. People on here love talking about the winners while conveniently ignoring the loser side.
And that's not even mentioning the skilling up issue to which older workers are more vulnerable. Who will fund the nation wide skill up costs for them?
Industrial revolution was all right as long as you owned the farm, as most farmers did. The losers were those farm workers who weren't needed in as great numbers anymore, and they weren't paid well in the first place. I agree on labour conditions though, there industrialisation was a clear downgrade for many.
What worries me about AI is how it's not obvious which new jobs it may create. Younger farm workers and children of farmers could just move and work in factories, where the employers mostly took care of their training. I can't see such opportunities here. I believe whole world needs either UBI, or governments paying for jobs that previously weren't seen worth paying for. Otherwise the economy will collapse due to mass unemployment and resulting lack of demand.
Plumbers and electricians will loose too when demand for their services will plummet because people who will loose jobs will have no money to pay for their services (even if the services are still needed).
Economy is a fine tuned system with lots of interdependent parts, any sadden change can have destructive effects. If many people will loose jobs at least two things will happen: 1. they'll stop buying products and services (may be except food), companies who provide these products and services will layoff employees even if their employees are not yet replaced by AI 2. people who lost jobs will default on mortgages causing financial crisis worse than 2008 one. UBI and wealth taxes will mitigate effects to some extent but it looks like political elites across the spectrum are controlled by wealthy people - even left of the center politicians prefer to tax middle class steering away from taxing the rich (both people and companies) more.
No I'm predicting a 90%+ loss.
This is getting ridiculous. Expect this to bust with prices crashing as soon as the combination of hype driven margin inflation and unfulfilled and easily cancelled purchasing agreements hit.
Great. The Cryptocurrency hype finally died down and prices for things (except GPUs) were MOSTLY going back to normal.
Now the sector has invented a new hype machine and hardware prices are all fucked again.
Is this going to happen every few years? The industry hypes up some nonsense that skyrockets all the hardware prices and causes everything to be on backorder, just to dump it and go to the next one?
Thank god I built my NAS in early October it seems like I got the proverbial last train out of town.
Same. Wish I bought 2x 12TB drives now rather than 2x8TB.
I bought 3x22TB WD Red Pros to put into a RaidZ1 vdev so I'll be good for a while. I have a clear upgrade path all the way up to 8 drives with 2x vdevs in a pool but that is something for far future me to consider.
Same here, though I did it a bit earlier. Data center SSD's I bought a while back for 50 euros each from Ebay now sell for over 200 euros. Similar increase for HDD's too.
Maybe in 3-5 years there will be a fantastic time to upgrade again, after AI bubble has burst and slightly used stuff gets dumped from data centers to online marketplaces.
I really hope the AI bubble bursts sooner rather than later. Sooner will impact the broad economy less severely (although it'll still be pretty bad) and curtail these supply chain shortages. If the bubble keeps inflating, the storage makers will have already mostly built out excess capacity and the crash will lead to even longer-term supply distortion.
I think it's still going to be a while. Go talk to anyone who isn't really tech savvy and you still hear all the same hype, now more than ever. None of them can tell you how they've actually used it to create much, if any value, but they still think it's the second coming of Christ.
I disagree, I think there's a huge bubble but I know at least 4 people using LLMs daily who are not techies, and I know what they use it for and it's valuable.
But none of them is paying for it.
Too bad both storage and video cards will be unsafe to buy used when the bubble bursts.
Maybe another round of cheap Aerons at least :)
> delivery times for enterprise-grade HDDs delayed by two years.
I sleep
> so hyperscalers are now switching to QLC NAND-based SSDs to avoid these backorders … This could lead to SSD prices rising worldwide
Real shit
The funniest thing about this is that, with high GPU prices, rising RAM costs, and now increasing SSD prices, Apple will end up producing the most affordable PCs.
If every other PC is more expensive, they will just increase prices.
Apple uses the same RAM, SSD, etc as everyone else does. They don't have a magic supply chain that is unaffected by the broader world.
They have a magic supply chain that is unaffected by the broader world which is one of the reasons why Tim Apple was chosen by Steve Jobs as his successor.
They don't use the same SSDs? They don't use the same RAM? They have their own supply chain in place? Whatcha talking about bud?
They use the same suppliers. The problem is the base chip, and also the wafer itself, all of which will impact apple.
(apple doesn't use hdds so not talking about that here).
Yes, but Apple tends to lock in their supply with long term contracts and prepayments, so they often are protected from supply shocks.
The problem is not chip supply it's manufacturing. Apple has their own manufacturing suppply. This is not the chip crisis of the last years. Hyper scalers are switching to consumer hardware because there is nothing in storage for Prosumer anymore and the manufacturing pipelines for these are smaller and harder to scale than consumer ones.
The NAND itself seems to come from companies like SK Hynix, which are currently struggling to meet demand.
Apple doesn't make their own NAND, just like they don't make their own screens. They did write their own NAND controller and stuck it into their CPUs, but the flash memory that actually stores data doesn't come directly from their factories.
It's not a fucking nand problem. The producers reduced capacity because of the slow consumer PC market in 2024. With hyperscalers shifting to consumer products the market is strained but everyone is scaling up their production now. This is not the same as the chip shortage we had in the last few years. The production capabilities are there they are just adjusting to new market realities. We will see a whiplash for a few months but we can and will produce enough chips for everyone which we were not able to do in the 20-22 crisis. Stop getting your "insights" from influencers and stop parroting them.
It's ABSOLUTELY a chip shortage. TSMC is booked out years in advance and jacking prices because they can.
On the dram side, because that's what I saw recently, sk hynix is down to TWO WEEKS of inventory: https://wccftech.com/sk-hynix-ddr5-inventory-down-to-just-2-...
I'd be curious what evidence there is to support its purely an end-product manufacturing shortage?
I don't see any evidence of a general chip shortage in your link either. That they are down in stock because of unprecedented demand after a downturn in the last 3 years in consumer demands of consumer products is something completely different than being down because of production capacity. They reduced production in 2023 and 2024. It's exactly what I wrote in my previous comment.
"As mentioned previously, the rip-roaring demand for HBM products is reducing the available wafer capacity for other DRAM and NAND products, reducing their supply in the process. We noted in a recent post that the global average delivery times for DDR5 are now quite stretched, currently hovering between 26 and 39 weeks.
Coming back, SK Hynix announced at its earnings call that it has fully booked its DRAM and NAND capacity through 2026, with HBM4 set to ship by the end of 2025."
Appple uses custom DRAM and NAND chips?
No, Apple has effectively promoted iCloud as the alternative to local storage as part of its product differentiation strategy in the lower price segment.
Apple will almost certainly introduce the same approach for the budget MacBook as well.