> Developed from design to production in nine months, accelerated by OpenAI’s models
> the use of OpenAI models to accelerate parts of the design and optimization process.
I wish there was more about this. As is I kind of have to assume that this is just meaningless marketing, like saying development was accelerated by Microsoft Office or their 5k LG Ultrafine 40-inch monitors.
Like, if this was as big a deal as it kind of vaguely implies, they would be making a bigger deal of it, right?
Chip CEO here. It really depends on what "design" or "production" means. Does "design" mean that the design was complete? Does "production" mean the beginning of production, i.e. tapeout? If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip. If measuring from concept (no RTL at all, block diagram of architecture) to tapeout, this is an amazing timeline. The truth is probably somewhere in between. A more concrete statement would use actual technical milestones and gates.
Not a chip CEO, but I read this article and thought that they're working on some kind of application specific chip only for serving models. Similar to how an FPGA can optimize certain tasks.
Given constant weights / biases of a Transformer / DNN you could use pipelining to feed forward calculations through the array one layer at a time. For DNN's with thousands of layers you might see 1:1 speed up per layer channel.
I doubt they would undergo this process for marginal gains.
In very crude terms, AFAICT, if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once.
IDK how the custom hardware exploits this; would love to hear any ideas!
> IDK how the custom hardware exploits this; would love to hear any ideas!
You might like this article [1], titled "FPGA-based CNN Acceleration using
Pattern-Aware Pruning". More context and details can be found in the PhD thesis of Léo Pradels [2].
>If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip.
Big companies are lumbering behemoth, crude assemblages of barely cobbled-together incentives and principal agent problems in a trenchcoat. Getting them to change direction, or worse, try something new at scale, is a massive undertaking
Nah, you just need to get the CEO behind it. Most coordination issues get solved when the CEO is breathing down your neck to get something done. Trouble is that they don't do this enough.
The Fire Phone was Jeff Bezos' personal baby, and we know how that went. Then there was the Apple G4 Cube with Steve Jobs, the Model X' Falcon Wing doors and Elon, and lets not even talk about the Metaverse and Zuck.
Actually, you've provided examples that prove the point. None of those were especially good (though everyone wanted the G4 Cube), and yet they made it to market anyway. Why?
Because the CEO was behind it, breathing down their necks.
The typical way a chip effort in a non-chip company works is that the "design" is the RTL (e.g. SystemVerilog that defines the behavior of the chip) and then this is handed off to a third-party "design house" (such as Broadcom) that turns that code into a real image of a chip, which is called a GDS (basically you can think of this as a very big layer by layer photoshop file) that can actually be sent to a fab. This is called "backend design", in contrast to the "frontend design" (the RTL itself).
As another commenter said, Broadcom is very experienced with backend design (as well as the supply chain management, testing, etc. that comes after the chip is taped out) and so this can't be regarded as a "first chip". Richard Ho (the head of hardware at OpenAI) is also extremely experienced and used to be the head of the Google TPU effort -- where he actually worked with Broadcom in a similar tapeout already. So yes, this is not a "first design"!
Yeah, "first chip" here likely means they contracted Broadcom (or a firm with similar experience) to do all the heavy lifting. Building out your own in-house teams for this sort of thing is a decade-long project - just look how much inside Apple's early chips was licensed ARM / PowerVR cores
Apple didn't have the talent in-house until they bought Intrincity who worked with Samsung on Apple's earlier Arm chips as well. https://en.wikipedia.org/wiki/Intrinsity
The way I heard it PA Semi was the singular driving force that led to Apple Silicon, but I'm not any kind of insider that's just the chatter I heard.
Whoever it was, whooo, that's hot shit. I remember an M1 MacBook Air just cleaning the clock of an Intel MacBook Pro and thinking "x86_64 has real competition again".
Great silicon. I'm over it with not having root on my own machine, so I've left the ecosystem, but it's really nice hardware, can't dispute that.
The hardware description languages (HDL) used in chip development are like programming languages. The existing models understand them and can do a lot with them. You don’t need to have separate, specialty models designed for this work to use LLMs in chip design workflows.
Design verification also involves a lot of traditional programming which benefits from LLMs.
So it’s not meaningless at all. You could download some of the open source chip design software today and the LLMs could even help you get started on your own tiny chip if you are so interested.
I tried making a button using Claude entirely (including the 3D printed enclosure) and it effed up pretty hard with the traces and the header spacing. The project was a big red arcade button that plays the "ah-my-groin.mp3" when pushed (from Simpsons). It did cool work on saving battery life, and the 3d enclosure was awesome, but yeah, I'm convinced I'd have to do another version or two of the custom chip until it came back right. I used a Blender MCP for the 3d modeling. I used a KiCAD MCP server for the chip design/validation.
I think we're not there yet. I've been meaning to look at this flux.ai to see if it has the prompts/workflow worked out better than what I was able to cobble together in a few hours. Maybe Alteryx's MCP server would have been better. I'll try that this weekend for another board I've got.
> I tried making a button using Claude entirely (including the 3D printed enclosure) and it effed up pretty hard with the traces and the header spacing.
PCB design and 3D CAD design are different topics.
Hardware Description Languages are closer to programming languages than CAD. Look at some Verilog to get an idea - https://en.wikipedia.org/wiki/Verilog
Designing circuit board and 3D models (even using something like OpenSCAD) is a very spatial process today. You are dealing with coordinates one way or another.
This is very unlike how FPGA and (I assume) ASIC is done. That is more like a traditional programming language but everything happens all at once (no sequence of statements outside tests, if you need that you have to write a state machine yourself). You define logic expressions between signal, add stateful latches, etc. But you never specify the physical layout.
Instead you feed your description to a tool that acts a constraint solver/optimiser that computes the layout for you (this is for FPGAs called synthesising IIRC, it is akin to a compiler). Typically quite slow, even for small circuts like we did at university it took minutes, and for large circuits it might easily days.
Now, this raises the question, what if you design a PCB net list using AI, but then use traditional autorouting and layout? I believe that can also be done, but I have no experience designing PCBs, so I don't know how well it works.
Meta: can we not downvote people who are clarifying what they're saying and asking questions, even if they're wrong about something, if the content isn't otherwise objectionable?
I didn’t downvote, but the OP is either a troll or someone who doesn’t want to notice he doesn’t know what he’s talking about. Either way we want less of that on HN.
I'll acknowledge that I don't know what I'm talking about. I really appreciated the clarity! Surely you find value in knowing that creating your own custom chips is almost doable by someone who doesn't know what they're talking about! (also, I am a troll, but in this case, just clueless)
Maybe the confusion stems from the word "chip". Creating a chip usually means designing and producing a microcontroller or a processor, not a printed circuit board that you populate with existing chips.
Ohhhhhh! Yes, that's exactly the problem. It all makes sense now. I was just piecing together an existing microcontroller and a mp3 module by printing a custom circuit board.
One (kicad) make the board, the other (blender) make the casing for it. Both are “hardware” but is electronics and the other is mechanical. Electronic one AI can do a good job, I can’t wait for it to fully built the whole circuit for you based on your specs.
PCB layout is an art, and doesn't seem to map well to LLMs (I tried for shits and giggles recently). Claude in general, kind of like code, does a lot of redundant belt and suspenders stuff in the schematics it generates (if it can generate them at all). It's one of those things that's really not there yet outside of the simplest designs.
DeepPCB has an AI autorouter [1] that uses reinforcment learning and works really well. Recently they also released an AI agent that analyzes your board, proposes plans and can route your board for you [2]. They have a KiCad plugin [3] and you can try it for free.
Most HDL code is locked up behind corporate firewalls and not available as training data. While LLMs can handle it to an extent there's a lot of room for improvement. I'll bet that OpenAI and their competitors are racing to license this IP from major hardware vendors in order to compete in the chip design vertical.
There is quite a lot of Verilog/SystemVerilog and VHDL code in the wild. And hardware description language code is very simple and straightforward relative to programming code.
And the two things that take up VAST amounts of time in ASIC design are testbenches and timing closure.
A LOT of hardware design is testbenches to verify things. AI is REALLY GOOD at generating things like testbenches. And nobody really cares if the quality of your testbench code sucks as long as it validates what it claims to.
I don't know how good AI is at timing closure, but I wouldn't necessarily be surprised if it is pretty good at it up to the physical point. That's lots of textual output which you can put a constraint on.
Everything involving physical design, though, tends to be a disaster waiting to happen if you let AI loose on it.
One day we can design our own pcb with chips, hardware and other io. Companies will accept these as files and you can collect your pcb the same day. I think in China they are doing this already
> The existing models understand them and can do a lot with them.
In my experience they are not especially good at SystemVerilog. There's a lot of knowledge about it that is locked behind paywalls and it's very niche.
My guess is the "from scratch" here is quite the exaggeration. Otherwise why did they need Broadcom?
Not having a free toolchain that can actually handle the real language has probably been pretty bad on the downstream public knowledgebase. Hopefully Verilator can eventually close that hole, and there can be more high-quality designs and codebases incorporated into future models. Claude is at least good enough to write SV that triggered a compiler crash or two. :)
Broadcom already has a ton of IP for AI SoCs. I'm guessing the hard parts of this inference chip was already designed by Broadcom and OpenAI simply told Broadcom what it wanted. It's likely very similar to Google's TPU.
Early testing shows that the first-generation accelerator will deliver performance per watt substantially better than current state-of-the-art
What is substantial here? Vera Rubin is shipping in volume later this year and it is expected to be 10x more power efficient for inference than Blackwell.[0] Even if they're already taped out the chip, getting bugs fixed, getting chips manufactured, getting HBM allocation, getting a rack design, hooking them up together, putting them in a data center will likely take at least another 12 months or likely more. By the time this chip is in data centers in volume, they're likely competing against Vera Rubin Ultra or maybe even Feynman.
Personally, I don't think OpenAI should have invested in this project. It's too early for them. They should have focused on models like Anthropic and win there. When they're profitable, they can take on these projects.
The risk here is very high for OpenAI because AI has a hard cap in energy. If you have a gigawatt, you should only install the best chips. If Nvidia's chips are better, then this is a wasted project and likely wasted billions.
They have. That's why OpenAI was able to get a working demo in 9 months. But going from a small scale system to a full fledged data center deployment is likely much harder.
I don't know how much of the things outside of the chip Broadcom has vs Google's proprietary tech that is not shared with Broadcom.
Nvidia's Vera Rubin has 6 unique chips working together in a single rack.[0]
I thought most of the Google tpu magic is on wiring up these chips into supercomputer like clusters with specialized interconnects and whatnot. The chips themselves are less interesting in isolation.
I’m just happy to see diversity here; sometimes I feel like Nvidia is going to eat the world, with buying other fabs and branching out - or up, I guess - from chips and racks to models, frameworks, and end user stuff.
Right. There are two possible meanings and shades in-between:
1) OpenAI genuinely have AI technologies that can improve chip design (bold, unlikely claim, needs evidence)
2) OpenAI designed test/verification models and kernels that could be run on the simulated hardware to test its performance
As you and others have said, it's hard to trust when they are happy to write something that could easily only mean the latter but sounds like the former.
at the hardware company I work at, people are now using claude code and developing skills for it to do basic stuff like triage or do initial debug on failing tests, search for potential causes in RTL, generate skeleton documentation for designs etc
Browsing openai's job postings in the past few months is enough to contirm that it's more than this. They are for sure making serious efforts at building ai for chip design.
Jesus. This is tinfoil hat territory now. Why would they fake something like that? ANY company in this field would try to become free from nvda. Goog has done it already, amazon has their own thing, so it can be done. Not saying they'll 0shot this vertical, but ffs, they don't need to fake anything. They are making an effort, and it would be insane to think they aren't. Might work, might not work, but to even think that the effort is fake is going too far.
I'm not saying this isn't a thing. I'm saying oAI doesn't need to fake trying to make a chip or hiring people to make AI better at chip making, or dogfooding or anything like this. It's obvious they're doing it. They'd have 0 reason to fake something like this "for the investors". Come on!
From time to time? Lol you must realize, frontier lab eng are using Codex/Claude-Code 99% in loops, on models the public doesn't have access to. Why? Because it works. Just a matter of time before humans are out of the loop and what comes next is a black hole
"The future is here, it's just not evenly distributed"
Or OpenAI accelerated the design and optimization process by summarizing emails exchanged during the design and optimization process, or made it possible to ask an AI questions about meeting notes
Yes, obviously. But do we think LLMs without access to proprietary information do a better job with them than Broadcom's human experts or existing proprietary tools at this level of operations?
It is still a bold claim and it still needs evidence.
We would obviously get a bit more of the evidence if it were to be more useful for the upcoming IPO than this rather open-ended, reinterpretable phrasing.
I've used GPT-5.5 and Opus both for FPGA design with good results. We built a lot of tooling around it to help the models, but even without that they're definitely capable of designing digital logic.
My guess: it is that those who KNOW the subject realize that LLMs suck at it, and those who do not, do not realize it, since their output is plausible, and sometimes even works.
This actually plays out across every field and is well documented. An expert can recognize the hallucinations and bullshit coming out of LLMs, while non-experts see plausible output and do not know enough to know it is BS.
Wrong. Myself and colleagues know the subject and they are useful in FPGA design. You should stop hallucinating about topics you don't have experience in.
> OpenAI genuinely have AI technologies that can improve chip design (bold, unlikely claim, needs evidence)
Why is that a bold and unlikely claim?
Are you saying that AI, which has been proven to cure diseases, solve our hardest math problems, write complex computer code and generate entire generated worlds and HD video from a simple prompt would somehow be like, my bad, I guess I can't design chips?
Because then they'd likely have stfu and outperformed Intel, Nvidia and AMD, or at least one of them.
They're burning more cash than pretty much anyone else and doesn't have anything public that looks like a matching revenue stream so they probably need one very badly.
There is a lot of verilog out there, it's pretty feasible that they had AI assistance writing more to design their chip.
It doesn't have to be revolutionary, it could just be AI-assisted design and lined up well enough with their operations for a custom ASIC to be worth it.
Also there's some much boilerplate around everything. Writing a testbench with codex is extremely feasible. This is the kind of verifiable feedback loop the agents shine at.
I feel like "the use of OpenAI models to accelerate parts of the design and optimization process" just means that engineers were using ChatGPT to sanity check their designs and suggest potential optimizations, though that's just my take (and I'm quite cynical about AI marketing in general!)
> the use of email, spam filters and spellchecker to accelerate parts of the design and optimization process
honestly you don't realise how much more efficient it is until you are stuck using the wrong flavour of outlook, the spam filter breaks or sloppy spelling, punctuation and grammar force you to clarify details needlessly.
My girlfriend works at Broadcom doing chip design, and based on what she's told me they JUST got claude code like 3 weeks ago, so I really doubt this means anything beyond them vibe coding some scripts or something...
VHDL, VLSI are well documented languages, with well build test and verification frameworks and harnesses. Even just by iteration you could get there if you have the money to pay for it.
I just read a claim on Twitter that the reason these companies (Google and Amazon as well as OpenAI) are using Broadcom isn't just for design expertise, but because Broadcom have allocation agreements in place with TSMC and the memory manufacturers.
Nope, not down. "total Personal Computing Device (PCD) market — comprising traditional PCs and tablets — posted 2.8% year-over-year growth in Q1 2026, with combined shipments reaching 103.3 million units. PC shipments grew 3% YoY with 65.6 million units" https://www.idc.com/promo/pcdforecast/
Q2 is forecasted to be negative, partly because of RAM prices like you said, but for the most part this is something that only price sensitive nerds care about. Broadcom sells a ton of server chips. Server sales are up 30% vs last year so I highly doubt they're desperate to use their allocation
I was actually thinking of smartphones first because they seem to be the best-selling "personal computing devices" (different definition from IDC) and come with a lot of RAM (8-16 GB or so? Mine has 12) these days. And there I confused Broadcom with Qualcomm - Qualcomm's biggest end customers seem to be smartphone buyers.
I thought of PCs second since most chip manufacturers make some thing or another that goes into them (Broadcom probably more than Qualcomm), and yes it's very suprising that PC sales don't seem to be down yet.
> the full-year 2026 [PCD] outlook has been revised to −10.4% year-over-year
because
> erosion of consumer purchasing power amid regional inflation and currency volatility in many key markets, compounded by memory and storage shortages that are proving more severe than anticipated in the previous forecast cycle.
The positive Q1 YoY growth
> was largely the product of pull-forward demand, as both consumer and commercial buyers accelerated purchases ahead of anticipated price increases and limited product availability.
The idea that only nerds care about the cost of things is... absurd.
Most design partners have allocation agreements. The thing is Broadcom is an absolute GIANT in the ASIC design space, and it's closest competitor Marvell is a fraction of it's size.
There are a lot of large tech companies that most of HN has never heard about that completely dominate entire segments.
Broadcom has become wealthy by being Google's TPU hardware partner, including sharing their TSMC capacity with Google, and evidently now they are doing the same thing with OpenAI. What a brilliant way to take advantage of the AI gold rush!
I wish they weren't using their piles of money to extort money out of the software industry like they are with VMWare and Bitnami.
> Broadcom has become wealthy by being Google's TPU hardware partner...
Kinda, but not exactly.
Broadcom cornered the enterprise infra and security market in the late 2010s and early 2020s after acquiring CA Technologies, BMC (EDIT: Did NOT acquire them, they were considering it back in 2018 but decided against it and KKR ended up acquiring them), Symantec (which they bought instead of BMC), and VMWare and were able to make a strong cybersecurity story during the late 2010s cybersecurity and SaaS boom.
That gave them plenty of cashflow that helped subsidize their hardware business when hardware was not viewed as hot as it is today.
Additionally, Broadcom is GCP's marquee customer and has been for a little under a decade so they were able to make a sweetheart deal where all that software businesses at Broadcom would be exclusively using GCP and in return GCP would working with Broadcom to design it's silicon and source infra needed for their DC buildouts.
Ironically, the DoJ blocking Broadcom's acquisition of Qualcomm was the best thing it ever could have done for Broadcom, because it gave Broadcom the dry powder to dominate the Enterprise SaaS and build a strong niche in the cybersecurity space.
> piles of money to extort money out of the software industry
From personal experience, executives and leadership who started off in the electronics and hardware industry are much more vicious and cutthroat than their peers who started in software.
Working in an industry that historically had to deal with high commodification, low margins, and long tail sales leads to leadership that can execute. Additionally, no one climbs the leadership ladder without having spent years as a line-level engineer, but that's true for software as well to an extent.
Edit: can't reply
> Did they acquire also BMC?
Nope.
Broadcom was considering acquiring them in 2018 but decided not to go through with the opportunity and KKR jumped in.
Good information, Broadcom is a playa, lots and lots of acquisitions! (a quick google search turns up a very eventful history for Broadcom)
> From personal experience, executives and leadership who started off in the electronics and hardware industry are much more vicious and cutthroat than their peers who started in software.
Only The Paranoid Survive is quite a name for a management book. It implies surviving in the world you are speaking about.
I wanna see an inference chip where the weights are part of the rom of the chip.
There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle.
That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second coming out the output bus.
Downside is this chip would be huuuuge - a whole wafer.
Wafer level faults probably won't matter though - neural nets are resistant to a few missing or wrong weights.
Due to the speed the industry moves, you'd want to race from model weights to production super fast, make 50 wafers, use them for a year, then bin them when that model is obsolete.
this appeared some time ago, https://taalas.com/, but I'm sure there's others thinking these same thoughts.
this would be best for small models imo, nothing frontier because that changes too fast
By the way, you've seen Cerebras? It's not gone as far as what you described - loads of cores and RAM but you still load up the weights onto it as software and they need to be streamed into the chip for large models - but it is a whole wafer.
Cerebras is a whole lot of SRAM, basically a ton more L1/L2 cache, hence increasing throughput.
They're pretty supply constrained right now though and their production costs seem prohibitive.
The interesting players at the moment are from Toronto: taalas (print the model onto the silicon) and tenstorrent (dataflow programming based hardware)
Not really that: you are pointing to Compute-In-Memory (CIM) - techniques where the data (here, a multiplier value) is part of the processor (here, the multiplying circuit).
The problem of "fetch and process" is bypassed completely architecturally: the data is there where the processing happens - it's not moved, there is no latency.
>> I wanna see an inference chip where the weights are part of the rom of the chip.
I've been wondering about that for a while now. For a lot of tasks putting weights in ROM is probably OK. OTOH:
>> There would be 1 multiplier per weight...
I'm not sure that is a good idea. Maybe if its quantized down to 2 bits... Otherwise maybe a small ROM near each multiplier (or row of them or whatever) so the multipliers could handle N distinct matrix operations without having to move the data from far away.
Another fun thought is to have a row of MAC units on DRAM so a DRAM row would be a vector. Row size might be 64Kbit or 8K weights if they're 8bit. This also keeps the weights and calcs on the same chip. I'm not sure this would put enough multipliers on one chip though. Systolic arrays can have tens or hundreds of thousands each doing one op per clock cycle.
analog chips could also be very interessting instead of using digital signals and processing them against the weights in the ROM. I have no idea if that scales with such big models though.
Supposedly memristors would be ideal for this (and it would be reprogrammable), but then again, memristors seem to be the carbon nanotubes of the computing world.
I've also been thinking about this. Although the forward pass of a transformer model also involves some heavier operations like normalization, reciprocals, exponentiations or other non-linearities (GeLU, SiLU) which may (though typically don't) involve learned weights as operands.
“ Wafer level faults probably won't matter though - neural nets are resistant to a few missing or wrong weights.”
Brain science people “love” traumatic brain injury cases because it can help explore what happens when bits of the “brain wafer” get damaged. We’ve learned a lot from such things.
I wonder if people are intentionally “destroying” parts of the model weights to learn more about what happens? Like could you strategically wipe a gig of the model so it’s “all zeros” and see what happens?
Of course tampering with chunks or nodes in the NNs is a way to study the "spawned" (through gradient descent etc.) configuration and "reverse-engineer the black box" to get "AI transparency".
Anthropic published an important work around one year ago.
This is called mechanistic interpretability. There is lots of fascinating insights already since you can do basically everything down to the neuron or weight level thousands of times. The human brain is many orders of magnitude harder to make sense of.
well its actually called ablation, and its one way to do mech interp. anthriopics got a bunch of work on mech interp here https://transformer-circuits.pub/, like SAEs and NLAs
This is another large leap in the great divide of haves and have-nots. Big tech AI labs will develop LLM accelerators and hardware LLMs that increase frontier model output to tens of thousands-hundreds of thousands of TPS.
These chips will be used internally for their own business goals, giving them the capability to iterate at such an insane pace they will be able to clone every software product and software company on Earth. Meanwhile they'll trickle out 100-300 tps access to the rest of subscription users to drain them of their cash and keep the beast fed with fresh training data.
We really need more people pursuing hardware LLMs from an open-weights perspective and consumer perspective, and we probably need laws around free and open access to AI, and some anti-AI monopoly laws, because otherwise society as a collective has no chance of keeping up with big tech and our rights and freedoms will be rapidly eroded.
How can any individual company building a product, with access to 100-300 TPS behind-frontier security-gated, censored and capability gated models expect to compete with a company like Anthropic or OpenAI with frontier, unrestricted, unlocked models that can produce 100-1000x the output? 3-5 of their employees working to clone your 500 staff business will likely be easy pickings for them.
> seems like soooo much efficiency waiting to be unlocked at the chip level
Well if you are exclusively using GPUs that are general purpose, of course you leave so much efficiency on the table. That’s why Google started making TPUs more than a decade ago. I remember that kerfuffle when Google fired Timnit Gebru when Gebru’s paper used GPUs to calculate the environment impact of LLMs while ignoring the efficiency of TPUs; this basically made Jeff Dean very angry due to that wide efficiency gap.
It kind of was. I really hate gaslighting, but GP is not inaccurate. Google claimed it did not meet their bar for publication because it ignored recent research on how to reduce the environmental and bias-related risks of LLMs. On the other hand, a large org is unlikely to subsidize high-profile research that makes it look bad. And Gebru was critical of Google’s internal culture and diversity efforts…
It'd be cool to see more of this type of thing, but I have to imagine the ability for it to be updated to a brand-new model as new models come out is limited. If that is the case, it's going to be an extremely hard sell.
It really depends on the pricepoint at which they can get a board. If they can do a ~32B model for 1k$ and a size of an external HDD, I'd buy one now, even knowing that it won't be upgradeable / the model remains fixed. The speeds they've shown are a quality of its own, and there's plenty you can do with such a model and faster than instant responses.
Maybe in 10 years when the tech matures, but IMO now seems a bit too early to have a tech like this. It is like intelligence without evolution or progress.. yes it can be used in some niche markets, but difficult to be generic.
There are plenty of applications that would be useful right now. Specialized models for tool use, like fine tuning for command line tools that are already well established and don’t change often. I’m sure there are many areas where the training data is essential crystallized and unlikely to change. Think of them more like delegated agents or coprocessors that another model could route to so instead of routing to a quantized or lesser model it could use a full fidelity model that is faster, almost instantaneous.
You don't need SOTA models for all tasks, and being able to do more routine tasks at something like 10% of the cost and 70x speed unlocks LLM use for things that are just unthinkable now (bulk classification tasks, real time speech interaction, etc)
Yes, but with current architectures world knowledge is baked into the weights. We might stop figuring out how to make models better, but the world keeps changing, science is going to keep making progress at understanding the world, etc. This creates a significant minimum rate of change and I'm pretty skeptical that it's worth baking weights into silicon as a result.
I think it would just be an opportunity to sell another chip a few years down the line. If the utility curve flattens out on the performance of models I can see a future where you are buying an up to date chip every few years to upgrade to the latest and greatest, while providing up to date context as part of the user input. Like if I have a programming task and I supply a copy of up-to-date documentation alongside my input, I would think that I could still get good output out of a dated model.
I mean it just depends on the price of the chip. You might just replace the chip like you would any other component. Like a video game cartridge or something.
What makes you think that? The rate of change seems to have been increasing and there is so many chip and model best in different directions at the moment.
I think the model they chose is out of date and hard to sell, but there are plenty of use cases where today's dumb small models are fine. A Qwen 3.5/3.6 or Gemma 3 model on silicon at those speeds would be genuinely world changing even if it's only 1-3B params. Such a model at those speeds will remain extremely useful even over a 5-6 year timespan, I think.
If you consider the places you could deploy it -- with no network access, and at those high speeds... very useful .. for adding vague "common sense" fuzzy thinking to all kinds of applications that right now piss consumers off with poor UX. Esp if the model can do voice-to-text and text-to-speech well (some of the smaller models can)
> I think the model they chose is out of date and hard to sell
I understood it as a proof-of-concept, not a for-mass-production single blueprint - i.e.: "if you need your NN in a CIM form on ASIC, we can do it".
Their next proof-of-concept was said to be meant to be about size: "we showed you we can do it with 8b, now we are working to show you we can do 24b or 32b". Then, "and we plan to go bigger and faster".
> Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM. It is expected in our labs this spring and will be integrated into our inference service shortly thereafter. // Following this, a frontier LLM will be fabricated using our second-generation silicon platform (HC2). HC2 offers considerably higher density and even faster execution. Deployment is planned for winter (19 Feb 2006)
I wouldn't be surprised if "fast, cheap, dumb" end us being the market for LLMs.
The state-of-the-art models aren't at "can fully replace knowledge worker" levels yet and I doubt they'll get there any time soon, so charging $2000 / month for access isn't going to happen. Right now everyone and their dog is being handed subsidized credits to play with AI, but the actual outcome is rarely good enough to be worth the money they'd need to charge for it. It might very well take another order of magnitude or two to get LLMs to be truly good (if it is even possible at all), and considering how much money is already being pumped into it I just don't see that happening.
On the other hand, the dumb models are more than adequate for simple noncritical tasks, like directing a user to the appropriate FAQ entry, or playing phone decision tree. There's a lot of money in making chatbot assistants actually useful, or in augmenting website search. Turning it into a glorified "language-to-API-call" translator doesn't take a lot of smarts, but as long as it's cheap you can make a killing in volume.
> On the other hand, the dumb models are more than adequate for simple noncritical tasks, like directing a user to the appropriate FAQ entry
This is a lane I’ve been experimenting in —- seeing what I can get out of models that work in 16GB VRAM for simple tasks (screen scraping, decision tree navigation, natural language queries). It’s interesting for sure (certainly reveals non-deterministic limits) and promising for low criticality review-opportunity tasks, but I also feel like I need better sources/community for understanding and reflection. Preferably those that aren’t hype channels. Any pointers?
It seems technically interesting, but they seem very sparse on details. I don't know if I like the idea of a single unchanging model forever on a chip. How much more expensive would the silicon be if they used rewritable ROM for the weights? Such an arrangement would permit fine-tunes of the model it was designed for, which might minimize concerns about the model becoming outdated.
In a chatbot, 17k tok/s is a neat but nearly useless showcase. In a coding agent it is a meaningful improvement. In robotics, it could be an absolute revolution.
8B models aren't useful in general, but for specific use cases they can provide an enourmous amount of intelligence - nVidia's Tesla/Waymo competitor is a 7B LLM with a 2B diffusion model, and running that at those speeds could be an order of magnitude cheaper than existing solutions.
17K tok/s is approaching realtime motor cortex needs for a robot with ~12 actuators (bipedal humanoid) and an IMU. I don't know how many parameters a motor cortex would need but 8B feels like it is within 2 orders of magnitude.
this is an LLM, not a motor cortex. it will output commands as text (json, ...), so comparing size is not very meaningful, especially considering neurons are highly complex and likely requires thousands of artificial simple neurons (weight+bias)
Bumping the speed of these things would be more than meaningful. It would be a massive game changer.
I assert like 80% of this “multi agent parallel workflow” business is simply a workaround to models being soooooo slow. Like as the dude driving these things… you kick it off and twiddle your thumbs waiting minutes to hours sometimes for all the inference and token generator to finish. So you dispatch multiple workstreams in parallel to be more efficient.
I assert that if the model was even 10x faster we’d be using these things radically different. You’d be doing things that are currently time prohibitive. At 100x, holy shit will software dev get crazy. You’d be kicking off hundreds of parallel workers attacking a problem from every angle and stuff. Who even knows!!!
And the thing is, 10x will absolutely come and probably even 100x. And it will be sold like a video game cartridge or something depending on how the actual model gets “baked” into the hardware. No remote inference at all.
Could you give me some example how in robotics it can be an absolute revolution?
My understanding is that robotics doesn't really rely much on LLM's in the first place but rather other things.
Is the thing that you are suggesting that it would ingest all real time data and then reason through it at an incredibly fast speed and then act on it and re-iterate? I might imagine some problems with this though I am not a robotics engineer and perhaps someone who deeply understands this topic can give more information.
LLM are very good at looking at images and reasoning about them. much more than just object recognition/segmentation, they can explain the physics in the image, the intents, plan actions, ...
That's because of posttraining optimizing for benchmarks that test that.
They tend to collapse into nonsense and hallucinations pretty quickly if you move slightly out of the envelope of the current visual reasoning benchmaxxing.
Disclaimer: I'm a robotics noob, but I've been working on robotics for a few months now.
I'd say virtually all robots you've seen in the real world today rely on classical approaches - you build a rudimentary map, then use classical algorithms to find paths/do area coverage. The robots do no reason or understand what they're looking for, they're more like in-game units. At most there's some bounded, lightweight image classification going on.
LLMs can understand and reason about the world natively. nVidia has a Tesla FSD/Waymo competitor which simply their 7B reasoning LLM but instead of outputting tokens directly, its outputs are fed to a 2B diffusion model that outputs 1.6 second long trajectory for the car, and this is enough for an L2 system. But to make this work, they need the model to run at 10Hz, so they use super high-end hardware to do it (Jetson Thor) and the car is still "blind" for 100ms at a time (they have a parallel classical safety system).
With on-chip LLMs you could run this loop at like 100Hz on a chip that costs a few hundred bucks, rather than 10Hz on a board that costs several thousand.
I think hardware like this is the future for LLM-providers once we reach a point where the models aren't advancing much any more. You could argue we're close now.
The hyperscalers like AWS will made great use of these to serve up models that will be relevant for several years. But right now, we're still seeing significant bumps in model quality every couple of months - especially with open-weight models like Deepseek/Kimi/GLM.
Until that point, though, I don't see how this is ever going to be cost effective vs general purpose hardware.
I also think we'll see miniature versions of this baked into mobile hardware for super fast and efficient on-device LLMs.
1. If LLMs keep improving, burning models onto silicon becomes obsolete too fast and is not worth doing. Outcome: We keep getting better LLMs.
2. If LLM improvements slow down, they will be burned onto silicon. Outcome: We get faster, cheaper and energy-efficient LLMs.
Either way sounds great to me. It will certainly be a mix so we can even get both.
Pretty huge move. Google and their TPUs are looking infinitely more prescient as I think they are on their 7th generation, along with the offshoots it inspired like the LPU and even others, perhaps like Cerebras and their Wafer Scale Engine.
However, based off first impressions, it seems like this is meant for inference side, and not training, which is also an interesting choice.
Training is pretty much a 1x cost, and efficiency there is already on the way down with architectural improvements. Inference though is an ongoing cost which over time takes orders of magnitude more resources, so focusing on making that far more efficient means way greater gains over time.
What makes you think this? With wider adoption the ratio shall shift in favor of inference. And API price is becoming more important than SOTA capability.
Cerebras's Codex Spark 5.3 has been a huge flop. Small context window and old model. But hopefully they can improve so that we can benefit from 1000 tokens/second with GPT 5.5.
> early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art
We're starting to see what really matters here, and though this is hand wavy the TPU makes similar claims.
I think googles memo about having no moat still stands (see: https://newsletter.semianalysis.com/p/google-we-have-no-moat... if you are unaware). It kind of makes sense that all of this is looking more like 60's to 90's IBM, DEC, Cray, Sun and the hardware race that happened then. History doesn't repeat but it often rhymes and I suspect that these efforts will follow the same trajectory.
To be clear, that is not "Google's memo". It's a memo by a guy who happened to work at Google. There is a diversity of opinions at a company that employs 180,000 people.
With the pace of AI, and with AI helping to pave the way for faster/better AI, I keep wondering if hardware like this will become obsolete well before it has a meaningful ROI. Huge AI models can be run with less resources already through quantization and offloading, but that's just the beginning.
One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop. Think that's crazy? Look at the size of the first hard drives. The IBM 350 was a disk with 50 platters, 24 inches in diameter, that held 3.5Mb, and was leased for today's equivalent of $35K.
Compare that to a multi-terabyte ssd. Now apply that improvement to how an LLM is architected and run now. With AI assisting, it won't be long before a leap occurs and these data centers with all their current ultra-cutting edge Nvidia cards are nearly obsolete overnight.
They do. Mythos kicked ass while it lasted. And what we know of the scaling law curves promises us even more gains in the future.
"The future" being "whenever training and inference at increased scale becomes economical". Which is probably bounded by new generations of hardware, but might also be pushed forward by algorithmic advances.
The likes of Mythos show that the scaling laws are real, and you can x5/x2 the total/active params and get meaningful gains. If "inference per param" gets cheaper? Up the params and get more intelligence for the same price.
Very true, and all I am basing my comment on is the improvement in speed AI has demonstrated when applied to software development, and inferring it might enable a similar 10X or 100X improvement in both hardware architecture as well LLM structure and/or interface methods. If that speed improvement applies to performance of AI, that could mean the 70 years it took for people to improve storage technology might be able to be compressed to achieve a step change in AI performance in a drastically shorter timeframe.
I think Jevons Paradox and scaling laws will make this not the case. If bigger models are always better (which seems they are), then will always need high-end hardware.
> One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop.
I think there will be specialized hardware (beside GPUs) that would be custom made for LLMs. Yes TPUs exist, but mainly for datacenter. GPUs exist, but they are adapted from mainly graphic application. Once all the demand from data center dries up, innovation will kick in.
True but as someone else pointed out; at that time we'd be interested in running 200T parameter model rather than 200B. Why, you might ask? Law of human laziness - a human will become as lazy as the technology allows it to. With the 200T or 20,000 T model - I'd be heavily incentivized to ask it to make the bread for me that I enjoy making now or create a movie for me (featuring myself) which will maximize the dopamine production in my brain.
Con or not it is an obvious thing they have to do. Might as well promise.
IIRC their biggest cost they're "hiding" in their financials by doing creative accounting is inference (putting it into marketing and whatnot, in the billions)... if they can't hide it in their S-1 then they have to rationalize it, either by a) increasing the prices (not gonna happen, with token based billing orgs are already watching their codex spends) or b) lowering the inference costs. You can lower that by "soft optimizing" (dumbing down) your models but then you have the other players breathing down your neck (see quick rise of Claude), or actually optimizing, in software and in hardware. We're like 5 years into the rise of LLMs, there's not THAT much left on the table unless you write to the metal you specifically designed for your models (and I'm pretty sure the lack of "nvidia tax" would help with covering most of the r&d costs of a custom solution, at least in the long term).
50% cheaper inference without losses in fidelity would unquestionably be a massive win for OpenAI.
So far, the accelerator is showing cost savings of roughly 50% compared with typical AI graphics processing units, Broadcom Chief Executive Officer Hock Tan said in an interview. - [0]
50% cost saving. The picture changes so quickly, there are still a lot of low hanging fruits, that I find any discussion about whether a vendor has moats, or if they can recoup investment, is moot and futile.
Microsoft, Google, and Amazon also do this, but they also have the hyperscaler datacenter infrastructure to host the chips. Designing and taping out the chip is one thing, packaging, cooling, deploying, powering, and managing the fleet is another stack entirely. Wonder where that will come from?
I had Opus 4.5 design an LLM inference engine in verilog, including firmware and automated verification a while ago: https://github.com/cpldcpu/smollm.c
It's of course far from optical. But lowering the implementation through the abstraction levels turned out to be extremely powerful.
Can you suggest some tutorials for Verilog and FPGAs in general?
I have a spare Tang Nano 9k but I don't feel confident about blindly asking Claude to vibecode me a solution and still would like to have at-least a basic level of understanding.
hm.. has been quite a while for me. The good thing about the Tang Nano is that it is supported by the Yosys open source toolchain. There are quite a few resources on the web when you search for the combination.
So I’ve been wondering about “one or two levels back” chip design. If I understand it, 28nm chips (pre EUV) is just about suitable to run (not train just inference) frontier models.
And so if I was a mid-level State would it be worth while to take my nascent chip industry and push it out to build a 28nm foundry and supporting eco-system.
The models will come but the real challenge of the future is having enough compute power for every one and every use. Even if LLMs don’t become AGI they will still be incredible tools - and as OpenAI seems to spend 8000 for each 200 monthly subscription building one’s own data centres seems sensible
You are underestimating how difficult it is even for a large nation state to attract the kind of talent and investment it would take to set up a chip industry. It is out of reach for anyone outside of the 3-5 largest national economies and a few big American/Chinese multinational corporations.
I am not sure how much of the work is done by OpenAI, or whether it is basically a Broadcom chip specifically built for OpenAI models. It is a necessary step, but building a high-performance chip is not easy. Look at companies like Groq, Amazon, and Google.
Both Google and Amazon also codesign heavily with Broadcomm (Amazon also with Marvell and Alchip)
Broadcomm does stuff like physical design, provides IP blocks, managing manufacturing process with TSMC, packaging and testing. Google and Amazon work with system architecture, performance targets, and requirements but Broadcomm as consultant.
Definitely has that smell... At the same time though, they NEED inference cost to drop substantially, and even better for them if it only happens for their models on their hardware.
I assume they're doing everything they can to make that happen model-side, but coming at it from the other end makes sense too if they can swing it.
Memory bandwidth is the bottleneck in the Spark. If you replace the SoC with an optimized ASIC but keep the same 256-bit LPDDR5 the performance will be the same. You can increase performance by using wider memory but that's also more expensive.
Personally, I doubt it. Apple hamstrung themselves with unified SOC memory, there are cheap dGPUs that smoke the M5's prefill speeds and even have faster decode too. Apple is running up against the limitations of putting a mobile integrated chipset up against the desktop form factor. An SOC stops looking like a smart decision at that scale.
The software side is still pretty sketchy, too. Apple's ecosystem is fractured between NPU, MPS and Accelerate BLAS, with libraries like MLX and CoreML built precariously overtop. Apple has to commit to a full rearchitecture of their GPU to challenge Nvidia, which fractures that ecosystem even further.
Very interested to know the distribution of effort between the two companies. Is this truly a brainchild of OpenAI engineers or did they pay to white label and use a new Broadcom chip?
This is just an uncut wafer - I don't think it's intended to be a wafer-scale chip.
Cerebras etch memory onto the wafer alongside the processing elements, but AFAIK OpenAI are going to be using HBM memory and a conventional chiplet design.
cheap token is more important now than ever. Chinese open weight model is getting pretty good. the real cost of AI adaption will come down to who (China or US) can provide cheap token for consumers and companies. Microsoft consider DeepSeek for their cowork is an example and now OpenAI with its own AI inference chip.
> May we scale smoothly, exponentially and uneventfully through A[SI]
That sentence sounds weird to me. I can't really put my finger on why, maybe the combination of adverbs, or just the fact of writing the desire of scaling as a company so directly. It feels (to me) like openly claiming their selfish goals. Or maybe I am just misinterpreting and they are referring to the whole humanity as "We" (but knowing Broadcom and in a lesser extent OpenAI doings, I am not convinced).
CA Technologies was much worse than Broadcom in its heyday.
Three of their top execs - CEO, CFO, and head of sales - went to federal prison on securities fraud, conspiracy, and other charges. The CEO, Sanjay Kumar, who was at least partly the fall guy for co-founder Charles Wang, served 10 years.
Being acquired by Broadcom could only have been an upgrade, as strange as that may sound.
I'm assuming they used LLMs to (help humans) do custom circuit design. Even pre LLM there were various computer optimizations that didn't require humans like genetic algorithms. It'd be cool to see a paper on how they did it.
I mean I'd love to be able to buy something like the 17k tps taalas chip as a pcie or m.2.
Imagine when we can roar along at that speed, low power. Can just have the model reason for a while about anything and everything. It reminds me of the "race to idle" for mcus etc.
The current taalas chip is for a 3.1B param model. I’m hope so much that they can get that up to the 30B range. Just imagine Gemma 4 or Qwen 3.6 at 17k tps.
It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually. And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases.
I wonder if it's being worked on in secret, if there's something about it that makes it infeasible, or if companies are really too nervous to lock in one model like that because the next one down the line could be a huge improvement. Re. infeasability, I have heard that the Taalas demonstration chip ran Llama 3.1 8B (a pretty horrible model) and that even that took a massive amount of transistors / die area. So it might just be the case that the good models are too big to fit on silicon?
The new chip sounds like it's vustom made to accelerate a few specific models they really need to run fast. The advantage is it's truly and ASIC, not a xPU. There are several new startups targeting EDA tooling automation, Chip Agents is the biggest one I can think of but their are smaller players too, Silimate is one I recall. These companies are focusing on building fast AI powered tools to speed up the tape out cycle.
They don't have true competition, what they lose out on is market share with hyperscalers, since OpenAI would have no plans to share inference hardware with any other company right now. Plus, I don't know how does NVIDIA's investment equation pans out long terms given OpenAI will be investing in more purpose built inference stack for the future.
Dang, I just checked and CBRS is in free-fall since the IPO.
Sucks, I think they're a cool company.
OTOH, I was the only person back then pushing hard during my time at KAUST (back in 2019) to buy one of their systems when they were nobody, eventually resulting in a partnership between the two.
Then I joined their online discourse, very few users, I was semi-active there but they didn't care much.
Then I came to Toronto and heard they were opening an office here, tried to get noticed several times but got mostly ignored. I asked about upcoming events several times, anything to get involved, "yeah man, maybe one day". Then they made an event during Toronto Tech Week and didn't even tell me ... idk.
I don't get schadenfreude as I still think they're a cool company.
My point is they put all the eggs in one basket (AI inference) and neglected everything else. They seem to be on shaky ground now ... sad.
They tested on spark model, i bet it's a mix of that with focus on inference speed. Whatever it is, hopefully it shows up with current models as faster. Token/s is as big thing as anything else, and thats where they can really gain some edge over the competition.
No information on how significant the reduction in energy per token is. No information on amortized price per request. Increasingly its clear OpenAI must demonstrate order of magnitude reductions in cost to not die, this is investor story time without that information.
Although this seems to be for inference itself only and not training but inference is a recurring cost and training is a one time cost and so to me, even if Nvidia still gets moat on training, I don't think that it could ever justify its massive evaluations because for example, some chinese models are actually trained on Non-Nvidia models. The moat in that is incredibly thin.
(at the moment), I think that if I were Nvidia, I would be a bit terrified and I imagine the stock to not be doing super great as I can just imagine everyone online might start talking about it for better or for worse.
I am a bit impressed by OpenAI but is this what can be classified as a plan for OAI to salvage itself and all the commitments it has made nearing a 1.4 Trillion dollars from my memory and this article[0] is from 2025
But could OpenAI simply walk out of its commitments when necessary (for example to Nvidia) if this chip works out or what exactly might happen in the future as these commitments are asked to be paid for, its still smart for OAI to diversify with this chip and to have more deeper ways of revenue than just being a simple middleman but I imagine that Nvidia and others have also invested in OpenAI and they must not be happy with this change.
The thing with AI deals are that they have become so complicated that it is hard for me to find the first order impact of things, let alone second or third order impacts and financial accountability seems to be impacted quite heavily because of all of it and there is some sense that it is done so intentionally.
im very excited that frontier models now have so much money and revenue they are releasing their own chips that could change the relationships and bottom line
A lot of companies that serve a single family of LLMs seem to prefer nvidia though. Why is that?
It's not just good drivers, which is what moats them for games and ML. It's a multi-decade work of making chips that are nice to program for and software infrastructure around them.
Apple and Google have excelent chips, yet they needed to invest a lot in long-tail software projects to make those chips do actual premium work. Still not state of the art for serving LLMs (although Google is strong in that, mostly because it piggybacked on previous chip-related software work for phones and so on).
What will people use to write for Jalapeño matters.
Nvidia has multi-decade heritage. Apple spent almost a decade in MLX. Snapdragon failed partly here. OpenAI announced nothing regarding to that, so this big moat that multiple companies have (nvidia the most prominent) is nil for them.
I call BS. It’s probably a white label around existing Broadcom IP, impossible to go from zero to this kind of chip in nine months. I doubt OpenAI had any significant contribution.
9 months to production is completely impossible anyway.
9 months from design to early samples is probably impossible given than TSMC takes 3 months after tape out to produce them. Then it’s up to the customer to qualify and revise for production. TSMC doesn’t do that.
One thing I don't like about California based companies is how cringe the names always are.
"Jalapeño" is such a bad name, having an "ñ" already makes it difficult and annoying to deal with in so many little ways. Good luck with that.
But also, theres the sort of "yes lets use Mexican related things because we're California" thought that I just really hate. I don't know, its like corporate Memphis to me. You see a product like this, you know it's an uppity califonia based firm that came up with it.
No worse, I suppose, than, the obsession with Lord of the Rings that the authoritarian surveillance companies have. Palantir, Anduril. Then we have the not defense/surveillance ones: Mithril, Valar, Narya, Erebor
None, probably. Just saying Jalapeño is no worse than any other non-descriptive company name. Although at least Palantir and Anduril are aptly named for what they do. The VC firms less so.
You are correct. I don't know why I thought there were 5 Rs in strawberry, and now I look properly I can count them correctly, there are indeed 6 Rs in strawberry.
I am sorry for initially giving an incorrect answer.
Don't worry, in Europe it's the same, but for insurances/lawyer stuff. Tons of companies have names based on Latin words such as Civitas/Insalus/Legalia/Legalitas or whatever which looks tacky/rancid/old fashioned kilometers away.
> Developed from design to production in nine months, accelerated by OpenAI’s models
> the use of OpenAI models to accelerate parts of the design and optimization process.
I wish there was more about this. As is I kind of have to assume that this is just meaningless marketing, like saying development was accelerated by Microsoft Office or their 5k LG Ultrafine 40-inch monitors.
Like, if this was as big a deal as it kind of vaguely implies, they would be making a bigger deal of it, right?
Chip CEO here. It really depends on what "design" or "production" means. Does "design" mean that the design was complete? Does "production" mean the beginning of production, i.e. tapeout? If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip. If measuring from concept (no RTL at all, block diagram of architecture) to tapeout, this is an amazing timeline. The truth is probably somewhere in between. A more concrete statement would use actual technical milestones and gates.
Not a chip CEO, but I read this article and thought that they're working on some kind of application specific chip only for serving models. Similar to how an FPGA can optimize certain tasks.
Given constant weights / biases of a Transformer / DNN you could use pipelining to feed forward calculations through the array one layer at a time. For DNN's with thousands of layers you might see 1:1 speed up per layer channel.
I doubt they would undergo this process for marginal gains.
Yes, my statement was not about the quality or performance of the chip -- simply the tapeout timeline that was stated, by itself.
i don't understand what the second paragraph is saying.
In very crude terms, AFAICT, if you have a bunch of matrix multiplications, but one of matrices (the one with model weights) doesn't change, you can seriously speed up the computation. One thing is that you don't need to re-fetch the elements of the constant matrix, you can keep it near the ALUs. Then you maybe can detect and ignore sparse / empty blocks by marking them once.
IDK how the custom hardware exploits this; would love to hear any ideas!
> IDK how the custom hardware exploits this; would love to hear any ideas!
You might like this article [1], titled "FPGA-based CNN Acceleration using Pattern-Aware Pruning". More context and details can be found in the PhD thesis of Léo Pradels [2].
[1]: https://inria.hal.science/hal-04689673/document
[2]: https://theses.hal.science/tel-05021575v1/file/PRADELS_Leo.p...
Random thought. Once models stabilise, could you possibly hardcode the model in gates? Or are they too large for a single chip?
https://www.anuragk.com/blog/posts/Taalas.html
Basically getting around the branch predictor problem with generalized compute architectures https://en.wikipedia.org/wiki/Branch_predictor
>If measuring from RTL-freeze to tapeout, this is a fairly typical (even somewhat unimpressive) timeline (accounting for some unexpected issues) for a large, complex 3nm chip.
Even for a company’s first design?
I don't think you get the newcomer novelty buff when your val approaches 13 digits.
Big companies are lumbering behemoth, crude assemblages of barely cobbled-together incentives and principal agent problems in a trenchcoat. Getting them to change direction, or worse, try something new at scale, is a massive undertaking
Nah, you just need to get the CEO behind it. Most coordination issues get solved when the CEO is breathing down your neck to get something done. Trouble is that they don't do this enough.
Eh, zero guarantees on that one.
The Fire Phone was Jeff Bezos' personal baby, and we know how that went. Then there was the Apple G4 Cube with Steve Jobs, the Model X' Falcon Wing doors and Elon, and lets not even talk about the Metaverse and Zuck.
> The Fire Phone was Jeff Bezos' personal baby, and we know how that went.
I'd rather guess that Jeff Bezos' opinion on what makes a good phone is/was different on the opinion of many potential buyers.
Actually, you've provided examples that prove the point. None of those were especially good (though everyone wanted the G4 Cube), and yet they made it to market anyway. Why?
Because the CEO was behind it, breathing down their necks.
The typical way a chip effort in a non-chip company works is that the "design" is the RTL (e.g. SystemVerilog that defines the behavior of the chip) and then this is handed off to a third-party "design house" (such as Broadcom) that turns that code into a real image of a chip, which is called a GDS (basically you can think of this as a very big layer by layer photoshop file) that can actually be sent to a fab. This is called "backend design", in contrast to the "frontend design" (the RTL itself).
As another commenter said, Broadcom is very experienced with backend design (as well as the supply chain management, testing, etc. that comes after the chip is taped out) and so this can't be regarded as a "first chip". Richard Ho (the head of hardware at OpenAI) is also extremely experienced and used to be the head of the Google TPU effort -- where he actually worked with Broadcom in a similar tapeout already. So yes, this is not a "first design"!
I wonder if broadcomm borrowed IP between the Google tpu and this design. How would you ever know it didn't happen?
More likely that the AI training set contained the IP of others, and we all know how that turns out.
This isn't Broadcom's first design.
Yeah, "first chip" here likely means they contracted Broadcom (or a firm with similar experience) to do all the heavy lifting. Building out your own in-house teams for this sort of thing is a decade-long project - just look how much inside Apple's early chips was licensed ARM / PowerVR cores
Apple didn't have the talent in-house until they bought Intrincity who worked with Samsung on Apple's earlier Arm chips as well. https://en.wikipedia.org/wiki/Intrinsity
I think the folks at PA Semi had some chops too.
PA Semi group did the logic designs. I think they're talking about physical design though.
The way I heard it PA Semi was the singular driving force that led to Apple Silicon, but I'm not any kind of insider that's just the chatter I heard.
Whoever it was, whooo, that's hot shit. I remember an M1 MacBook Air just cleaning the clock of an Intel MacBook Pro and thinking "x86_64 has real competition again".
Great silicon. I'm over it with not having root on my own machine, so I've left the ecosystem, but it's really nice hardware, can't dispute that.
The hardware description languages (HDL) used in chip development are like programming languages. The existing models understand them and can do a lot with them. You don’t need to have separate, specialty models designed for this work to use LLMs in chip design workflows.
Design verification also involves a lot of traditional programming which benefits from LLMs.
So it’s not meaningless at all. You could download some of the open source chip design software today and the LLMs could even help you get started on your own tiny chip if you are so interested.
I tried making a button using Claude entirely (including the 3D printed enclosure) and it effed up pretty hard with the traces and the header spacing. The project was a big red arcade button that plays the "ah-my-groin.mp3" when pushed (from Simpsons). It did cool work on saving battery life, and the 3d enclosure was awesome, but yeah, I'm convinced I'd have to do another version or two of the custom chip until it came back right. I used a Blender MCP for the 3d modeling. I used a KiCAD MCP server for the chip design/validation.
I think we're not there yet. I've been meaning to look at this flux.ai to see if it has the prompts/workflow worked out better than what I was able to cobble together in a few hours. Maybe Alteryx's MCP server would have been better. I'll try that this weekend for another board I've got.
> I tried making a button using Claude entirely (including the 3D printed enclosure) and it effed up pretty hard with the traces and the header spacing.
PCB design and 3D CAD design are different topics.
Hardware Description Languages are closer to programming languages than CAD. Look at some Verilog to get an idea - https://en.wikipedia.org/wiki/Verilog
Right. KiCAD for PCB design. Blender for 3D CAD. Oh, are you saying I should have used something other than the KiCAD MCP server for better results?
Designing circuit board and 3D models (even using something like OpenSCAD) is a very spatial process today. You are dealing with coordinates one way or another.
This is very unlike how FPGA and (I assume) ASIC is done. That is more like a traditional programming language but everything happens all at once (no sequence of statements outside tests, if you need that you have to write a state machine yourself). You define logic expressions between signal, add stateful latches, etc. But you never specify the physical layout.
Instead you feed your description to a tool that acts a constraint solver/optimiser that computes the layout for you (this is for FPGAs called synthesising IIRC, it is akin to a compiler). Typically quite slow, even for small circuts like we did at university it took minutes, and for large circuits it might easily days.
Now, this raises the question, what if you design a PCB net list using AI, but then use traditional autorouting and layout? I believe that can also be done, but I have no experience designing PCBs, so I don't know how well it works.
For digital chip design, research Verilog and/or SystemVerilog, and for tools, check out verilator and the OSS cad suite: https://github.com/YosysHQ/oss-cad-suite-build
VHDL is not a language for spatial design. Its more akin to a programming language with circuit semantics.
They’re saying that VHDL is an entirely different concept than physical modeling.
You're comparing apples and oranges.
Meta: can we not downvote people who are clarifying what they're saying and asking questions, even if they're wrong about something, if the content isn't otherwise objectionable?
I didn’t downvote, but the OP is either a troll or someone who doesn’t want to notice he doesn’t know what he’s talking about. Either way we want less of that on HN.
I'll acknowledge that I don't know what I'm talking about. I really appreciated the clarity! Surely you find value in knowing that creating your own custom chips is almost doable by someone who doesn't know what they're talking about! (also, I am a troll, but in this case, just clueless)
Maybe the confusion stems from the word "chip". Creating a chip usually means designing and producing a microcontroller or a processor, not a printed circuit board that you populate with existing chips.
Ohhhhhh! Yes, that's exactly the problem. It all makes sense now. I was just piecing together an existing microcontroller and a mp3 module by printing a custom circuit board.
One (kicad) make the board, the other (blender) make the casing for it. Both are “hardware” but is electronics and the other is mechanical. Electronic one AI can do a good job, I can’t wait for it to fully built the whole circuit for you based on your specs.
PCB layout is an art, and doesn't seem to map well to LLMs (I tried for shits and giggles recently). Claude in general, kind of like code, does a lot of redundant belt and suspenders stuff in the schematics it generates (if it can generate them at all). It's one of those things that's really not there yet outside of the simplest designs.
DeepPCB has an AI autorouter [1] that uses reinforcment learning and works really well. Recently they also released an AI agent that analyzes your board, proposes plans and can route your board for you [2]. They have a KiCad plugin [3] and you can try it for free.
[1] https://deeppcb.ai/reinforcement-learning-pcb-routing-explai... [2] https://deeppcb.ai/cooper/ [3] https://deeppcb.ai/deeppcb-kicad-plugin-ai-pcb-routing/
Disclaimer: I work at InstaDeep, the company behind DeepPCB, but I don't work on this product.
The question isn’t whether or not they employed a particular tool, the question is how big of an impact did it have.
Most HDL code is locked up behind corporate firewalls and not available as training data. While LLMs can handle it to an extent there's a lot of room for improvement. I'll bet that OpenAI and their competitors are racing to license this IP from major hardware vendors in order to compete in the chip design vertical.
Does it work better when using compiler based ecosystem (e.g. https://github.com/llvm/circt)
There is quite a lot of Verilog/SystemVerilog and VHDL code in the wild. And hardware description language code is very simple and straightforward relative to programming code.
And the two things that take up VAST amounts of time in ASIC design are testbenches and timing closure.
A LOT of hardware design is testbenches to verify things. AI is REALLY GOOD at generating things like testbenches. And nobody really cares if the quality of your testbench code sucks as long as it validates what it claims to.
I don't know how good AI is at timing closure, but I wouldn't necessarily be surprised if it is pretty good at it up to the physical point. That's lots of textual output which you can put a constraint on.
Everything involving physical design, though, tends to be a disaster waiting to happen if you let AI loose on it.
This reminds me of the dude on youtube building a chip fab in his shed.
One day we can design our own pcb with chips, hardware and other io. Companies will accept these as files and you can collect your pcb the same day. I think in China they are doing this already
> The existing models understand them and can do a lot with them.
In my experience they are not especially good at SystemVerilog. There's a lot of knowledge about it that is locked behind paywalls and it's very niche.
My guess is the "from scratch" here is quite the exaggeration. Otherwise why did they need Broadcom?
Doesn’t Broadcom bring a lot more to bear here than just Verilog? Including relationships with the actual fabricators.
Not having a free toolchain that can actually handle the real language has probably been pretty bad on the downstream public knowledgebase. Hopefully Verilator can eventually close that hole, and there can be more high-quality designs and codebases incorporated into future models. Claude is at least good enough to write SV that triggered a compiler crash or two. :)
Broadcom already has a ton of IP for AI SoCs. I'm guessing the hard parts of this inference chip was already designed by Broadcom and OpenAI simply told Broadcom what it wanted. It's likely very similar to Google's TPU.
What is substantial here? Vera Rubin is shipping in volume later this year and it is expected to be 10x more power efficient for inference than Blackwell.[0] Even if they're already taped out the chip, getting bugs fixed, getting chips manufactured, getting HBM allocation, getting a rack design, hooking them up together, putting them in a data center will likely take at least another 12 months or likely more. By the time this chip is in data centers in volume, they're likely competing against Vera Rubin Ultra or maybe even Feynman.Personally, I don't think OpenAI should have invested in this project. It's too early for them. They should have focused on models like Anthropic and win there. When they're profitable, they can take on these projects.
The risk here is very high for OpenAI because AI has a hard cap in energy. If you have a gigawatt, you should only install the best chips. If Nvidia's chips are better, then this is a wasted project and likely wasted billions.
[0]https://developer.nvidia.com/blog/scaling-token-factory-reve...
Why do you assume Broadcom has a ton of IP for AI SoCs but hasn't done any of the other work around data center scale deployments?
They have. That's why OpenAI was able to get a working demo in 9 months. But going from a small scale system to a full fledged data center deployment is likely much harder.
I don't know how much of the things outside of the chip Broadcom has vs Google's proprietary tech that is not shared with Broadcom.
Nvidia's Vera Rubin has 6 unique chips working together in a single rack.[0]
[0]https://developer-blogs.nvidia.com/wp-content/uploads/2026/0...
I thought most of the Google tpu magic is on wiring up these chips into supercomputer like clusters with specialized interconnects and whatnot. The chips themselves are less interesting in isolation.
I’m just happy to see diversity here; sometimes I feel like Nvidia is going to eat the world, with buying other fabs and branching out - or up, I guess - from chips and racks to models, frameworks, and end user stuff.
Right. There are two possible meanings and shades in-between:
1) OpenAI genuinely have AI technologies that can improve chip design (bold, unlikely claim, needs evidence)
2) OpenAI designed test/verification models and kernels that could be run on the simulated hardware to test its performance
As you and others have said, it's hard to trust when they are happy to write something that could easily only mean the latter but sounds like the former.
3) The engineers working on the chip used ChatGPT from time to time.
at the hardware company I work at, people are now using claude code and developing skills for it to do basic stuff like triage or do initial debug on failing tests, search for potential causes in RTL, generate skeleton documentation for designs etc
But isn't this rather the ordinary product of an LLM, now?
Is it worth the claim that they are making in a press release?
> Is it worth the claim that they are making in a press release?
Definitely, yes, because being vague about it like they have been lets investors fill-in-the-blanks with whatever they want it to mean.
I'd be shocked if it was anything more than this.
Browsing openai's job postings in the past few months is enough to contirm that it's more than this. They are for sure making serious efforts at building ai for chip design.
Impossible to know. Could be fake/aspirational roles to impress investors with their grand vision.
Jesus. This is tinfoil hat territory now. Why would they fake something like that? ANY company in this field would try to become free from nvda. Goog has done it already, amazon has their own thing, so it can be done. Not saying they'll 0shot this vertical, but ffs, they don't need to fake anything. They are making an effort, and it would be insane to think they aren't. Might work, might not work, but to even think that the effort is fake is going too far.
https://antoniocortes.com/en/2026/03/10/ghost-jobs-the-econo...
I'm not saying this isn't a thing. I'm saying oAI doesn't need to fake trying to make a chip or hiring people to make AI better at chip making, or dogfooding or anything like this. It's obvious they're doing it. They'd have 0 reason to fake something like this "for the investors". Come on!
It kinda depends on what your prior is. Some companies do a press release and I immediately pay attention or even take action.
Other companies? Fool me once Altman, let's see the thing at scale making money.
Near frontier AI is clearly relevant to some kinds of logic design, I'm learning some Hardcaml at the moment and yeah, AI is super helpful.
Can it leapfrog a company without hardware experience to near the front of the pack of companies with decades of hardware experience? Less obvious.
Unrelatedly, would OpenAI dramatically overstate something to manipulate the press and public and capital markets?
It's arguably their core competency .
AI is going to matter in logic design and synthesis. How much, how soon, and where are open questions.
They have a history of lying and making grandiose claims. It's unreasonable to extend them the benefit of the doubt again.
Do you have inside knowledge?
From time to time? Lol you must realize, frontier lab eng are using Codex/Claude-Code 99% in loops, on models the public doesn't have access to. Why? Because it works. Just a matter of time before humans are out of the loop and what comes next is a black hole
"The future is here, it's just not evenly distributed"
Or OpenAI accelerated the design and optimization process by summarizing emails exchanged during the design and optimization process, or made it possible to ask an AI questions about meeting notes
> 1) OpenAI genuinely have AI technologies that can improve chip design (bold, unlikely claim, needs evidence)
Chip design languages (HDLs like Verilog or VHDL) are well understood by LLMs. They don’t need specialty tools to use GPT-5.5 or other LLMs with them.
You could even try it yourself with open source chip design tooling if you wanted to see it.
Yes, obviously. But do we think LLMs without access to proprietary information do a better job with them than Broadcom's human experts or existing proprietary tools at this level of operations?
It is still a bold claim and it still needs evidence.
We would obviously get a bit more of the evidence if it were to be more useful for the upcoming IPO than this rather open-ended, reinterpretable phrasing.
> do a better job with them than Broadcom's human experts or existing proprietary tools
No, obviously. They'd be expected to do a substantially worse job and yet still drastically accelerate the design process.
LLMs make all sorts of dumb mistakes when writing c++ or python yet are nonetheless massively beneficial.
I don't understand why you're getting downvoted.
I've used GPT-5.5 and Opus both for FPGA design with good results. We built a lot of tooling around it to help the models, but even without that they're definitely capable of designing digital logic.
My guess: it is that those who KNOW the subject realize that LLMs suck at it, and those who do not, do not realize it, since their output is plausible, and sometimes even works.
This actually plays out across every field and is well documented. An expert can recognize the hallucinations and bullshit coming out of LLMs, while non-experts see plausible output and do not know enough to know it is BS.
Wrong. Myself and colleagues know the subject and they are useful in FPGA design. You should stop hallucinating about topics you don't have experience in.
https://dl.acm.org/doi/10.1145/3785362
https://developer.nvidia.com/culitho
https://www.synopsys.com/blogs/chip-design/analog-layout-syn...
https://arxiv.org/abs/2302.06415
I feel like they would be very specific if it was no.1.
> OpenAI genuinely have AI technologies that can improve chip design (bold, unlikely claim, needs evidence)
Why is that a bold and unlikely claim?
Are you saying that AI, which has been proven to cure diseases, solve our hardest math problems, write complex computer code and generate entire generated worlds and HD video from a simple prompt would somehow be like, my bad, I guess I can't design chips?
> solve our hardest math problems
We're not quite there yet :)
https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_m...
> Why is that a bold and unlikely claim?
Because they could have offered even slightly more evidence.
Because then they'd likely have stfu and outperformed Intel, Nvidia and AMD, or at least one of them.
They're burning more cash than pretty much anyone else and doesn't have anything public that looks like a matching revenue stream so they probably need one very badly.
Perhaps they used gpt 5.5 mini to draft emails. Create a coffee schedule.
There is a lot of verilog out there, it's pretty feasible that they had AI assistance writing more to design their chip.
It doesn't have to be revolutionary, it could just be AI-assisted design and lined up well enough with their operations for a custom ASIC to be worth it.
Also there's some much boilerplate around everything. Writing a testbench with codex is extremely feasible. This is the kind of verifiable feedback loop the agents shine at.
I feel like "the use of OpenAI models to accelerate parts of the design and optimization process" just means that engineers were using ChatGPT to sanity check their designs and suggest potential optimizations, though that's just my take (and I'm quite cynical about AI marketing in general!)
> the use of email, spam filters and spellchecker to accelerate parts of the design and optimization process
honestly you don't realise how much more efficient it is until you are stuck using the wrong flavour of outlook, the spam filter breaks or sloppy spelling, punctuation and grammar force you to clarify details needlessly.
My girlfriend works at Broadcom doing chip design, and based on what she's told me they JUST got claude code like 3 weeks ago, so I really doubt this means anything beyond them vibe coding some scripts or something...
VHDL, VLSI are well documented languages, with well build test and verification frameworks and harnesses. Even just by iteration you could get there if you have the money to pay for it.
NVIDIA already designs most of their chips using AI. Why would you assume it's meaningless marketing?
Perhaps because they are suggesting what they are doing is novel.
novel to whom, the reader or the industry?
something can be non-novel in the industry, yet novel to the reader, at which point it is useful ... for such readers.
realistically, how hard are AI accelerators to design?
I would assume they've already made as big a deal of it as they can without outright lying too much. Read the rest of the press release.
FWIW, Google is now on their 8th generation TPU, having put out the last 4 generations on a 1-year cadence.
AlphaChip is what a chip design with AI is. I'm very suspicious that OpenAI has anything like this or they would be bragging about it.
https://deepmind.google/blog/how-alphachip-transformed-compu...
Probably obvious but still omitted in the OpenAI post: chips are being made by TSMC [1]. Wasn't sure if Intel got it.
1. https://www.investing.com/news/stock-market-news/openai-unve...
I just read a claim on Twitter that the reason these companies (Google and Amazon as well as OpenAI) are using Broadcom isn't just for design expertise, but because Broadcom have allocation agreements in place with TSMC and the memory manufacturers.
...and because most hardware sales except AI accelerators are down due to RAM prices, Broadcom probably can't otherwise use their allocation at TSMC.
Nope, not down. "total Personal Computing Device (PCD) market — comprising traditional PCs and tablets — posted 2.8% year-over-year growth in Q1 2026, with combined shipments reaching 103.3 million units. PC shipments grew 3% YoY with 65.6 million units" https://www.idc.com/promo/pcdforecast/
Q2 is forecasted to be negative, partly because of RAM prices like you said, but for the most part this is something that only price sensitive nerds care about. Broadcom sells a ton of server chips. Server sales are up 30% vs last year so I highly doubt they're desperate to use their allocation
I was actually thinking of smartphones first because they seem to be the best-selling "personal computing devices" (different definition from IDC) and come with a lot of RAM (8-16 GB or so? Mine has 12) these days. And there I confused Broadcom with Qualcomm - Qualcomm's biggest end customers seem to be smartphone buyers.
I thought of PCs second since most chip manufacturers make some thing or another that goes into them (Broadcom probably more than Qualcomm), and yes it's very suprising that PC sales don't seem to be down yet.
According to your own source
> the full-year 2026 [PCD] outlook has been revised to −10.4% year-over-year
because
> erosion of consumer purchasing power amid regional inflation and currency volatility in many key markets, compounded by memory and storage shortages that are proving more severe than anticipated in the previous forecast cycle.
The positive Q1 YoY growth
> was largely the product of pull-forward demand, as both consumer and commercial buyers accelerated purchases ahead of anticipated price increases and limited product availability.
The idea that only nerds care about the cost of things is... absurd.
Most design partners have allocation agreements. The thing is Broadcom is an absolute GIANT in the ASIC design space, and it's closest competitor Marvell is a fraction of it's size.
There are a lot of large tech companies that most of HN has never heard about that completely dominate entire segments.
I recently put 2+2 together.
Broadcom has become wealthy by being Google's TPU hardware partner, including sharing their TSMC capacity with Google, and evidently now they are doing the same thing with OpenAI. What a brilliant way to take advantage of the AI gold rush!
I wish they weren't using their piles of money to extort money out of the software industry like they are with VMWare and Bitnami.
Well Google has reduced reliance on Broadcom already. They found a new hardware partner, MediaTek, that’s probably much, much cheaper than Broadcom.
https://finance.yahoo.com/sectors/technology/articles/broadc...
> Well Google has reduced reliance on Broadcom already. They found a new hardware partner, MediaTek
Oh dear god. I'm actually feeling sorry for Google at that point. Good luck, you'll need it...
MediaTek was spun out of UMC. UMC was a powerhouse of ASIC design.
My hunch is that this change is driven by bean counters.
> Broadcom has become wealthy by being Google's TPU hardware partner...
Kinda, but not exactly.
Broadcom cornered the enterprise infra and security market in the late 2010s and early 2020s after acquiring CA Technologies, BMC (EDIT: Did NOT acquire them, they were considering it back in 2018 but decided against it and KKR ended up acquiring them), Symantec (which they bought instead of BMC), and VMWare and were able to make a strong cybersecurity story during the late 2010s cybersecurity and SaaS boom.
That gave them plenty of cashflow that helped subsidize their hardware business when hardware was not viewed as hot as it is today.
Additionally, Broadcom is GCP's marquee customer and has been for a little under a decade so they were able to make a sweetheart deal where all that software businesses at Broadcom would be exclusively using GCP and in return GCP would working with Broadcom to design it's silicon and source infra needed for their DC buildouts.
Ironically, the DoJ blocking Broadcom's acquisition of Qualcomm was the best thing it ever could have done for Broadcom, because it gave Broadcom the dry powder to dominate the Enterprise SaaS and build a strong niche in the cybersecurity space.
> piles of money to extort money out of the software industry
From personal experience, executives and leadership who started off in the electronics and hardware industry are much more vicious and cutthroat than their peers who started in software.
Working in an industry that historically had to deal with high commodification, low margins, and long tail sales leads to leadership that can execute. Additionally, no one climbs the leadership ladder without having spent years as a line-level engineer, but that's true for software as well to an extent.
Edit: can't reply
> Did they acquire also BMC?
Nope.
Broadcom was considering acquiring them in 2018 but decided not to go through with the opportunity and KKR jumped in.
Did they acquire also BMC?
Good information, Broadcom is a playa, lots and lots of acquisitions! (a quick google search turns up a very eventful history for Broadcom)
> From personal experience, executives and leadership who started off in the electronics and hardware industry are much more vicious and cutthroat than their peers who started in software.
Only The Paranoid Survive is quite a name for a management book. It implies surviving in the world you are speaking about.
[0] https://www.goodreads.com/book/show/66863.Only_the_Paranoid_...
I wanna see an inference chip where the weights are part of the rom of the chip.
There would be 1 multiplier per weight (and since they're constant, the whole thing turns into a bunch of simple adders), and the total pipelined system throughput would be one token per clock cycle.
That means you can probably have millions of users simultaneously using a single bit of silicon, with perhaps 500 million tokens per second coming out the output bus.
Downside is this chip would be huuuuge - a whole wafer.
Wafer level faults probably won't matter though - neural nets are resistant to a few missing or wrong weights.
Due to the speed the industry moves, you'd want to race from model weights to production super fast, make 50 wafers, use them for a year, then bin them when that model is obsolete.
this appeared some time ago, https://taalas.com/, but I'm sure there's others thinking these same thoughts. this would be best for small models imo, nothing frontier because that changes too fast
you can try it out here: https://chatjimmy.ai/
By the way, you've seen Cerebras? It's not gone as far as what you described - loads of cores and RAM but you still load up the weights onto it as software and they need to be streamed into the chip for large models - but it is a whole wafer.
Cerebras is a whole lot of SRAM, basically a ton more L1/L2 cache, hence increasing throughput.
They're pretty supply constrained right now though and their production costs seem prohibitive.
The interesting players at the moment are from Toronto: taalas (print the model onto the silicon) and tenstorrent (dataflow programming based hardware)
There is a huge downside to weights being modifiable - it means you need to have multipliers (not simply adders), and SRAM to store those weights.
I suspect for equal performance, that's probably a 5x increase in silicon area (and therefore cost).
> weights [as] part of the rom of the chip
Not really that: you are pointing to Compute-In-Memory (CIM) - techniques where the data (here, a multiplier value) is part of the processor (here, the multiplying circuit).
The problem of "fetch and process" is bypassed completely architecturally: the data is there where the processing happens - it's not moved, there is no latency.
>> I wanna see an inference chip where the weights are part of the rom of the chip.
I've been wondering about that for a while now. For a lot of tasks putting weights in ROM is probably OK. OTOH:
>> There would be 1 multiplier per weight...
I'm not sure that is a good idea. Maybe if its quantized down to 2 bits... Otherwise maybe a small ROM near each multiplier (or row of them or whatever) so the multipliers could handle N distinct matrix operations without having to move the data from far away.
Another fun thought is to have a row of MAC units on DRAM so a DRAM row would be a vector. Row size might be 64Kbit or 8K weights if they're 8bit. This also keeps the weights and calcs on the same chip. I'm not sure this would put enough multipliers on one chip though. Systolic arrays can have tens or hundreds of thousands each doing one op per clock cycle.
analog chips could also be very interessting instead of using digital signals and processing them against the weights in the ROM. I have no idea if that scales with such big models though.
The drawback is in keeping signal fidelity (e.g. dissipation, temperature etc.) and in the conversion between analogue and digital.
Nonetheless, yes, there are already implemented solutions for small NNs (I understand mostly acting as triggers).
Supposedly memristors would be ideal for this (and it would be reprogrammable), but then again, memristors seem to be the carbon nanotubes of the computing world.
I've also been thinking about this. Although the forward pass of a transformer model also involves some heavier operations like normalization, reciprocals, exponentiations or other non-linearities (GeLU, SiLU) which may (though typically don't) involve learned weights as operands.
firmware upgrade would mean flashing a huge BIN file.
“ Wafer level faults probably won't matter though - neural nets are resistant to a few missing or wrong weights.”
Brain science people “love” traumatic brain injury cases because it can help explore what happens when bits of the “brain wafer” get damaged. We’ve learned a lot from such things.
I wonder if people are intentionally “destroying” parts of the model weights to learn more about what happens? Like could you strategically wipe a gig of the model so it’s “all zeros” and see what happens?
I have to wonder
Of course tampering with chunks or nodes in the NNs is a way to study the "spawned" (through gradient descent etc.) configuration and "reverse-engineer the black box" to get "AI transparency".
Anthropic published an important work around one year ago.
This is called mechanistic interpretability. There is lots of fascinating insights already since you can do basically everything down to the neuron or weight level thousands of times. The human brain is many orders of magnitude harder to make sense of.
well its actually called ablation, and its one way to do mech interp. anthriopics got a bunch of work on mech interp here https://transformer-circuits.pub/, like SAEs and NLAs
Somehow related:
https://github.com/elder-plinius/OBLITERATUS
Reminds me of Golden Gate Claude (https://www.anthropic.com/news/golden-gate-claude)
This is another large leap in the great divide of haves and have-nots. Big tech AI labs will develop LLM accelerators and hardware LLMs that increase frontier model output to tens of thousands-hundreds of thousands of TPS.
These chips will be used internally for their own business goals, giving them the capability to iterate at such an insane pace they will be able to clone every software product and software company on Earth. Meanwhile they'll trickle out 100-300 tps access to the rest of subscription users to drain them of their cash and keep the beast fed with fresh training data.
We really need more people pursuing hardware LLMs from an open-weights perspective and consumer perspective, and we probably need laws around free and open access to AI, and some anti-AI monopoly laws, because otherwise society as a collective has no chance of keeping up with big tech and our rights and freedoms will be rapidly eroded.
How can any individual company building a product, with access to 100-300 TPS behind-frontier security-gated, censored and capability gated models expect to compete with a company like Anthropic or OpenAI with frontier, unrestricted, unlocked models that can produce 100-1000x the output? 3-5 of their employees working to clone your 500 staff business will likely be easy pickings for them.
This should concern everyone.
This is very cool to see - seems like soooo much efficiency waiting to be unlocked at the chip level.
What's everyone think of Taalas?
They're actually burning the LLM model into the silicon, with some onboard memory for fine-tuning. They claim huge cost / latency wins.
Super fast demo live at: https://chatjimmy.ai/
https://taalas.com/
https://www.reddit.com/r/singularity/comments/1r9frzk/taalas...
> seems like soooo much efficiency waiting to be unlocked at the chip level
Well if you are exclusively using GPUs that are general purpose, of course you leave so much efficiency on the table. That’s why Google started making TPUs more than a decade ago. I remember that kerfuffle when Google fired Timnit Gebru when Gebru’s paper used GPUs to calculate the environment impact of LLMs while ignoring the efficiency of TPUs; this basically made Jeff Dean very angry due to that wide efficiency gap.
That ... wasn't the kerfuffle
She wrote the stochastic parrots paper.
Google’s internal review blocked it from publication. Stated reasons were about paper quality. You can speculate whether that was the real reason.
Gebru issued an ultimatum email and said she would resign if some list of conditions weren’t met.
Google said “thanks, we accept your resignation”.
She claims it is retaliation, but it seems more like an own-goal if you ask me. She basically handed Google the solution to their problem.
Practical lesson: don’t tell your employer you might quit before you’re ok with leaving.
It kind of was. I really hate gaslighting, but GP is not inaccurate. Google claimed it did not meet their bar for publication because it ignored recent research on how to reduce the environmental and bias-related risks of LLMs. On the other hand, a large org is unlikely to subsidize high-profile research that makes it look bad. And Gebru was critical of Google’s internal culture and diversity efforts…
It'd be cool to see more of this type of thing, but I have to imagine the ability for it to be updated to a brand-new model as new models come out is limited. If that is the case, it's going to be an extremely hard sell.
> extremely hard sell.
It really depends on the pricepoint at which they can get a board. If they can do a ~32B model for 1k$ and a size of an external HDD, I'd buy one now, even knowing that it won't be upgradeable / the model remains fixed. The speeds they've shown are a quality of its own, and there's plenty you can do with such a model and faster than instant responses.
Maybe in 10 years when the tech matures, but IMO now seems a bit too early to have a tech like this. It is like intelligence without evolution or progress.. yes it can be used in some niche markets, but difficult to be generic.
There are plenty of applications that would be useful right now. Specialized models for tool use, like fine tuning for command line tools that are already well established and don’t change often. I’m sure there are many areas where the training data is essential crystallized and unlikely to change. Think of them more like delegated agents or coprocessors that another model could route to so instead of routing to a quantized or lesser model it could use a full fidelity model that is faster, almost instantaneous.
You don't need SOTA models for all tasks, and being able to do more routine tasks at something like 10% of the cost and 70x speed unlocks LLM use for things that are just unthinkable now (bulk classification tasks, real time speech interaction, etc)
A hard sell right now. The rate of change will slow down
Yes, but with current architectures world knowledge is baked into the weights. We might stop figuring out how to make models better, but the world keeps changing, science is going to keep making progress at understanding the world, etc. This creates a significant minimum rate of change and I'm pretty skeptical that it's worth baking weights into silicon as a result.
I think it would just be an opportunity to sell another chip a few years down the line. If the utility curve flattens out on the performance of models I can see a future where you are buying an up to date chip every few years to upgrade to the latest and greatest, while providing up to date context as part of the user input. Like if I have a programming task and I supply a copy of up-to-date documentation alongside my input, I would think that I could still get good output out of a dated model.
That's why we have reasoning/CoT LLMs that can use tools to get updated information.
I mean it just depends on the price of the chip. You might just replace the chip like you would any other component. Like a video game cartridge or something.
What makes you think that? The rate of change seems to have been increasing and there is so many chip and model best in different directions at the moment.
I think the model they chose is out of date and hard to sell, but there are plenty of use cases where today's dumb small models are fine. A Qwen 3.5/3.6 or Gemma 3 model on silicon at those speeds would be genuinely world changing even if it's only 1-3B params. Such a model at those speeds will remain extremely useful even over a 5-6 year timespan, I think.
If you consider the places you could deploy it -- with no network access, and at those high speeds... very useful .. for adding vague "common sense" fuzzy thinking to all kinds of applications that right now piss consumers off with poor UX. Esp if the model can do voice-to-text and text-to-speech well (some of the smaller models can)
> I think the model they chose is out of date and hard to sell
I understood it as a proof-of-concept, not a for-mass-production single blueprint - i.e.: "if you need your NN in a CIM form on ASIC, we can do it".
Their next proof-of-concept was said to be meant to be about size: "we showed you we can do it with 8b, now we are working to show you we can do 24b or 32b". Then, "and we plan to go bigger and faster".
> Our second model, still based on Taalas’ first-generation silicon platform (HC1), will be a mid-sized reasoning LLM. It is expected in our labs this spring and will be integrated into our inference service shortly thereafter. // Following this, a frontier LLM will be fabricated using our second-generation silicon platform (HC2). HC2 offers considerably higher density and even faster execution. Deployment is planned for winter (19 Feb 2006)
I wouldn't be surprised if "fast, cheap, dumb" end us being the market for LLMs.
The state-of-the-art models aren't at "can fully replace knowledge worker" levels yet and I doubt they'll get there any time soon, so charging $2000 / month for access isn't going to happen. Right now everyone and their dog is being handed subsidized credits to play with AI, but the actual outcome is rarely good enough to be worth the money they'd need to charge for it. It might very well take another order of magnitude or two to get LLMs to be truly good (if it is even possible at all), and considering how much money is already being pumped into it I just don't see that happening.
On the other hand, the dumb models are more than adequate for simple noncritical tasks, like directing a user to the appropriate FAQ entry, or playing phone decision tree. There's a lot of money in making chatbot assistants actually useful, or in augmenting website search. Turning it into a glorified "language-to-API-call" translator doesn't take a lot of smarts, but as long as it's cheap you can make a killing in volume.
> On the other hand, the dumb models are more than adequate for simple noncritical tasks, like directing a user to the appropriate FAQ entry
This is a lane I’ve been experimenting in —- seeing what I can get out of models that work in 16GB VRAM for simple tasks (screen scraping, decision tree navigation, natural language queries). It’s interesting for sure (certainly reveals non-deterministic limits) and promising for low criticality review-opportunity tasks, but I also feel like I need better sources/community for understanding and reflection. Preferably those that aren’t hype channels. Any pointers?
It seems technically interesting, but they seem very sparse on details. I don't know if I like the idea of a single unchanging model forever on a chip. How much more expensive would the silicon be if they used rewritable ROM for the weights? Such an arrangement would permit fine-tunes of the model it was designed for, which might minimize concerns about the model becoming outdated.
There is no memory storage of weights in the Taalas cards but translation of the weight multiplier into a circuit.
In a chatbot, 17k tok/s is a neat but nearly useless showcase. In a coding agent it is a meaningful improvement. In robotics, it could be an absolute revolution.
8B models aren't useful in general, but for specific use cases they can provide an enourmous amount of intelligence - nVidia's Tesla/Waymo competitor is a 7B LLM with a 2B diffusion model, and running that at those speeds could be an order of magnitude cheaper than existing solutions.
17K tok/s is approaching realtime motor cortex needs for a robot with ~12 actuators (bipedal humanoid) and an IMU. I don't know how many parameters a motor cortex would need but 8B feels like it is within 2 orders of magnitude.
this is an LLM, not a motor cortex. it will output commands as text (json, ...), so comparing size is not very meaningful, especially considering neurons are highly complex and likely requires thousands of artificial simple neurons (weight+bias)
There's nothing about Taalas that is specific to an LLM
Bumping the speed of these things would be more than meaningful. It would be a massive game changer.
I assert like 80% of this “multi agent parallel workflow” business is simply a workaround to models being soooooo slow. Like as the dude driving these things… you kick it off and twiddle your thumbs waiting minutes to hours sometimes for all the inference and token generator to finish. So you dispatch multiple workstreams in parallel to be more efficient.
I assert that if the model was even 10x faster we’d be using these things radically different. You’d be doing things that are currently time prohibitive. At 100x, holy shit will software dev get crazy. You’d be kicking off hundreds of parallel workers attacking a problem from every angle and stuff. Who even knows!!!
And the thing is, 10x will absolutely come and probably even 100x. And it will be sold like a video game cartridge or something depending on how the actual model gets “baked” into the hardware. No remote inference at all.
Could you give me some example how in robotics it can be an absolute revolution?
My understanding is that robotics doesn't really rely much on LLM's in the first place but rather other things.
Is the thing that you are suggesting that it would ingest all real time data and then reason through it at an incredibly fast speed and then act on it and re-iterate? I might imagine some problems with this though I am not a robotics engineer and perhaps someone who deeply understands this topic can give more information.
LLM are very good at looking at images and reasoning about them. much more than just object recognition/segmentation, they can explain the physics in the image, the intents, plan actions, ...
That's because of posttraining optimizing for benchmarks that test that.
They tend to collapse into nonsense and hallucinations pretty quickly if you move slightly out of the envelope of the current visual reasoning benchmaxxing.
Disclaimer: I'm a robotics noob, but I've been working on robotics for a few months now.
I'd say virtually all robots you've seen in the real world today rely on classical approaches - you build a rudimentary map, then use classical algorithms to find paths/do area coverage. The robots do no reason or understand what they're looking for, they're more like in-game units. At most there's some bounded, lightweight image classification going on.
LLMs can understand and reason about the world natively. nVidia has a Tesla FSD/Waymo competitor which simply their 7B reasoning LLM but instead of outputting tokens directly, its outputs are fed to a 2B diffusion model that outputs 1.6 second long trajectory for the car, and this is enough for an L2 system. But to make this work, they need the model to run at 10Hz, so they use super high-end hardware to do it (Jetson Thor) and the car is still "blind" for 100ms at a time (they have a parallel classical safety system).
With on-chip LLMs you could run this loop at like 100Hz on a chip that costs a few hundred bucks, rather than 10Hz on a board that costs several thousand.
I think hardware like this is the future for LLM-providers once we reach a point where the models aren't advancing much any more. You could argue we're close now.
The hyperscalers like AWS will made great use of these to serve up models that will be relevant for several years. But right now, we're still seeing significant bumps in model quality every couple of months - especially with open-weight models like Deepseek/Kimi/GLM.
Until that point, though, I don't see how this is ever going to be cost effective vs general purpose hardware.
I also think we'll see miniature versions of this baked into mobile hardware for super fast and efficient on-device LLMs.
I see only these two possibilities:
1. If LLMs keep improving, burning models onto silicon becomes obsolete too fast and is not worth doing. Outcome: We keep getting better LLMs. 2. If LLM improvements slow down, they will be burned onto silicon. Outcome: We get faster, cheaper and energy-efficient LLMs.
Either way sounds great to me. It will certainly be a mix so we can even get both.
Pretty huge move. Google and their TPUs are looking infinitely more prescient as I think they are on their 7th generation, along with the offshoots it inspired like the LPU and even others, perhaps like Cerebras and their Wafer Scale Engine.
However, based off first impressions, it seems like this is meant for inference side, and not training, which is also an interesting choice.
Training is pretty much a 1x cost, and efficiency there is already on the way down with architectural improvements. Inference though is an ongoing cost which over time takes orders of magnitude more resources, so focusing on making that far more efficient means way greater gains over time.
Inference costs are higher than training now. I think.
Nvidia is king of general purpose training chips. But inferences can be specialized.
What makes you think this? With wider adoption the ratio shall shift in favor of inference. And API price is becoming more important than SOTA capability.
> With wider adoption the ratio shall shift in favor of inference
Yes? That’s why more money will be spent on inference than training?
I’m talking absolute cost. As the number of people using AI and burning tokens goes up the amount of spend on inference goes up.
I am fairly confident that Anthropic has way way more GPUs serving Claude Code to users than they have training models. They’ve got a lot of users!!
> API price is becoming more important than SOTA capability.
Also yes? This is why custom silicon for efficient inference makes sense!
I think we’re in total agreement here :)
Cerebras's Codex Spark 5.3 has been a huge flop. Small context window and old model. But hopefully they can improve so that we can benefit from 1000 tokens/second with GPT 5.5.
> early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art
We're starting to see what really matters here, and though this is hand wavy the TPU makes similar claims.
I think googles memo about having no moat still stands (see: https://newsletter.semianalysis.com/p/google-we-have-no-moat... if you are unaware). It kind of makes sense that all of this is looking more like 60's to 90's IBM, DEC, Cray, Sun and the hardware race that happened then. History doesn't repeat but it often rhymes and I suspect that these efforts will follow the same trajectory.
To be clear, that is not "Google's memo". It's a memo by a guy who happened to work at Google. There is a diversity of opinions at a company that employs 180,000 people.
With the pace of AI, and with AI helping to pave the way for faster/better AI, I keep wondering if hardware like this will become obsolete well before it has a meaningful ROI. Huge AI models can be run with less resources already through quantization and offloading, but that's just the beginning. One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop. Think that's crazy? Look at the size of the first hard drives. The IBM 350 was a disk with 50 platters, 24 inches in diameter, that held 3.5Mb, and was leased for today's equivalent of $35K.
https://www.computerhistory.org/storageengine/first-commerci...
Compare that to a multi-terabyte ssd. Now apply that improvement to how an LLM is architected and run now. With AI assisting, it won't be long before a leap occurs and these data centers with all their current ultra-cutting edge Nvidia cards are nearly obsolete overnight.
> One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop.
But if you have such a breakthrough could you not also apply it and run 200T models on todays datacenters?
That assumes scaling laws still hold up. A bigger model might end up only incrementally more intelligent.
They do. Mythos kicked ass while it lasted. And what we know of the scaling law curves promises us even more gains in the future.
"The future" being "whenever training and inference at increased scale becomes economical". Which is probably bounded by new generations of hardware, but might also be pushed forward by algorithmic advances.
I think they're out of training data though...
Synthetics are often used for "data amplification" nowadays. Extra compute covers a multitude of sins.
Not only you could: you would also want to.
The likes of Mythos show that the scaling laws are real, and you can x5/x2 the total/active params and get meaningful gains. If "inference per param" gets cheaper? Up the params and get more intelligence for the same price.
Quite true
Interesting comment, but the comparison with hard disk drives is probably unfair.
The IBM 350 was commercialized 70 years ago; it took 70 years for someone like you to be able to compare that to a multi-TB SSD.
Furthermore, nothing says that Moore's Law will necessarily apply to LLMs, for decades to come.
Very true, and all I am basing my comment on is the improvement in speed AI has demonstrated when applied to software development, and inferring it might enable a similar 10X or 100X improvement in both hardware architecture as well LLM structure and/or interface methods. If that speed improvement applies to performance of AI, that could mean the 70 years it took for people to improve storage technology might be able to be compressed to achieve a step change in AI performance in a drastically shorter timeframe.
I think Jevons Paradox and scaling laws will make this not the case. If bigger models are always better (which seems they are), then will always need high-end hardware.
Usually breakthroughs in computing lead to more usage of computing, not less.
> One day, maybe not far from now, a breakthrough will allow huge LLMs (say 200B in size) to run well on an old 5 year old Dell desktop.
I think there will be specialized hardware (beside GPUs) that would be custom made for LLMs. Yes TPUs exist, but mainly for datacenter. GPUs exist, but they are adapted from mainly graphic application. Once all the demand from data center dries up, innovation will kick in.
> I keep wondering if hardware like this will become obsolete well before it has a meaningful ROI
it will build expertise/infra/know-how foundation for next generation of hardware
True but as someone else pointed out; at that time we'd be interested in running 200T parameter model rather than 200B. Why, you might ask? Law of human laziness - a human will become as lazy as the technology allows it to. With the 200T or 20,000 T model - I'd be heavily incentivized to ask it to make the bread for me that I enjoy making now or create a movie for me (featuring myself) which will maximize the dopamine production in my brain.
I agree with you. Stepping stones are still a part of getting there, if only to be briefly useful.
Looking at the development of memory bandwidth, capacity and prices over the last 10 years there is little indication that’s likely.
>designed for initial deployment by the end of 2026 and expanding in the years ahead,
So after the IPO and will be featured heavily in the IPO sales brochure as a future promise?
I'm sceptical over any pre-IPO announcements.
Yeah, the narrative feels like pre-IPO shenanigans, and it looks like the lid on my laundry basket. I wouldn’t be surprised if this is a con.
Con or not it is an obvious thing they have to do. Might as well promise.
IIRC their biggest cost they're "hiding" in their financials by doing creative accounting is inference (putting it into marketing and whatnot, in the billions)... if they can't hide it in their S-1 then they have to rationalize it, either by a) increasing the prices (not gonna happen, with token based billing orgs are already watching their codex spends) or b) lowering the inference costs. You can lower that by "soft optimizing" (dumbing down) your models but then you have the other players breathing down your neck (see quick rise of Claude), or actually optimizing, in software and in hardware. We're like 5 years into the rise of LLMs, there's not THAT much left on the table unless you write to the metal you specifically designed for your models (and I'm pretty sure the lack of "nvidia tax" would help with covering most of the r&d costs of a custom solution, at least in the long term).
50% cheaper inference without losses in fidelity would unquestionably be a massive win for OpenAI.
Who's IPO? Broadcom and Google are already listed, obviously.
OpenAI's upcoming mega IPO
OpenAI, the non profit organization, is going to become a publically traded profit maximizing corporation
> OpenAI, the non profit organization
No, the nonprofit org stays nonprofit, while the for-profit org it owns will become publically traded.
See https://openai.com/index/evolving-our-structure/
> OpenAI was founded as a nonprofit, and is today overseen and controlled by that nonprofit.
Does anybody actually believe that?
I haven't seen this discussed here:
So far, the accelerator is showing cost savings of roughly 50% compared with typical AI graphics processing units, Broadcom Chief Executive Officer Hock Tan said in an interview. - [0]
50% cost saving. The picture changes so quickly, there are still a lot of low hanging fruits, that I find any discussion about whether a vendor has moats, or if they can recoup investment, is moot and futile.
[0] - https://www.bloomberg.com/news/articles/2026-06-24/openai-an...
If GPUs have 75% margin then 50% cheaper is no surprise.
Operational costs far outweight hardware cost.
Do they? Genuinely ansking.
Yep, I was surprised to learn that too.
"Typical" is doing a lot of work there. That could mean much older chips than Nvidia is currently selling.
Microsoft, Google, and Amazon also do this, but they also have the hyperscaler datacenter infrastructure to host the chips. Designing and taping out the chip is one thing, packaging, cooling, deploying, powering, and managing the fleet is another stack entirely. Wonder where that will come from?
Don't forget Stargate.
Update: Somebody on Twitter said it's going to be hosted 50/50 at Microsoft and Oracle.
I forgot Stargate
We’ve entered the “if you care about software, build hardware” phase of AI
I have been eyeing what Taalas is doing [1] by making pure hardware models. The speed is absurd.
[1] https://taalas.com/products/
They talk about products, but they don't sell the hardware, thus they don't really have a product, just a service.
I know, it's nick picking, but when people can just reach in and take services away, like Fable/Mythos, hardware is the only thing worth buying.
I'm sure they'll have a product for you if you have millions to invest in a partnership with them.
"Nitpicking"
I have and it was wild. Paradoxically it made me realize that I actually like reading the stream as it's generating.
crazy product. their test chatbot feels a db query.
https://chatjimmy.ai
“People who are really serious about software should make their own hardware.” ― Alan Kay
What are the other phases. Or what are you referring to in general?
I had Opus 4.5 design an LLM inference engine in verilog, including firmware and automated verification a while ago: https://github.com/cpldcpu/smollm.c
It's of course far from optical. But lowering the implementation through the abstraction levels turned out to be extremely powerful.
Can you suggest some tutorials for Verilog and FPGAs in general?
I have a spare Tang Nano 9k but I don't feel confident about blindly asking Claude to vibecode me a solution and still would like to have at-least a basic level of understanding.
hm.. has been quite a while for me. The good thing about the Tang Nano is that it is supported by the Yosys open source toolchain. There are quite a few resources on the web when you search for the combination.
the hdlbits course is really good imo
So I’ve been wondering about “one or two levels back” chip design. If I understand it, 28nm chips (pre EUV) is just about suitable to run (not train just inference) frontier models.
And so if I was a mid-level State would it be worth while to take my nascent chip industry and push it out to build a 28nm foundry and supporting eco-system.
The models will come but the real challenge of the future is having enough compute power for every one and every use. Even if LLMs don’t become AGI they will still be incredible tools - and as OpenAI seems to spend 8000 for each 200 monthly subscription building one’s own data centres seems sensible
You are underestimating how difficult it is even for a large nation state to attract the kind of talent and investment it would take to set up a chip industry. It is out of reach for anyone outside of the 3-5 largest national economies and a few big American/Chinese multinational corporations.
I am not sure how much of the work is done by OpenAI, or whether it is basically a Broadcom chip specifically built for OpenAI models. It is a necessary step, but building a high-performance chip is not easy. Look at companies like Groq, Amazon, and Google.
Both Google and Amazon also codesign heavily with Broadcomm (Amazon also with Marvell and Alchip)
Broadcomm does stuff like physical design, provides IP blocks, managing manufacturing process with TSMC, packaging and testing. Google and Amazon work with system architecture, performance targets, and requirements but Broadcomm as consultant.
This is starting to sound like startup scope creep. Instead of making the AI model it’s now custom silicon, web browsers, and consumer electronics?
Definitely has that smell... At the same time though, they NEED inference cost to drop substantially, and even better for them if it only happens for their models on their hardware.
I assume they're doing everything they can to make that happen model-side, but coming at it from the other end makes sense too if they can swing it.
Maybe, but they’re also a massive company. At some point Google stopped being a startup and become a massive company with margins to look after
Nearly all those initiatives have failed though
I hope to see something like this, but in a small form factor like the NVIDIA spark.
I want a super fast LLM that is Opus 4.6+, like, in ability.
Memory bandwidth is the bottleneck in the Spark. If you replace the SoC with an optimized ASIC but keep the same 256-bit LPDDR5 the performance will be the same. You can increase performance by using wider memory but that's also more expensive.
M3 Ultra has a 1024 bit memory bus (819 GB/s) and starts at $3,999 (96GB of RAM). It can be done....
The tradeoff is that the M3 Ultra's GPU loses to laptop GPUs in compute benchmarks. All of that bandwidth is wasted idling for token prefill.
For inference workloads, it makes a lot more sense to optimize for prefill/ttft before maxing out memory bandwidth.
With the M6 theoretically coming later this year, Apple seems to be realizing they need to catch up with more lanes of GPU.
Personally, I doubt it. Apple hamstrung themselves with unified SOC memory, there are cheap dGPUs that smoke the M5's prefill speeds and even have faster decode too. Apple is running up against the limitations of putting a mobile integrated chipset up against the desktop form factor. An SOC stops looking like a smart decision at that scale.
The software side is still pretty sketchy, too. Apple's ecosystem is fractured between NPU, MPS and Accelerate BLAS, with libraries like MLX and CoreML built precariously overtop. Apple has to commit to a full rearchitecture of their GPU to challenge Nvidia, which fractures that ecosystem even further.
I love those 128 GB dGPUs.
Unfortunately Sam Altman won't be the one to deliver us at-home hardware that can run Opus-level models
I wonder what is happening with the OpenAI / Jony Ive crossover episode.
Forget about it. Datacenter class hardware is getting farther and farther from desktop use. It’s not PCIe GPUs anymore.
Very interested to know the distribution of effort between the two companies. Is this truly a brainchild of OpenAI engineers or did they pay to white label and use a new Broadcom chip?
The only surprising thing about this is that they didn't do it three years ago.
This seems like more competition for Cerebras? Am I understanding correctly?
This is just an uncut wafer - I don't think it's intended to be a wafer-scale chip.
Cerebras etch memory onto the wafer alongside the processing elements, but AFAIK OpenAI are going to be using HBM memory and a conventional chiplet design.
Still competition for cerebras. Seems quite unlikely they will get an OpenAI deal anytime soon.
They have an OpenAI deal right now. https://openai.com/index/cerebras-partnership/
No - this is OpenAI trying to complete with Google (TPU) and Amazon/Anthropic (Trainium) on cost.
Cerebras are addressing very specific use cases, not general purpose LLM serving, and OpenAI does already partner with them.
cheap token is more important now than ever. Chinese open weight model is getting pretty good. the real cost of AI adaption will come down to who (China or US) can provide cheap token for consumers and companies. Microsoft consider DeepSeek for their cowork is an example and now OpenAI with its own AI inference chip.
I’m not understanding. If cost per token hits the floor that does not mean that you want a model that uses tokens.
If the Chinese are optimizing for token usage, that’s also speed.
Why use more token if few do trick?
If it’s really a differentiator, why announce it? Why not keep it secret and make it a competitive advantage?
Investors, I'm sure everyone's had the idea and they're doing it.
> May we scale smoothly, exponentially and uneventfully through A[SI]
That sentence sounds weird to me. I can't really put my finger on why, maybe the combination of adverbs, or just the fact of writing the desire of scaling as a company so directly. It feels (to me) like openly claiming their selfish goals. Or maybe I am just misinterpreting and they are referring to the whole humanity as "We" (but knowing Broadcom and in a lesser extent OpenAI doings, I am not convinced).
Word of Advice for OpenAI:
Never underestimate Broadcom’s ability to shaft their own customers
- VMware
- CA Technologies
- Symantec Enterprise Security
- Brocade
- LSI Corporation
I don’t know. I’m kind of glad that two of my least favorite companies are working together.
CA Technologies was much worse than Broadcom in its heyday.
Three of their top execs - CEO, CFO, and head of sales - went to federal prison on securities fraud, conspiracy, and other charges. The CEO, Sanjay Kumar, who was at least partly the fall guy for co-founder Charles Wang, served 10 years.
Being acquired by Broadcom could only have been an upgrade, as strange as that may sound.
Why the hell Broadcom of all companies?
I'm assuming they used LLMs to (help humans) do custom circuit design. Even pre LLM there were various computer optimizations that didn't require humans like genetic algorithms. It'd be cool to see a paper on how they did it.
I mean I'd love to be able to buy something like the 17k tps taalas chip as a pcie or m.2.
Imagine when we can roar along at that speed, low power. Can just have the model reason for a while about anything and everything. It reminds me of the "race to idle" for mcus etc.
The current taalas chip is for a 3.1B param model. I’m hope so much that they can get that up to the 30B range. Just imagine Gemma 4 or Qwen 3.6 at 17k tps.
> 17k tps taalas chip
It's odd to me that I haven't heard anything about this approach (baking LLMs/weights into silicon directly) since. It seems almost common-sense that we're going to end up there eventually. And it feels like that point is drawing ever closer now that model capabilities, if not quite plateauing out, are at least getting to a "good enough" point for a LOT of use cases.
I wonder if it's being worked on in secret, if there's something about it that makes it infeasible, or if companies are really too nervous to lock in one model like that because the next one down the line could be a huge improvement. Re. infeasability, I have heard that the Taalas demonstration chip ran Llama 3.1 8B (a pretty horrible model) and that even that took a massive amount of transistors / die area. So it might just be the case that the good models are too big to fit on silicon?
I have also been thinking about this a lot, and share your belief that this is inevitable.
Taalas has a running demo here: https://chatjimmy.ai/
It's eye opening: generated an AVX-512 optimized Mersenne Twister in C in 0.076s, 13,706 tok/s. Too fast for the tok/s to be terribly accurate.
Good models will require multiple Taalas chips but Groq and Cerebras also require a lot of chips and that hasn't stopped them.
The new chip sounds like it's vustom made to accelerate a few specific models they really need to run fast. The advantage is it's truly and ASIC, not a xPU. There are several new startups targeting EDA tooling automation, Chip Agents is the biggest one I can think of but their are smaller players too, Silimate is one I recall. These companies are focusing on building fast AI powered tools to speed up the tape out cycle.
Fucking Broadcom?
The only time I've ever seen that name before is when trying to solve driver issues, on both Linux and Windows.
Are they especially stingy with their IP related to drivers or something?
There is a never ending torrent of money coming, so why not make custom chips.
Whoo ... party!
*requires VMWare license.
aw shucks nvda has some spicy competition
Make sure you all use that fancy ñ
They don't have true competition, what they lose out on is market share with hyperscalers, since OpenAI would have no plans to share inference hardware with any other company right now. Plus, I don't know how does NVIDIA's investment equation pans out long terms given OpenAI will be investing in more purpose built inference stack for the future.
they're still kings for training, though I've heard Anthropic is training now on JAX+TPU setup, so might not be a monopoly in that segment.
Look at the SIZE of that chip.
Cerebras stock is down nearly 20% today.
Not only is approach overlapping, OpenAI is also Cerebras's only major customer.
If you're referring to the big circle of silicon, that's a wafer, generally contains many chips (100-1000s).
The alt text of the first image describes it as the "Jalapeño inference chip".
As a non-RTFA-er. I'm assuming it's a wafer-scale chip, similar to the ones made by Cerebras.
EDIT: From TechRadar[0]: "The 300mm wafer that both CEOs are holding will generate about 50 to 60 ASICs."
[0] https://www.techradar.com/pro/broadcom-and-openai-debut-jala...
That made me chuckle but I guess if you have never seen one I could see how that assumption could be made.
If this photo is real I wonder what can be revealed about the approach they have taken by analyzing the architecture of what we can see.
For reticle-limit chips, it's on the order of 100. And less than that once you filter out bad dies.
Everybody here knows that.
What some don't know (including you) is that the industry is doing wafer-sized chips nowadays, of which Cerebras is the flagship company.
That's why the stock movement could be related, and that is why GP wrote that comment.
I think Cerebras stock going down could also be partly caused by the lock-up period ending today for 200k shares (page 73 of their prospectus) - https://www.sec.gov/Archives/edgar/data/2021728/000162828026...
It doesn’t seem like it? Unless I am misunderstanding these Nasdaq insider trading reports: https://www.nasdaq.com/market-activity/stocks/cbrs/insider-a...
That's just the wafer disc. Looks like it was presented to Sam Altman for ceremonial purposes.
The wafer disc is what the CPU gets "printed" on.
Dang, I just checked and CBRS is in free-fall since the IPO.
Sucks, I think they're a cool company.
OTOH, I was the only person back then pushing hard during my time at KAUST (back in 2019) to buy one of their systems when they were nobody, eventually resulting in a partnership between the two.
Then I joined their online discourse, very few users, I was semi-active there but they didn't care much.
Then I came to Toronto and heard they were opening an office here, tried to get noticed several times but got mostly ignored. I asked about upcoming events several times, anything to get involved, "yeah man, maybe one day". Then they made an event during Toronto Tech Week and didn't even tell me ... idk.
I don't get schadenfreude as I still think they're a cool company.
My point is they put all the eggs in one basket (AI inference) and neglected everything else. They seem to be on shaky ground now ... sad.
my friend briefly worked there and then got hit by layoffs, as a result, I am enjoying the schadenfreude.
So this is where all the memory they bought is going to.
that's not really how it works
'braodcom' ha ha ... it's not OpenAI's chip then ...
NVidia stocks are red now
Because of Micron, no? I don't think it's related to OpenAI's announcement
What happened with Micron?
I wonder how close OpenAI is getting to using the memory they purchased. Are they planning to stack a huge amount of HBM2 into these chips?
I assume OpenAI has been buying memory and "giving" it to Nvidia in exchange for a discount.
They tested on spark model, i bet it's a mix of that with focus on inference speed. Whatever it is, hopefully it shows up with current models as faster. Token/s is as big thing as anything else, and thats where they can really gain some edge over the competition.
If this is something that will hurt Nvidia, I'm all for it
No information on how significant the reduction in energy per token is. No information on amortized price per request. Increasingly its clear OpenAI must demonstrate order of magnitude reductions in cost to not die, this is investor story time without that information.
how much does this chip help with inference speed?
It's probably the same speed but cheaper.
No surprise here. [0]
[0] https://news.ycombinator.com/item?id=45429514
Although this seems to be for inference itself only and not training but inference is a recurring cost and training is a one time cost and so to me, even if Nvidia still gets moat on training, I don't think that it could ever justify its massive evaluations because for example, some chinese models are actually trained on Non-Nvidia models. The moat in that is incredibly thin.
(at the moment), I think that if I were Nvidia, I would be a bit terrified and I imagine the stock to not be doing super great as I can just imagine everyone online might start talking about it for better or for worse.
I am a bit impressed by OpenAI but is this what can be classified as a plan for OAI to salvage itself and all the commitments it has made nearing a 1.4 Trillion dollars from my memory and this article[0] is from 2025
But could OpenAI simply walk out of its commitments when necessary (for example to Nvidia) if this chip works out or what exactly might happen in the future as these commitments are asked to be paid for, its still smart for OAI to diversify with this chip and to have more deeper ways of revenue than just being a simple middleman but I imagine that Nvidia and others have also invested in OpenAI and they must not be happy with this change.
The thing with AI deals are that they have become so complicated that it is hard for me to find the first order impact of things, let alone second or third order impacts and financial accountability seems to be impacted quite heavily because of all of it and there is some sense that it is done so intentionally.
https://techcrunch.com/2025/11/06/sam-altman-says-openai-has...
> significantly better performance-per-watt than current state-of-the-art alternatives
An interesting example of how the current market dynamics incentivize low cost and therefore power efficiency and therefore lowering resource use.
im very excited that frontier models now have so much money and revenue they are releasing their own chips that could change the relationships and bottom line
This is why ram prices are fucked. Cause altman doesnt give a shit about normal people as long as openAI suceeds
But nvidia's moat is software support, isn't it?
You don't need a whole lot of software support if you just want to serve a single family of LLMs.
A lot of companies that serve a single family of LLMs seem to prefer nvidia though. Why is that?
It's not just good drivers, which is what moats them for games and ML. It's a multi-decade work of making chips that are nice to program for and software infrastructure around them.
Apple and Google have excelent chips, yet they needed to invest a lot in long-tail software projects to make those chips do actual premium work. Still not state of the art for serving LLMs (although Google is strong in that, mostly because it piggybacked on previous chip-related software work for phones and so on).
> A lot of companies that serve a single family of LLMs seem to prefer nvidia though. Why is that?
If you write your tools for CUDA, you’re going to prefer hardware the runs CUSA.
How is there anything more to it than this?
Cool. That's it.
What will people use to write for Jalapeño matters.
Nvidia has multi-decade heritage. Apple spent almost a decade in MLX. Snapdragon failed partly here. OpenAI announced nothing regarding to that, so this big moat that multiple companies have (nvidia the most prominent) is nil for them.
Is there any actual content on what the chips are?
You can't purchase Microsoft or AWS chips, but both of them do pretty good write-ups on what they've done. https://blogs.microsoft.com/blog/2026/01/26/maia-200-the-ai-...
This seems utterly empty of actual substance.
lol
I call BS. It’s probably a white label around existing Broadcom IP, impossible to go from zero to this kind of chip in nine months. I doubt OpenAI had any significant contribution.
That’s exactly what this is.
9 months to production is completely impossible anyway.
9 months from design to early samples is probably impossible given than TSMC takes 3 months after tape out to produce them. Then it’s up to the customer to qualify and revise for production. TSMC doesn’t do that.
There’s no AI that makes this happen in 9 months.
The similarities between the AI world and the crypto world are so much closer than any AI fanboy would ever admit.
Wow thats sounds tempting to use open ai newest chips
One thing I don't like about California based companies is how cringe the names always are.
"Jalapeño" is such a bad name, having an "ñ" already makes it difficult and annoying to deal with in so many little ways. Good luck with that.
But also, theres the sort of "yes lets use Mexican related things because we're California" thought that I just really hate. I don't know, its like corporate Memphis to me. You see a product like this, you know it's an uppity califonia based firm that came up with it.
No worse, I suppose, than, the obsession with Lord of the Rings that the authoritarian surveillance companies have. Palantir, Anduril. Then we have the not defense/surveillance ones: Mithril, Valar, Narya, Erebor
What kinds of names would you suggest?
None, probably. Just saying Jalapeño is no worse than any other non-descriptive company name. Although at least Palantir and Anduril are aptly named for what they do. The VC firms less so.
Strawberry was too complicated as a codename.
Too many Rs.
Too many? But there are only two Rs in strawberry, how can that be too many?
You are correct. I don't know why I thought there were 5 Rs in strawberry, and now I look properly I can count them correctly, there are indeed 6 Rs in strawberry.
I am sorry for initially giving an incorrect answer.
Don't worry, in Europe it's the same, but for insurances/lawyer stuff. Tons of companies have names based on Latin words such as Civitas/Insalus/Legalia/Legalitas or whatever which looks tacky/rancid/old fashioned kilometers away.
Jalapeño
Jalapeño
Jalapeño
Really has a… ring to it