This is an outstanding blog post. Initially, the title did little to captivate me, but the blog post was so well written that I got nerd-sniped. Who knew this little adapter was so fascinating! I wonder if the manufacturer is buying the Mellanox cards used from data center tear-downs. The author claims they can be had for only 20 USD online. That seems too good to be true!
I cannot find anything for less than 285 USD. The blog post gave a price of 174 USD. I have no reason to disbelieve the author, but a bummer to see the current price is 110 USD more!
> The author claims they can be had for only 20 USD online. That seems too good to be true!
In my experience, the cheap eBay MLX cards are DellEMC/HPE/etc OEM cards. However I also encountered zero problems cross-flashing those cards back to generic Mellanox firmware. I'm running several of those cross-flashed CX-4 Lx cards going on six or seven years now and they've been totally bulletproof.
I believe the author is talking about the OCP (2.0) network card itself, that these adapters internally. The OCP nics are quite cheap compared to pcie - here’s 100GBE for 100!
https://ebay.us/m/HMQAph
This 100GbE card is an OCP 2.0 type 2 adapter, which will _probably_ not work with the PX PCB since that NIC has two of these mezzanine connectors, and PX only one.
What also may not work are Dell rNDC cards. They look like they have OCP 2.0 type 1 connectors, but may not quite fit (please correct me if I'm wrong). They do however have a nice cooling solution, which could be retrofitted to one of the OCP 2.0 cards.
I've also ordered a Chelsio T6225-OCP cards out of curiosity. These should fit in the PX adapter but require a 3rd-party driver on macOS (which then supports jumbo frames, etc.)
What also fits physically is a Broadcom BCM957304M3040C, but there are no drivers on macOS, and I couldn't get the firmware updated on Linux either.
That’s a good point to note! I think the stacking height would matter, but in theory the single connector is still 8x pcie and should link without the upper 8x lanes connected.
Ha! Been running these for years on both linux and windows (on lenovo x1 laptops). Using cheap chinese thunderbolt-to-nvme adapters + nvme-to-pcie boards + mellanox cx4 cards (recently got one cx5 and a solarflare x2).
If you don’t mind me asking, what are you using these for? Saturating these seems like it would have reasonably few workloads outside of like cdn or multi-tenant scenarios. Curious what my lack of imagination is hiding here.
Officially: to access NAS, get raw market-data files (tens to hundreds of gigabytes a day), not needed on laptop every day, but only once in a while to fix or analyze something.
Really: because I can, and it is fun. I upgraded my home lan to 10G, because used 10G hardware is cheap (and now 25G enters the same price range).
"because I can, and it is fun." The best answer! I am most of the way done with upgrading most of my homelab to 100G from 10G, but there really isn't a practical reason for it. 100G has dropped in price so much as datacenters are all about 400/800G now.
I'm using ConnectX5s for most, and some ConnectX4s in the older servers. Both of those cards have really come down in price in the used/ebay market. I have been playing around with some different optics - I have a bunch of CWDM4s which are very inexpensive and use a single SingleMode pair.... but of course they run hot so if you have them in servers without good air flow you might have problems.
I'm using mostly fiber just because the servers are connected to Cisco 9305 with 72 100g ports.
Oh they are both loud and eat a ton of power. I think the 9305 is at least 800 watts at idle. That is the biggest downside of the retired datacenter gear... yo u really need a dedicated room with power, cooling, and sound isolation.
I wouldn't really call 100Gbit overkill, if you compare it to modern disk drives is about where we should be relative to shared storage/NAS/etc infrastructure people used to run. So yes, being able to share my /home directory across a few dozen machines at my house without a huge perf impact vs using a local drive seems a pretty reasonable use case. Sure its faster than my WAN access, but who cares?
Frankly, 10Gbit is fully 25 years old with, 10GbaseT being 20 years old this year.
Thats ridiculously ancient technology. There is/was a 25/40GbaseT spec too (now 10 years old), which basically no one implemented because like ECC ram (and tape drives, and seem to be trying to do it with harddrives and GPUs) the MBA's have taken over parts of the computer industry and decided that they can milk huge profit margins from technologies which are incrementally more difficult because smaller users just don't matter to their bottom lines. The only reason those MBAs are allowing us to have it now, is because a pretty decent percentage of us can now get 5Gbit+ internet access and our wifi routers can do 1Gbit+ wireless, and the weak link is being able to attach the two.
I did a bit of back of the napkin math/simulation about a possible variable rate Ethernet (ex like NBbaseT, where it has multiple speeds and selects faster one based on line conditions), and concluded that 80+Gbit using modern PHY/DSP's and high symbol rate, multiple bands, techology which is dirt cheap thanks to wifi/bt/etc on fairly short cable distances (ex 30-50M) on CAT8 is entirely possible. And this isn't even fantasy, short cat7 runs are an entire diffrent ballpark from a phone pair, and these days mg.fast/etc have shown 10Gbit+ over that junk.
Agreed - the big thing is 100g is much much cheaper now as so much 100g gear is coming out of datacenters. So many of those older ConnectX4s and 5s, plus lots of switches and optics. 100g really is the new 10g for homelabs.
I do media production, and sometimes move giant files (like ggufs) around my network, so 25 Gbps is more useful than 10 Gbps, if it's no too expensive.
The linux implementation is quite poor. Among other issues, your answer is linux treats it as a TCP/IP link- so packets don't have any offloading for checksums, etc. it's also incomplete (ex- you have to physically unplug and replug the cord every time one side loses link- even a reboot).
This is my firsthand experence trying to get some tablet motherboards to link up and work as a proxmox cluster w/ TB3 as the link between nodes.
I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.
I figured this was Jeff’s CTO Laboratory. I enjoy your channel. Are you kicking yourself for pulling the trigger on MS-01s now that MS-02 Ultras released?
Thunderbolt is basically external PCIe, so this is not so surprising. High speed NICs do consume a relatively large amount of power. I have a feeling I've seen that logo on the board before.
I don't know how to measure the direct power impact on a MacBook Pro (since it's got a battery), but the typical power consumption of these cards is 9 W, not much more than Aquantia 10 GBit cards.
Also, if you remember where you saw that logo, please let me know!
JFYI, for measuring power draw, you might be able to use `macmon`[0] to see the total system power consumption. The values reported by the internal current sensor seem to be quite accurate.
Speaking of hardware, the RTL8159 (10Gbps) hit the market late last year and is said to consume only about 2–3W. It apparently runs very cool compared to older chips. (Though it would need to be bonded to reach 25Gbps ;-)
I got me one of these adapters (RTL8127AF TXA403, with SFP+ cage); I haven't properly benchmarked it yet.
There's no driver support on macOS, and for Linux you'd need a bleeding edge kernel. Just trying to physically connect it (along with a connected SFP28 transceiver) to my Mac's Thunderbolt port using an external PCIe-to-TB adapter, macmon tells me a power draw of around 4.3 W, so it's not significantly less for half the bandwidth, but the card doesn't get hot at all.
Plus 1-2.5w per active cable. You need the heatsinks as the cx4 cards expect active airflow, and active transceivers as well.
I have a 10gbit dual port card in a Lenovo mini pc. There is no normal way to get any heat out of there so I put a 12v small radial fan in there as support. It works great at 5v: silent and cool. It is a fan though so might not suit your purpose.
It isn't. There is no sense in which "Thunderbolt is basically external PCIe". Thunderbolt provides a means of encapsulating PCIe over its native protocols, which puts PCIe on the same footing as other encapsulated things like DisplayPort and, for TB4 and later, USB.
I'm surprised you are only getting 20gbit/s. I did not expect PCIe to be be the limiting factor here. I've got a 100gbit cx4 card currently in a PCIe3 X4 slot (for reasons, don't judge) and it easily maxes that out. I would have expected the 25g cx4 cards to be at least able to get everything out of it. RDMA is required to achieve that in a useful way though.
Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.
The new low-power Realtek chipsets will definitely push 10 GbE forward because the chipset won't be much more expensive to integrate and run than the 2.5Gbps packages.
It all comes down to performance per Watt, the availability of cheap switching gear, and the actual utility in an office / home environment.
For 10 Gbps, cabling can be an issue. Existing "RJ45"-style Cat 6 cables could still work, but maybe not all of them.
Higher speeds will most likely demand a switch to fiber (for anything longer than a few meters) or Twinax DAC (for inter-device connects). Since Wifi already provides higher speeds, one may be inclined to upgrade just for that (because at some point, Wireless becomes Wired, too).
That change comes with the complexity of running new cabling, fiber splicing, worrying about different connectors (SFP+, SFP28, SFP56, QSFP28, ...), incompatible transceiver certifications, vendor lock-in, etc. Not a problem in the datacenter, but try to explain this to a layman.
Lastly, without a faster pipe to the Internet, what can you do other than NAS and AI? The computers will still get faster chips but most folks won't be able to make use of the bandwidth because they're still stuck on 1Gbps Internet or less.
But that will change. Swiss Init7 has shown that 25GBps Internet at home is not only feasible but also affordable, and China seems to be adding lots of 10G, and fiber in general.
> Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
wifi is not faster.
However ethernet is not as critical as it used to be, even at the office. People like the conveniency of having laptops they can move around. Unless you are working from home, having a dedicated office space is now seen as a waste of space. If the speed of the wifi is good enough when you are in a meeting room or in your kitchen, there is no reason to plug your laptop when you move back in another place, especially if most connections are to the internet and not the local network. In the workplace, most NAS have been replaced by onedrive / gdrive, at home NAS use has always been limited to a niche population: nerds/techies, photographers, music or video producers...
We have 400Gbe which is certainly faster than USB.. but;
On consumer devices, I think part of the issue is that we’re still wedded to four-pair twisted copper as the physical medium. That worked well for Gigabit Ethernet, but once you push to 5 or 10 Gb/s it becomes inherently expensive. Twisted pair is simply a poor medium at those data rates, so you end up needing a large amount of complex silicon to compensate for attenuation, crosstalk, and noise.
That's doable but the double whammy is that most people use the network for 'internet' and 1G is simply more than enough, 10G therefore becomes quite niche so there's no enormous volume to overcome the inherent issues at low cost.
Wireless happened, I'd think. People started using wifi and cellular data for everything, so applications had to adapt to this lowest common denominator, and consumer broadband demand for faster-than-wifi speeds isn't there. Plus operators put all their money into cellular infra leaving no money to update broadband infra.
Wifi now can pretty realistically beat 2.5gbit/s while most Ethernet is still gigabit. It just seems strange to live in a world where the average laptop will get a faster connection speed over wifi than plugged in to Ethernet.
Ethernet is following suit to 2.5G which is otherwise a nonsensical step for ethernet speeds, I think this is further evidence that everything just follows wifi now.
Ethernet did not stagnate. Ethernet on UTP did stagnate due to reaching the limits of the technology, but Ethernet continues to advance over fiber.
For 10 Gbps I find it simpler and cheaper to use fiber or DACs, but motherboards don't provide SFP+, only RJ45 ports. Over 10 Gbps copper is a no go. SFP28 and above would be nice to have on motherboards, but that's a dream with almost zero chances to happen. For most people RJ45 + WiFi 7 is good enough, computer manufacturers will not put SFP+ or SFP28 for a small minority of people.
> Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
Practically spoken, a lot of the transfer speed advertised by wifi is marketing hogwash barely backed by reality, especially in congested environments.
> Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.
For both laptops and desktops, PCI lanes. Intel doesn't provide many lanes, so manufacturers don't want to waste valuable lanes permanently for capabilities most people don't ever need.
For laptops in particular, power draw. The faster you push copper, the more power you need. And laptops have even less PCIe lanes available to waste.
For desktops, it's a question of market demand. Again - most applications don't need ultra high transfer rate, most household connectivity is DSL and (G)PON so 1 GBit/s is enough to max out the uplink. And those few users that do need higher transfer rates can always install a PCIe card, especially as there is a multitude of different options to provide high bandwidth connectivity.
> Practically spoken, a lot of the transfer speed advertised by wifi is marketing hogwash barely backed by reality, especially in congested environments.
Yes but a hogwash of several gigabits sometimes does give you real-world performance of more than a gigabit.
> Intel doesn't provide many lanes, so manufacturers don't want to waste valuable lanes permanently for capabilities most people don't ever need.
It's been a bunch of years that a single lane could do 10Gbps, and a bunch more years that a single lane could do 5Gbps.
Also don't ethernet ports tend to be fed by the chipset? So they don't really take lanes.
Yeah, it's because the network card adapter's heatsink is sandwiched between two PCBs. Not great, not terrible, works for me.
The placement is mostly determined by the design of the OCP 2.0 connector. OCP 3.0 has a connector at the short edge of the card, which allows exposing/extending the heat sink directly to the outer case.
If somebody has the talent, designing a Thunderbolt 5 adapter for OCP 3.0 cards could be a worthwhile project.
A Flex PCB connecting the OCP2 connector would allow to put the converter board behind the NIC board, allowing the NIC board to be exposed to the aluminum case to use the case itself as a heatsink (would need a split case so the NIC board can be screwed to one side of the case, pressing the main chip against it via a thermal pad).
As a stop-gap, I'd see if there was any way to get airflow into the case - I'd expect even a tiny fan would do much more than those two large heatsinks stuck onto the case (since the case itself has no thermal connection to the chip heatsink).
I've had a lot of problems with even 10GbE via Thunderbolt 3/4. Bandwidth-wise it works fine, but latency and jitter are issues. This means that stuff like high-speed cameras that need to be synchronized over Ethernet using Precision Time Protocol (PTP) tend to simply fail with these devices.
I’d heard similar complaints re: TB networking latency & jitter. Did some investigations and tuning on a pair of machines with USB4 ports connected via short TB5-rated cables. Eventually got the thunderbolt links to consistently beat the ether ones on both latency and jitter. And not just switched Ethernet either - even a direct Ethernet P2P link lost out to TB, though the difference there was small.
I used to have an SFP28 Mellanox card in my home server, but went back to a simple 2.5G Ethernet port for the LAN side. The Mellanox card ran hot and needed an extra fan near it to dissipate the heat. It was cool but there was no real benefit other than occasionally when transferring some large files.
Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors like the one presented here are unreliable and add several ms of latency.
For reference, I'm seeing pings from my Mac to my Linux boxes (Lenovo Tiny5) at well under 1ms, not much worse than between them directly. But yeah, your mileage may vary.
Yep, these cards need a fan (or any kind of directed air flow).
Where did you get "several ms of latency" figure from? I have not measured external card, but may be I should do it... Because cards themselves have latency in range of microseconds, not millis.
I haven't tested this particular Thunderbolt SFP adapter, but my experience with a TP-Link 1Gbps USB adapter is that it adds about 4ms of latency. Far from being unusable and similar to WiFi perhaps, but worse than PCIe cards that should be <1ms.
it's all just driver/options crap if I were to take a guess.
there are a lot of usb options that matter, and tp-link ships lots of realtek chipsets that require very special driver incantations that a lot of the linux drivers simply don't replicate.
two+ layers of bad options will surely add 4ms quick.
I think there's definitely something with that specific setup. For me, pinging between two cheap Realtek 2.5 GbE USB dongles (one is on a Mac one is on a 7 year old Intel Atom Synology) is still sub-ms (hovering around 0.7-0.8ms) so it's not an inherent problem to USB dongles.
USB itself can have a lot of issues anywhere in the chain. I have a Thunderbolt dock where half of the USB ports adds latency and reduced throughput just because the USB chipset that powers them is terrible (it has two separate USB chipsets from different brands). Switch to a different port on the exact same dock and it's fine.
Does this manufacturer's practice pattern of repackaging data center components (e.g. Mellanox) imply any up and coming product creation opportunities?
That is really cool to read. And here I am, still running my home network on a measly 1Gbit Ethernet. I considered upgrading, but the equipment power consumption even when idle makes it an expensive proposition to consider just for fun.
Muscle memory for folks who have been doing it since before -i was an option. I still instinctively type `sudo su -` because it worked consistently on older deployments. When you have to operate a fleet of varying ages and distributions, you tend to quickly learn [if only out of frustration] what works everywhere vs only on the newer stuff.
`sudo su - <user>` also seems easier for me to type than `sudo -i -u <user>`
Thanks! Have you tried the boltctl/rescan setup I mentioned in the post? It should get you going, as long as your Thunderbolt/USB4 setup is correct.
If you're using an adapter card to add Thunderbolt functionality, then your mainboard needs to support that, and the card must be connected to a PCIe bus that's wired to the Intel PCH, not to the CPU.
Yes, rescan, re-enroll too. But it still shows as disconnected.
I don't know if the firmware is completely incompatible, but it is weird that under windows works and in Linux doesn't
It's embedded in the laptop (a HP ZBook from work). Disconnected as in network. Laptop charges, but signal doesn't work. With Thunderbolt 3 devices, it works. (The card itself is T4).
Ha. I got one of the 10G Thunderbolt adapters a several years ago. And eventually started having problems with Zoom calls around noon. With dropped connections and stuttering. Zoom restarts usually fixed the problem.
After it happened 3-4 times, I started debugging. It turned out that we usually get at least a bit of sunlight around noon, as it burns away the morning clouds. And my Thunderbolt box was in direct sunlight, and eventually started overheating.
And a Zoom restart made it fall back onto the Wifi connection instead of wired.
I fixed that by adding a small USB-powered fan to the Thunderbolt box as a temporary workaround. I just realized that it's been like this for the last 3 years: https://pics.ealex.net/s/overheat
Please forgive me for my ignorance, but are there currently any ways of being able to write data down at that speed? I see 2026 PCIe 5.0 NVMe advertising theoretical 14gb/s but not sure how feasible even that is.
PCIe 5 drives max out at about 14 GBps (bytes), which is ~112 gbps (bits).
Displayport 2.1 UHBR20 is 80 gbps.
USB4 maxes out at 80 gbps.
As you can see, 1gbps ethernet is starting to look like stone age technology. 2.5gbps becoming the next step seems a bit strange when we were jumping orders of magnitude every few years before. But also, ethernet tends to be used on longer cables than DP or USB, and trying to push it much faster results in exponentially increasing losses to resistance and radiation, the cable starts acting like an antenna even with the twisted pairs. Fiber optics are much better suited to high speed long distance, but too expensive and fragile for consumer use.
> the biggest downside of the PX adapter is that it gets really hot, like not touchable hot. Sometimes, either the network connection silently disappeared or (sadly) my Mac crashed with a kernel panic in the network driver. Apple has assured me that this was not a security issue. Other than that, the PX seems to do the job.
> All other 25 GbE adapter solutions I’ve found so far ... have a spinning fan. ... the biggest downside of the PX adapter is that it gets really hot, like not touchable hot. Sometimes, either the network connection silently disappeared or (sadly) my Mac crashed with a kernel panic in the network driver. ... Other than that, the PX seems to do the job
I don't know about the Ethernet part but it bothers me that even wifi has become faster than the wired USB port on our phones.
All I want to do is copy over all the photos and videos from my phone to my computer but I have to baby sit the process and think whether I want to skip or retry a failed copy. And it is so slow. USB 2.0 slow. I guess everybody has given up on the idea of saving their photos and videos over USB?
Wifi is fast but the latency is terrible and the reliability is even worse. It can go up and down like a yo-yo. USB is far more predictable even if it is a bit slower.
I have a cluster of 4 RPi Zero Ws and network reliability is not great. Since it is for the chaos, it’s fine, but it’s very common to have a node be offline at any given time.
Even worse, the control plane is exposed, but for something that runs 3 Hercules mainframe emulation and two Altairs with MP/M, it’s fine.
Not sure why this happens to you. I have HA with several dozens WiFi devices and I have only 2 devices (one relay, one sensor) that disconnect regularly, they have both poor WiFi signal, one in a basement and one far from the AP. Almost all are on 2.4 GHz, not by choice, but they work well.
I feel like this is an artifact from the late 2010s when the talk was of removing the port completely from phones, where that was being touted alongside swapping speakers with haptic screen audio as a way to make them completely waterproof.
As wireless charging never quite reached the level hoped – see AirPower – and Google/Apple seemingly bought and never did anything with a bunch of haptic audio startups, I figure that idea died....but they never cared enough to make sure the USB port remained top end.
I'd usually be against losing ports and user serviceable stuff but if the device could actually be properly sealed up (ie no speakers, mics, charge ports, etc) that would be legitimately useful.
If the photos on the phone are visible as files on a mounted filesystem, you can use rsync to copy them. If the connection drops but recovers by itself, you can put rsync inside a while true loop until it’s doing nothing.
I’m using Dropbox for syncing photos from phone to Linux laptop, and mounting the SDcard locally for cameras, so this is a guess.
> but I have to baby sit the process and think whether I want to skip or retry a failed copy
Do you import originals or do you have the "most compatible" setting turned on?
I always assumed apple simply hated people that use windows/linux desktops so the occasional broken file was caused by the driver being sort-of working and if people complain, well, they can fuck off and pay for icloud or a mac. After upgrading to 15 pro which has 10 gbps usb-c it still took forever to import photos and the occasional broken photos kept happening, and after some research it turns out that the speed was limited by the phone converting the .heic originals into .jpg when transferring to a desktop. Not only does it limit the speed, it also degrades the quality of the photos and deletes a bunch of metadata.
After changing the setting to export original files the transfer is much faster and I haven’t had a single broken file / video. The files are also higher quality and lower filesize, although .heic is fairly computationally-demanding.
Idk about Android but I suspect it might have a similar behavior
Wouldn’t this be useful for clustering Macs over TB5? Wasn’t the maximum bandwidth over USB-cables 5Gbps? With a switch, you could cluster more than just 4 Mac Studios and have a couple terabytes for very large models to work with.
I was hoping somebody would suggest that (and eventually try it out).
With TB5, and deep pockets, you might probably also benchmark it against a setup with dedicated TB5 enclosures (e.g., Mercury Helios 5S).
TB5 has PCIe 4.0 x4 instead of PCIe 3.0 x4 -- that should give you 50 GbE half-duplex instead of 25 GbE. You would need a different network card though (ConnectX-5, for example).
Pragmatically though, you could also aggregate (bond) multiple 25 GbE network card ports (with Mac Studio, you have up to 6 Thunderbolt buses, so more than enough to saturate a 100GbE connection).
Oddly enough, that’s exactly what I’ve been benchmarking - different ways of linking Strix Halo machines - with respect to throughput & latency.
Posted a little bit re: the TB side of things on the Framework and Level1Techs forums but haven’t pulled everything together yet because the higher-speed Ethernet and Infiniband data is still being collected.
So far my observations re: TB is that, on Strix Halo specifically, while latency can be excellent there seem to be some limits on throughput. My tests cap out at ~11Gbps unidir (Tx|Rx), ~22Gbps bidi (Tx+Rx). Which is wierd because the USB4 ports are advertised at 40Gbps bidi, the links report as 2x20Gbs, and are stable with no errors/flapping - so not a cabling problem.
The issue seems rather specific to TB networking on Strix Halo using the USB4 links between machines.
Emphasis to exclude common exceptions - other platforms eg Intel users getting well over 20Gbps; other mini PCs eg MS-1 Max USB4v2; local network eg I’ve measured loopback >100Gbps; or external storage where folk are seeing 18Gbps+ / numbers that align with their devices.
Emd goal is to get hard data on all reasonably achievable link types. Already have data on TB & lower-speed Ethernet (switched & P2P), currently doing setup & tuning on some Mellanox cards to collect data for higher-speed Ethernet and IB. P2P-only for now; 100GbE switching is becoming mainstream but IB switches are still rather nutty.
Happy to collaborate with any other folk interested in this topic. Reach out to (username at pm dot me).
This is an outstanding blog post. Initially, the title did little to captivate me, but the blog post was so well written that I got nerd-sniped. Who knew this little adapter was so fascinating! I wonder if the manufacturer is buying the Mellanox cards used from data center tear-downs. The author claims they can be had for only 20 USD online. That seems too good to be true!
Small thing: I just checked Amazon.com: https://www.amazon.com/s?k=thunderbolt+25G&crid=2RHL4ZJL96Z9...
I cannot find anything for less than 285 USD. The blog post gave a price of 174 USD. I have no reason to disbelieve the author, but a bummer to see the current price is 110 USD more!
Thank you!
I think, tragically, the blog post has caused this price increase.
The offers on Amazon are most likely all drop shippers trying to gauge a price that works for them.
You might have better luck ordering directly from China for a fraction of the price: https://detail.1688.com/offer/836680468489.html
AliExpress sells them, too, which is much more accessible for anyone living in the EU.
> The author claims they can be had for only 20 USD online. That seems too good to be true!
In my experience, the cheap eBay MLX cards are DellEMC/HPE/etc OEM cards. However I also encountered zero problems cross-flashing those cards back to generic Mellanox firmware. I'm running several of those cross-flashed CX-4 Lx cards going on six or seven years now and they've been totally bulletproof.
I saw the blog post last week and immediately bought the last one on that Amazon listing for the original price... hopefully they restock soon!
I'm going to try a couple other fan assisted cooling options, as I'd like to keep the setup reasonably compact.
I just ran fiber to my desk and I have a more expensive QNAP unit that does 10G SFP+, but this will let me max out the connection to my NAS.
Be sure to test this adapter on iPad Pro, just for kicks (yes it works!)
Although I managed to panic the kernel a couple of times without the extra heatsinks on...
Have you tried eBay? I bought my Mellanox Connect X3 10 Gb cards off there for like $25 years back to build a 10Gb network at home with a Mikrotik.
I believe the author is talking about the OCP (2.0) network card itself, that these adapters internally. The OCP nics are quite cheap compared to pcie - here’s 100GBE for 100! https://ebay.us/m/HMQAph
This 100GbE card is an OCP 2.0 type 2 adapter, which will _probably_ not work with the PX PCB since that NIC has two of these mezzanine connectors, and PX only one.
What also may not work are Dell rNDC cards. They look like they have OCP 2.0 type 1 connectors, but may not quite fit (please correct me if I'm wrong). They do however have a nice cooling solution, which could be retrofitted to one of the OCP 2.0 cards.
I've also ordered a Chelsio T6225-OCP cards out of curiosity. These should fit in the PX adapter but require a 3rd-party driver on macOS (which then supports jumbo frames, etc.)
What also fits physically is a Broadcom BCM957304M3040C, but there are no drivers on macOS, and I couldn't get the firmware updated on Linux either.
That’s a good point to note! I think the stacking height would matter, but in theory the single connector is still 8x pcie and should link without the upper 8x lanes connected.
Spec for reference, I’m not 100% sure. https://docs.nvidia.com/nvidia-connectx-5-ethernet-adapter-c...
you can get a 100Gb normal pcie card like a MCX416A for less than $100 if you're willing to flash them
Not much to add here but wanted to agree, this is post was actually hacker news (tm)
You can say that again, classic HN post with Jeff confessing to have bought literally the last adapter with the original posted price.
$285 is still an AMAZING price for 25GbE ethernet over TB4. I paid $200 for the Sonnet TB4 10GbE adapter.
Ha! Been running these for years on both linux and windows (on lenovo x1 laptops). Using cheap chinese thunderbolt-to-nvme adapters + nvme-to-pcie boards + mellanox cx4 cards (recently got one cx5 and a solarflare x2).
Pic of a previous cx3 (10 gig on tb3) setup: https://habrastorage.org/r/w780/getpro/habr/upload_files/d3c...
10gig can saturate full speed, 25G in my experience rarely reaches same 20G as the author observed.
If you don’t mind me asking, what are you using these for? Saturating these seems like it would have reasonably few workloads outside of like cdn or multi-tenant scenarios. Curious what my lack of imagination is hiding here.
Officially: to access NAS, get raw market-data files (tens to hundreds of gigabytes a day), not needed on laptop every day, but only once in a while to fix or analyze something.
Really: because I can, and it is fun. I upgraded my home lan to 10G, because used 10G hardware is cheap (and now 25G enters the same price range).
"because I can, and it is fun." The best answer! I am most of the way done with upgrading most of my homelab to 100G from 10G, but there really isn't a practical reason for it. 100G has dropped in price so much as datacenters are all about 400/800G now.
Nice! Cool to hear from a fellow admirer of overkill-lan setups ;)
Which cards do you prefer for 100G, and what is the situation with dacs/optics?
I'm using ConnectX5s for most, and some ConnectX4s in the older servers. Both of those cards have really come down in price in the used/ebay market. I have been playing around with some different optics - I have a bunch of CWDM4s which are very inexpensive and use a single SingleMode pair.... but of course they run hot so if you have them in servers without good air flow you might have problems.
I'm using mostly fiber just because the servers are connected to Cisco 9305 with 72 100g ports.
Thanks! That cisco sounds like a hell of equipment. Is it loud? And how much power does it draw?
And thanks for pointing at CWDM4, these are quite cheap on ebay now
Oh they are both loud and eat a ton of power. I think the 9305 is at least 800 watts at idle. That is the biggest downside of the retired datacenter gear... yo u really need a dedicated room with power, cooling, and sound isolation.
I wouldn't really call 100Gbit overkill, if you compare it to modern disk drives is about where we should be relative to shared storage/NAS/etc infrastructure people used to run. So yes, being able to share my /home directory across a few dozen machines at my house without a huge perf impact vs using a local drive seems a pretty reasonable use case. Sure its faster than my WAN access, but who cares?
Frankly, 10Gbit is fully 25 years old with, 10GbaseT being 20 years old this year.
Thats ridiculously ancient technology. There is/was a 25/40GbaseT spec too (now 10 years old), which basically no one implemented because like ECC ram (and tape drives, and seem to be trying to do it with harddrives and GPUs) the MBA's have taken over parts of the computer industry and decided that they can milk huge profit margins from technologies which are incrementally more difficult because smaller users just don't matter to their bottom lines. The only reason those MBAs are allowing us to have it now, is because a pretty decent percentage of us can now get 5Gbit+ internet access and our wifi routers can do 1Gbit+ wireless, and the weak link is being able to attach the two.
I did a bit of back of the napkin math/simulation about a possible variable rate Ethernet (ex like NBbaseT, where it has multiple speeds and selects faster one based on line conditions), and concluded that 80+Gbit using modern PHY/DSP's and high symbol rate, multiple bands, techology which is dirt cheap thanks to wifi/bt/etc on fairly short cable distances (ex 30-50M) on CAT8 is entirely possible. And this isn't even fantasy, short cat7 runs are an entire diffrent ballpark from a phone pair, and these days mg.fast/etc have shown 10Gbit+ over that junk.
Agreed - the big thing is 100g is much much cheaper now as so much 100g gear is coming out of datacenters. So many of those older ConnectX4s and 5s, plus lots of switches and optics. 100g really is the new 10g for homelabs.
I do media production, and sometimes move giant files (like ggufs) around my network, so 25 Gbps is more useful than 10 Gbps, if it's no too expensive.
[flagged]
Note that you can do point-to-point network links directly with thunderbolt (and usb4).
https://support.apple.com/guide/mac-help/ip-thunderbolt-conn... etc
Yes! However, I got around 15 Gbps with a Thunderbolt-only setup (TB3/TB4) = only 75% of the Ethernet setup.
You'd also mostly be limited to short cables (1-2m) and a ring topology.
Any idea why that would be the case?
The linux implementation is quite poor. Among other issues, your answer is linux treats it as a TCP/IP link- so packets don't have any offloading for checksums, etc. it's also incomplete (ex- you have to physically unplug and replug the cord every time one side loses link- even a reboot).
This is my firsthand experence trying to get some tablet motherboards to link up and work as a proxmox cluster w/ TB3 as the link between nodes.
I have a 3 node proxmox setup on MS-01s using a 25G Thunderbolt ring for Ceph, and indeed it took a lot of hoops to get it working correctly and reliably. I did manage to get it such that nodes can go up and down without needing to unplug anything, and the dynamic routing works if a node disappears. Performance is pretty good, with a more realistic 20ish gbit/sec.
AWESOME!! Do you mind linking references? I put the project aside because of these issues.
Yea I'll dig up the link I used - there was a great reference about getting the thunderbolt working after reboots/etc.
I figured this was Jeff’s CTO Laboratory. I enjoy your channel. Are you kicking yourself for pulling the trigger on MS-01s now that MS-02 Ultras released?
Yea! Those MS-O2s look great, so I may need to upgrade! I did get a couple of DGX Sparks to play with.
Thunderbolt is basically external PCIe, so this is not so surprising. High speed NICs do consume a relatively large amount of power. I have a feeling I've seen that logo on the board before.
I don't know how to measure the direct power impact on a MacBook Pro (since it's got a battery), but the typical power consumption of these cards is 9 W, not much more than Aquantia 10 GBit cards.
Also, if you remember where you saw that logo, please let me know!
JFYI, for measuring power draw, you might be able to use `macmon`[0] to see the total system power consumption. The values reported by the internal current sensor seem to be quite accurate.
[0] https://github.com/vladkens/macmon
Speaking of hardware, the RTL8159 (10Gbps) hit the market late last year and is said to consume only about 2–3W. It apparently runs very cool compared to older chips. (Though it would need to be bonded to reach 25Gbps ;-)
I got me one of these adapters (RTL8127AF TXA403, with SFP+ cage); I haven't properly benchmarked it yet.
There's no driver support on macOS, and for Linux you'd need a bleeding edge kernel. Just trying to physically connect it (along with a connected SFP28 transceiver) to my Mac's Thunderbolt port using an external PCIe-to-TB adapter, macmon tells me a power draw of around 4.3 W, so it's not significantly less for half the bandwidth, but the card doesn't get hot at all.
Very nice tip, thank you!
I measure around +11W idle. While running a speed test, I read ca. +15W.
Thanks for the measurements! 15W under load definitely justifies those massive heatsinks.
I’m looking forward to your writeup on the RTL8127AF as well. Your blog is awesome!
Plus 1-2.5w per active cable. You need the heatsinks as the cx4 cards expect active airflow, and active transceivers as well.
I have a 10gbit dual port card in a Lenovo mini pc. There is no normal way to get any heat out of there so I put a 12v small radial fan in there as support. It works great at 5v: silent and cool. It is a fan though so might not suit your purpose.
Do you mean active Thunderbolt cable? Short Thunderbolt cables (0.8m) are passive.
https://www.reddit.com/r/UsbCHardware/comments/y5uokj/commen...
Active Ethernet cables.
The PCI-E logo or the “octopus in a chip” logo? I’m more interested in the latter.
The octopus logo. Might've been on Aliexpess or one of the other Chinese marketplaces.
It isn't. There is no sense in which "Thunderbolt is basically external PCIe". Thunderbolt provides a means of encapsulating PCIe over its native protocols, which puts PCIe on the same footing as other encapsulated things like DisplayPort and, for TB4 and later, USB.
I'm surprised you are only getting 20gbit/s. I did not expect PCIe to be be the limiting factor here. I've got a 100gbit cx4 card currently in a PCIe3 X4 slot (for reasons, don't judge) and it easily maxes that out. I would have expected the 25g cx4 cards to be at least able to get everything out of it. RDMA is required to achieve that in a useful way though.
Edit: forgot is isn't "true" PCIe but tunneled.
The limitation is Thunderbolt (32 Gbps theoretical limit for PCIe 3 tunneling).
Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.
The new low-power Realtek chipsets will definitely push 10 GbE forward because the chipset won't be much more expensive to integrate and run than the 2.5Gbps packages.
It all comes down to performance per Watt, the availability of cheap switching gear, and the actual utility in an office / home environment.
For 10 Gbps, cabling can be an issue. Existing "RJ45"-style Cat 6 cables could still work, but maybe not all of them.
Higher speeds will most likely demand a switch to fiber (for anything longer than a few meters) or Twinax DAC (for inter-device connects). Since Wifi already provides higher speeds, one may be inclined to upgrade just for that (because at some point, Wireless becomes Wired, too).
That change comes with the complexity of running new cabling, fiber splicing, worrying about different connectors (SFP+, SFP28, SFP56, QSFP28, ...), incompatible transceiver certifications, vendor lock-in, etc. Not a problem in the datacenter, but try to explain this to a layman.
Lastly, without a faster pipe to the Internet, what can you do other than NAS and AI? The computers will still get faster chips but most folks won't be able to make use of the bandwidth because they're still stuck on 1Gbps Internet or less.
But that will change. Swiss Init7 has shown that 25GBps Internet at home is not only feasible but also affordable, and China seems to be adding lots of 10G, and fiber in general.
Fun times ahead.
But why is the gear progressing so very slowly? Why a 25 year gap between reasonable power 1Gbps and 10Gbps?
And while not every cat6 will do 10, it would still be worth a shot, and devices aren't using 5 instead they're using even less.
Not to mention that cat8 will happily do 40Gbps as long as you can get from your switch to your end devices in 30 meters.
Perhaps because internet speeds have been under 1gbps most places until recently and the average person doesn't care about file transfers over LAN.
> Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
wifi is not faster.
However ethernet is not as critical as it used to be, even at the office. People like the conveniency of having laptops they can move around. Unless you are working from home, having a dedicated office space is now seen as a waste of space. If the speed of the wifi is good enough when you are in a meeting room or in your kitchen, there is no reason to plug your laptop when you move back in another place, especially if most connections are to the internet and not the local network. In the workplace, most NAS have been replaced by onedrive / gdrive, at home NAS use has always been limited to a niche population: nerds/techies, photographers, music or video producers...
PCI-E lanes for consumers. Gigabit would saturate the PCI bus, but once you're on PCI-E you only need to give it 1 lane, usually off the chipset.
Servers had a reason to spend for the 10G, 25G and 40G cards which used 4 lanes.
There are 10 Gigabit chips that can run off of one PCI-E 4.0 lane now and the 2.5G and 5G speeds are supported(802.3bz).
We have 400Gbe which is certainly faster than USB.. but;
On consumer devices, I think part of the issue is that we’re still wedded to four-pair twisted copper as the physical medium. That worked well for Gigabit Ethernet, but once you push to 5 or 10 Gb/s it becomes inherently expensive. Twisted pair is simply a poor medium at those data rates, so you end up needing a large amount of complex silicon to compensate for attenuation, crosstalk, and noise.
That's doable but the double whammy is that most people use the network for 'internet' and 1G is simply more than enough, 10G therefore becomes quite niche so there's no enormous volume to overcome the inherent issues at low cost.
Wireless happened, I'd think. People started using wifi and cellular data for everything, so applications had to adapt to this lowest common denominator, and consumer broadband demand for faster-than-wifi speeds isn't there. Plus operators put all their money into cellular infra leaving no money to update broadband infra.
Wifi now can pretty realistically beat 2.5gbit/s while most Ethernet is still gigabit. It just seems strange to live in a world where the average laptop will get a faster connection speed over wifi than plugged in to Ethernet.
Ethernet is following suit to 2.5G which is otherwise a nonsensical step for ethernet speeds, I think this is further evidence that everything just follows wifi now.
10Gbase-T requires a lot of transistors and power (maybe over 10x more than 1G) so it just wasn't worth the cost.
Ethernet did not stagnate. Ethernet on UTP did stagnate due to reaching the limits of the technology, but Ethernet continues to advance over fiber.
For 10 Gbps I find it simpler and cheaper to use fiber or DACs, but motherboards don't provide SFP+, only RJ45 ports. Over 10 Gbps copper is a no go. SFP28 and above would be nice to have on motherboards, but that's a dream with almost zero chances to happen. For most people RJ45 + WiFi 7 is good enough, computer manufacturers will not put SFP+ or SFP28 for a small minority of people.
> Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
Practically spoken, a lot of the transfer speed advertised by wifi is marketing hogwash barely backed by reality, especially in congested environments.
> Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.
For both laptops and desktops, PCI lanes. Intel doesn't provide many lanes, so manufacturers don't want to waste valuable lanes permanently for capabilities most people don't ever need.
For laptops in particular, power draw. The faster you push copper, the more power you need. And laptops have even less PCIe lanes available to waste.
For desktops, it's a question of market demand. Again - most applications don't need ultra high transfer rate, most household connectivity is DSL and (G)PON so 1 GBit/s is enough to max out the uplink. And those few users that do need higher transfer rates can always install a PCIe card, especially as there is a multitude of different options to provide high bandwidth connectivity.
> Practically spoken, a lot of the transfer speed advertised by wifi is marketing hogwash barely backed by reality, especially in congested environments.
Yes but a hogwash of several gigabits sometimes does give you real-world performance of more than a gigabit.
> Intel doesn't provide many lanes, so manufacturers don't want to waste valuable lanes permanently for capabilities most people don't ever need.
It's been a bunch of years that a single lane could do 10Gbps, and a bunch more years that a single lane could do 5Gbps.
Also don't ethernet ports tend to be fed by the chipset? So they don't really take lanes.
Neat, but the thermal design is absolutely terrible. Sticking that heatsink inside the aluminum case without any air circulation is awful.
Yeah, it's because the network card adapter's heatsink is sandwiched between two PCBs. Not great, not terrible, works for me.
The placement is mostly determined by the design of the OCP 2.0 connector. OCP 3.0 has a connector at the short edge of the card, which allows exposing/extending the heat sink directly to the outer case.
If somebody has the talent, designing a Thunderbolt 5 adapter for OCP 3.0 cards could be a worthwhile project.
A Flex PCB connecting the OCP2 connector would allow to put the converter board behind the NIC board, allowing the NIC board to be exposed to the aluminum case to use the case itself as a heatsink (would need a split case so the NIC board can be screwed to one side of the case, pressing the main chip against it via a thermal pad).
As a stop-gap, I'd see if there was any way to get airflow into the case - I'd expect even a tiny fan would do much more than those two large heatsinks stuck onto the case (since the case itself has no thermal connection to the chip heatsink).
My goal was to get a fanless setup (for a quiet office).
If that's not a requirement just get the Raiden Digit Light One, which does have a fan (and otherwise the same network card).
If I could design an adapter PCB myself, I would go straight to OCP 3.0, which allows for a much simpler construction, and TB5 speeds.
Alternatively, there are DELL CX422A rNDC cards (R887V) that appear to have an OCP 2.0 connector but a better heatsink design.
I'd be more worried about cooling the transceivers properly.
My optical transceiver gets to around 52 °C (measured via IR camera), well below its design limit, so that's not bad.
If truly concerned, one could use SFP28 to SFP28 cage adapters to have the heat outside the case, and slap on some extra heatsinks there.
I've had a lot of problems with even 10GbE via Thunderbolt 3/4. Bandwidth-wise it works fine, but latency and jitter are issues. This means that stuff like high-speed cameras that need to be synchronized over Ethernet using Precision Time Protocol (PTP) tend to simply fail with these devices.
I’d heard similar complaints re: TB networking latency & jitter. Did some investigations and tuning on a pair of machines with USB4 ports connected via short TB5-rated cables. Eventually got the thunderbolt links to consistently beat the ether ones on both latency and jitter. And not just switched Ethernet either - even a direct Ethernet P2P link lost out to TB, though the difference there was small.
Interesting - I'll try some other methods
I used to have an SFP28 Mellanox card in my home server, but went back to a simple 2.5G Ethernet port for the LAN side. The Mellanox card ran hot and needed an extra fan near it to dissipate the heat. It was cool but there was no real benefit other than occasionally when transferring some large files.
Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors like the one presented here are unreliable and add several ms of latency.
> Until motherboards include SFP ports […]
A micro-ATX motherboard with on-board 2xSFP28 (Intel E810):
* https://download-2.msi.com/archive/mnu_exe/server/D3052-data...
* https://www.techradar.com/pro/this-amd-motherboard-has-a-uni...
For reference, I'm seeing pings from my Mac to my Linux boxes (Lenovo Tiny5) at well under 1ms, not much worse than between them directly. But yeah, your mileage may vary.
Yep, these cards need a fan (or any kind of directed air flow).
Where did you get "several ms of latency" figure from? I have not measured external card, but may be I should do it... Because cards themselves have latency in range of microseconds, not millis.
I haven't tested this particular Thunderbolt SFP adapter, but my experience with a TP-Link 1Gbps USB adapter is that it adds about 4ms of latency. Far from being unusable and similar to WiFi perhaps, but worse than PCIe cards that should be <1ms.
it's all just driver/options crap if I were to take a guess.
there are a lot of usb options that matter, and tp-link ships lots of realtek chipsets that require very special driver incantations that a lot of the linux drivers simply don't replicate.
two+ layers of bad options will surely add 4ms quick.
I think there's definitely something with that specific setup. For me, pinging between two cheap Realtek 2.5 GbE USB dongles (one is on a Mac one is on a 7 year old Intel Atom Synology) is still sub-ms (hovering around 0.7-0.8ms) so it's not an inherent problem to USB dongles.
USB itself can have a lot of issues anywhere in the chain. I have a Thunderbolt dock where half of the USB ports adds latency and reduced throughput just because the USB chipset that powers them is terrible (it has two separate USB chipsets from different brands). Switch to a different port on the exact same dock and it's fine.
Does this manufacturer's practice pattern of repackaging data center components (e.g. Mellanox) imply any up and coming product creation opportunities?
That is really cool to read. And here I am, still running my home network on a measly 1Gbit Ethernet. I considered upgrading, but the equipment power consumption even when idle makes it an expensive proposition to consider just for fun.
nitpicking but why would someone type `sudo su` vs `sudo -i`
Muscle memory for folks who have been doing it since before -i was an option. I still instinctively type `sudo su -` because it worked consistently on older deployments. When you have to operate a fleet of varying ages and distributions, you tend to quickly learn [if only out of frustration] what works everywhere vs only on the newer stuff.
`sudo su - <user>` also seems easier for me to type than `sudo -i -u <user>`
I've mostly only ever seen `sudo su` in tutorials, so someone who's only familiar with the command through those is one possible reason why.
I still have issues under Linux (Kernel 6.14) and Thinderboldt 4 docking stations. The simply don't get recognised.
But this is a cool solution
Thanks! Have you tried the boltctl/rescan setup I mentioned in the post? It should get you going, as long as your Thunderbolt/USB4 setup is correct.
If you're using an adapter card to add Thunderbolt functionality, then your mainboard needs to support that, and the card must be connected to a PCIe bus that's wired to the Intel PCH, not to the CPU.
Yes, rescan, re-enroll too. But it still shows as disconnected. I don't know if the firmware is completely incompatible, but it is weird that under windows works and in Linux doesn't
Disconnected as in "network"? What PCIe card do you use? Can you update the firmware (maybe from Windows)?
Also check the BIOS settings (try setting TB security to "No Security" or "User Authorization")
Some OEM Mellanox cards can be cross-flashed to NVIDIA's stock firmware, maybe that's also relevant.
It's embedded in the laptop (a HP ZBook from work). Disconnected as in network. Laptop charges, but signal doesn't work. With Thunderbolt 3 devices, it works. (The card itself is T4).
Ha. I got one of the 10G Thunderbolt adapters a several years ago. And eventually started having problems with Zoom calls around noon. With dropped connections and stuttering. Zoom restarts usually fixed the problem.
After it happened 3-4 times, I started debugging. It turned out that we usually get at least a bit of sunlight around noon, as it burns away the morning clouds. And my Thunderbolt box was in direct sunlight, and eventually started overheating.
And a Zoom restart made it fall back onto the Wifi connection instead of wired.
I fixed that by adding a small USB-powered fan to the Thunderbolt box as a temporary workaround. I just realized that it's been like this for the last 3 years: https://pics.ealex.net/s/overheat
As my old grandad said, there’s nothing so permanent as a temporary repair!
Please forgive me for my ignorance, but are there currently any ways of being able to write data down at that speed? I see 2026 PCIe 5.0 NVMe advertising theoretical 14gb/s but not sure how feasible even that is.
PCIe 5 drives max out at about 14 GBps (bytes), which is ~112 gbps (bits).
Displayport 2.1 UHBR20 is 80 gbps.
USB4 maxes out at 80 gbps.
As you can see, 1gbps ethernet is starting to look like stone age technology. 2.5gbps becoming the next step seems a bit strange when we were jumping orders of magnitude every few years before. But also, ethernet tends to be used on longer cables than DP or USB, and trying to push it much faster results in exponentially increasing losses to resistance and radiation, the cable starts acting like an antenna even with the twisted pairs. Fiber optics are much better suited to high speed long distance, but too expensive and fragile for consumer use.
25 gigabit/s is 3125 megabyte/s, the SSD in my 4 year old laptop can write at nearly 6000 megabyte/s.
> the biggest downside of the PX adapter is that it gets really hot, like not touchable hot. Sometimes, either the network connection silently disappeared or (sadly) my Mac crashed with a kernel panic in the network driver. Apple has assured me that this was not a security issue. Other than that, the PX seems to do the job.
Made me chuckle.
Is this satire?
> All other 25 GbE adapter solutions I’ve found so far ... have a spinning fan. ... the biggest downside of the PX adapter is that it gets really hot, like not touchable hot. Sometimes, either the network connection silently disappeared or (sadly) my Mac crashed with a kernel panic in the network driver. ... Other than that, the PX seems to do the job
Now I just have to contrive the circumstances where this is useful to me. :)
I don't know about the Ethernet part but it bothers me that even wifi has become faster than the wired USB port on our phones.
All I want to do is copy over all the photos and videos from my phone to my computer but I have to baby sit the process and think whether I want to skip or retry a failed copy. And it is so slow. USB 2.0 slow. I guess everybody has given up on the idea of saving their photos and videos over USB?
> USB 2.0 slow
Many phones indeed only support USB 2.0. For example the base iPhone 17. The Pro does support USB 3.2, however.
> I guess everybody has given up on the idea of saving their photos and videos over USB?
Correct.
Wifi is fast but the latency is terrible and the reliability is even worse. It can go up and down like a yo-yo. USB is far more predictable even if it is a bit slower.
I have a cluster of 4 RPi Zero Ws and network reliability is not great. Since it is for the chaos, it’s fine, but it’s very common to have a node be offline at any given time.
Even worse, the control plane is exposed, but for something that runs 3 Hercules mainframe emulation and two Altairs with MP/M, it’s fine.
I have a bunch of HA wifi connected sensors, I see them drop off and reconnect all the time it is most annoying.
Not sure why this happens to you. I have HA with several dozens WiFi devices and I have only 2 devices (one relay, one sensor) that disconnect regularly, they have both poor WiFi signal, one in a basement and one far from the AP. Almost all are on 2.4 GHz, not by choice, but they work well.
Why don't you get a phone with 3.0+ USB?
My last two phones in the last 4 years had at least USB 3.1
I feel like this is an artifact from the late 2010s when the talk was of removing the port completely from phones, where that was being touted alongside swapping speakers with haptic screen audio as a way to make them completely waterproof.
As wireless charging never quite reached the level hoped – see AirPower – and Google/Apple seemingly bought and never did anything with a bunch of haptic audio startups, I figure that idea died....but they never cared enough to make sure the USB port remained top end.
I'd usually be against losing ports and user serviceable stuff but if the device could actually be properly sealed up (ie no speakers, mics, charge ports, etc) that would be legitimately useful.
If the photos on the phone are visible as files on a mounted filesystem, you can use rsync to copy them. If the connection drops but recovers by itself, you can put rsync inside a while true loop until it’s doing nothing.
I’m using Dropbox for syncing photos from phone to Linux laptop, and mounting the SDcard locally for cameras, so this is a guess.
> but I have to baby sit the process and think whether I want to skip or retry a failed copy
Do you import originals or do you have the "most compatible" setting turned on?
I always assumed apple simply hated people that use windows/linux desktops so the occasional broken file was caused by the driver being sort-of working and if people complain, well, they can fuck off and pay for icloud or a mac. After upgrading to 15 pro which has 10 gbps usb-c it still took forever to import photos and the occasional broken photos kept happening, and after some research it turns out that the speed was limited by the phone converting the .heic originals into .jpg when transferring to a desktop. Not only does it limit the speed, it also degrades the quality of the photos and deletes a bunch of metadata.
After changing the setting to export original files the transfer is much faster and I haven’t had a single broken file / video. The files are also higher quality and lower filesize, although .heic is fairly computationally-demanding.
Idk about Android but I suspect it might have a similar behavior
> given up on the idea of saving their photos and videos over USB?
Until USB has monthly service business to compete with cloud storage revenue.
Wouldn’t this be useful for clustering Macs over TB5? Wasn’t the maximum bandwidth over USB-cables 5Gbps? With a switch, you could cluster more than just 4 Mac Studios and have a couple terabytes for very large models to work with.
I was hoping somebody would suggest that (and eventually try it out).
With TB5, and deep pockets, you might probably also benchmark it against a setup with dedicated TB5 enclosures (e.g., Mercury Helios 5S).
TB5 has PCIe 4.0 x4 instead of PCIe 3.0 x4 -- that should give you 50 GbE half-duplex instead of 25 GbE. You would need a different network card though (ConnectX-5, for example).
Pragmatically though, you could also aggregate (bond) multiple 25 GbE network card ports (with Mac Studio, you have up to 6 Thunderbolt buses, so more than enough to saturate a 100GbE connection).
Too bad Jeff Geerling returned his Mac Studios to Apple. Would be lovely to see how 5x faster RDMA impacts the performance.
25 Gbps isn't all that much. It would be good, but would be below the 40+ Gbps I was getting on the TB5 ring network.
I think where it would show more significant speed up is on the AMD Strix Halo cluster.
Except I haven't been able to get RDMA over Thunderbolt on there to work, so it'd be apples to oranges comparing ConnectX to Thunderbolt on Linux.
Oddly enough, that’s exactly what I’ve been benchmarking - different ways of linking Strix Halo machines - with respect to throughput & latency.
Posted a little bit re: the TB side of things on the Framework and Level1Techs forums but haven’t pulled everything together yet because the higher-speed Ethernet and Infiniband data is still being collected.
So far my observations re: TB is that, on Strix Halo specifically, while latency can be excellent there seem to be some limits on throughput. My tests cap out at ~11Gbps unidir (Tx|Rx), ~22Gbps bidi (Tx+Rx). Which is wierd because the USB4 ports are advertised at 40Gbps bidi, the links report as 2x20Gbs, and are stable with no errors/flapping - so not a cabling problem.
The issue seems rather specific to TB networking on Strix Halo using the USB4 links between machines.
Emphasis to exclude common exceptions - other platforms eg Intel users getting well over 20Gbps; other mini PCs eg MS-1 Max USB4v2; local network eg I’ve measured loopback >100Gbps; or external storage where folk are seeing 18Gbps+ / numbers that align with their devices.
Emd goal is to get hard data on all reasonably achievable link types. Already have data on TB & lower-speed Ethernet (switched & P2P), currently doing setup & tuning on some Mellanox cards to collect data for higher-speed Ethernet and IB. P2P-only for now; 100GbE switching is becoming mainstream but IB switches are still rather nutty.
Happy to collaborate with any other folk interested in this topic. Reach out to (username at pm dot me).
I recently did a complete disk backup/clone which only took 15 minutes instead of hours. Maxed the SSD which was being backed up at about 2.5GB/s.
rsync...grsync...a solution for broken partial batch transfers since forever
Remote Time Machine backups are snappier than ever before :)
Pretty much anywhere you have networked storage? Gigabit is about on-par with pre-sata ATA133.
Would be useful if I had to debug my internet link and I only had a laptop.
It also works on iPad Pro :)
Porn?
What kind of porn requires 25 gigabits ?
As a guess, large-scale volumetric or photogrammetric "datasets" could be difficult to stream over lesser interconnects.
A lot of porn.
Uncompressed 8k ones
Mind-blowing 3D stereoscopic 480 FPS with 11.2 audio channels with 768 kbps rate and spatial sound stage :D
> reduces temperatures by at least 15 Kelvin, bringing the ambient enclosure temperature below 40 °C,
I had to do a double-take when it mentioned Kelvin since That is physically impossible.
Isn't "reduces temperatures by 15 Kelvin" the same as "reduces temperatures by 15 Celsius"?
Yes, Kelvin is only a linear offset from Celsius. (273.15 for anyone who doesn’t already know).
It’s a little bit funny/coy to use it mixed with Celsius.
reduces temperatures by at least 15 Kelvin == the same as reduces temperatures by at least 15 Celcius.
It 'reduces it by' ... not reduces it TO