1. Add an IP that has been freed from another use to a CG-NAT Pool.
2. Get complaints from customers about being hard banned from things like Netflix, Sport Streaming and VPNs or other utilities.
3. Investigate, no IP reputation issues. Find some random GEO IP database that has a side business in selling lists of VPNS or other geo breakout tools. They have listed this IP for some random reason. Almost never nefarious.
4. Give it 3 weeks for the Geoip nard to update from the wrong classification (harmful) to some kind of also wrong but unharmful classification like "Datacentre"
5. Customers can stream The Witcher again. Yay.
Really while ipv6 should be a solution here, another very good solution would be the removal of such useless middlemen from the face of the earth.
As someone who read all books and played the third game: People should totally be banned from watching Netflix's Witcher.
On a more serious note, I used GeoIP when it was free and it was a godsend to reduce malicious connection attempts to my webserver without impacting 99 % of my "clients" (not paying customers).
These kind of services *are* helpful. Wehterh you should rely on them when you have millions of customers is a different story altogether.
>On a more serious note, I used GeoIP when it was free and it was a godsend to reduce malicious connection attempts to my webserver without impacting 99 % of my "clients" (not paying customers).
Yeah, I think the issue is that, if you have 30 paying customers with, lets say, a canadian ip address and a canadian credit card, and then one bloke connects from the UK by that ip address, its daft that you just blacklist the ip and take all those customers offline. Personally I would start with a polite notice to the ISP, as its likely someones running a proxy and they should theoretically be as interested to remove it as you are.
Few if any of them permit outside contact. I remember having to get a customer to escalate their PSN case after removing the GEOIP configuration because PSN didnt reflect the change immediately. Its really difficult from a customer service perspective, because the error messages tend to pin the blame on the ISP, despite it wholly being a circumstance of the content owner.
You do get lucky, IIRC theres a few blog posts where people go into detail about how to contact some providers, but I found that some of them rotated those emails pretty regularly.
I use cloudflare to make my weather station available over T-Mobile. They don’t filter inbound ipv6 on regular phone lines (they do for TMHI) so you can host a simple page on ipv6, only set the AAAA record in cloudflare, and they will proxy it for ipv4 users so I can ignore being CGNAT’d for ipv4. Make sure if you do this setup with a tool like ddclient to keep the record current as T-mobile rotates ipv6 frequently
> Isn’t essentially the entire US on CG-NAT for IPv4 on mobile data?
T-Mobile, for one, has had their handsets IPv6-only for a few years, so if your Android/iOS does a DNS lookup and gets an AAAA record back, it will skip CG-NAT. T-Mobile presenting at NANOG in 2018 on IPv6:
> Chances are if you use the Internet on your smartphone, you are connecting via IPv6. According to the Internet Society’s 2018 State of IPv6 Deployment,[1] 80% of smartphones in the US on the major cellular network operators use IPv6 and major mobile networks are driving IPv6 adoption with Verizon Wireless at 84%, Sprint at 70%, T-Mobile USA at 93%, and AT&T Wireless at 57%. Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs.
Though some folks aren't happy with the implementation:
> But from my own experience, neither T-Mobile nor AT&T allows inbound traffic to the phone's IPv6 address. This negates some of the advantages of having a globally routable IPv6 address.
> Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs.
I think this has also already happened. You don't really notice it as a user, because the phone itself can act as a 4-to-6 translator (similar to how DS-Lite in DOCSIS transports CG-NATted v4 traffic in v6) for both native apps doing incompatible things with literal v4 addresses and tethering clients.
I’ve had IPv4 CG-NAT on mobile LTE since ever and for a decade on residential cable in europe. Cloudflare is being lazy, Google too. I get served “Captacha’s” at least 10 times a day.
Even when i have IPv6 assigned, iOS and macOS seem to prefer A TXT RR and proceed just using IPv4 almost always. On LAN mDNS link-local IPv6 is always prefered.
They need to come up with an ip solution that is useful enough that people actually want to upgrade to it.
When you compare it to other technologies like https, tls1.3, unicode, 5g cellular, wifi 6, wifi 5 or bluetooth versions, etc. It’s clear that ipv6 adoption is not what it should be if they launched a protocol with clearer benefits to the end user.
> It’s clear that ipv6 adoption is not what it should be if they launched a protocol with clearer benefits to the end user.
The "end user" has no idea about TLS 1.3 or many other things. It's the techies that work behind the scenes that make the changes 'on behalf' of everyone else.
And IPv6 traffic is, according to Google, the majority of traffic it sees in many countries (including the US at >52%):
The 'real' holdouts are enterprise companies and corporate networks as evidenced by the fact that IPv6 usage goes up on weekends (i.e., when most people aren't at work on said corporate networks). See also:
> Chances are if you use the Internet on your smartphone, you are connecting via IPv6. According to the Internet Society’s 2018 State of IPv6 Deployment,[1] 80% of smartphones in the US on the major cellular network operators use IPv6 and major mobile networks are driving IPv6 adoption with Verizon Wireless at 84%, Sprint at 70%, T-Mobile USA at 93%, and AT&T Wireless at 57%. Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs.
Being able to connect your smartphone to the Internet seems like a clear benefit to the end user IMHO. Would hate to see what every mobile phone being behind CG-NAT would be like.
I think smartphones are a special case, because they generally cannot run public facing services that open up ports and are specifically designed to be hardened for ipv6 and designed to work in conjunction with the carrier's ipv6 firewall/network.
Compare that to a home network where, printers are shared, iot devices have open ports, computers and nas share drives. IPv6 may address the needs of cellular carriers and devices, but it doesn't adequately address the needs of small local networks connecting to the internet.
Yeah, IPv6 is heavily tuned to the needs of the large-scale network operators, and is actively worse for the regular user and small networks.
From user/small admin standpoint, the goal is to re-use as much admin knowledge as possible - and what's on the wire does not really matter. So the ideal IPv4 upgrade _for users_ is IPv4 with larger addresses, but otherwise behaving identically. Ideally all the admin tooling stays the same, and the software needs changing some struct names, and tweaking IP regex. And sure, it'll all be different on the wire and all the OS'es need to be upgraded - but that is not a problem, consumer OS'es live only for a few years anyway.
From large network operator standpoint, the goal is improve efficiency of the huge networks. So lets eliminate NAT everywhere, completely redo host addressing, get rid of DHCP, and so on - redesign everything from scratch so it's "better". Sure, it's a huge learning curve but they have departments full of network engineers, they can do it. They are not some part-time sysadmins who just want their network to keep functioning.
I have MultiWAN on OPNsense. My PC IP is always 192.168.0.12. My router decides which upstream it should go. If I go full IPv6, router should derive double IPv6 from both WANs and if main upstream goes down, stop advertising IPv6 from main upstream. Or stop advertising gateway. I don't know what is the right IPv6 way of doing MultiWAN.
Not only PC may change IP, but also servers. Legacy IPv4 DNS can be extended to IPv6, but that mechanical action is not flexible enough. With IPv6 we need to be able to mass replace IPv6 /64 prefix leaving all suffixes intact. We probably need /64 prefix alias system. Software is not prepared for this. In IPv4 SNAT and DNAT were being these "aliases". If NAT is not an option anymore, then DNS must step in.
For many server software it just not possible to listen on multiple IPv6 address. Last time I tried MySQL, it just could not listen on multiple addresses. I could not make it listen on IPv4 and IPv6, specifying two addresses. MySQL server wanted just one address. This address could be [::], which means all interfaces and all protocols. And Linux implements some stupid hack to accept IPv4 connections to IPv6 socket. And Windows Vista also adopted this brainrot. But this is all wrong. Servers have to learn to listen to multiple IPs. This is normal. And for good IPv6 servers should learn to not only listen on multiple IPs, most wanted multiple IPv6, but also rebind listeners on the fly. If I got disconnected from ISP, reconnected by DHCPv6, and ISP assigned another IPv6 prefix, then DynDNS should update all my zones to new /64 prefix, and all servers in my network should rebind listeners.
Or else we may abandon all that TRUE IPv6 philosophy and do SNAT in DNAT in IPv6 just like in IPv4, but with wider address space. But then again, software (another software) is not quite ready for this. Software is expecting public IPv6 address to be just reachable. And private IPv6 address to be just unreachable.
I grew up in Australia, and have spent a fair bit of time in India for over a decade, and now live in India (1⅓ years).
Every ISP that I have experienced, mobile and broadband, is using CGNAT. The easiest way I’ve seen this on broadband is https://iknowwhatyoudownload.com/ showing several movie downloads per day.
Cloudflare isn’t the only problem, but they are the worst, probably by dint of popularity. I get blocked outright occasionally (presented dishonestly as because my request matched attack patterns due to things like SQL injection in query string parameters, when I’m actually just trying to load any regular page), and blocked with hCAPTCHA frequently (normally presented dishonestly with their stock page as “example.com needs to review the security of your connection before proceeding”, though a few like blender.org customise it). It’s draining.
In Cloudflare’s actual article, they claim their bot detection to be resilient to CGNAT <https://blog.cloudflare.com/detecting-cgn-to-reduce-collater...>. Frankly, if it is so, I wonder if they just have a rule that amounts to “is user in India”. I definitely feel prejudged and discriminated against. I am idly curious if leasing a static IP from my ISP would help anything, in the short or long term.
In Australia, I think I experienced Cloudflare’s blocking page once in my life, and no others.
>Every ISP that I have experienced, mobile and broadband, is using CGNAT. The easiest way I’ve seen this on broadband is https://iknowwhatyoudownload.com/ showing several movie downloads per day.
From memory, APNIC was handing out a /22 to every new member, then a /23, then a /23 worth. Now it asks you to submit a plan on how you would allocate a /23 if you received those ips.
>and blocked with hCAPTCHA frequently (normally presented dishonestly with their stock page as “example.com needs to review the security of your connection before proceeding”
I meant, in the context: blocked by Cloudflare with hCAPTCHA recourse. But as I consider it more carefully, I don’t think they don’t use hCAPTCHA in their challenges any more. They used reCAPTCHA at first, moved to hCAPTCHA around 2020, then they made their own thing Turnstile in ~2022 and migrated challenges to that at probably? the same time.
The Register is adding very little on top of https://blog.cloudflare.com/detecting-cgn-to-reduce-collater...
Previously discussed (a bit) at https://news.ycombinator.com/item?id=45746509
The usual story is:
1. Add an IP that has been freed from another use to a CG-NAT Pool.
2. Get complaints from customers about being hard banned from things like Netflix, Sport Streaming and VPNs or other utilities.
3. Investigate, no IP reputation issues. Find some random GEO IP database that has a side business in selling lists of VPNS or other geo breakout tools. They have listed this IP for some random reason. Almost never nefarious.
4. Give it 3 weeks for the Geoip nard to update from the wrong classification (harmful) to some kind of also wrong but unharmful classification like "Datacentre"
5. Customers can stream The Witcher again. Yay.
Really while ipv6 should be a solution here, another very good solution would be the removal of such useless middlemen from the face of the earth.
As someone who read all books and played the third game: People should totally be banned from watching Netflix's Witcher.
On a more serious note, I used GeoIP when it was free and it was a godsend to reduce malicious connection attempts to my webserver without impacting 99 % of my "clients" (not paying customers).
These kind of services *are* helpful. Wehterh you should rely on them when you have millions of customers is a different story altogether.
>On a more serious note, I used GeoIP when it was free and it was a godsend to reduce malicious connection attempts to my webserver without impacting 99 % of my "clients" (not paying customers).
Yeah, I think the issue is that, if you have 30 paying customers with, lets say, a canadian ip address and a canadian credit card, and then one bloke connects from the UK by that ip address, its daft that you just blacklist the ip and take all those customers offline. Personally I would start with a polite notice to the ISP, as its likely someones running a proxy and they should theoretically be as interested to remove it as you are.
I'm talking about Indian, Pakistani, Chinese IPs. Countries I don't do business with.
What I'd give at #2 to have some kind of backdoor ISP contact to these vendors to be able to ask "what exactly the hell is going on here?".
Having to troubleshoot things like Playstation Network bans ("or is it a ban?") behind CGNAT is an interesting adventure that typically leads nowhere.
Few if any of them permit outside contact. I remember having to get a customer to escalate their PSN case after removing the GEOIP configuration because PSN didnt reflect the change immediately. Its really difficult from a customer service perspective, because the error messages tend to pin the blame on the ISP, despite it wholly being a circumstance of the content owner.
You do get lucky, IIRC theres a few blog posts where people go into detail about how to contact some providers, but I found that some of them rotated those emails pretty regularly.
> Because CGNAT is more prominent, and more heavily used, in Africa and Asia […]
Isn’t essentially the entire US on CG-NAT for IPv4 on mobile data?
I’ve also had DOCSIS connections, i.e., fixed lines, with only CG-NAT in Europe years ago.
I wonder if that's not very visible in Cloudflare's data because those mobile devices will likely use IPv6 to connect to Cloudflare-hosted sites.
That’s what I was thinking. Anyone coming from Cloudflare will end up getting there via IPv6.
I use cloudflare to make my weather station available over T-Mobile. They don’t filter inbound ipv6 on regular phone lines (they do for TMHI) so you can host a simple page on ipv6, only set the AAAA record in cloudflare, and they will proxy it for ipv4 users so I can ignore being CGNAT’d for ipv4. Make sure if you do this setup with a tool like ddclient to keep the record current as T-mobile rotates ipv6 frequently
Well, what I mean to say was “anyone coming from Verizon will get to Cloudflare via IPv6”.
My current endpoint lacks IPv6, so I use Cloudflare so IPv6 clients can get to it. Verizon’s IPv6 is noticeably faster than their IPv4 CG-NAT.
mobile devices dont get ip6 do they? last i looked my cheapo gateway only provided v4 cgnat
Phones have been on IPv6 for years.
> Isn’t essentially the entire US on CG-NAT for IPv4 on mobile data?
T-Mobile, for one, has had their handsets IPv6-only for a few years, so if your Android/iOS does a DNS lookup and gets an AAAA record back, it will skip CG-NAT. T-Mobile presenting at NANOG in 2018 on IPv6:
* https://www.youtube.com/watch?v=d6oBCYHzrTA
And Rocky Mountain IPv6 Taskforce in 2017:
* https://www.youtube.com/watch?v=nNMNglk_CvE
Further data:
> Chances are if you use the Internet on your smartphone, you are connecting via IPv6. According to the Internet Society’s 2018 State of IPv6 Deployment,[1] 80% of smartphones in the US on the major cellular network operators use IPv6 and major mobile networks are driving IPv6 adoption with Verizon Wireless at 84%, Sprint at 70%, T-Mobile USA at 93%, and AT&T Wireless at 57%. Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs.
* https://www.arin.net/blog/2020/01/16/mobile-edge-of-the-inte...
Though some folks aren't happy with the implementation:
> But from my own experience, neither T-Mobile nor AT&T allows inbound traffic to the phone's IPv6 address. This negates some of the advantages of having a globally routable IPv6 address.
* https://isc.sans.edu/diary/27814
> Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs.
I think this has also already happened. You don't really notice it as a user, because the phone itself can act as a 4-to-6 translator (similar to how DS-Lite in DOCSIS transports CG-NATted v4 traffic in v6) for both native apps doing incompatible things with literal v4 addresses and tethering clients.
In Brazil I think basically more than half of fixed broadband should be CGNAT.
Basically only one single ISP don't use CGNAT...
Would be interesting if Cloudflare could give this info!
For those of us who don't have many options other than satellite internet, it generally uses CG-NAT, specifically Starlink.
AFAIK all mobile networks use NAT unless you pay a lot more for a special service with a public static IP.
I’ve had IPv4 CG-NAT on mobile LTE since ever and for a decade on residential cable in europe. Cloudflare is being lazy, Google too. I get served “Captacha’s” at least 10 times a day.
Even when i have IPv6 assigned, iOS and macOS seem to prefer A TXT RR and proceed just using IPv4 almost always. On LAN mDNS link-local IPv6 is always prefered.
Isn’t that just a correlation with crappy internet?
If you’re on non cg Nat it’s likely a pretty high end connection
They need to come up with an ip solution that is useful enough that people actually want to upgrade to it.
When you compare it to other technologies like https, tls1.3, unicode, 5g cellular, wifi 6, wifi 5 or bluetooth versions, etc. It’s clear that ipv6 adoption is not what it should be if they launched a protocol with clearer benefits to the end user.
> It’s clear that ipv6 adoption is not what it should be if they launched a protocol with clearer benefits to the end user.
The "end user" has no idea about TLS 1.3 or many other things. It's the techies that work behind the scenes that make the changes 'on behalf' of everyone else.
And IPv6 traffic is, according to Google, the majority of traffic it sees in many countries (including the US at >52%):
* https://www.google.com/intl/en/ipv6/statistics.html#tab=per-...
The 'real' holdouts are enterprise companies and corporate networks as evidenced by the fact that IPv6 usage goes up on weekends (i.e., when most people aren't at work on said corporate networks). See also:
> Chances are if you use the Internet on your smartphone, you are connecting via IPv6. According to the Internet Society’s 2018 State of IPv6 Deployment,[1] 80% of smartphones in the US on the major cellular network operators use IPv6 and major mobile networks are driving IPv6 adoption with Verizon Wireless at 84%, Sprint at 70%, T-Mobile USA at 93%, and AT&T Wireless at 57%. Plus, some mobile networks are taking the step to run IPv6-only to simplify network operations and reduce costs.
* https://www.arin.net/blog/2020/01/16/mobile-edge-of-the-inte...
Being able to connect your smartphone to the Internet seems like a clear benefit to the end user IMHO. Would hate to see what every mobile phone being behind CG-NAT would be like.
I think smartphones are a special case, because they generally cannot run public facing services that open up ports and are specifically designed to be hardened for ipv6 and designed to work in conjunction with the carrier's ipv6 firewall/network.
Compare that to a home network where, printers are shared, iot devices have open ports, computers and nas share drives. IPv6 may address the needs of cellular carriers and devices, but it doesn't adequately address the needs of small local networks connecting to the internet.
It's the Internet protocol. End users are not supposed to interact with it directly.
What exactly would replace IPv6? It's just an implementation detail, but an important one if you want to make the rest of the stack suck less.
Yeah, IPv6 is heavily tuned to the needs of the large-scale network operators, and is actively worse for the regular user and small networks.
From user/small admin standpoint, the goal is to re-use as much admin knowledge as possible - and what's on the wire does not really matter. So the ideal IPv4 upgrade _for users_ is IPv4 with larger addresses, but otherwise behaving identically. Ideally all the admin tooling stays the same, and the software needs changing some struct names, and tweaking IP regex. And sure, it'll all be different on the wire and all the OS'es need to be upgraded - but that is not a problem, consumer OS'es live only for a few years anyway.
From large network operator standpoint, the goal is improve efficiency of the huge networks. So lets eliminate NAT everywhere, completely redo host addressing, get rid of DHCP, and so on - redesign everything from scratch so it's "better". Sure, it's a huge learning curve but they have departments full of network engineers, they can do it. They are not some part-time sysadmins who just want their network to keep functioning.
It does not behave identically.
I have MultiWAN on OPNsense. My PC IP is always 192.168.0.12. My router decides which upstream it should go. If I go full IPv6, router should derive double IPv6 from both WANs and if main upstream goes down, stop advertising IPv6 from main upstream. Or stop advertising gateway. I don't know what is the right IPv6 way of doing MultiWAN.
Not only PC may change IP, but also servers. Legacy IPv4 DNS can be extended to IPv6, but that mechanical action is not flexible enough. With IPv6 we need to be able to mass replace IPv6 /64 prefix leaving all suffixes intact. We probably need /64 prefix alias system. Software is not prepared for this. In IPv4 SNAT and DNAT were being these "aliases". If NAT is not an option anymore, then DNS must step in.
For many server software it just not possible to listen on multiple IPv6 address. Last time I tried MySQL, it just could not listen on multiple addresses. I could not make it listen on IPv4 and IPv6, specifying two addresses. MySQL server wanted just one address. This address could be [::], which means all interfaces and all protocols. And Linux implements some stupid hack to accept IPv4 connections to IPv6 socket. And Windows Vista also adopted this brainrot. But this is all wrong. Servers have to learn to listen to multiple IPs. This is normal. And for good IPv6 servers should learn to not only listen on multiple IPs, most wanted multiple IPv6, but also rebind listeners on the fly. If I got disconnected from ISP, reconnected by DHCPv6, and ISP assigned another IPv6 prefix, then DynDNS should update all my zones to new /64 prefix, and all servers in my network should rebind listeners.
Or else we may abandon all that TRUE IPv6 philosophy and do SNAT in DNAT in IPv6 just like in IPv4, but with wider address space. But then again, software (another software) is not quite ready for this. Software is expecting public IPv6 address to be just reachable. And private IPv6 address to be just unreachable.
[delayed]
I grew up in Australia, and have spent a fair bit of time in India for over a decade, and now live in India (1⅓ years).
Every ISP that I have experienced, mobile and broadband, is using CGNAT. The easiest way I’ve seen this on broadband is https://iknowwhatyoudownload.com/ showing several movie downloads per day.
Cloudflare isn’t the only problem, but they are the worst, probably by dint of popularity. I get blocked outright occasionally (presented dishonestly as because my request matched attack patterns due to things like SQL injection in query string parameters, when I’m actually just trying to load any regular page), and blocked with hCAPTCHA frequently (normally presented dishonestly with their stock page as “example.com needs to review the security of your connection before proceeding”, though a few like blender.org customise it). It’s draining.
In Cloudflare’s actual article, they claim their bot detection to be resilient to CGNAT <https://blog.cloudflare.com/detecting-cgn-to-reduce-collater...>. Frankly, if it is so, I wonder if they just have a rule that amounts to “is user in India”. I definitely feel prejudged and discriminated against. I am idly curious if leasing a static IP from my ISP would help anything, in the short or long term.
In Australia, I think I experienced Cloudflare’s blocking page once in my life, and no others.
>Every ISP that I have experienced, mobile and broadband, is using CGNAT. The easiest way I’ve seen this on broadband is https://iknowwhatyoudownload.com/ showing several movie downloads per day.
From memory, APNIC was handing out a /22 to every new member, then a /23, then a /23 worth. Now it asks you to submit a plan on how you would allocate a /23 if you received those ips.
>and blocked with hCAPTCHA frequently (normally presented dishonestly with their stock page as “example.com needs to review the security of your connection before proceeding”
Isn't that from cloudflare, not hcaptcha?
I meant, in the context: blocked by Cloudflare with hCAPTCHA recourse. But as I consider it more carefully, I don’t think they don’t use hCAPTCHA in their challenges any more. They used reCAPTCHA at first, moved to hCAPTCHA around 2020, then they made their own thing Turnstile in ~2022 and migrated challenges to that at probably? the same time.
The same applies to us rich folk on mobile. Not sure what the point of this article is.
I agree. A bunch of platforms use CGNAT, like Tailscale: https://tailscale.com/kb/1015/100.x-addresses
My understanding is that they use the IP range assigned for CGNAT as their private address range to avoid conflicts, but do they use CGNAT?