> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
It doesn't make the economics any different. In a browser environment, you're maybe looking at the acceptable lease being 100MB for 1 second. Much more than that, and you start hitting limits of what browsers will let you do on low-end phones. Longer than that, and we're back to the user-observable latencies being too long.
100MB for 1 second just is not much of a deterrent.
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.
I implemented something similar for my bot defences. If headless chrome is detected you still get the same anubis-style PoW but even if you submit the right answer you get rejected.
The captcha itself (matching pictures to text) is mostly for ML training data. I think pass/fail is mostly based on heuristics like how you moved your mouse which could get analyzed before you complete the captcha. https://www.techradar.com/news/captcha-if-you-can-how-youve-...
reason why is 1. Google and others really needed the training data, and 2. it probably helped justify the cost of providing the captcha service for free worldwide (old free tier was 1M/mo)
A person's testimony is a source. I can add mine: you can tell when you're truly blocked because if you click for the accessibility audio-based captcha it will actually tell you you're blocked (but, if you did the visual captcha, would simply loop forever while telling you you did it wrong).
I don't think you'll find an article by Google saying "yes, we sometimes completely block users while making it look like they're not blocked and wasting their time".
A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.
I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.
Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.
Most CF / Recaptcha problems are users going "off the golden path", and not realizing that their config changes are at fault.
If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing for.
At least for me, CF is fine; recaptcha is the only one I really have problems with.
I dont care what recaptcha wants to optimize for. I dont think that using a vpn is that a rare thing anyway. If others have figured out how to do it without requiring spending 30 seconds to solve a captcha, I dont see why websites still use recaptcha/captchas for that.
And that it is "my fault" not being logged into google I was least expecting to see here.
A lot of users also run cheap cracked fire sticks and other low reputation hardware that's proxying their residential traffic for nefarious reasons which makes all the big providers put up their guard.
From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.
Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.
I doubt they are doing it. You just have to get on a VPN to and see yourself being flooded with captchas despite browsing the web like a normal human and solving dozens of captchas along the way.
Supposedly, but not really. I regularly encounter sites where cloudflare serves me with an ambiguous ban notice rather than a proof of work. What's worse is that these apparent IP bans take effect even if I already had a valid active session (ie previously passed the check).
Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult challenge) in the exact same scenario.
Regardless of the precise logic it's no excuse for the policy. Simply hand out a sufficiently difficult PoW to prevent widespread abuse.
I'm quite certain it isn't a generic "datacenter" list though because a given VPN exit that was working will suddenly stop. Meanwhile I have a valid cookie yet that is disregarded.
Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
Putting aside the question of whether it will continue to work, even somewhat, against botnets, I find your first paragraph confusing.
Reload loops, or being able to "bypass" anubis (unless you merely mean bypassing it for the token validity period by solving a challenge), sound like misconfigurations. There's no reason for anubis itself to cause reload loops; it's tricky to configure a webserver to use it in some scenarios.
Any ability to bypass anubis probably means the site is using it in auth/challenge mode only, and then misconfigured their webserver's auth checking. Or it's a bug. If you mean the double-spend tavis mentioned in his blog post which previously made the HN frontpage, that was patched right after it was reported to the maintainer almost a year ago.
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
Most users of residential proxies just get a SOCKS5 address and routing, they don't actually get computational resources of the infected systems beyond that. The user of the proxies, the operator of what the article describes as a control node, would be the device responsible for the PoW.
Do you have any evidence that AI providers aren't using residential proxies?
If you pop my machine and use it to route 100 MBit/s, I might not notice for months.
If I hear the fan spinning at night, you're probably getting caught immediately.
If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV playback will start to stutter, which will be resolved by taking the compromised TV box, and removing the malware using advanced mechanical means called "a trash compactor".
>chances are her TV playback will start to stutter
video decoding is hardware accelerated, and there's probably enough excess compute to be able to do some sort of PoW challenge. Besides, unlike humans, bots aren't in a hurry, so they can spread out the work across a long time to minimize disruption.
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).
Anubis's default 1-week token lifetime may not be nearly enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine.
We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
If that happens the browser engines (all what, 3 of them?) can add a PoW API to call into native code. Or a pathologically scalar algorithm can be adopted so that wasm is good enough. RandomX or something close to it probably qualifies.
It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).
Uh, is it resistant to parallelization across multiple sites? Because that's the situation for the scrapers. They're not trying to solve a single PoW challenge across many cores.
Typically, the machine doing the content processing, including solving PoW, is the centralized "control" node described in the article, not the machines who's IP addresses are being used. In typical residential proxy networks, the residential proxies are exposed to the customer (the person paying for and using the proxies) as just SOCKS5 addresses, and no computational power from those compromised devices is made available for the scraper besides that used to power the SOCKS5 server itself, the customer is just paying for the transport and address (and indeed, is often billed on either a per-GB or per-IP basis).
In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.
Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.
These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.
This isn't to say that true botnets built with actual malware delivered through either software exploits, phishing attacks, or watering hole attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.
Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
Can you not design a PoW that is most efficient in a browser? Don't brute force hashes like Hashcash/Bitcoin, do something similar to RandomX instead but in JS. Browsers ought to run the fastest JS interpreters already so if interpreting JS becomes the bulk of the work, that attack might not work. Maybe even involve the DOM or whatever else makes sense.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
On a small enough site (even LWN might qualify) the chance of two random sets of client IPs intersecting can be quite low.
Private trackers do this. If they ban a user that geolocates to a certain city and ISP, they'll ban new signups from that city and ISP because there's probably only a few users from the same city and ISP. And then report to their friends at other trackers, that a user with that city and ISP is trying to evade a ban.
PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
But why? Obviously an unjustified cryptominer is bad, like unnecessarily slow JavaScript, but this one has a good purpose and to the user is no different than PoW.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
I had meta's crawler hitting the pi searcher at something like 5qps for days on end, just ... querying for substrings of pi, ignoring robots.txt, etc. it wasn't enough to break anything but it triggered a lot of alerts.
I can imagine that sites with dynamic content and potentially unbounded query types or pathnames are in danger from particularly stupid crawlers.
Grok actually shows a number of sources used for an answer. Once I asked it something simple and it apparently scanned 200 different websites. And it was just a short prompt. Now imagine millions of users asking for something multiple times a day.
That's really a huge issue right now (to some extent even before the AI hype) that almost everywhere google is effectively the only entity explicitly allowed to scrape.
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
Maybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
It should really just be called DDoS, at a certain level of incompetence intent doesn’t matter. You’re right that there’s zero (information gathering) benefit over reasonable scraping which wouldn’t cripple the site.
750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour
Millions per hour is tens per second though; perhaps the fix is performance improvements
We have put in a number of performance improvements, yes. The nice thing about those is that they also make the site snappier when it's not under load. Right now we have just over a million items in our CMS, plus our publicly available mailing list archives, which are much larger, even if they're less frequently referenced.
Hah. I have a homelab with a couple of sites, including a personal Forgejo installation.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
The paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.
It's not AI companies scraping these websites, it's AI companies creating a massively profitable need for data, and every random Joe with vibe coded scrapers tries to make a buck out of it.
It isn't. AI scraping has nothing to do with it, for the reasons you said. Someone wants the web to go offline, they are DDoSing the entire web, and it's working. For some reason we are tackling the symptom instead of finding out who that person is. Come on, it can't be that hard to subpoena Bright Data. The law enforcement system knows how to track down someone who's trying to be anonymous on the internet.
Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.
I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.
Appreciate your kind words! Many people have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
Well this is not what is happening in practice, Wikipedia / Wikidata, OpenStreetMap, OpenFoodFacts... All provide APIs and even a full dump of their database available to download for free, but no, the stupid bots still DDoS them 24h/24.
How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.
I was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.
The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone.
10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access.
But really, this finally explains the hordes of really basic boring games that just advertise other boring games. Idle games and the like that really just want you to keep your phone unlocked and open. Millions of downloads on the app stores for entirely offline content (and ads) and no way to block the network access.
> 10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what.
Why does network access need to be a binary, all or nothing?
When you install an app, the app should request permissions to specific DNS names, i.e. pointing to the servers that the app's authors operate. If I install Todoist, the app should only ask for access to Todoist's servers. If I install Netflix, the app should only ask for access to Netflix's servers. The OS can then put a DNS firewall in and block any network access that wasn't granted when the app was installed.
> And both ios and android make it hard to deny apps network access
The list of apps that genuinely need "any" network access (web browsers, VPN apps, stuff like Termux...) is incredibly small compared to the list of apps that need access to a small number of VPN targets (these days, most apps). Apple / Google could even decide, if they really want to make it easy for apps to request network access, to basically allow apps to automatically get network access, so long as the list of domains the app needs access to is no more than a handful. The security value of isolating "all" network access permissions to only the relative handful of apps that actually need to request it, would be huge.
The Bright Data “free” VPN they’re talking about requires the user to go through steps to enable it.
These aren’t as simple as downloading a free game and then the phone is compromised as long as it’s installed.
The users who install these things don’t care about permissions prompts. They’ll follow instructions to tap any prompt the instructions ask. They want the free thing and don’t care what they have to do to get it.
GrapheneOS allows you to deny network access per app pretty trivially. Google Play services make it a bit more difficult because the app might marshall the network request through that; I'm not sure how to verify that behavior when it happens.
Thanks for mentioning GrapheneOS. I'll just explain to others what "pretty trivially" means here. GrapheneOS adds a huge checkmark "Network access?" when you install an app. It's impossible to miss.
The problem is the yes/no options - I want to give access to specific endpoints - but now that’s “all of Cloudflare for everything” which means the web entire.
It used to be you could let Onkyo App™ access the five IPs for onkyo.com and be done with it.
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.
I have a medium-sized Discord server of web sysadmin people (mostly wiki operators) that came together after I wrote a similar blog post [1] about how the LLM scrapers/resproxies are making it suck to run wikis. Not sure if there's other private communities out there, but feel free to email me if you want to join
Disclaimer: this works for my very small number of personal services that I run. I have no idea how this would (or probably wouldn't) scale at all. Also, the methodology I describe below is based on what I'm able to do technically, which is pretty much limited to bash scripting.
On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and therefore there is no reason for the existence of that traffic (at least as far as I'm concerned), and therefore I treat it as malicious.
The address is recorded and goes into a database.
Periodically, the database is dumped to a file in a format that the firewall reads, and all the 'malicious' addresses detected above are added to a list so that those addresses are blocked from accessing the legitimate service ports. (analogy: if you throw an egg at my outdoor wall, I'm not going to let you into my house through the door because I don't want egg on my furniture).
I have a blocking period of about 3 months - because the things I run are important to exactly a single person. A blocking period much shorter would be recommended to prevent the gross-overblocking of legitimate users who may have un-lucked into being assigned a residential IP address that was previously used in a proxy-scan-scam.
Discard this if it's a stupid idea at-scale, but I quite the like the 'idea' of it, and I made it work, mainly for the technical challenge.
It’s a great idea but not sure for larger websites when these residential proxy platforms are using innocent user ip addresses. Then you’re left blocking innocent users. It’s a tough call.
I did something like this using fail2ban for some time, but 1) it didn't help much due to the larger number of IPs, 2) it blocked widely used VPN services.
I came up with the idea and helped build Grub, the distributed crawler. Looksmart bought it, ran it for a time, then sold it to Wikimedia. I reclaimed the name recently (abandoned mark) and have a new crawler now that is agentic. I use it for my own research runs, and it's not my main focus at this point, nor am I trying to get it attention. A lot of LLMs and coding agents can easily fetch content if it is needed and we're blind to how they do it. See the Claude plugin for Chrome as an example of using it in a user-in-the-loop solution. That said, I've been spending a little time thinking about how to bake the contract into the crawler, as opposed to expecting someone else to act ethically using it.
Grub was, in a very real way, a botnet. And, we harmed site owners when we were operating at full capacity. There were a few bugs in the early days where we would reschedule a site because the ingestion in the server broke, which then caused the page to be rescheduled. Stupid error, and we fixed it, but it's illustrative of the fact even good intentions isn't enough here.
What I've come up with over the years is similar to the idea Cloudflare is implementing with payments to site owners by charging the crawlers. My objection to Cloudflare's implementation is based on a personal opinion about Cloudflare being a single point of failure and also a decrypted choke point. Their ideas about how to handle crawlers, and pay for the load on the sites is solid. It presumes to use the 402 response to demand payment. I'm clearly biased about Cloudflare, but that's my prerogative here.
It may be possible to solve this with cryptocurrency, in a distributed way, and I've prototyped a system that uses the Lightning Network to handle the payments from a 402 response. Lightning Labs also worked on a project called Apeture for a time that did something similar.
HN's site knows every item ID, and it knows fresh IDs get read in a predictable distribution while old ones mostly sleep. Sustained access outside that is itself the scraper signal. No IP reputation needed, which matters now that residential proxies burn an address after a handful of requests.
Karma gives you a clean way to let humans through. Issue logged-in accounts with decent karma a token whose cold-content budget scales with it (the karma), so an account with history scrolling back through a 2014 thread just reads it. Karma should gate the tier, not be spent as currency, or upvote rings become a crawling business.
Anonymous readers who deep link into one old thread from a search engine get the first fetch or two free (and you watch the article IDs, not the IPs). What remains after those carve-outs is bulk traversal of cold IDs with no identity attached, and that traffic gets rate limited and answered with a 402: pay per page over Lightning, priced at a healthy multiple of what residential proxy bandwidth already costs, or come back slowly for free.
There are probably holes in these thoughts. It's one of the harder problems to solve, for sure.
LI've emailed there in May and June about different topics and gotten no reply to a request to confirm receipt. Is there a backup method for when your algorithm throws people's email away without even informing the sender with a delivery failure notification?
We get thousands of emails - far too many to even read them all, let alone respond the way we would like to.
There are other reasons why we might not respond, but overload is the main one.
Edit: I only see one email in the archive related to your account. It was from Sept 2024 and we responded to it. Are you talking about a different account?
Oh, okay I didn't know you weren't getting around to all incoming emails, that makes sense then. Previously the responses were fast and reliable and I thought I was being dropped by a silent bot/spam detection algorithm (as is becoming frustratingly common).
It's indeed from two different email addresses from the one on this account. They're not that important so never mind, thanks for checking and for the reply!
Not dang and not the person you are asking but there is no CDN. HN is just two servers running BSD one active and one standby. HN is all text so there is not much bandwidth usage.
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
Google might not register it, but in some related testing, even a few hundred requests through the API (serial, not concurrent) takes much longer than a single "Past" page request would, even if the latter were substantially rate-limited. Of course, if those requests (to HN itself rather than the API) are blocked entirely, that's a moot issue.
On dead/flagged items, there's some value. Whilst the title/URL context aren't available, just knowing what fraction of submissions and comments are moderated is interesting data, and it is possible to construct patterns against specific accounts.
I'm frequently encountering what appear to be banned accounts. Being able to trace those through the API to see where and when they were banned, or now much moderated activity they're generating, can be useful. I'm relying heavily on the "/replies?id=<UID>&by=<moderator>" search endpoint (generally dang, tomhow, sctb, or pg as mod) currently to find out if there was a specific ban admonishment from a moderator. That's often but not always the case.
But it's not possible, say, to tell through the API what sites are banned. Looking at site history with "showdead" enabled can tell you that though, e.g.:
... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:
It's the collection of 30 items from that date I'm interested in. For my scraping, I don't actually need to further query the individual posts as I've got the elements I'm interested in (title, date, story position, URL, votes, comment count, submitter, site/domain) from the index page itself, parsed out of the HTML. The "Past" entries alone are a significant (though not huge) request load. To update the past three years would be about another 1,000 requests, which, if fulfilled and modestly rate-limited would hopefully not keel the servers over.
Once I've pulled in those "Past" pages, I could of course do further API queries, though at this point I don't see any specific need to do so.
I suppose that requesting the "past" links be included in the API set could be a request I might make of HN, or the ability to request, say, all submissions (or comments!) for a given date.
There are groups which have done HN analytics in the past using the API, for example Whaly.io:
I could look more into their methodology to see if I can use similar approaches.
The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)
That's missing the title and URL, as I suspected it would, though the submitter UID is available.
To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
I was involved in both sides of this battle over ten years ago. Things haven't changed all that much.
It's important to note that neither side has moral legitimacy. Not everyone who carries a rifle is a enemy. Not everyone wearing body armor is a saint.
I have given up on the idea that "human vs bot" matters at all when it comes to anything other than voting (which should only be done in person with paper and pen, by the way.)
You could make an argument that "likes" are a form of voting, but you shouldn't. We need to abandon the idea of supposedly democratized algorithms and focus instead on actual democracy.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
Caches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
On my sites, I see ClaudeBot consistently (and has been like this for over a year) ask for "${SITE}.com/base_dir1" and then get the redirect (Caddy does this automatically) to get "${SITE}.com/base1/base_dir1/" (trailing slash).
The hrefs on my sites include the trailing slashes for directories, so looks like ClaudeBot's internal code is stripping them off before requesting them, and therefore essentially makes almost 2x the requests to my sites for directories, half of them ending up being redirects back to the same url but with the trailing slash.
At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.
> I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
And it's made necessary because another group of people thought that selling IP blocking services would be a good idea. One party sells walls, another party sells ladders.
Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
Mmm... your quote (IDK where it's from) mentions them having consent from device owners, but your FBI link cautions on how to avoid getting infected by malware.
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 address for all their employees, and most VPN services.
And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP.
Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.
Thank you. Gmail must not be like our fellow HN users we see here, quoting a couple:
“I'm tiny and only run little personal stuff. I just block vast IP address blocks.”
“Apologies. :( Since you say you've never visited the website before, then that means you're either in one of the countries or in one of the residential IP ranges that I've had to block.”
Although Google isn’t afraid to completely block iCloud private relay from Google scholar. Other sites may reject the first iCloud private relay visit, then reopen site in new tab and often automatically assigns new unblocked IP. Anyway, I would’ve thought it’s an acceptable cost from e.g. Google’s perspective to block ranges that did something bad once in spite of collateral damage.
No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.
The article was called "The one we're commenting on"
> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.
Does that sound like your typical self-hosted blog?
The Bright Data mentioned in the article, as well as other similarly malicious but even harder to identify parties, most certainly do fire up residential proxies, no matter what the site, no matter how useless or duplicate the data which they're trying to get. Not at first - they start by trying to get your content from cheap data center connections - but as soon as some kind of bot-mitigation appears, they move to residential proxies to try and evade that, with a first tier coming from "global south" residential proxies, and then scaling up to (presumably more expensive / less widely available) proxies from the USA (I've seen some from Europe, but very few in relative terms). Each tier also appears to have the option to run JavaScript.
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
ive seen unscrupulous founders fake traction during diligence, which is my day job
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
In theory, proof of work that is used to mine a cryptocurrency could be a solution.
Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.
The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.
I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.
Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.
Proof of work, even "custom", where the user does not need a particular interaction with the page, does not work. The scrapers are running headless Chrome and solving the work. They do not care, they do not pay the bill, the compromised system's owner pays the bill.
I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The PoW challenge prevents less than 10% of the bots.
First of all I am very critical of the $100 per consumer device per month figure.
Also all major browsers block crypto miners on webpages now (for good reasons) so it may prove difficult to allow "good" mining scripts while still blocking "bad" ones.
Better skip the PoW part as it'll be wildly inefficient for most of the work done.
Instead, exchange web traffic for actual $. Say, some kind of tokens that are easily turned back into hard cash through a 3rd party.
Requesting a 100KB file? Okay, that'll be a $0.00002 token, please! (visitor's user agent provides it in a manner transparent to regular web users). Requesting a 3MB image? Okay, that'll be a $0.0005 token, please!
Result: niche websites earn hard cash. It doesn't matter much if you're hammered as long as the hammering comes with a corresponding flow of tokens (read: $). No token(s)? No service.
Regular web users would pay for those tokens through their normal internet service fees, and otherwise not be bothered. Massive scrapers would have to pay somehow for the tokens to be served web data at all.
In effect: put the bulk of public web sites behind a paywall. But with the bar low enough & in a manner that it's transparent for regular web users. Clicked "reload" by accident? Oops, internet service bill got upped by 2 micro-$.
> partly because it causes annoying delays for those trying to get to the site
This is true but usually a small issue. It’s further alleviated by cached tokens so you only have to solve the challenge once in a while per site, and a login token may let you skip it.
> partly because it seems inevitable that the scrapers will eventually find their way around it…A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
Opnsense has a traffic capture feature in the interface diagnostics menu, if you want to spot check what servers the devices are currently talking to.
Should be pretty obvious: client devices and internal services will have no traffic >95% of the time, just NTP for timekeeping, DHCP lease renewal, and associated ARP (running total: two dozen packets if you monitor them for a full 24h), then any system updaters (readily identifiable by the initial DNS requests), and finally of course you'll see the traffic of the service that the device hosts, if any, which can be easily dismissed by not looking at incoming connections (scraping uses outgoing connections)
> Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fair trade-off. Unsurprisingly, I have nothing but negative associations with their mascot.
The FSF has the right idea about all this:
> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.
> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.
I think all of these mitigations are unfortunate. They hurt one of the things that makes the web cool: it's a stable, stateless, idempotent way to access data.
This makes it a prime target for aggressive scraping by LLM companies, but it also makes it accessible and fast, and a prime target for benign use (like archive.org or "read later" services).
For my own sites, I'll eat the cost of the crawlers (mitigated by making the sites as efficient as possible) and keep them available to everyone.
Ever since bots became a problem on the internet 10-20 years ago, it has seemed like the common-sense solution is some kind of micropayment. Pay $0.01 to view the page. When money is on the line, scrapers are likely to be more well-behaved, even if they do pay. The problem is, and has always been, the friction of payment. How do you pay $0.01? The credit card processors will tack on a $6 surcharge. We need a trusted third-party that turn money into "internet article credits" that you can spend in small increments, like a video game. But I suspect that thousands of people have already though of this system, and tried it, but ran into some roadblock. I'm guessing there's some egregious regulation that makes micropayments impossible.
> I'm guessing there's some egregious regulation that makes micropayments impossible.
More likely there isn't any kind of universal standard that's easy to implement for browser makers, has low overhead, and preserves internet users' anonymity as much as possible.
The currently existing friction of using micropayments is the problem here, I suspect.
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?
The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").
> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.
In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
Blocking in ipv6 works roughly the same way as in ipv4, just that the scale is different. Instead of blocking something like a company's /24 or an ISP's /16 when they don't respond to abuse messages, you block the company's /48 or the ISP's /32. It'll vary per organisation how large a range they got exactly but you can see that in WHOIS. End users are no longer at a /32 (v4) but at /64 (v6), or some prosumers might have a /29 (v4) and /56 (v6). Same concept, just a different prefix length
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
What’s more, they will scale up with increased resources on the site.
If you redline at 20 searches a sec, and put in 4 more workers, suddenly you’re serving 100r/sec to the bots, paying 5x for it, and your users are still seeing shit qos. I've seen multiple cores of nginx saturated just dealing with one dos/crawl run on a somewhat high profile site.
I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
The excessive scraping and ignoring robots.txt only breaks the informal social contract established over the past decades of the open internet.
The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is little, but is distributed over millions of residential devices all over the world, making it difficult to categorize or block. That should be illegal, and that's what Google et al can be expected to be policing on their app stores.
On what grounds would it be illegal though? Things don't become illegal just because you don't like them. They may become illegal just because the president doesn't like them, but I don't think you're him, and in the absence of that, there has to be a majority of Congress and most of them want a reason.
It's not illegal to run code on a device without informed consent to everything the code does. The CFAA may be excessively broad but it isn't that broad.
I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
I ran a gitweb server which was battered by bots so I eventually had to take it down. Gitweb! You can just connect using the git protocol and download everything vastly more efficiently!
In other words, they don't care at all. For them, residential bandwidth is completely free.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.
Sorry, I wasn't clear. In order to know you have an up to date copy of the page you need two pieces of information:
1. When the page itself was last updated
2. When the crawled copy was last updated
In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.
The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.
This is a predictable consequence of age verification laws and social media bans. Formerly VPNs were a nice to have but now they are a necessity in many countries to navigate the modern internet.
The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
> The moment it does you'll be able to tell because Bitcoin will crash to 0.
The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now.
Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized hardware this computation only costs fractions of a penny, so they are barely hindered by it.
The SHA256 based PoW scheme has a very heavy ASIC advantage. People have tried to design PoW scheme that minimize the custom hardware advantage, but I'm not sure if they managed to close the gap far enough to make PoW feasible for this application.
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
Edit: If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call.
Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.
Again: Not a one-note flute, though fairly focused of late on AI and poisoning.
Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (<https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.
(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)
The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.
I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.
Jon's been around a while and some of the piss and vinegar of youth may have subsided. He does tend to show up with LWN comes up, whether as a topic of discussion (or more often) from submitted articles. That's his baliwick, and again, I don't fault him at all for it.
Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
I'm confused why everyone is pretending that a ban holds any meaning here. he probably already has a new account, it might have made sense to just silently ban him in the hope of imposing a minor cost on what you believe he is doing, but wasting your time addressing it imposed a far greater cost on you than him.
I understand how it can be confusing. The key factors in doing it this way are (1) the community regards older accounts, especially ones that have significant posting history, as more credible; and (2) doing it publicly rather than silently has transparency value.
It's not a sign about that project in any way. I had never heard of it and have no opinion about it one way or the other.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue.
The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).
>sell access to the user's internal residential network
That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space):
if (addr.isSiteLocalAddress() || addr.isLoopbackAddress()) {
LogUtils.e("PopaTunnelAsyncThread", "Hacking? The Host Resolved Ip is " + addr + " on tunnel id:" + tunnelId);
throw new IllegalArgumentException("Hacking? The tunnel host resolved ip is internal");
}
Local and loopback addresses like 10.0.0.0, 172.16.0.0, 192.168.0.0, and 127.0.0.0 do not work. It will not connect to people's private file servers on their network.
It'll still connect to IPv6 addresses and bypass any firewalls.
Also users might become part (victim?) of a police investigation because of illegal actions that seem to originate from their local residential connection.
So still good to take down such backdoors. Would be nice to go after the botnet operators as well...
Sorry, I understand scraping is a problem, but talking about open Internet while simultaneously complaining you can no longer discriminate datacenter IPs like you used to is hypocrisy.
I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.
Noone complained we can't discriminate DC IPs, though to be fair some (imo bad) operators did just that. This is not even about preventing bots, which has perfectly legitimate usecases (eg. Internet Archive).
This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple rate-limiting, they have to employ so-called residential VPNs. They're residential in that they route through real user connections, and so you can't block the IP/subnet without dropping a certain amount of legitimate human-driven traffic.
Personal example: some time ago, i had to disable a wordpress plugin on a site that was causing 100% CPU usage on the whole box (hosting dozens of wordpress instances). That plugin was a simple calendar, but a bot was repeatedly scraping non-existent (or rather, "no event planned for this day") pages for every date in the calendar that you can represent in the DB timestamp, clearing the cache as it went to try and find new events for 1000 years ago. Whoever operates this IP space doesn't matter to me, i'd just like to block them because they don't respect robot.txt… but i can't because they use a "residential proxy" and will change IP address every hour or so.
This is definitely an issue, where data centre addresses are, by default, second class citizens or even persona non grata. The problem is that this reputation has been, unfortunately, well-earned.
There's no easy answer here. The ephemerality and pseudonymity of VPS address usage screams untrustworthy, and the only way to reign that in is better identification of who is using the VPS/address or significantly more restrictive rules applied to data/port usage. And I'm not sure if I like the general direction that points towards - away from the "open internet".
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
> It trades something fungible and incredibly cheap (CPU)
it could be RAM-bound, which is very much NOT cheap nowadays :)
Yes, but the people with the RAM nowadays are the data centers, not the end users.
It doesn't make the economics any different. In a browser environment, you're maybe looking at the acceptable lease being 100MB for 1 second. Much more than that, and you start hitting limits of what browsers will let you do on low-end phones. Longer than that, and we're back to the user-observable latencies being too long.
100MB for 1 second just is not much of a deterrent.
Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.
A fun fact about Google captchas: they've often decided whether you will succeed or fail the captcha before you do the captcha.
I implemented something similar for my bot defences. If headless chrome is detected you still get the same anubis-style PoW but even if you submit the right answer you get rejected.
This seems nonsensical. Care to elaborate?
The captcha itself (matching pictures to text) is mostly for ML training data. I think pass/fail is mostly based on heuristics like how you moved your mouse which could get analyzed before you complete the captcha. https://www.techradar.com/news/captcha-if-you-can-how-youve-...
reason why is 1. Google and others really needed the training data, and 2. it probably helped justify the cost of providing the captcha service for free worldwide (old free tier was 1M/mo)
If Google determines you're an undesirable user, doing the captcha is just an exercise to waste your time.
I certainly experienced this (the vicious try-again cycle) but curious if you have any sources for this?
A person's testimony is a source. I can add mine: you can tell when you're truly blocked because if you click for the accessibility audio-based captcha it will actually tell you you're blocked (but, if you did the visual captcha, would simply loop forever while telling you you did it wrong).
I don't think you'll find an article by Google saying "yes, we sometimes completely block users while making it look like they're not blocked and wasting their time".
That explains why I can never get past Google's captchas! I don't even automate Google searches, I wonder why they don't like me.
Depending on the IP reputation as well as the kind of IP address you have, this can happen.
Google also prefers if you have a Google account logged in.
I've usually been more annoyed at the surprise dissonance of "was that an anime girl on kernel.org?" than annoyed at the delay.
> just crank it up when you get a flood,
A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.
I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.
For me Cloudflare is worse, it takes more than 5 seconds, where as anubius take 1-2 secs.
funny with all the IP information they have, cloudflare cannot do a better job. (I am on IPv6)
and most of the time, its on marketing product pages like in framework main site, which can be cached.
Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.
Most CF / Recaptcha problems are users going "off the golden path", and not realizing that their config changes are at fault.
If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing for.
At least for me, CF is fine; recaptcha is the only one I really have problems with.
I dont care what recaptcha wants to optimize for. I dont think that using a vpn is that a rare thing anyway. If others have figured out how to do it without requiring spending 30 seconds to solve a captcha, I dont see why websites still use recaptcha/captchas for that.
And that it is "my fault" not being logged into google I was least expecting to see here.
> that it is "my fault" not being logged into google I was least expecting to see here.
Parent was just starting a fact of how our digital overlords determine the probability of your browser being a bot. Why take it personal?
If you don't live in a first world country then you will also find yourself on the "too bad, don't care" list.
A lot of users also run cheap cracked fire sticks and other low reputation hardware that's proxying their residential traffic for nefarious reasons which makes all the big providers put up their guard.
From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.
Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.
I doubt they are doing it. You just have to get on a VPN to and see yourself being flooded with captchas despite browsing the web like a normal human and solving dozens of captchas along the way.
I guess that raises the bot score given that the vpn IP is a data Center IP and thus in a lot of ban lists
Also some of the free VPN apps support themselves by proxying this kind of traffic:
https://www.kaspersky.com/blog/what-is-wrong-with-free-vpn-s...
Which also involves detecting entropy across sites I guess
Supposedly, but not really. I regularly encounter sites where cloudflare serves me with an ambiguous ban notice rather than a proof of work. What's worse is that these apparent IP bans take effect even if I already had a valid active session (ie previously passed the check).
Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult challenge) in the exact same scenario.
There's lists of data center IPs. You're probably in them and That's why you're getting banned
Regardless of the precise logic it's no excuse for the policy. Simply hand out a sufficiently difficult PoW to prevent widespread abuse.
I'm quite certain it isn't a generic "datacenter" list though because a given VPN exit that was working will suddenly stop. Meanwhile I have a valid cookie yet that is disregarded.
Cloudflare often just straight up blocks me or makes me do a captcha. IMO those are both much worse than Anubis
Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
Putting aside the question of whether it will continue to work, even somewhat, against botnets, I find your first paragraph confusing.
Reload loops, or being able to "bypass" anubis (unless you merely mean bypassing it for the token validity period by solving a challenge), sound like misconfigurations. There's no reason for anubis itself to cause reload loops; it's tricky to configure a webserver to use it in some scenarios.
Any ability to bypass anubis probably means the site is using it in auth/challenge mode only, and then misconfigured their webserver's auth checking. Or it's a bug. If you mean the double-spend tavis mentioned in his blog post which previously made the HN frontpage, that was patched right after it was reported to the maintainer almost a year ago.
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
Residential proxy bandwidth is extremely expensive, comparatively speaking. It can be up to $1 per GB but is more typically about $0.20 per GB.
Most users of residential proxies just get a SOCKS5 address and routing, they don't actually get computational resources of the infected systems beyond that. The user of the proxies, the operator of what the article describes as a control node, would be the device responsible for the PoW.
Do you have any evidence that AI providers aren't using residential proxies?
If you pop my machine and use it to route 100 MBit/s, I might not notice for months.
If I hear the fan spinning at night, you're probably getting caught immediately.
If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV playback will start to stutter, which will be resolved by taking the compromised TV box, and removing the malware using advanced mechanical means called "a trash compactor".
>chances are her TV playback will start to stutter
video decoding is hardware accelerated, and there's probably enough excess compute to be able to do some sort of PoW challenge. Besides, unlike humans, bots aren't in a hurry, so they can spread out the work across a long time to minimize disruption.
the device acts as a proxy. i don't think any browser is running on the device, it is just forwarding packets.
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Oh, but you can PoW every page.
Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).
> Oh, but you can PoW every page.
They meant you can’t PoW every page transition.
If clicking every link on your website throws you back to another Anubis page for 2-3 seconds, users will bounce.
That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.
Anubis's default 1-week token lifetime may not be nearly enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine.
We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
If that happens the browser engines (all what, 3 of them?) can add a PoW API to call into native code. Or a pathologically scalar algorithm can be adopted so that wasm is good enough. RandomX or something close to it probably qualifies.
There are PoW approaches that even the playing field between data centers and desktops. RandomX is my favorite.
For what it's worth I'm working on hashx support. It's just going to take a bit to ship while I do browser testing with broken browser configs.
Interesting. How do they tell the difference between legitimate and forged ip owner records?
It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).
Uh, is it resistant to parallelization across multiple sites? Because that's the situation for the scrapers. They're not trying to solve a single PoW challenge across many cores.
Typically, the machine doing the content processing, including solving PoW, is the centralized "control" node described in the article, not the machines who's IP addresses are being used. In typical residential proxy networks, the residential proxies are exposed to the customer (the person paying for and using the proxies) as just SOCKS5 addresses, and no computational power from those compromised devices is made available for the scraper besides that used to power the SOCKS5 server itself, the customer is just paying for the transport and address (and indeed, is often billed on either a per-GB or per-IP basis).
In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.
Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.
These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.
This isn't to say that true botnets built with actual malware delivered through either software exploits, phishing attacks, or watering hole attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.
That's a completely different question, your claim was about parallelism.
At least anubis works for me. (I run umatrix)
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
> Unfortunately whatever HN is using routinely blocks my login with "Sorry."
I believe that's the HN application itself, not a WAF in front of it.
HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented.
Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.
Why is that good?
It's relatively difficult to enter and edit any significant amount of text on a phone, so phone-based discussions tend to be shallow.
HN accessibility isn't bad, since it's all just text.
i personally like using http://hcker.news as a reader, its much nicer
Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.
> but PoW scales
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
[1]: https://news.ycombinator.com/item?id=30175269
Can you not design a PoW that is most efficient in a browser? Don't brute force hashes like Hashcash/Bitcoin, do something similar to RandomX instead but in JS. Browsers ought to run the fastest JS interpreters already so if interpreting JS becomes the bulk of the work, that attack might not work. Maybe even involve the DOM or whatever else makes sense.
Or you can go full Reddit and just block anything that seems even remotely suspicious.
Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.
Using VPN? Blocked.
Your iPhone is too old, blocked.
Your screen brightness too low? Believe or not, blocked.
My forum got scraped so hard that the ISP blackholed the IPv4 several times a week.
I've ended up putting only IPv6 on the domain. It's running this way for 2 years already.
Even worse, not blocked, shadowbanned.
Parks and Rec reference?
> Your screen brightness too low? Believe or not, blocked.
... What?!
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
On a small enough site (even LWN might qualify) the chance of two random sets of client IPs intersecting can be quite low.
Private trackers do this. If they ban a user that geolocates to a certain city and ISP, they'll ban new signups from that city and ISP because there's probably only a few users from the same city and ISP. And then report to their friends at other trackers, that a user with that city and ISP is trying to evade a ban.
PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
This is called “installing a cryptominer on your web page” and is generally considered illegitimate.
> generally considered illegitimate
But why? Obviously an unjustified cryptominer is bad, like unnecessarily slow JavaScript, but this one has a good purpose and to the user is no different than PoW.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
I had meta's crawler hitting the pi searcher at something like 5qps for days on end, just ... querying for substrings of pi, ignoring robots.txt, etc. it wasn't enough to break anything but it triggered a lot of alerts.
I can imagine that sites with dynamic content and potentially unbounded query types or pathnames are in danger from particularly stupid crawlers.
Meta has hit me with over 1,000 requests per second. Luckily it tripped my rate limiting but geez.
I always thought it's the web search tool.
Grok actually shows a number of sources used for an answer. Once I asked it something simple and it apparently scanned 200 different websites. And it was just a short prompt. Now imagine millions of users asking for something multiple times a day.
I kind of wish the recent Google monopoly court ruling had forced Google to open up their index to anyone, not just Perplexity/other big players.
That's really a huge issue right now (to some extent even before the AI hype) that almost everywhere google is effectively the only entity explicitly allowed to scrape.
Yeah. It was always like that. It's like that comic Know The Work Rules ;)
> a badly configured scraping setup?
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
Maybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
It should really just be called DDoS, at a certain level of incompetence intent doesn’t matter. You’re right that there’s zero (information gathering) benefit over reasonable scraping which wouldn’t cripple the site.
Who’s doing it, are they even using the data?
750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour
Millions per hour is tens per second though; perhaps the fix is performance improvements
We have put in a number of performance improvements, yes. The nice thing about those is that they also make the site snappier when it's not under load. Right now we have just over a million items in our CMS, plus our publicly available mailing list archives, which are much larger, even if they're less frequently referenced.
Yeah we can just rewrite the web servers in Rust ;)
That'll be great until.. they rewrite the scrapers in Rust! Then we're really hosed!
1,000,000 / 3,600 = ~278
Maybe someone is paid per scrape, without reduction in payment for duplicates.
Maybe every web query for Linux commands in $LARGE_COUNTRY checks all the Linux websites again.
Hah. I have a homelab with a couple of sites, including a personal Forgejo installation.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
The paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.
Maybe they are aggressively scanning for updates on the page
It's not AI companies scraping these websites, it's AI companies creating a massively profitable need for data, and every random Joe with vibe coded scrapers tries to make a buck out of it.
It isn't. AI scraping has nothing to do with it, for the reasons you said. Someone wants the web to go offline, they are DDoSing the entire web, and it's working. For some reason we are tackling the symptom instead of finding out who that person is. Come on, it can't be that hard to subpoena Bright Data. The law enforcement system knows how to track down someone who's trying to be anonymous on the internet.
So who would benefit from the entire web going offline?
Which powerful entities have historically hated a free and open internet?
...all of them??
Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.
I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.
Thanks for making common crawl as good as it is. It’s a really important part of making the Internet better
Appreciate your kind words! Many people have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
Well this is not what is happening in practice, Wikipedia / Wikidata, OpenStreetMap, OpenFoodFacts... All provide APIs and even a full dump of their database available to download for free, but no, the stupid bots still DDoS them 24h/24.
Why don't they take legal action?
How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
I have essentially this at work, but the scrapers ignore it. (Or at least many, many scrapers ignore it.)
that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.
I was thinking you scrape your own website every day in the middle of the night when traffic is low, and make that available. They can come and collect it every day if they want to.
Yeah. Though I guess the point I thought of was like a deals site. That would have infinite pages and content
Feels like it would be a good time for freenet and the like to catch on.
The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone.
10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access.
But really, this finally explains the hordes of really basic boring games that just advertise other boring games. Idle games and the like that really just want you to keep your phone unlocked and open. Millions of downloads on the app stores for entirely offline content (and ads) and no way to block the network access.
> 10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what.
Why does network access need to be a binary, all or nothing?
When you install an app, the app should request permissions to specific DNS names, i.e. pointing to the servers that the app's authors operate. If I install Todoist, the app should only ask for access to Todoist's servers. If I install Netflix, the app should only ask for access to Netflix's servers. The OS can then put a DNS firewall in and block any network access that wasn't granted when the app was installed.
> And both ios and android make it hard to deny apps network access
The list of apps that genuinely need "any" network access (web browsers, VPN apps, stuff like Termux...) is incredibly small compared to the list of apps that need access to a small number of VPN targets (these days, most apps). Apple / Google could even decide, if they really want to make it easy for apps to request network access, to basically allow apps to automatically get network access, so long as the list of domains the app needs access to is no more than a handful. The security value of isolating "all" network access permissions to only the relative handful of apps that actually need to request it, would be huge.
The Bright Data “free” VPN they’re talking about requires the user to go through steps to enable it.
These aren’t as simple as downloading a free game and then the phone is compromised as long as it’s installed.
The users who install these things don’t care about permissions prompts. They’ll follow instructions to tap any prompt the instructions ask. They want the free thing and don’t care what they have to do to get it.
The article also talks about NetNut, which was embed in many apps released into the app stores.
GrapheneOS allows you to deny network access per app pretty trivially. Google Play services make it a bit more difficult because the app might marshall the network request through that; I'm not sure how to verify that behavior when it happens.
Thanks for mentioning GrapheneOS. I'll just explain to others what "pretty trivially" means here. GrapheneOS adds a huge checkmark "Network access?" when you install an app. It's impossible to miss.
The problem is the yes/no options - I want to give access to specific endpoints - but now that’s “all of Cloudflare for everything” which means the web entire.
It used to be you could let Onkyo App™ access the five IPs for onkyo.com and be done with it.
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
cloudflare routinely denies my human-piloted browser now, on many sites.
you're not a bot thats irrelevant
> I think that nobody would care if I use wget or curl for few pages
If only you were the only one doing it...
One article mentioned in the OP was discussed here:
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?
It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.
As someone dealing with similar on a large site, I'd love to see a private community to discuss some of these issues.
I have a medium-sized Discord server of web sysadmin people (mostly wiki operators) that came together after I wrote a similar blog post [1] about how the LLM scrapers/resproxies are making it suck to run wikis. Not sure if there's other private communities out there, but feel free to email me if you want to join
[1] - https://weirdgloop.org/blog/clankers
Disclaimer: this works for my very small number of personal services that I run. I have no idea how this would (or probably wouldn't) scale at all. Also, the methodology I describe below is based on what I'm able to do technically, which is pretty much limited to bash scripting.
On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and therefore there is no reason for the existence of that traffic (at least as far as I'm concerned), and therefore I treat it as malicious.
The address is recorded and goes into a database.
Periodically, the database is dumped to a file in a format that the firewall reads, and all the 'malicious' addresses detected above are added to a list so that those addresses are blocked from accessing the legitimate service ports. (analogy: if you throw an egg at my outdoor wall, I'm not going to let you into my house through the door because I don't want egg on my furniture).
I have a blocking period of about 3 months - because the things I run are important to exactly a single person. A blocking period much shorter would be recommended to prevent the gross-overblocking of legitimate users who may have un-lucked into being assigned a residential IP address that was previously used in a proxy-scan-scam.
Discard this if it's a stupid idea at-scale, but I quite the like the 'idea' of it, and I made it work, mainly for the technical challenge.
Project is here on Github: https://github.com/UninvitedActivity/UninvitedActivity
It’s a great idea but not sure for larger websites when these residential proxy platforms are using innocent user ip addresses. Then you’re left blocking innocent users. It’s a tough call.
I did something like this using fail2ban for some time, but 1) it didn't help much due to the larger number of IPs, 2) it blocked widely used VPN services.
Most residential users change their IP address every 24 hours.
I came up with the idea and helped build Grub, the distributed crawler. Looksmart bought it, ran it for a time, then sold it to Wikimedia. I reclaimed the name recently (abandoned mark) and have a new crawler now that is agentic. I use it for my own research runs, and it's not my main focus at this point, nor am I trying to get it attention. A lot of LLMs and coding agents can easily fetch content if it is needed and we're blind to how they do it. See the Claude plugin for Chrome as an example of using it in a user-in-the-loop solution. That said, I've been spending a little time thinking about how to bake the contract into the crawler, as opposed to expecting someone else to act ethically using it.
Grub was, in a very real way, a botnet. And, we harmed site owners when we were operating at full capacity. There were a few bugs in the early days where we would reschedule a site because the ingestion in the server broke, which then caused the page to be rescheduled. Stupid error, and we fixed it, but it's illustrative of the fact even good intentions isn't enough here.
What I've come up with over the years is similar to the idea Cloudflare is implementing with payments to site owners by charging the crawlers. My objection to Cloudflare's implementation is based on a personal opinion about Cloudflare being a single point of failure and also a decrypted choke point. Their ideas about how to handle crawlers, and pay for the load on the sites is solid. It presumes to use the 402 response to demand payment. I'm clearly biased about Cloudflare, but that's my prerogative here.
It may be possible to solve this with cryptocurrency, in a distributed way, and I've prototyped a system that uses the Lightning Network to handle the payments from a 402 response. Lightning Labs also worked on a project called Apeture for a time that did something similar.
HN's site knows every item ID, and it knows fresh IDs get read in a predictable distribution while old ones mostly sleep. Sustained access outside that is itself the scraper signal. No IP reputation needed, which matters now that residential proxies burn an address after a handful of requests.
Karma gives you a clean way to let humans through. Issue logged-in accounts with decent karma a token whose cold-content budget scales with it (the karma), so an account with history scrolling back through a 2014 thread just reads it. Karma should gate the tier, not be spent as currency, or upvote rings become a crawling business.
Anonymous readers who deep link into one old thread from a search engine get the first fetch or two free (and you watch the article IDs, not the IPs). What remains after those carve-outs is bulk traversal of cold IDs with no identity attached, and that traffic gets rate limited and answered with a 402: pay per page over Lightning, priced at a healthy multiple of what residential proxy bandwidth already costs, or come back slowly for free.
There are probably holes in these thoughts. It's one of the harder problems to solve, for sure.
Ive been seeing a 'sorry' message occasionally when accessing older pages.
Is that a side effect of whatever you are doing?
It was, but there were too many legit users getting affected, so we turned most of that off a few days ago. Are you still seeing it?
Not in last day no.
Ok that's good - if you get it again can you let us know at hn@ycombinator.com? We definitely don't want to exclude legit users.
Yes will do
LI've emailed there in May and June about different topics and gotten no reply to a request to confirm receipt. Is there a backup method for when your algorithm throws people's email away without even informing the sender with a delivery failure notification?
We get thousands of emails - far too many to even read them all, let alone respond the way we would like to.
There are other reasons why we might not respond, but overload is the main one.
Edit: I only see one email in the archive related to your account. It was from Sept 2024 and we responded to it. Are you talking about a different account?
Oh, okay I didn't know you weren't getting around to all incoming emails, that makes sense then. Previously the responses were fast and reliable and I thought I was being dropped by a silent bot/spam detection algorithm (as is becoming frustratingly common).
It's indeed from two different email addresses from the one on this account. They're not that important so never mind, thanks for checking and for the reply!
Not dang and not the person you are asking but there is no CDN. HN is just two servers running BSD one active and one standby. HN is all text so there is not much bandwidth usage.
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
[1] - https://news.ycombinator.com/item?id=48416693
And the backup is field-tested to fall over within a few hours of the primary ;-)
<https://news.ycombinator.com/item?id=32048148>
<https://news.ycombinator.com/item?id=32031243>
(AFAIK that specific failure mode has in fact been addressed.)
For one datapoint ...
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
<https://news.ycombinator.com/item?id=36078578>
<https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
HN is exported to firebase, which you can hit directly, for that sort of purpose
https://github.com/HackerNews/API
I know that.
I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard.
API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites.
That said, a standard, say, JSON packaging of website contents available on request might be fun to have.
I feel less bad hammering firebase in a "while True:" loop vs hitting HN's servers.
12,000 times less bad? <https://news.ycombinator.com/item?id=48868910>
Between me and you, I don't think Google would register that amongst the noise of just running the service anyway on their monitoring systems.
You're right to point out that if you're trying to get the contents of dead objects the API is of no use though.
Google might not register it, but in some related testing, even a few hundred requests through the API (serial, not concurrent) takes much longer than a single "Past" page request would, even if the latter were substantially rate-limited. Of course, if those requests (to HN itself rather than the API) are blocked entirely, that's a moot issue.
On dead/flagged items, there's some value. Whilst the title/URL context aren't available, just knowing what fraction of submissions and comments are moderated is interesting data, and it is possible to construct patterns against specific accounts.
I'm frequently encountering what appear to be banned accounts. Being able to trace those through the API to see where and when they were banned, or now much moderated activity they're generating, can be useful. I'm relying heavily on the "/replies?id=<UID>&by=<moderator>" search endpoint (generally dang, tomhow, sctb, or pg as mod) currently to find out if there was a specific ban admonishment from a moderator. That's often but not always the case.
But it's not possible, say, to tell through the API what sites are banned. Looking at site history with "showdead" enabled can tell you that though, e.g.:
<https://news.ycombinator.com/from?site=synthetica.cloud>
(From the New queue, one of several "dead" submissions not flagged, suggesting a site ban.)
Hypothesizing an undocumented "site" API endpoint ... doesn't seem to check out:
Returns: (Similarly for "domain", "url", and "URL".)And there's the "day" endpoint doesn't seem to work either, though it conspicuously does not report "not found", e.g.:
(I've tried a few other date format variants, including Unix time (seconds) without success.)Looking at the API ...
... it's starting to make sense, but ...
... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:
<https://news.ycombinator.com/front?day=2008-11-05>
It's the collection of 30 items from that date I'm interested in. For my scraping, I don't actually need to further query the individual posts as I've got the elements I'm interested in (title, date, story position, URL, votes, comment count, submitter, site/domain) from the index page itself, parsed out of the HTML. The "Past" entries alone are a significant (though not huge) request load. To update the past three years would be about another 1,000 requests, which, if fulfilled and modestly rate-limited would hopefully not keel the servers over.
Once I've pulled in those "Past" pages, I could of course do further API queries, though at this point I don't see any specific need to do so.
I suppose that requesting the "past" links be included in the API set could be a request I might make of HN, or the ability to request, say, all submissions (or comments!) for a given date.
There are groups which have done HN analytics in the past using the API, for example Whaly.io:
"A Year on Hacker News" <https://news.ycombinator.com/item?id=31295219>
"Top Hacker News commenters of 2021" <https://news.ycombinator.com/item?id=29778994>
"What Happened This Year on Hacker News (2021)" <https://news.ycombinator.com/item?id=29769470>
I could look more into their methodology to see if I can use similar approaches.
The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)
OK, looking at a recent dead atomic128 comment:
So userID is visible.And from a current dead submission in the New queue:
That's missing the title and URL, as I suspected it would, though the submitter UID is available.To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.
If you scrape 6TB from across the web or grab it from one place, it's still 6TB
I was involved in both sides of this battle over ten years ago. Things haven't changed all that much.
It's important to note that neither side has moral legitimacy. Not everyone who carries a rifle is a enemy. Not everyone wearing body armor is a saint.
I have given up on the idea that "human vs bot" matters at all when it comes to anything other than voting (which should only be done in person with paper and pen, by the way.)
You could make an argument that "likes" are a form of voting, but you shouldn't. We need to abandon the idea of supposedly democratized algorithms and focus instead on actual democracy.
I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
Caches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.
As someone who operates a wiki, this does not solve the problem.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.
Semi-off-topic but...
On my sites, I see ClaudeBot consistently (and has been like this for over a year) ask for "${SITE}.com/base_dir1" and then get the redirect (Caddy does this automatically) to get "${SITE}.com/base1/base_dir1/" (trailing slash).
The hrefs on my sites include the trailing slashes for directories, so looks like ClaudeBot's internal code is stripping them off before requesting them, and therefore essentially makes almost 2x the requests to my sites for directories, half of them ending up being redirects back to the same url but with the trailing slash.
At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.
> I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
If you block it, it comes back with a residential proxy network and headless Chrome. Better not to block based on the obvious signs.
Blocking is too obvious. I would prefer to feed back false information but only to LLM crawlers.
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
https://fzakaria.com/2026/07/09/who-does-anubis-actually-sto...
I wrote about this recently as well.
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
And it's made necessary because another group of people thought that selling IP blocking services would be a good idea. One party sells walls, another party sells ladders.
Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.
I think they also have to operate in countries that don't mind shady things like this.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
TIL:
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
WTF. That's just botnets.
Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...
Mmm... your quote (IDK where it's from) mentions them having consent from device owners, but your FBI link cautions on how to avoid getting infected by malware.
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 address for all their employees, and most VPN services.
by consent they mean a dialog/EULA with careful wording was display and the user clicked ok.
And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
There are absolutely no actual drawbacks for most users.
> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP.
Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.
Thank you. Gmail must not be like our fellow HN users we see here, quoting a couple:
Although Google isn’t afraid to completely block iCloud private relay from Google scholar. Other sites may reject the first iCloud private relay visit, then reopen site in new tab and often automatically assigns new unblocked IP. Anyway, I would’ve thought it’s an acceptable cost from e.g. Google’s perspective to block ranges that did something bad once in spite of collateral damage.Thank god for residential proxies.
Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet
By providing a way for corporate AI scrapers to operate with impunity and force the last few independently-run websites to move to the cloud?
By providing a way for independent hackers to extract data from closed commercial websites.
No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.
The article was called "The one we're commenting on"
> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.
Does that sound like your typical self-hosted blog?
> Does that sound like your typical self-hosted blog?
When compared to
> They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
yes, absolutely.
The Bright Data mentioned in the article, as well as other similarly malicious but even harder to identify parties, most certainly do fire up residential proxies, no matter what the site, no matter how useless or duplicate the data which they're trying to get. Not at first - they start by trying to get your content from cheap data center connections - but as soon as some kind of bot-mitigation appears, they move to residential proxies to try and evade that, with a first tier coming from "global south" residential proxies, and then scaling up to (presumably more expensive / less widely available) proxies from the USA (I've seen some from Europe, but very few in relative terms). Each tier also appears to have the option to run JavaScript.
i know a few very large startups that used it to fake their way into an exit
unethical yes but really raises the question as to what we see is real or not
Money is real. DAU that don't pay subscriptions, or don't lead to paid conversions on hosted ads, are worthless.
"Raises the question of what we see is real"
No they really don't, dishonest founders do that.
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
ive seen unscrupulous founders fake traction during diligence, which is my day job
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
In theory, proof of work that is used to mine a cryptocurrency could be a solution.
Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.
The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.
I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.
Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.
Proof of work, even "custom", where the user does not need a particular interaction with the page, does not work. The scrapers are running headless Chrome and solving the work. They do not care, they do not pay the bill, the compromised system's owner pays the bill.
I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The PoW challenge prevents less than 10% of the bots.
Thats why I said proof of work that is used to mine a cryptocurrency to pay the bills of websites that serve information.
As long as the website gets paid more than the cost of serving the pages, it does not matter if a human or a bot did the POW.
Securing signup forms is another issue. Maybe related. But not what I was referring to.
Very good point, I missed the point that the PoW is paying the publisher. I will have to dig into this, this is a pretty nice idea.
Residential proxy users don't get to run computation on the proxies.
First of all I am very critical of the $100 per consumer device per month figure.
Also all major browsers block crypto miners on webpages now (for good reasons) so it may prove difficult to allow "good" mining scripts while still blocking "bad" ones.
I don't think this is a practical solution
Better skip the PoW part as it'll be wildly inefficient for most of the work done.
Instead, exchange web traffic for actual $. Say, some kind of tokens that are easily turned back into hard cash through a 3rd party.
Requesting a 100KB file? Okay, that'll be a $0.00002 token, please! (visitor's user agent provides it in a manner transparent to regular web users). Requesting a 3MB image? Okay, that'll be a $0.0005 token, please!
Result: niche websites earn hard cash. It doesn't matter much if you're hammered as long as the hammering comes with a corresponding flow of tokens (read: $). No token(s)? No service.
Regular web users would pay for those tokens through their normal internet service fees, and otherwise not be bothered. Massive scrapers would have to pay somehow for the tokens to be served web data at all.
In effect: put the bulk of public web sites behind a paywall. But with the bar low enough & in a manner that it's transparent for regular web users. Clicked "reload" by accident? Oops, internet service bill got upped by 2 micro-$.
Proof of work captchas are widely deployed, especially on more niche sites. Kiwiflare is one (used for a harassment forum)
You could've just said Anubis to avoid giving those guys advertisement.
Proof of work doesn't help if the abuser is massively decentralized and is using other peoples moneys.
They discuss why this might be a bad idea in the article.
They discuss why proof of work is bad, not crypto
> partly because it causes annoying delays for those trying to get to the site
This is true but usually a small issue. It’s further alleviated by cached tokens so you only have to solve the challenge once in a while per site, and a login token may let you skip it.
> partly because it seems inevitable that the scrapers will eventually find their way around it…A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
Solved by making money off it.
From the article:
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
Opnsense has a traffic capture feature in the interface diagnostics menu, if you want to spot check what servers the devices are currently talking to.
Should be pretty obvious: client devices and internal services will have no traffic >95% of the time, just NTP for timekeeping, DHCP lease renewal, and associated ARP (running total: two dozen packets if you monitor them for a full 24h), then any system updaters (readily identifiable by the initial DNS requests), and finally of course you'll see the traffic of the service that the device hosts, if any, which can be easily dismissed by not looking at incoming connections (scraping uses outgoing connections)
> Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
[0]: https://github.com/jow-/nlbwmon
[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...
[2]: https://docs.mossdef.org/adblock-fast/
What a pity. Mostly I just want personal archives of things so that I can search them much faster than commercial solutions and the like.
I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themselves into believing that creating massive barriers to access their sites is a fair trade-off. Unsurprisingly, I have nothing but negative associations with their mascot.
The FSF has the right idea about all this:
> Some web developers have started integrating a program called Anubis to decrease the amount of requests that automated systems send and therefore help the website avoid being DDoSed. The problem is that Anubis makes the website send out a free JavaScript program that acts like malware. A website using Anubis will respond to a request for a webpage with a free JavaScript program and not the page that was requested. If you run the JavaScript program sent through Anubis, it will do some useless computations on random numbers and keep one CPU entirely busy. It could take less than a second or over a minute. When it is done, it sends the computation results back to the website. The website will verify that the useless computation was done by looking at the results and only then give access to the originally requested page.
> At the FSF, we do not support this scheme because it conflicts with the principles of software freedom. The Anubis JavaScript program's calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware. Proprietary software is often malware, and people often run it not because they want to, but because they have been pressured into it. If we made our website use Anubis, we would be pressuring users into running malware. Even though it is free software, it is part of a scheme that is far too similar to proprietary software to be acceptable. We want users to control their own computing and to have autonomy, independence, and freedom.
https://www.fsf.org/blogs/sysadmin/our-small-team-vs-million...
I think all of these mitigations are unfortunate. They hurt one of the things that makes the web cool: it's a stable, stateless, idempotent way to access data.
This makes it a prime target for aggressive scraping by LLM companies, but it also makes it accessible and fast, and a prime target for benign use (like archive.org or "read later" services).
For my own sites, I'll eat the cost of the crawlers (mitigated by making the sites as efficient as possible) and keep them available to everyone.
https://archive.fo/PAcF5
Ever since bots became a problem on the internet 10-20 years ago, it has seemed like the common-sense solution is some kind of micropayment. Pay $0.01 to view the page. When money is on the line, scrapers are likely to be more well-behaved, even if they do pay. The problem is, and has always been, the friction of payment. How do you pay $0.01? The credit card processors will tack on a $6 surcharge. We need a trusted third-party that turn money into "internet article credits" that you can spend in small increments, like a video game. But I suspect that thousands of people have already though of this system, and tried it, but ran into some roadblock. I'm guessing there's some egregious regulation that makes micropayments impossible.
> I'm guessing there's some egregious regulation that makes micropayments impossible.
More likely there isn't any kind of universal standard that's easy to implement for browser makers, has low overhead, and preserves internet users' anonymity as much as possible.
The currently existing friction of using micropayments is the problem here, I suspect.
> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?
The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").
> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.
In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:
brdtnet.com
luminatinet.com
bright-sdk.com
luminati.io
As listed in this article, on HN's front page 34 days ago: https://news.ycombinator.com/item?id=48422993 (https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...)
Well the argument appears to be, people put them in their apps instead of ads. (Or more likely on top of ads.) The argument is money.
The users presumably don't know about this, or you know, they clicked, "I agree."
Nearly Half of LG Smart TV Apps Contain Residential Proxies
https://news.ycombinator.com/item?id=48635954
Of course the issue with blocking residential IPs is that then they would be prevented from doing normal things on the internet.
At which point, millions of people will be forced to complain to their local representatives and... hey presto? :)
Why would anyone who doesn't have a use for a residential proxy have an argument for residential proxies?
I use them to scrape closed sites to make the information more open. For example YouTube.
How much does routing traffic though residential proxies cost?
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
Blocking in ipv6 works roughly the same way as in ipv4, just that the scale is different. Instead of blocking something like a company's /24 or an ISP's /16 when they don't respond to abuse messages, you block the company's /48 or the ISP's /32. It'll vary per organisation how large a range they got exactly but you can see that in WHOIS. End users are no longer at a /32 (v4) but at /64 (v6), or some prosumers might have a /29 (v4) and /56 (v6). Same concept, just a different prefix length
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
Yes that was my point. We should just block all of them.
Has no one noticed their miniflux instance failing to fetch feeds because of this?
So far I’ve only encountered one site that blocked automated access to its web feed. I assume that was just an oversight.
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
Is reasonable! Many others are not
> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
What’s more, they will scale up with increased resources on the site.
If you redline at 20 searches a sec, and put in 4 more workers, suddenly you’re serving 100r/sec to the bots, paying 5x for it, and your users are still seeing shit qos. I've seen multiple cores of nginx saturated just dealing with one dos/crawl run on a somewhat high profile site.
I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.
If it weren't a real problem, these types of articles and services wouldn't exist.
Plenty of complaints exist about things that are not real problems.
Well that's not what's happening here, lol.
Why the air quotes? Evading a ban and using (potentially ill gotten) residential ips to circumvent that refusal of service, is a bad actor.
Surely that depends on the motivation for the ban.
No. When someone says you’re not welcome, you’re not welcome. Regardless of the reason.
If I get kicked out of Epstein island because I refuse to **** a child, that doesn't make me a bad actor.
And then going there 1 million times under fake identities? Yeah, I'm sure that's not a bad actor.
mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
"He who has the gold makes the rules" is older than the pyramids.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?
The excessive scraping and ignoring robots.txt only breaks the informal social contract established over the past decades of the open internet.
The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is little, but is distributed over millions of residential devices all over the world, making it difficult to categorize or block. That should be illegal, and that's what Google et al can be expected to be policing on their app stores.
On what grounds would it be illegal though? Things don't become illegal just because you don't like them. They may become illegal just because the president doesn't like them, but I don't think you're him, and in the absence of that, there has to be a majority of Congress and most of them want a reason.
Because they don't have the informed consent* of the owner of the device wich ends up running the code?
* no, small print in a click-through agreement doesn't count.
It's not illegal to run code on a device without informed consent to everything the code does. The CFAA may be excessively broad but it isn't that broad.
Google itself is a huge database.Who makes these rules depends on who's leading the market.
I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
I ran a gitweb server which was battered by bots so I eventually had to take it down. Gitweb! You can just connect using the git protocol and download everything vastly more efficiently!
In other words, they don't care at all. For them, residential bandwidth is completely free.
Right. This whole controversy makes no sense as a pure scraping thing. It seems more like someone is trying to take the web offline.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.
Common Crawl's dataset was downloaded in full 100 times in 2025.
We agree that it would be great if it was even more widely used.
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
Common Crawl's archive has metadata that says when each record (html file) was crawled.
But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.
We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.
Sorry, I wasn't clear. In order to know you have an up to date copy of the page you need two pieces of information:
1. When the page itself was last updated
2. When the crawled copy was last updated
In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.
The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.
This is a predictable consequence of age verification laws and social media bans. Formerly VPNs were a nice to have but now they are a necessity in many countries to navigate the modern internet.
The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.
That does not explain why so many residential VPNs operate with so many IPs in countries where there are no social media bans. Here in France:
- i know many people who buy shady IPTV boxes from stores/markets for like 50€/year
- i know some people who use "smart lightbulbs" and other nonsense
- almost everyone i know plays free smartphone games, which as LWN reminded, may contain a shady SDK
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
> The moment it does you'll be able to tell because Bitcoin will crash to 0.
The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now.
Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized hardware this computation only costs fractions of a penny, so they are barely hindered by it.
The SHA256 based PoW scheme has a very heavy ASIC advantage. People have tried to design PoW scheme that minimize the custom hardware advantage, but I'm not sure if they managed to close the gap far enough to make PoW feasible for this application.
Residential proxy users don't have the ability to run compute on their proxies.
https://salad.com/salad-gateway-service
https://salad.com/earn
> widespread scraping of web sites in search of training data for large language models and related projects
This is a good thing, thanks to this we have powerful open source LLMs.
> This activity overwhelms sites with traffic.
When LLMs get good enough, we won't need those sites anymore :)
[not satire, this is what I think, without self-censorship]
There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
https://news.ycombinator.com/newsguidelines.html
Edit: If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
Just curious dang, did you warn them before banning?
Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.
No, but if they have a change of heart and genuinely want to use HN as intended, they're welcome to let us know. I've added that in an edit now.
Seriously, dang?
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call.
I made it all the way back to https://news.ycombinator.com/posts?id=atomic128&next=4628060... (6 months ago) before seeing posts about anything else, only to find that there was a different agenda before that. Not cool.
Edit: and before all that, there was this: https://news.ycombinator.com/posts?id=atomic128&next=4164795.... This is obviously not using HN as intended.
Exceptions?
<https://news.ycombinator.com/item?id=47116093> (LLM but not PF).
<https://news.ycombinator.com/item?id=47095664> (a16h)
<https://news.ycombinator.com/item?id=46695693> (vuln exploits) 2026-1-20
<https://news.ycombinator.com/item?id=46280602> (???, but not PF)
<https://news.ycombinator.com/item?id=46195234> (Monero / Dark Web) 2025-12-8
<https://news.ycombinator.com/item?id=45894305> and <https://news.ycombinator.com/item?id=45826273> (Tor hidden service) Nov 2025
That's from the past 20 comments.
Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.
Again: Not a one-note flute, though fairly focused of late on AI and poisoning.
Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (<https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.
(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)
The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.
I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.
> Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I took a look at the most recent comments from both accounts and they don't look similar to me in this respect.
I think there are two questions here though:
1. Was the violation egregious?
2. Did it deserve an immediate ban, or did they deserve a warning etc.?
Seems to me the answer to (1) is yes, but the answer to (2) I'm less sure about.
Jon's been around a while and some of the piss and vinegar of youth may have subsided. He does tend to show up with LWN comes up, whether as a topic of discussion (or more often) from submitted articles. That's his baliwick, and again, I don't fault him at all for it.
Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
I'm pretty confident it's not that.
HN's prime directive is "anything that gratifies one's intellectual curiosity": <https://news.ycombinator.com/newsguidelines.html> and many, many, many dang comments.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Occasional alts are explicitly permitted, though not to engage in abuse (e.g., mutual admiration societies, sock-puppetry). See: <https://news.ycombinator.com/item?id=9963551> <https://news.ycombinator.com/item?id=9823379> (both against sock puppetry) and <https://news.ycombinator.com/item?id=9122086> and <https://news.ycombinator.com/item?id=7504621> (on where throwaways are/aren't permitted).
Where HN does favour persistent accounts the stated claim is to foster community, rather than for nefarious tracking purposes: <https://news.ycombinator.com/item?id=18082346> and <https://news.ycombinator.com/newsguidelines.html>. From that last:
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
I'm confused why everyone is pretending that a ban holds any meaning here. he probably already has a new account, it might have made sense to just silently ban him in the hope of imposing a minor cost on what you believe he is doing, but wasting your time addressing it imposed a far greater cost on you than him.
I understand how it can be confusing. The key factors in doing it this way are (1) the community regards older accounts, especially ones that have significant posting history, as more credible; and (2) doing it publicly rather than silently has transparency value.
This is a strong positive sign that poison fountain works.
I wasn't aware of this project. Thanks for the heads up.
It's not a sign about that project in any way. I had never heard of it and have no opinion about it one way or the other.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
Yeah. A fun thing to do is to try and actually read common crawl!
Really makes you think, what we're feeding them...
exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
>types of operator running residential-proxy networks to attack web sites.
This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product.
>paid for hijacking their users' network connections
Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors.
>Recently, LWN was subjected what was, by far, the heaviest scraper attack yet.
LWN is a static site. To me it seems more expensive to use Anubis than just serve the actual page.
>will now check for NetNut-infected apps
Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).
>sell access to the user's internal residential network
That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space):
Local and loopback addresses like 10.0.0.0, 172.16.0.0, 192.168.0.0, and 127.0.0.0 do not work. It will not connect to people's private file servers on their network.It'll still connect to IPv6 addresses and bypass any firewalls.
Also users might become part (victim?) of a police investigation because of illegal actions that seem to originate from their local residential connection.
So still good to take down such backdoors. Would be nice to go after the botnet operators as well...
Again, why do we allow China on the Internet?
Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.
Sorry, I understand scraping is a problem, but talking about open Internet while simultaneously complaining you can no longer discriminate datacenter IPs like you used to is hypocrisy.
I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.
Noone complained we can't discriminate DC IPs, though to be fair some (imo bad) operators did just that. This is not even about preventing bots, which has perfectly legitimate usecases (eg. Internet Archive).
This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple rate-limiting, they have to employ so-called residential VPNs. They're residential in that they route through real user connections, and so you can't block the IP/subnet without dropping a certain amount of legitimate human-driven traffic.
Personal example: some time ago, i had to disable a wordpress plugin on a site that was causing 100% CPU usage on the whole box (hosting dozens of wordpress instances). That plugin was a simple calendar, but a bot was repeatedly scraping non-existent (or rather, "no event planned for this day") pages for every date in the calendar that you can represent in the DB timestamp, clearing the cache as it went to try and find new events for 1000 years ago. Whoever operates this IP space doesn't matter to me, i'd just like to block them because they don't respect robot.txt… but i can't because they use a "residential proxy" and will change IP address every hour or so.
This is definitely an issue, where data centre addresses are, by default, second class citizens or even persona non grata. The problem is that this reputation has been, unfortunately, well-earned.
There's no easy answer here. The ephemerality and pseudonymity of VPS address usage screams untrustworthy, and the only way to reign that in is better identification of who is using the VPS/address or significantly more restrictive rules applied to data/port usage. And I'm not sure if I like the general direction that points towards - away from the "open internet".