If an automated service is pulling the top 100 domains from CF and naively trusting them, why can't it also pull the categorization information that's right there and make sure none of the categories are "Malware"??? Who would write something like that? It's absolutely believable that the top 100 domains could contain malware domains...because of the nature of botnets and malware.
People make mistakes. Security engineers need to understand what sort of mistakes people are making and mitigate that risk. Brushing it under the rug as silly users making mistakes doesn't protect anyone.
The automated services using this for security-related purposes are presumably built by "security engineers", if they're making mistakes like this they're obviously woefully underqualified.
I'm a security engineer, I have built things like this, and I made the original comment. A lot of my job revolves around developing automation for security needs.
Also, many of the top 100 domains serve user-generated content (like AWS/S3). Blindly trusting anything from them just because they are big is so woefully misguided it boggles my mind; I seriously doubt that anyone is actually doing what is described in the article.
Idk, I have done security audits for startups and small tech companies. They won't have a security engineer on staff and are "moving fast and breaking things". I've seen things much more misguided than this.
Yeah, I'm not sure that baby-proofing everything as proposed here is going to result in a working society.
If we expected airplanes or cars to be able to be safely operated by people with zero understanding of how such vehicles work, nobody would be getting anywhere.
You eventually reach a level of stupidity and/or incompetence after which trying to alter the product to coddle those users becomes counterproductive.
Why not include them? What's wrong with have the most resolved domain being the top domain. I think it's interesting to know the actual most resolved domain, than the top of some editorialized list.
As discussed in the article, threat actors are using a botnet to game the system by repeatedly issuing queries for the domains; the list is intended to represent the top 100 domains resolved by legitimate users (and legitimate bots, I assume), not just "who can make the most queries to CloudFlare for a domain".
> We should have two rankings: one representing trust and real human use, and another derived from raw DNS volume.
Isn't identifying real humans an unsolved problem? I'm not sure efforts to hide the truth that these domain are actually the most requested domains does anyone any favors. Is there something using these rankings as an authoritative list or are they just vanity metrics similar to the Alexa Top Site rankings of yore? If they are authoritative, then Cloudflare defining "trusted" is going to be problematic as I would expect them to hide that logic to avoid gaming.
> Isn't identifying real humans an unsolved problem?
I'm not sure this was ever a problem to begin with. The obsession with "confirm you are human" has created a lot of "bureaucracy" on technical level without actually protecting websites from unauthorised use. Why not actually bite the bullet and allow automations to interact with web resources instead of bothering humans to solve puzzles 10 times per day?
> Cloudflare defining "trusted"
They would love to monetise the opportunity, no doubt
> I'm not sure this was ever a problem to begin with. The obsession with "confirm you are human" has created a lot of "bureaucracy" on technical level without actually protecting websites from unauthorised use. Why not actually bite the bullet and allow automations to interact with web resources instead of bothering humans to solve puzzles 10 times per day?
I mostly just let the bots have my sites, but I also don't have anything popular enough that it costs me money to do so. If I was paying for extra compute or bandwidth to accommodate bots, I may have a stronger stance.
I do feel a burden with my private site that has a request an account form that has no captcha or bot blocking technology. Fake account requests are 100 to 1 real account, but this is my burden as a site owner, not my users' burden. Currently the fake account requests are easy enough to scan and I think I do a good job of picking out the humans, but I can't be sure and I fear this works because I run small software.
I send them on endless redirect loops with very slow responses. Cost me very little bandwidth and it effectively traps one bot process that then isn't available for useful work. Multiply by suitably large 'n' and they might even decide to start to play nice.
>"Why not actually bite the bullet and allow automations to interact with web resources instead of bothering humans to solve puzzles 10 times per day?"
This is a great idea if you've developed your 'full-stack', but if you're interfacing with others, it often doesn't work well. For example, if you use an external payment processor, and allow bots to constantly test stolen credit card data, you will eventually get booted from the service.
I think the comment means we have these “institutional” problems that we’re constantly protecting with tricks like captchas instead of actually addressing why a payment processor would have a problem with that or be unable to handle it in their own way.
The average normal user would go months to years between needing to update payment info, so why would that require them to solve puzzles 10 times a day?
That is also notably a completely unnecessary dumpster fire created by the credit card companies. Hey guys, how about an API that will request the credit card company to send a text/email to the cardholder asking them to confirm they want to make a payment to Your Company, and then let your company know in real time whether they said yes? Use that once when they first add the card and you're not going to be a very useful service for card testing.
What you need is for all card issuers to be required to implement it by the network. Otherwise you'll still have people showing up to test all the cards that don't support it and the payment processors would still kick you off for that.
> Aisuru switched to invoking Cloudflare’s main DNS server — 1.1.1.1
I don’t suppose they use DNS to find their command-and-control servers? It’d be funny if Cloudflare could steal the botnet that way. (For the public good. I know that actually doing such a thing would raise serious concerns. Never know, maybe there would be a revival of interest in DNSSEC.) I remember reading a case within the last few years of finding expired domains in some malware’s list of C2 servers, and registering them in order to administer disinfectant. Sadly, IoT nonsense probably can’t be properly fixed, so they could probably reinfect it even if you disinfected it.
I wonder whether by now the botnets moved on to authenticating C2 server and using fallbacks methods if the malware discovers an endpoint to be "compromised"
This wouldn't raise serious concerns. Ask the customers/community if doing it before hand is something they agree with in some form of poll, then just do it. At the end of the day DNS is a million years old, out-dated and the mission is to help make a better internet. If Cloudflare straight up asked us all if it was cool to modify their DNS servers to identify / disrupt malicious use from botnets I'd agree. People not using DoH or internal things like dnscrypt-proxy need to get with the times.
I'm not saying I agree with it but we're all engineers, the internet and everything built on it was engineered, to put up with script kiddies and hacked computers and not-so-tech-savvy internet citizens using their devices and installing Infatica, and other malware/proxy services on their devices because it came within the agreement for installing some free app where their kids could 'pop bubbles' on their parents phones or some free desktop app included it; then distinguishing their IP addresses and IP-scores as they blend in with their regular human traffic makes it hard to block it. Ain't nobody got time for whack-a-mole internet, families and businesses will need to secure their networks.
Honestly I'd be ok with an up-to-date live list of all known infected IP addresses and their last timestamp for what, and who detected them as a bot/malicious IP address so I could just use some simple ipsets and iptables, or make a simple script to disallow things like posting, interactions while still allowing them to see content on websites would be ideal. Add a little banner 'you're infected, or somebody on your network is infected, this is how to fix it and practice best security, and more info on the subject'
These services switched from DDoS/attacks to renting out their hacked network spaces. They don't need to be making bank at our expense.
My ISP shares its residential IP pool with a middle east country (I can't remember which) users. God knows what those users are doing, but whenever "our" part of the pool is switched with "theirs", I get many more captchas, blocked websites and strange content suggestions.
"We" could pay for VPN access, but paying for the connection twice (local ISP and vpn ISP) adds up. And now the ball is in the VPN provider court.
Find it a pity that the decision was to remove them instead of owning how the ranking is made. it’s going to be very very hard to differentiate between bots and humans at the dns query level, if possible at all. DNS magnitude is about the dispersion of source ips/networks and if this is a distributed bot net, it makes sense. and it’s about 1.1.1.1 queries, not overall dns queries, which they couldn’t possibly obtain. See this more as proof of the impact of these botnets, which I found super interesting. Removing them just makes it poorer imho.
If an automated service is pulling the top 100 domains from CF and naively trusting them, why can't it also pull the categorization information that's right there and make sure none of the categories are "Malware"??? Who would write something like that? It's absolutely believable that the top 100 domains could contain malware domains...because of the nature of botnets and malware.
That's PEBCAK.
People make mistakes. Security engineers need to understand what sort of mistakes people are making and mitigate that risk. Brushing it under the rug as silly users making mistakes doesn't protect anyone.
The automated services using this for security-related purposes are presumably built by "security engineers", if they're making mistakes like this they're obviously woefully underqualified.
Almost nothing is built by security engineers, including security features of security products at security companies.
I'm a security engineer, I have built things like this, and I made the original comment. A lot of my job revolves around developing automation for security needs.
Also, many of the top 100 domains serve user-generated content (like AWS/S3). Blindly trusting anything from them just because they are big is so woefully misguided it boggles my mind; I seriously doubt that anyone is actually doing what is described in the article.
Idk, I have done security audits for startups and small tech companies. They won't have a security engineer on staff and are "moving fast and breaking things". I've seen things much more misguided than this.
I just finished working at a small company like what you are probably describing. It was...horrific. But I try not to think about that anymore!
[dead]
True masters of security realize all software is flawed, and therefore write none.
Use none, too.
Many people are woefully under qualified, we need to have a working society anyway.
Yeah, I'm not sure that baby-proofing everything as proposed here is going to result in a working society.
If we expected airplanes or cars to be able to be safely operated by people with zero understanding of how such vehicles work, nobody would be getting anywhere.
You eventually reach a level of stupidity and/or incompetence after which trying to alter the product to coddle those users becomes counterproductive.
Why not include them? What's wrong with have the most resolved domain being the top domain. I think it's interesting to know the actual most resolved domain, than the top of some editorialized list.
As discussed in the article, threat actors are using a botnet to game the system by repeatedly issuing queries for the domains; the list is intended to represent the top 100 domains resolved by legitimate users (and legitimate bots, I assume), not just "who can make the most queries to CloudFlare for a domain".
So why not get rid of "gamed" requests? Why would gaming it be fine as long as your domain isn't malware related?
Why not just disqualify these obviously not in the top 100 domains that are cheating and also host malicious content which has the same effect...??
From TFA:
"We should have two rankings: one representing trust and real human use, and another derived from raw DNS volume." (citing Mr Greenland)
> We should have two rankings: one representing trust and real human use, and another derived from raw DNS volume.
Isn't identifying real humans an unsolved problem? I'm not sure efforts to hide the truth that these domain are actually the most requested domains does anyone any favors. Is there something using these rankings as an authoritative list or are they just vanity metrics similar to the Alexa Top Site rankings of yore? If they are authoritative, then Cloudflare defining "trusted" is going to be problematic as I would expect them to hide that logic to avoid gaming.
> Isn't identifying real humans an unsolved problem?
I'm not sure this was ever a problem to begin with. The obsession with "confirm you are human" has created a lot of "bureaucracy" on technical level without actually protecting websites from unauthorised use. Why not actually bite the bullet and allow automations to interact with web resources instead of bothering humans to solve puzzles 10 times per day?
> Cloudflare defining "trusted"
They would love to monetise the opportunity, no doubt
> I'm not sure this was ever a problem to begin with. The obsession with "confirm you are human" has created a lot of "bureaucracy" on technical level without actually protecting websites from unauthorised use. Why not actually bite the bullet and allow automations to interact with web resources instead of bothering humans to solve puzzles 10 times per day?
I mostly just let the bots have my sites, but I also don't have anything popular enough that it costs me money to do so. If I was paying for extra compute or bandwidth to accommodate bots, I may have a stronger stance.
I do feel a burden with my private site that has a request an account form that has no captcha or bot blocking technology. Fake account requests are 100 to 1 real account, but this is my burden as a site owner, not my users' burden. Currently the fake account requests are easy enough to scan and I think I do a good job of picking out the humans, but I can't be sure and I fear this works because I run small software.
I send them on endless redirect loops with very slow responses. Cost me very little bandwidth and it effectively traps one bot process that then isn't available for useful work. Multiply by suitably large 'n' and they might even decide to start to play nice.
>"Why not actually bite the bullet and allow automations to interact with web resources instead of bothering humans to solve puzzles 10 times per day?"
This is a great idea if you've developed your 'full-stack', but if you're interfacing with others, it often doesn't work well. For example, if you use an external payment processor, and allow bots to constantly test stolen credit card data, you will eventually get booted from the service.
I think the comment means we have these “institutional” problems that we’re constantly protecting with tricks like captchas instead of actually addressing why a payment processor would have a problem with that or be unable to handle it in their own way.
The average normal user would go months to years between needing to update payment info, so why would that require them to solve puzzles 10 times a day?
That is also notably a completely unnecessary dumpster fire created by the credit card companies. Hey guys, how about an API that will request the credit card company to send a text/email to the cardholder asking them to confirm they want to make a payment to Your Company, and then let your company know in real time whether they said yes? Use that once when they first add the card and you're not going to be a very useful service for card testing.
Isn't that basically 3DSecure / Verified by Visa?
It's what those things should have been.
What you need is for all card issuers to be required to implement it by the network. Otherwise you'll still have people showing up to test all the cards that don't support it and the payment processors would still kick you off for that.
[dead]
Some web resources cost money or have implications beyond sending a 200 back.
> Aisuru switched to invoking Cloudflare’s main DNS server — 1.1.1.1
I don’t suppose they use DNS to find their command-and-control servers? It’d be funny if Cloudflare could steal the botnet that way. (For the public good. I know that actually doing such a thing would raise serious concerns. Never know, maybe there would be a revival of interest in DNSSEC.) I remember reading a case within the last few years of finding expired domains in some malware’s list of C2 servers, and registering them in order to administer disinfectant. Sadly, IoT nonsense probably can’t be properly fixed, so they could probably reinfect it even if you disinfected it.
I wonder whether by now the botnets moved on to authenticating C2 server and using fallbacks methods if the malware discovers an endpoint to be "compromised"
That's been happening for well over 20 years, and I'm sure there are even earlier examples.
This wouldn't raise serious concerns. Ask the customers/community if doing it before hand is something they agree with in some form of poll, then just do it. At the end of the day DNS is a million years old, out-dated and the mission is to help make a better internet. If Cloudflare straight up asked us all if it was cool to modify their DNS servers to identify / disrupt malicious use from botnets I'd agree. People not using DoH or internal things like dnscrypt-proxy need to get with the times.
There's ethical ways to do things: https://www.justice.gov/archives/opa/pr/court-authorized-ope...
I'm not saying I agree with it but we're all engineers, the internet and everything built on it was engineered, to put up with script kiddies and hacked computers and not-so-tech-savvy internet citizens using their devices and installing Infatica, and other malware/proxy services on their devices because it came within the agreement for installing some free app where their kids could 'pop bubbles' on their parents phones or some free desktop app included it; then distinguishing their IP addresses and IP-scores as they blend in with their regular human traffic makes it hard to block it. Ain't nobody got time for whack-a-mole internet, families and businesses will need to secure their networks.
Honestly I'd be ok with an up-to-date live list of all known infected IP addresses and their last timestamp for what, and who detected them as a bot/malicious IP address so I could just use some simple ipsets and iptables, or make a simple script to disallow things like posting, interactions while still allowing them to see content on websites would be ideal. Add a little banner 'you're infected, or somebody on your network is infected, this is how to fix it and practice best security, and more info on the subject'
These services switched from DDoS/attacks to renting out their hacked network spaces. They don't need to be making bank at our expense.
My ISP shares its residential IP pool with a middle east country (I can't remember which) users. God knows what those users are doing, but whenever "our" part of the pool is switched with "theirs", I get many more captchas, blocked websites and strange content suggestions.
"We" could pay for VPN access, but paying for the connection twice (local ISP and vpn ISP) adds up. And now the ball is in the VPN provider court.
Find it a pity that the decision was to remove them instead of owning how the ranking is made. it’s going to be very very hard to differentiate between bots and humans at the dns query level, if possible at all. DNS magnitude is about the dispersion of source ips/networks and if this is a distributed bot net, it makes sense. and it’s about 1.1.1.1 queries, not overall dns queries, which they couldn’t possibly obtain. See this more as proof of the impact of these botnets, which I found super interesting. Removing them just makes it poorer imho.
given the anti-user behaviour of modern Windows, shouldn't microsoft.com be down as malware too?
after yesterday's reveal[1]: facebook should certainly be down as "scams"
[1]: https://news.ycombinator.com/item?id=45845772
If sentiment and personal bias were a factor in classifying malware then I'd be rid of all of faang and social media.
[flagged]