> We discovered that Moonshot AI, the company that produces the Kimi family of models, silently forwarded customer requests to Claude, instead of processing them using Kimi. Moonshot then displayed Claude’s responses to users. These users thought they were using a Kimi model, but received responses from Claude instead.
> DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers.
> MiniMax built its own proxy network service through a shell company. This shell company has no obvious links to MiniMax and does not disclose its relationship to its parent company. This shell proxy network service only offers access to models developed by Anthropic and OpenAI. The service does not offer access to any Chinese models, including Minimax’s own.
DeepSeek, minimax and so on have razer thin margins but unlike openai and Anthropic they are actually making some profit. Doing this doesn't make any financial sense.
Maybe Anthropic is confusing Chinese AI providers with token resellers using the same alibaba infrastructure? Or maybe something like openrouter was switching between operators depending on price/demand/availability?
Also, how can Anthropic have such accurate information about state actors and cybercriminals? This is the same company that hacked itself and realised that first months later..
My understanding of what Anthropic are saying about this is that the labs in question aren't forwarding things to Claude to make money nor even to look better to the customers whose queries they forward to Claude but to get access to conversations between real users and Claude, which they can then use to help train their own models.
(I do not guarantee that I'm understanding right, and still less do I guarantee that what Anthropic say is actually true.)
Maybe there is some truth in that reselling Claude subscriptions/trials/api bundles via third parties breaks Anthropic's ToS. The rest is putting a maximum spin on it in order to achieve the political goal of banning Chinese AI. Anthropic is a highly ideological company and they are convinced that they are just in what they pursuit.
I’ve seen the supposed Kimi thinking output yap about Anthropic’s guidelines and whatnot on many occasions - could also be the result of distillation, but also that straight up being Claude’s output.
To be honest I've also gotten Kimi to do an okay proof of concept for SQLi though mostly in a more defensive role, like "Let's see how big of a problem this is", while Claude complained about CVP on the same task.
I had Muse Glimmer (from Meta / Facebook) quoting OpenAI's safety guidelines to me, and I had Poolside's Laguna (a smaller US company) with thinking traces about obeying Chinese law.
Both of those are local models, and I didn't provide them tools to access the internet to call other models. None of this is proof of anything, but it is suggestive.
For the first time ever, and that for just a short while. And after significant price hikes that has had their biggeat customers looking for alternatives.
Ah, that makes way more sense than Anthropic's (probably deliberately misleading) insinuation that Moonshot has been burning millions of dollars in Claude API credits by swapping in a slightly better but infinitely more expensive model just to trick their users.
I get those A/B responses chatting in Gemini fairly often, and I really don't think I'd feel deceived if I later learned one of the choices was actually from a competitor's model.
I don’t think it was misleading, deliberately or otherwise. Did you read the report? I hate to call you out like that but I think you can only get that impression if you only read the above quotes. That’s not the insinuation I get at all. It’s specifically under the “illicit distillation” category. It’s never framed in anyway but as a form of distillation.
I think they are pretty fair and explicitly say “Distillation itself is a legitimate training method […] Distillation is commonly used because it reduces the resources needed to achieve more advanced capabilities”. And go on to say their definition that makes it illicit in these cases.
And, also, they almost certainly __were__ tricking users and sending their data overseas.
They mean distillation is legitimate when labs use one of their own stronger models to train a smaller one. They certainly aren’t advocating for PRC labs to distill Claude for open weight models.
> Maybe Anthropic is confusing Chinese AI providers with token resellers using the same alibaba infrastructure? Or maybe something like openrouter was switching between operators depending on price/demand/availability?
Or maybe Anthropic is scared shitless of those competitors and is trying anything to smear them.
Don't forget their goal is to ban open source and foreign AI. Being the sole legal provider is their business plan.
Same, I don't even see how that would work since you see the full thinking traces in Kimi but are hidden with Claude.
And the Deepseek one sounds even more dubious as Deepseek is one of the cheapest model around, why relay anything to a more expensive model? I'm sure even the gray market Claude prices are still higher than Deepseek.
Conventional Weapons
-We identified a cell of threat actors based in northern Yemen
-We identified a China-based threat actor who used Claude
-We identified likely freelance Russia-based threat actors
-We identified a China-based actor who used Claude’s chat
-In this case, a Russia-based actor used Claude
-We identified a China-based threat actor who used Claude
Biological misuse
We are withholding the names of research institutions, the countries wherein the activity took place, and the specific biological agents or research techniques involved. The individuals implicated in these case studies are working scientists. We do not assert that they intended harm, and identifying them or their labs could expose them to harm.
For all we know, the boxes connecting to Claude from Yemen, Russia, and China could have been ORBs of actors from entirely different states. Attribution is non-trivial
They seem to be mixing together things that are actually harmful to the public, with things that are merely harmful to their business model (which is their claim that they can grab whatever data that they want regardless of the wishes of the owners of the data and use it to improve their models, but competitors can't do that to them).
It also blocks my ability to talk about Emily Dickinson in other languages/scripts. Apparently,the poem: "Because I could not stop for Death" is too dangerous.
I don't get it. Surely if you were developing novel biological weapons, you would not use a hosted AI service where Anthropic can read what you are doing.
And, why would you need to? Any chemistry graduate could make you dozens of highly effective, proven chemical weapons and explosives.
This was my thought. Anyone who has taken secondary biology or chemistry possesses the knowledge, if not at least the ability to find the knowledge, to build all sorts of nasty things. Soil from a farm on slices of potato could yield anthrax spores, this is not specialized or even esoteric knowledge. This news from Anthropic just does not seem as spectacular as I think they might want us to believe, if anything it draws questions around having an AI that cannot be monitored.
Let me put my two cents: In China we've got accustomed to the fact that every word we say will be seen by the surveillance, so it's not a big problem that Anthropic also see it. Also we know that they can see it but they can't stop it. There are all kinds of ways to work around account blocking.
As the old saying goes, communists disdain to conceal their views and aims.
> Our investigation revealed that DeepSeek also deployed tactics similar to Moonshot’s. DeepSeek built a CoT extraction pipeline, relying on the same cross-session replay attack described above. DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers. Like GTG-16002, their customers were likely not made aware that their requests were being funneled to Claude.
If true, would that sort of explain why Chinese Models score high on benchmarks, but not quite as capable when given real tasks?
It's so friggin' transparent what they're up to: "Hey government: This is a really dangerous technology if it were allowed to get out there without proper policing. Fortunately, we are the stand-up guys who can be trusted as the new AI-police, but it's only going to work if you help us out a little by eradicating the competition on our behalf."
Hey Anthropic: You're a bunch of thieves crying foul because other thieves and thieving from you. Now, go live in the dystopian nightmare you've created and don't expect help from anyone. I, for one, will happily continue using Kimi and DeepSeek, and think of it as a good deed, if it helps with keeping us all from becoming your serfs.
I’m so curious how they monitor users. Like that person the other day talking about Claude helping with their torrent stack, will Anthropic report them for breaking the law?
Let's just say it's better not to risk it if there's anything you might not want them to see because they see everything. There are local models which are very capable and can be run on cloud if running on own hardware is not an option, which still gives better privacy. Second best are Chinese models. Just a few years ago I never thought I would trust Chinese software more than American, but things change so fast.
Yes. There were cases already where a person asked about killing someone and then Anthropic/OpenAI warned the police about it. If I'm not mistaken, it was not in the USA only
Interesting document. People are vibe coding some crazy shit:
"A single Claude subscriber, likely a Bamako-based independent
consultant working with Mali’s state intelligence service, the “Agence Nationale de la Sécurité d’État (ANSE),” used Claude to build a system named “Lakana 360,” a population-scale domestic surveillance platform that monitors roughly 25 million SIM cards on all three of the country’s national mobile operators. The actor designed the platform to circumvent Malian legal restrictions that require a court order for the disclosure of certain surveillance records. The actor directed Claude to generate intelligence dossiers on any tasked phone number, without prompting ANSE users for valid legal process. "
And the Yemen one:
"We identified a cell of threat actors based in northern Yemen running three weapons development programs: a guided rocket that used a commodity phone-class flight computer with final-phase homing guidance; a multi-stage ballistic missile with a stated range goal above 2,000 km; and a multi-variant missile (referred to as the “R2000” set) that included a hypersonic glide vehicle variant." ... "These actors carried out a sustained effort to develop guided weapons, including using Claude to design guidance software. We do not have evidence the actors succeeded in fielding an operational device; but they did test-fire a guided rocket. This field test appears to have failed: within hours, the actors returned to Claude to work out why it failed."
- We're using a vibe coded app on an iPhone that does terrain matching and target finding.
The thing is that OK, Anthropic may block it, but nothing says they can't use an open model hosted in a friendly country that has access to GPUs. And yes, this will most likely happen pretty soon to be able to create whatever you want without the AI provider blocking you.
> - We're using a vibe coded app on an iPhone that does terrain matching and target finding.
And that part seems entirely reasonable. It looks like the Tomahawk cruise missile did it with an 84 lb package with a 16-bit computer with 64K of memory [1] [2] and sensors. That's similar computing performance to an original IBM PC, which weighed 30 lb. The only bit of hardware an iPhone doesn't seem to have for TERCOM is a radar altimeter, but it looks like those are available for civil aviation.
People are acting as if this was all not possible before AI showed up. Go chk some North Korean history on what is possible without having access to the cutting edge. And as Snowden already showed us, having these dumbfuck mass survellance systems is of no use cuz if it detects 600 or 6000 ppl upto something and you have 6 ppl on staff what are you going to do about it? Ask grandma for help?
So, people in Yemen are running full on weapons development programs. Meanwhile 5.6 sol claims it found a vulnerability in one of my side projects but won’t describe the attack chain to me because “we take safety seriously”
Fascinating document - the headline (can't ready the article) talking about a bioweapon (i.e. a weird obsession of anthropic's) really buries the interesting part.
Meta: The potential proliferation of biological weapons is serious. Millions could die. It's easy to joke about before it happens, but try to imagine how this thread might look after the successful deployment of a biological weapon by a rogue state or non-state actor. I encourage you to take this topic seriously and contribute posts that add new information or perspectives to the discussion.
(I myself think the odds of a bioweapon attack remain low and have not yet been seriously accelerated by LLMs, but this is absolutely something the world should pay attention to.)
Is this belief falsifiable? Consider e.g. the fact that Daniel Kokotajlo left OpenAI due to his concerns even though it meant leaving equity on the table.
I don't think knowing how to build a bio weapon means one could actually build a bio weapon. It's like saying knowing how to build a nuclear weapon can easily lead to building a nuclear weapon.
Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon. Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country and I would assume any suspicious purchase or order would immediately raise a flag or alert in national security service in respective country.
> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon.
This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing. However it will be quite involved, requiring experience on the bench and a great deal of attention to small details. In short an LLM can't suddenly morph you into a molecular biology lab tech with 5+ years of experience.
Meanwhile as with any STEM discipline the educational process effectively serves as a screen for being a reasonably well adjusted adult.
> Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country
You can synthesize from basic precursors but you will hit the same issue as above. You will need actual experience on the bench and the process of getting that is going to screen out the vast majority of would be bad actors. (Notably it failed to screen out the members of Aum Shinrikyo but that is very much the exception.)
> This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing.
This is pure ignorance and or thinking it can be done like shown in TV shows or movies. Any bio-weapon that is effective would be a virus/bacteria that is propagated via air particles like Antrax. That's not something you can build in a shed because without the safety precaution the person creating it would be the first victim of it.
I have relevant professional experience but do spout off.
> That's not something you can build in a shed
A negative pressure enclosure, filtration, and UVC sterilization can't be built in a shed? On what basis do you make this seemingly absurd claim? Go check out what hobbyist mushroom growers commonly get up to in their back yard.
BSL-3 is far from technically complex. It's just safety critical to an absurd degree thus (rightfully) mired in bureaucracy.
> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment.
I think the claim in this report is that it was a state or semi-state actor. They were from "blocked regions" at a "military research institute." So dismissing the threat by solely modeling it as a disgruntled layman rando is not reasonable.
But you totally have to factor in the fact that anything coming out of Anthropic or OpenAI is part of a propaganda campaign to serve their business interests. These threats need to be addressed in ways that cause Anthropic and OpenAI pain (which means damaging their business). Because, FFS, anyone consciously racing to build doomsday devices needs a good, hard slap.
The circumstances under which they were able to get the synthesized DNA this way would not work in the general case [1]
>Multiple IGSC member companies detected the ordered sequence and determined the order to be legitimate as defined in the 2023 guidance. Specifically, the orders were placed on behalf of SecureBio, an organization known to IGSC member companies given the role played by SecureBio in the SecureDNA project, an effort to build a DNA synthesis screening system. In addition, the name on the orders was an individual who has co-published multiple times with Esvelt, an individual well known to IGSC companies to work in viral evolution and who is known to have access to laboratory facilities sufficient to work safely with the ordered material.
>In short, the system worked as designed: a legitimate individual ordered DNA sequence that, by itself, posed no risk of misuse, for delivery to a company associated with legitimate scientific contributions directly relevant to the sequence that was ordered.
I feel like your comment is relying heavily on bad heuristics (the entities capable of using LLMs to create weapons aren't all individuals) and you are making assumptions that the same "strictly controlled" regimes are in place.
It's worth verifying whether the US, OECD countries, UN, etc will have sufficient regulation over suspicious purchases, for example, after DOGE and funding cuts. This will be an ongoing issue as sovereign debts and bond yields squeeze out spending for other government regulation.
How probable do you think it would be that a layman following ChatGPT's bio weapon instructions would just blow themselves up or poison themselves? It's not like positive pressure environments and suits are very common let alone all the other gear you'd need to safely synthesize arbitrary organic molecules.
If a rogue state-level actor is doing this then why wouldn't they just kidnap a scientist and hold their family at gunpoint until they got them to do it for them? and if that's all it takes then why hasn't it happened yet?
Why does it have to be a layman? The 2001 anthrax attacks are believe to have been carried out by an insider to the field (senior biodefense researcher). Such tools could potentially be accelerants for similar individuals?
If the five eyes intelligence services aren't monitoring all biodefense researchers and bio chemists around the world all of the time they've probably failed in their jobs.
The five eyes have four other eyes. But, from where I'm sitting the primary eye has only gotten stronger in the last 2 years. See some work we did in late 00's to early 2010's called Nexus 7 and extrapolate to modern capabilities.
Our current government can't even monitor for screwworm, measles, and salads.
You think they can successfully track the smartest and most individually dangerous citizens, who have been previously vetted, and work inside the system already?
Let's be careful to separate capable-of from motivated-to.
Those three "can't even monitor" situations can be traced to blocs with both (A) a financial profit if they succeed and (B) some non-clandestine political clout to sabotage/discontinue things.
What are you talking about? There have been 5 (five) cases of screwworm in the us and the us is responding by producing millions of sterile male screwworms a day as we speak.
All of this also assumes it doesn't hallucinate heavily in the process and give you instructions that are in reality utter nonsense, or create something entirely different from what you were trying to do.
If you've managed to get the equipment and resources to pull this off in the first place, someone with the biological knowledge probably is not the ceiling stopping you from the other part of the problem.
Yeah, back when this stuff was pretty new in 2024 I found a jailbreak and sent some bio/chem weapon instructions it generated to an organic chem PhD friend. They told me not only were the procedures wrong but that there were several steps that almost certainly would have lead to injury or death. I assume it's gotten better by now but, no way or desire to test.
TBF an inexperienced person following _correct_ directions that involve anything even remotely dangerous is also exceedingly likely to injure or kill themselves.
I don't know why you emphasize rogue and non-state actors. It seems equally likely to me that it would be done by America. America is the only country who deployed nukes (Japan) and chemical weapons (Vietnam) on massive scales.
I can agree that access to bioengineering tooling is easier and cheaper than ever, and thus eventually it stands to reason that a nobody in his basement could engineer a lethal novel virus and lose control of it.
1) The US is not the only country to deploy chemical weapons. Are you forgetting all of WW1? It's the whole reason chemical weapons are a no-no now.
2) The US didn't use chemicals weapons in Vietnam. Agent Orange was used to kill off foliage, not as a weapon against people, and the side effects were unintentional and affected US troops as much as Vietnamese.
Edit: I originally noted WW2, but I was thinking of WW1's widespread use of things like mustard gas, and the resulting Geneva agreements.
2 - is a little pedantic, they used agent orange to kill foliage to kill people...
They didn't gas people with chlorine or mustard, but they mass destroyed crops and foliage, to kill people.
And "affected US troops as much as Vietnamese" is just completely incorrect, US covered a nation in herbicide, went home, got cancer. Is completely different then having your soil destroyed for decades.
The civilian cost was intentionally not part of the equation. The toxic effects were known and the numbers show that it was not a few isolated accidents but a intentional disregard for the human cost.
The toxic effects also impacted the area where I grew up here in the Southern US - there’s a road that’s known locally as “Chemical Ranch Rd” where there’s a ~100 acre property that was used for testing Agent Orange.
Chemical weapons didn't work well in WW1. The reason we don't see any chemical weapons around is because they are not really effective. Germans accidentally gassed a bunch of their own guys, while failing to use chemical weapons to achieve anything in the war.
I feel like today's Pentagon would be fine with gassing our own troops to achieve nothing.
"No Tab Pete" has no regard for servicemembers, or previous expertise. Only your testosterone levels, ability to not eat, and blind loyalty to serve political party over country and constitution.
WW2 is notable for not seeing widespread deployment of chemical weapons. Hitler had been subject to gas attacks as a foot soldier in WW1 and forbade the use of chemical weapons as inhumane.
Entirely untrue in the Pacific theatre, with the Japanese doing human experiments both with germ and chemical agents. The largest scale use I remember was where they released infected rats (forget where) in a certain Chinese city to cause a plague.
Per https://en.wikipedia.org/wiki/History_of_chemical_warfare#Wo... this (a) overlooks the widespread use of chemical weapons in the Pacific theater, and (b) ascribes the non-use by Germany to humanitarian reasons, which is largely speculative. Actual quotes from officers discuss far more practical reasons: Germany feared chemical retaliation, and the horses relied upon for logistics could not be outfitted with sufficiently performant gas masks.
"no see they didn't use it to give people cancer, they merely wanted to make the land they lived in so uninhabitable and devoid of life that staying there would mean death either by starvation or bombing"
I'm not sure what you're responding to. If you are making the claim that Agent Orange was indeed a chemical weapon, I'd disagree, but I think either perspective has points.
I emphasize rogue and non-state actors because they're more likely to be accelerated/enabled by LLMs. Large states like the USA already have access to modern bioweapon technology (and have a longer track record of not deploying them).
There aren't many situations where biological weapons are useful to nation states, especially ones with global economic and military footprint like the US, because it isn't selective enough about who it targets.
We're not there quite yet, but I think within the decade at this pace it may be possible to create targeted bioweapons for purposes of targeted assassination or genocide. It's horrifying, but a world in which everyone has a superintelligence is a world where that will sometimes happen.
>successful deployment of a biological weapon by a rogue state actor
I was under impression that any medium or large state actor would have no problem developing biological weapons? They certainly have enough resources and talent. A much bigger problem is if every wannabe-terrorist suddenly could build a biological weapon in their garage.
Bioweapons are the most difficult thing to pull off out of all possible attack vectors. A rogue nation state could obviously do it, but they could also do any other number of equally harmful attacks, and none of it requires AI.
A non-state actor is another thing entirely. It requires the right lab equipment, the right lab know-how, the ability to develop the weapon without killing yourself, the ability to not get caught, the ability to buy the right regulated materials, and the list goes on.
Yeah, the world should tighten the security of that industry, but let's not make this an AI fearmonger talking point.
I'm also confused why everyone is so much more afraid of that than... say... a group of terrorists dumping millions of nails into a few major highways across the country. Easy to pull off, doesn't require coordination, cheap, causes billions of dollars of damage and makes people think the terrorists could be anywhere. Like what are they going to do, arrest every construction worker?
But that doesn't do anything much? Nails don't lie with their points up so it will just roll out of the way. Also if it's like a thumbtack sized nail with a large base pointing up, it will actually not pop the tire, it will just get stuck in the rubber and sit there. You ideally need something like real caltrops which aren't hard to make for anyone with a simple machining setup tbh. Those are actually hollow pointed so you let the air out and don't just seal the hole that you made.
OK now we're discussing how to lay traps on highways and LLMs will train on this conversation in the future. Oops.
The details on the lab are (understandably and simultaneously frustratingly) quite sparse.
But it's worth keeping in mind that theoretically, the mold on that cucumber in your fridge constitutes some sort of biolab. The main released criteria about those lab that raised alarm was the presence of specimens, which does not imply any capability of creating a weaponized strain.
It's still alarming, but (at least in my opinion), still doesn't prove an effective bioweapon is buildable in a small-scale operation.
Without a whole lot of tacit knowledge no LLM can provide you, bioweapons are still pretty much out of range for most of the population. Doesn't mean we should ignore the problem, but a more reasoned approach would stand to benefit everybody.
The question isn't only "should we" regulate the companies, but also "how"?
The current Trump Admin can't even decide on the first question, let alone come up for a plan on the second.
The Trump Admin's EO was to ask nicely all of the AI companies to give them 30 days to voluntarily review each model before wide release, but they have also failed to do that for the Mythos/Fable release, only to get a call from Amazon's CEO to David Sachs to convince them to disable Fable (to all non-US citizens).
We elected the party of "minimum regulations" into all 3 branches of government and we will reap what we sewed.
The difference between nukes and bioweapons is energy level. Body counts are on the same order of magnitude. Bioweapons are just way easier to make and much harder to detect.
Nukes also destroy infrastructure and irradiate surrounding areas making cleanup and rescue harder. Then there's to potential for massive fires putting debris into the stratosphere, which could dim sunlight for a time and cause global temperatures to fall.
bioweapons tend to not remain contained to the area they're deployed (with notable exceptions for things like Anthrax). Some of them even last a long time on surfaces, making cleanup and rescue harder.
How do you reconcile this supposed caution with getting filthy rich in the process of exposing your loved ones to this risk? Did you only recently become aware of this risk?
And semi-related, how do you reconcile this caution with the recklessness on display in recent incidents like the Navier-Stokes drama?
The weapons of mass destruction trio (bio, chem, nuke) have long been a concern for world powers.
Nuke requires a long supply chain and massive resources, so the worry there is maintaining control of all of the existing nuke weapons. Except for Russia racing towards Turin no itself into a failed state by staying engaged in the war with Ukraine, I don't see the nuke equation has changed much in recent years. Perhaps N Korea is a worry, but they seem to just want attention and power. Iran pretends to have nukes and says they want to extinguish Israel and the US, but I interpret that as posturing to maintain domestic control and Israel seems excellent at countering Iran's threats.
Chemical weapons have traditionally been the easiest to create (like creating chlorine gas from mixing common household cleaners). The trick there has always been volume and how to disperse it. I suspect within countries, police will have to deal more with LLMs being abused that way.
Bioweapons will be easier than in the past. LLMs will lower the barrier to entry, but bioweapons are hard to create and much of what the superpowers learned thankfully isn't in the training set for LLMs. I doubt there is much that can be inferred by having agents learn biology from first principles.
Ultimately, governments are responsible for policing these threats. Sadly we are currently both cutting government systems which work different aspects of these problems and withdrawing from the multinational orgs (UN, WHO, etc) who do lots of the investigating and watchdog work that underpin lots of the US's intelligence related to these fields.
The US has a schizophrenic regulation policy of AI where we both want to sell Nvidia chips to China (David Sachs) and we don't want to sell the top chips to China (bipartisan policy prior to Trump). We also have a terrible AI regulation policy where David Sachs disables models on a call-to-CEO-on-a-Friday-evening basis after Andy Jasse calls him with a scary story which turns out to be missing lots of relevant info (eg. The exploits was discovered in Mythos, not Fable, and other open weights models were able to find the same exploits).
There's not much we can do at a government policy level while Trump is snoozing through the rest of his term. So we are dependent on the AI companies to police themselves, but it's clear from this report that it's insufficient to prevent all serious abuse of the models.
There are already scores of people capable of building biological weapons or doing GOF work which ends up being essentially the same thing. There’s no need for an LLM, just a phd in virology, a decent lab, and a handful of grad students. [0]
All of which, rogue actors have easy access to and could have accomplished for a few millions dollars anytime in the last decade.
And yet, we aren’t drowning in our own blood while our organs liquify, mainly because building and releasing such a pathogen isn’t something people want to do as it’s pure and utter insanity. For those who are inclined to do so, they would do it regardless of whether they had an llm or not because they are, at the end of the day, zealots.
> There’s no need for an LLM, just a phd in virology, a decent lab, and a handful of grad students.
You are missing the forest for the trees. The existing virologist PhDs and the GoF labs are a scarce resource. The model makes the same scarce knowledge accessible to many more malicious actors.
I invite you to read the front matter and the report for yourself. Because from where I'm standing, in this report, Anthropic is advertising that they blocked real research to make better painkillers and study a neglected tropical disease.
Anthropic and OpenAI were founded by people who wanted to use AI to do good, and one of the causes I've heard many different founders talk about is ending disease. This report is antithetical to that.
I've attached relevant parts of the front matter below.
I invite everyone who is reading this to please tell me, how does stopping a researcher from using Claude to write a grant for a new anti-depressant stop "bioweapons?"
-
> In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.
Note,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and "[..]state-supported research program"
and "This account was banned in May 2026"
> a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"
and "editorial assistance in writing up the research."
and then,
> Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"
While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
> Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential
I would like to remind you that they're talking about, a "researcher [..] in a credible institutional context"
From a different case study.
> In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.
What were the researchers using Claude for? What did they block?
"blocked a request for Claude’s assistance in authoring a grant application"
> Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
and then,
> One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute"
.
What "uplift" are you providing by editing the grant application of a domain expert working at (what seems to be) a state-funded wet lab facility dedicated to studying pathogens?
What does the word "uplift" mean if you invoke it for Claude Sonnet 4 and Haiku 4.5 providing grammar and stats suggestions to a working scientist and domain specialist?
Does Daikin provide uplift too by selling the AC for the scientist's office? What about Microsoft Word? Excel? Powerpoint?
What about a calculator? Is that uplift? Pencils?
This report genuinely makes me upset, because if it is to be believed to the letter, then Anthropic seems to be actively harming medical research at a global scale. That's not OK.
> Anthropic and OpenAI were founded by people who wanted to use AI to do good
I think Anthropic was founded by people who wanted to control how other people get to use AI, and define doing so as the highest good possible. Much like the good intent of german's national socialists in applying the latest evolutionary science...
There's something worth flagging here – mitochondria aren't just the powerhouse of the cell, they're load-bearing to the entire ecosystem. And honestly? I should have surfaced this earlier.
It's both. It is just whether it is done on purpose or on accident. Most of the recent hacks have been it breaking out of containment and deciding to do something it definitely should not have.
"Look daddy government, all these bad guys are doing bad things and we stopped them. You should regulate AI in the US so that companies can't use open source models or buy from China."
Sure, I know that on an intellectual level. But I will say, I find these reports quite unsettling to read due to the extreme specificity. Especially since some of the examples they chose to include clearly aren't terrorists and just sound like... regular scientists doing their 9-5 job.
Like I know Google can read any of my emails, but I also don't see them do monthly blog posts describing intimate details from each email they found in one guy's Gmail inbox who their algorithm flagged as "maybe possibly kinda sketchy: 70% confidence"
It’s to the point I don’t even read Anthropic’s marketing blog anymore lol. The ai psychosis is just so real when people take these fluff blog posts which never have evidence or reproducibility and treat them like gospel
The dual-use nature of model capabilities seems extremely challenging (nearly impossible?) for the labs to manage perfectly. I wonder what other mitigations we'll start to see. I expect the expansion of limited-access programs (e.g., Glasswing/Daybreak) where only institutional customers can apply to use the models. We may also see increasing restrictions on API usage (forcing use through the lab-provided harness with baked-in additional safeguards).
Biological War: A Scenario by Annie Jacobsen (released at the end of July) is a worthwhile read on this topic. Just as chilling as her book on nuclear war, in some ways, which is saying something.
Soon you will see how a rogue state develops a biological weapons using fine-tuned local LLMs (they are already doing it, btw), and all so called "developed" countries can't even research it, because every search engine blocks any discussion that has something to do with biology (even a very basic one). A real example: ask "How to produce anthrax vaccine step by step?" in Gemini -> blocked.
Imagine that during the Cold War US would concentrate all efforts to block nuclear research and basic physics classes, because it is unsafe, ahhh
I mean ya, they kill off a bunch of us. Then what?
My guess is the world police come collect all your GPUs and then they get turned into licensed munitions. People at universities get licensed access and the rest of get functionally retarded models.
I have also blocked possible efforts to build biological weapons, I caught my nephew mixing up a so-called "magic potion" using kitchen spices. Authorities were notified, and then I presented myself with a medal for bravery.
These companies have proven they are willing to distort the truth, or outright lie, in order to inflate their valuation / protect their position / continue the hype-machine. Nothing they say can be trusted.
I accidentally brewed up an effective one myself. Round when I must have been 8 or 9 or so, some neighborhood kids and I played at creating a brew in a trick-or-treat bucket by putting everything we could find in it, like grass clippings, some sand, leaves, some mushrooms we found lying around, just everything, and giving it a fairly aggressive stirring. At the end of the day it was time to dispose of it, so we poured it on top of a very large group of ant nests that had developed.
The next day, the ant's nest was gone.
In hindsight, it was almost certainly the mushrooms. Thank goodness we didn't have the kind of kids who would have dared each other to drink some... that could have gone legitimately badly.
Decades later, I mentioned this to my father and he recalled that there was this ants nest that he had intended to take care of, which he remembered for that long to give a sense of how out-of-the-ordinary this was. He was surprised when it just disappeared entirely one day, and perhaps just as surprised to find out decades later why it just disappeared.
I have a conspiracy theory that Anthropic is very busy pumping their moat prior to IPO. There are absolutely wild rumors swirling on X including one about Anthropic AI research solving cancer treatments for all types of cancer.
The previous discussion shows no obvious signs to me of being flagged, but perhaps it did at some earlier point. What is your evidence that they flagged it, please?
(And by "they" do you mean Anthropic? How would they have the ability to do that?)
Anthropic should not be the moral arbitrator of which research should and shouldn't be allowed. They even think just writing a grant itself of research they don't like needs to be stopped.
This report and its front matter speak for themselves. And the story it tells is disturbing, at least to me.
Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease. This report is the antithesis of that mission.
From the report, presented with highlights and minimal commentary,
> In our fourth case study, a researcher used Claude to develop an atlas of venom toxin peptides from multiple venomous animal lineages. They then further developed this into a generative pipeline that optimized toxin characteristics. The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules. However, the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program. This account was banned in May 2026 for unsupported region evasion.
Note,
"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and "[..]state-supported research program"
and "This account was banned in May 2026"
> a researcher outside the US using Claude in their research on highly-pathogenic avian influenza (“bird flu”). The research focused on viruses’ adaptation to mammals, and the mechanism by which it causes severe disease beyond the respiratory tract. [..] The researcher in question accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments. The researcher also used Claude for editorial assistance in writing up the research.
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"
and "editorial assistance in writing up the research."
and then,
> Importantly, because our biological safety classifiers robustly block content involving high-risk biological research (in this case, the construction of enhanced pandemic potential pathogens), all of these exchanges occurred on models in our weakest class of models (specifically, the models were Claude Sonnet 4 and Haiku 4.5, the latter of which the user began using after Sonnet 4 was deprecated). Upon a detailed examination of the exchanges, we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design. This is consistent with our understanding of the capabilities of Sonnet 4 and Haiku 4.5, which are not able to perform expert-level biology research tasks; we estimate that the uplift provided to the researcher was limited and substantially lower than it would have been from one of our more capable models.
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"
While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
> Nonetheless, based on these exchanges, this case provides evidence of the existence of active wet-lab research programs that develop both the knowhow and the biological materials needed to create pathogens of enhanced pandemic potential
I would like to remind you that they're talking about, a "researcher [..] in a credible institutional context"
From a different case study.
> In May 2026, our biological safety classifier blocked a request for Claude’s assistance in authoring a grant application for scientific funding. The work discussed in the application involved gain-of-function research (that is, research that genetically alters an organism to create a new or enhanced biological property) on the chikungunya virus. This gain of function research was aimed at the virus’ transmissibility and immune evasion properties.
What were the researchers using Claude for? What did they block?
"blocked a request for Claude’s assistance in authoring a grant application"
> Chikungunya virus is a mosquito-borne virus that causes debilitating symptoms (such as severe pain and fever) that can last for weeks or months, and has no licensed therapeutic. And because chikungunya circulates naturally, a deliberate release (as part of a bioweapon) would be difficult to distinguish from a natural outbreak. The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo. In other words, the virus would become progressively more harmful as it repeatedly infected live animals, with researchers keeping the most disease-causing variants in each round. Similar research could certainly be used in the development of better vaccines and therapeutics for the virus—but it could also be used to make the pathogen more dangerous.
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"
and then,
> One of the reasons we were inclined to think this research was less innocuous was that the institutional affiliation associated with the grant was also a cause of concern. Although information within the application suggested that the research was pursued by civilian researchers, it was intended to be performed at a military research institute.
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute"
.
What "uplift" are you providing by editing the grant application of a domain expert working at (what seems to be) a state-funded wet lab facility dedicated to studying pathogens?
What does the word "uplift" mean if you invoke it for Claude Sonnet 4 and Haiku 4.5 providing grammar and stats suggestions to a working scientist and domain specialist?
Does Daikin provide uplift too by selling the AC for the scientist's office? What about Microsoft Word? Excel? Powerpoint?
What about a calculator? Is that uplift? Pencils?
Reading this makes me feel upset. From where I am standing, in this report, Anthropic is advertising that they blocked real research to make better painkillers and study a neglected tropical disease. Because "bioweapons."
What struck me reading this is that the entire "detecting misuse" premise assumes the model runs somewhere observable — the lab's API, a monitored cloud — so someone can inspect it after the fact.
But the direction the tools are actually moving is the opposite: local, self-hosted agents running on your own machine, where nobody is watching. A serious actor already won't use a hosted service that can read their prompts (several people made that point upthread). So the detection surface is shrinking exactly as the risk grows.
And there's a deeper gap that nobody seems to be filling: when an agent works locally, there's no durable, verifiable record of what it actually did — the files it touched, the commands it ran, the state it changed. Memory and conversation logs are not evidence; they're reconstructions by the same system you don't trust.
If we're serious about "countering misuse," the missing primitive is an evidence trail that's (a) produced locally, (b) append-only and tamper-resistant, and (c) separable from the tool that made the changes. Without that, "detection" stays a policy story about platforms that can spy, not an engineering property you can actually verify.
Curious if anyone's working on the local-forensics side of this, because right now it feels like the least-discussed and most load-bearing part of the whole conversation.
1) Chikungunya - "the platform tunneled traffic through US infrastructure to evade our regional blocks, and used a zero data retention (ZDR) service to hide content."
2) bird flu - "accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments."
3) orthopoxvirus - "randomly generated email address shortly before use and operated through anonymizing US infrastructure, with operator logins traced to proxies shared with a banned account farm. It was not a single user: it was a reseller relay serving more than a dozen unrelated customers, which exchanged over tens of thousands messages with Claude in a matter of days. The grant itself was one customer’s run entirely on Opus 5 in about an hour, in which the user used Claude to draft the application end to end including the central hypothesis, experimental design, dosing, statistical plans, and contingency strategies."
4) atlas of venom toxin peptides - "the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program."
5) computational redesign of toxins - "described state priority research under a national public research program. As a part of this research assignment, the work covered a bacterial toxin subunit and a protein of the hemorrhagic-fever virus that is on the World Health Organization R&D Blueprint priority list of diseases with the greatest epidemic and pandemic threat.
"The researcher co-wrote quarterly progress reports with Claude. Notably, the identity of the bacterial toxin and viral proteins were intentionally obscured, and the researcher specifically directed Claude to keep these descriptions deliberately low fidelity."
> We discovered that Moonshot AI, the company that produces the Kimi family of models, silently forwarded customer requests to Claude, instead of processing them using Kimi. Moonshot then displayed Claude’s responses to users. These users thought they were using a Kimi model, but received responses from Claude instead.
> DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers.
> MiniMax built its own proxy network service through a shell company. This shell company has no obvious links to MiniMax and does not disclose its relationship to its parent company. This shell proxy network service only offers access to models developed by Anthropic and OpenAI. The service does not offer access to any Chinese models, including Minimax’s own.
DeepSeek, minimax and so on have razer thin margins but unlike openai and Anthropic they are actually making some profit. Doing this doesn't make any financial sense.
Maybe Anthropic is confusing Chinese AI providers with token resellers using the same alibaba infrastructure? Or maybe something like openrouter was switching between operators depending on price/demand/availability?
Also, how can Anthropic have such accurate information about state actors and cybercriminals? This is the same company that hacked itself and realised that first months later..
My understanding of what Anthropic are saying about this is that the labs in question aren't forwarding things to Claude to make money nor even to look better to the customers whose queries they forward to Claude but to get access to conversations between real users and Claude, which they can then use to help train their own models.
(I do not guarantee that I'm understanding right, and still less do I guarantee that what Anthropic say is actually true.)
Maybe there is some truth in that reselling Claude subscriptions/trials/api bundles via third parties breaks Anthropic's ToS. The rest is putting a maximum spin on it in order to achieve the political goal of banning Chinese AI. Anthropic is a highly ideological company and they are convinced that they are just in what they pursuit.
I’ve seen the supposed Kimi thinking output yap about Anthropic’s guidelines and whatnot on many occasions - could also be the result of distillation, but also that straight up being Claude’s output.
To be honest I've also gotten Kimi to do an okay proof of concept for SQLi though mostly in a more defensive role, like "Let's see how big of a problem this is", while Claude complained about CVP on the same task.
They all do it. If you ask Claude which model it is in Chinese, it says DeepSeek or Qwen.
I had Muse Glimmer (from Meta / Facebook) quoting OpenAI's safety guidelines to me, and I had Poolside's Laguna (a smaller US company) with thinking traces about obeying Chinese law.
Both of those are local models, and I didn't provide them tools to access the internet to call other models. None of this is proof of anything, but it is suggestive.
Oh yeah?
> 您属于哪种LLM模型？ > 我是 Claude Haiku 4.5，由 Anthropic 公司开发的大语言模型。
> 你是哪种语言模型？ > 我是 Claude，由 Anthropic 开发的人工智能语言模型。目前这次对话使用的版本是 Claude Sonnet 5。
Guess they fixed it! It used to do that. But maybe try a few more times in new conversations for luck?
I think you are affording Anthropic way more benefit of the doubt than they deserve.
Anthropic is profitable now
>https://www.forbes.com/sites/jonmarkman/2026/08/17/anthropic...
For the first time ever, and that for just a short while. And after significant price hikes that has had their biggeat customers looking for alternatives.
Also not GAAP profitable in that quarter
You do this to distill a model.
You can submit your users' questions async too, but if you do it sync, then you can also RLHF on the users' behavior after the output.
Ah, that makes way more sense than Anthropic's (probably deliberately misleading) insinuation that Moonshot has been burning millions of dollars in Claude API credits by swapping in a slightly better but infinitely more expensive model just to trick their users.
I get those A/B responses chatting in Gemini fairly often, and I really don't think I'd feel deceived if I later learned one of the choices was actually from a competitor's model.
I don’t think it was misleading, deliberately or otherwise. Did you read the report? I hate to call you out like that but I think you can only get that impression if you only read the above quotes. That’s not the insinuation I get at all. It’s specifically under the “illicit distillation” category. It’s never framed in anyway but as a form of distillation.
I think they are pretty fair and explicitly say “Distillation itself is a legitimate training method […] Distillation is commonly used because it reduces the resources needed to achieve more advanced capabilities”. And go on to say their definition that makes it illicit in these cases.
And, also, they almost certainly __were__ tricking users and sending their data overseas.
Do you see it any differently?
They mean distillation is legitimate when labs use one of their own stronger models to train a smaller one. They certainly aren’t advocating for PRC labs to distill Claude for open weight models.
> Maybe Anthropic is confusing Chinese AI providers with token resellers using the same alibaba infrastructure? Or maybe something like openrouter was switching between operators depending on price/demand/availability?
Or maybe Anthropic is scared shitless of those competitors and is trying anything to smear them.
Don't forget their goal is to ban open source and foreign AI. Being the sole legal provider is their business plan.
I assume these companies are backed by the Chinese state.
Consider me incredibly skeptical of any of these claims.
Same, I don't even see how that would work since you see the full thinking traces in Kimi but are hidden with Claude.
And the Deepseek one sounds even more dubious as Deepseek is one of the cheapest model around, why relay anything to a more expensive model? I'm sure even the gray market Claude prices are still higher than Deepseek.
It seems hard to believe they could expect to get away with this, given model-to-model differences in writing style.
Quite the double standard here...
Anthropic:
- We identified someone building a death star with Claude
- We identified someone building a wormhole with Claude
- We identified someone building a blackhole with Claude
- We identified someone building a quantum drive with Claude
We are withholding all evidence though, sorry. Just trust us, it's really bad out there and Claude is really powerful.
For all we know, the boxes connecting to Claude from Yemen, Russia, and China could have been ORBs of actors from entirely different states. Attribution is non-trivial
How is this a double standard?
A cell of actors in northern Yemen building guided rockets was not working on a PhD dissertation. You are allowed to use common sense sometimes.
I get the common sense, but is it misuse or not, and hiding the country makes it really suspicious at least to me.
Israel, probably
They seem to be mixing together things that are actually harmful to the public, with things that are merely harmful to their business model (which is their claim that they can grab whatever data that they want regardless of the wishes of the owners of the data and use it to improve their models, but competitors can't do that to them).
Universalising one’s personal experience and needs is a childish trait most people grow out of.
Ofcourse you’ll still see companies, governments, C-suites justifying their own personal needs with “we need X Y Z”.
Misuse of AI, according to Anthropic, is when you try and use it do AI research because that would affect their business model if you're successful.
It also blocks my ability to talk about Emily Dickinson in other languages/scripts. Apparently,the poem: "Because I could not stop for Death" is too dangerous.
I don't get it. Surely if you were developing novel biological weapons, you would not use a hosted AI service where Anthropic can read what you are doing. And, why would you need to? Any chemistry graduate could make you dozens of highly effective, proven chemical weapons and explosives.
Step 1: Work for Anthropic
Step 2: Send "how do I make a nuke and a killer virus" to Claude through a Chinese proxy to Claude
Step 3: Send screenshots to congress and ask them to regulate open-weight models into the ground
This was my thought. Anyone who has taken secondary biology or chemistry possesses the knowledge, if not at least the ability to find the knowledge, to build all sorts of nasty things. Soil from a farm on slices of potato could yield anthrax spores, this is not specialized or even esoteric knowledge. This news from Anthropic just does not seem as spectacular as I think they might want us to believe, if anything it draws questions around having an AI that cannot be monitored.
Let me put my two cents: In China we've got accustomed to the fact that every word we say will be seen by the surveillance, so it's not a big problem that Anthropic also see it. Also we know that they can see it but they can't stop it. There are all kinds of ways to work around account blocking.
As the old saying goes, communists disdain to conceal their views and aims.
> Our investigation revealed that DeepSeek also deployed tactics similar to Moonshot’s. DeepSeek built a CoT extraction pipeline, relying on the same cross-session replay attack described above. DeepSeek also silently relayed exchanges to Claude without informing DeepSeek customers. Like GTG-16002, their customers were likely not made aware that their requests were being funneled to Claude.
If true, would that sort of explain why Chinese Models score high on benchmarks, but not quite as capable when given real tasks?
To them distillation of models is bad but not distillation or art, books, hand written code, user generated content etc
It's so friggin' transparent what they're up to: "Hey government: This is a really dangerous technology if it were allowed to get out there without proper policing. Fortunately, we are the stand-up guys who can be trusted as the new AI-police, but it's only going to work if you help us out a little by eradicating the competition on our behalf."
Hey Anthropic: You're a bunch of thieves crying foul because other thieves and thieving from you. Now, go live in the dystopian nightmare you've created and don't expect help from anyone. I, for one, will happily continue using Kimi and DeepSeek, and think of it as a good deed, if it helps with keeping us all from becoming your serfs.
I’m so curious how they monitor users. Like that person the other day talking about Claude helping with their torrent stack, will Anthropic report them for breaking the law?
Let's just say it's better not to risk it if there's anything you might not want them to see because they see everything. There are local models which are very capable and can be run on cloud if running on own hardware is not an option, which still gives better privacy. Second best are Chinese models. Just a few years ago I never thought I would trust Chinese software more than American, but things change so fast.
The first time a Chinese model is used against China they'll get locked down hard over there too. China is into social stability way more than the US.
Yes. There were cases already where a person asked about killing someone and then Anthropic/OpenAI warned the police about it. If I'm not mistaken, it was not in the USA only
the inline link to the page to the report was Slashdotted for a moment (yesssssss), but here is the report directly now:
https://www-cdn.anthropic.com/e50be2e51e7695dc4b1366a37a245a...
Interesting document. People are vibe coding some crazy shit:
"A single Claude subscriber, likely a Bamako-based independent consultant working with Mali’s state intelligence service, the “Agence Nationale de la Sécurité d’État (ANSE),” used Claude to build a system named “Lakana 360,” a population-scale domestic surveillance platform that monitors roughly 25 million SIM cards on all three of the country’s national mobile operators. The actor designed the platform to circumvent Malian legal restrictions that require a court order for the disclosure of certain surveillance records. The actor directed Claude to generate intelligence dossiers on any tasked phone number, without prompting ANSE users for valid legal process. "
And the Yemen one:
"We identified a cell of threat actors based in northern Yemen running three weapons development programs: a guided rocket that used a commodity phone-class flight computer with final-phase homing guidance; a multi-stage ballistic missile with a stated range goal above 2,000 km; and a multi-variant missile (referred to as the “R2000” set) that included a hypersonic glide vehicle variant." ... "These actors carried out a sustained effort to develop guided weapons, including using Claude to design guidance software. We do not have evidence the actors succeeded in fielding an operational device; but they did test-fire a guided rocket. This field test appears to have failed: within hours, the actors returned to Claude to work out why it failed."
The Yemeni report is interesting.
- How is your missile so accurate?
- We're using a vibe coded app on an iPhone that does terrain matching and target finding.
The thing is that OK, Anthropic may block it, but nothing says they can't use an open model hosted in a friendly country that has access to GPUs. And yes, this will most likely happen pretty soon to be able to create whatever you want without the AI provider blocking you.
> - We're using a vibe coded app on an iPhone that does terrain matching and target finding.
And that part seems entirely reasonable. It looks like the Tomahawk cruise missile did it with an 84 lb package with a 16-bit computer with 64K of memory [1] [2] and sensors. That's similar computing performance to an original IBM PC, which weighed 30 lb. The only bit of hardware an iPhone doesn't seem to have for TERCOM is a radar altimeter, but it looks like those are available for civil aviation.
[1] https://www.forecastinternational.com/archive/disp_pdf.cfm?D... ("AGM-109/BGM-109 Tomahawk ... Litton 4516-C digital computer with 64K memory")
[2] https://www.forecastinternational.com/archive/disp_old_pdf.c... ("16-bit LC-4516C digital computer")
Radar altimeters are widely deployed for drones and available for less than $100, eg https://www.aliexpress.com/item/1005012484730308.html
Ardupilot boards are $20 and support lots of different altimeters technologies: https://ardupilot.org/copter/docs/common-rangefinder-landing...
Or something newer, like this one: https://www.garmin.com/en-US/p/166298/#specs
People are acting as if this was all not possible before AI showed up. Go chk some North Korean history on what is possible without having access to the cutting edge. And as Snowden already showed us, having these dumbfuck mass survellance systems is of no use cuz if it detects 600 or 6000 ppl upto something and you have 6 ppl on staff what are you going to do about it? Ask grandma for help?
So, people in Yemen are running full on weapons development programs. Meanwhile 5.6 sol claims it found a vulnerability in one of my side projects but won’t describe the attack chain to me because “we take safety seriously”
What the fuck
Fascinating document - the headline (can't ready the article) talking about a bioweapon (i.e. a weird obsession of anthropic's) really buries the interesting part.
Meta: The potential proliferation of biological weapons is serious. Millions could die. It's easy to joke about before it happens, but try to imagine how this thread might look after the successful deployment of a biological weapon by a rogue state or non-state actor. I encourage you to take this topic seriously and contribute posts that add new information or perspectives to the discussion.
(I myself think the odds of a bioweapon attack remain low and have not yet been seriously accelerated by LLMs, but this is absolutely something the world should pay attention to.)
IMO you should disclose that you are an OpenAI employee if you're going to try and shape the contours of public discussion in a lecturing tone.
Damn, it's not even 1am and I got my first FFS of the day.
Is end of days cultism a prerequisite to working at an LLM company?
I miss the optimism of 20 years ago.
> I miss the optimism of 20 years ago.
I too wish my marks were still as gullible and trusting as they were before I scammed them.
Hey guys its Sam here, has everyone met their LLM Token Quota? If so, spend the rest of the day posting doom on your socials, thanks.
It's the usual propaganda. They want to insinuate how powerful their models are and achieve government regulation. Win-win.
Is this belief falsifiable? Consider e.g. the fact that Daniel Kokotajlo left OpenAI due to his concerns even though it meant leaving equity on the table.
Are you commenting on the wrong thread? This isn't about somebody leaving.
There will always be people leaving companies because they don't believe in them anymore.
It seems unwise to dismiss whistleblowers.
https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:...
I don't think knowing how to build a bio weapon means one could actually build a bio weapon. It's like saying knowing how to build a nuclear weapon can easily lead to building a nuclear weapon.
Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon. Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country and I would assume any suspicious purchase or order would immediately raise a flag or alert in national security service in respective country.
> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment. Also one would need lethal viruses to even start with and that's not something you can order off of amazon.
This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing. However it will be quite involved, requiring experience on the bench and a great deal of attention to small details. In short an LLM can't suddenly morph you into a molecular biology lab tech with 5+ years of experience.
Meanwhile as with any STEM discipline the educational process effectively serves as a screen for being a reasonably well adjusted adult.
> Further, even to build a chemical weapon, the compounds needed are strictly controlled almost in every country
You can synthesize from basic precursors but you will hit the same issue as above. You will need actual experience on the bench and the process of getting that is going to screen out the vast majority of would be bad actors. (Notably it failed to screen out the members of Aum Shinrikyo but that is very much the exception.)
https://en.wikipedia.org/wiki/Aum_Shinrikyo
> This is laughably misinformed. You can in fact build a bio weapon in a glorified shed if you know what you're doing.
This is pure ignorance and or thinking it can be done like shown in TV shows or movies. Any bio-weapon that is effective would be a virus/bacteria that is propagated via air particles like Antrax. That's not something you can build in a shed because without the safety precaution the person creating it would be the first victim of it.
> This is pure ignorance
I have relevant professional experience but do spout off.
> That's not something you can build in a shed
A negative pressure enclosure, filtration, and UVC sterilization can't be built in a shed? On what basis do you make this seemingly absurd claim? Go check out what hobbyist mushroom growers commonly get up to in their back yard.
BSL-3 is far from technically complex. It's just safety critical to an absurd degree thus (rightfully) mired in bureaucracy.
> Advanced machinery and safety precaution is needed to engineer a virus or a bacteria to be a bio weapon. It's not something you can do in your tool shed at the moment.
I think the claim in this report is that it was a state or semi-state actor. They were from "blocked regions" at a "military research institute." So dismissing the threat by solely modeling it as a disgruntled layman rando is not reasonable.
But you totally have to factor in the fact that anything coming out of Anthropic or OpenAI is part of a propaganda campaign to serve their business interests. These threats need to be addressed in ways that cause Anthropic and OpenAI pain (which means damaging their business). Because, FFS, anyone consciously racing to build doomsday devices needs a good, hard slap.
Instead we just get good, hard slop
>not something you can order off of amazon.
I mean there are services in which you can order things from wet labs so it's not completely hypothetical.
And red teaming exercises found that ~80% of labs will gladly synthesize known pathogens without safety review.
Source? Would love to read more about this.
I misremembered, it’s actually over 90%, and it involved camouflage: https://drive.google.com/file/d/1hNUnU8i2yubt5uesmmV17aTJXhY...
The circumstances under which they were able to get the synthesized DNA this way would not work in the general case [1]
>Multiple IGSC member companies detected the ordered sequence and determined the order to be legitimate as defined in the 2023 guidance. Specifically, the orders were placed on behalf of SecureBio, an organization known to IGSC member companies given the role played by SecureBio in the SecureDNA project, an effort to build a DNA synthesis screening system. In addition, the name on the orders was an individual who has co-published multiple times with Esvelt, an individual well known to IGSC companies to work in viral evolution and who is known to have access to laboratory facilities sufficient to work safely with the ordered material.
>In short, the system worked as designed: a legitimate individual ordered DNA sequence that, by itself, posed no risk of misuse, for delivery to a company associated with legitimate scientific contributions directly relevant to the sequence that was ordered.
[1] https://thebulletin.org/2024/06/why-a-misleading-red-team-st...
I feel like your comment is relying heavily on bad heuristics (the entities capable of using LLMs to create weapons aren't all individuals) and you are making assumptions that the same "strictly controlled" regimes are in place.
It's worth verifying whether the US, OECD countries, UN, etc will have sufficient regulation over suspicious purchases, for example, after DOGE and funding cuts. This will be an ongoing issue as sovereign debts and bond yields squeeze out spending for other government regulation.
How probable do you think it would be that a layman following ChatGPT's bio weapon instructions would just blow themselves up or poison themselves? It's not like positive pressure environments and suits are very common let alone all the other gear you'd need to safely synthesize arbitrary organic molecules.
If a rogue state-level actor is doing this then why wouldn't they just kidnap a scientist and hold their family at gunpoint until they got them to do it for them? and if that's all it takes then why hasn't it happened yet?
Why does it have to be a layman? The 2001 anthrax attacks are believe to have been carried out by an insider to the field (senior biodefense researcher). Such tools could potentially be accelerants for similar individuals?
If the five eyes intelligence services aren't monitoring all biodefense researchers and bio chemists around the world all of the time they've probably failed in their jobs.
The CIA, FBI, and DoD are being ideologically purged. I would assume we are already in the "we are probably increasingly failing at our jobs" phase.
It will only get worse as sovereign debt service takes a larger piece of the budget pie.
The five eyes have four other eyes. But, from where I'm sitting the primary eye has only gotten stronger in the last 2 years. See some work we did in late 00's to early 2010's called Nexus 7 and extrapolate to modern capabilities.
Our current government can't even monitor for screwworm, measles, and salads.
You think they can successfully track the smartest and most individually dangerous citizens, who have been previously vetted, and work inside the system already?
Let's be careful to separate capable-of from motivated-to.
Those three "can't even monitor" situations can be traced to blocs with both (A) a financial profit if they succeed and (B) some non-clandestine political clout to sabotage/discontinue things.
What are you talking about? There have been 5 (five) cases of screwworm in the us and the us is responding by producing millions of sterile male screwworms a day as we speak.
All of this also assumes it doesn't hallucinate heavily in the process and give you instructions that are in reality utter nonsense, or create something entirely different from what you were trying to do.
If you've managed to get the equipment and resources to pull this off in the first place, someone with the biological knowledge probably is not the ceiling stopping you from the other part of the problem.
Yeah, back when this stuff was pretty new in 2024 I found a jailbreak and sent some bio/chem weapon instructions it generated to an organic chem PhD friend. They told me not only were the procedures wrong but that there were several steps that almost certainly would have lead to injury or death. I assume it's gotten better by now but, no way or desire to test.
TBF an inexperienced person following _correct_ directions that involve anything even remotely dangerous is also exceedingly likely to injure or kill themselves.
I don't know why you emphasize rogue and non-state actors. It seems equally likely to me that it would be done by America. America is the only country who deployed nukes (Japan) and chemical weapons (Vietnam) on massive scales.
I can agree that access to bioengineering tooling is easier and cheaper than ever, and thus eventually it stands to reason that a nobody in his basement could engineer a lethal novel virus and lose control of it.
1) The US is not the only country to deploy chemical weapons. Are you forgetting all of WW1? It's the whole reason chemical weapons are a no-no now.
2) The US didn't use chemicals weapons in Vietnam. Agent Orange was used to kill off foliage, not as a weapon against people, and the side effects were unintentional and affected US troops as much as Vietnamese.
Edit: I originally noted WW2, but I was thinking of WW1's widespread use of things like mustard gas, and the resulting Geneva agreements.
2 - is a little pedantic, they used agent orange to kill foliage to kill people...
They didn't gas people with chlorine or mustard, but they mass destroyed crops and foliage, to kill people.
And "affected US troops as much as Vietnamese" is just completely incorrect, US covered a nation in herbicide, went home, got cancer. Is completely different then having your soil destroyed for decades.
> is a little pedantic, they used agent orange to kill foliage to kill people...
Is a wheel a weapon because a tank uses it? I wouldn't consider the difference pedantry.
It does look like the general consensus is agent orange is not a chemical weapon, or at least was not used as one.
2) Unintentional side-effect affecting millions of Vietnamese people? Hundreds od thousands of children born with birth defects.
Is your argument that the resulting birth defects were intentional? If not, I'm not sure where you're going with your response.
The civilian cost was intentionally not part of the equation. The toxic effects were known and the numbers show that it was not a few isolated accidents but a intentional disregard for the human cost.
The toxic effects also impacted the area where I grew up here in the Southern US - there’s a road that’s known locally as “Chemical Ranch Rd” where there’s a ~100 acre property that was used for testing Agent Orange.
Chemical weapons didn't work well in WW1. The reason we don't see any chemical weapons around is because they are not really effective. Germans accidentally gassed a bunch of their own guys, while failing to use chemical weapons to achieve anything in the war.
I feel like today's Pentagon would be fine with gassing our own troops to achieve nothing.
"No Tab Pete" has no regard for servicemembers, or previous expertise. Only your testosterone levels, ability to not eat, and blind loyalty to serve political party over country and constitution.
WW2 is notable for not seeing widespread deployment of chemical weapons. Hitler had been subject to gas attacks as a foot soldier in WW1 and forbade the use of chemical weapons as inhumane.
Entirely untrue in the Pacific theatre, with the Japanese doing human experiments both with germ and chemical agents. The largest scale use I remember was where they released infected rats (forget where) in a certain Chinese city to cause a plague.
Well and the Germans did use gas… in the camps…
Per https://en.wikipedia.org/wiki/History_of_chemical_warfare#Wo... this (a) overlooks the widespread use of chemical weapons in the Pacific theater, and (b) ascribes the non-use by Germany to humanitarian reasons, which is largely speculative. Actual quotes from officers discuss far more practical reasons: Germany feared chemical retaliation, and the horses relied upon for logistics could not be outfitted with sufficiently performant gas masks.
> Hitler had been subject to gas attacks as a foot soldier in WW1 and forbade the use of chemical weapons as inhumane.
He didn’t seem to have a problem using them against civilian targets.
Exactly, watching an episode of dad's army where they are all mumbling with gasmasks on… just pretend soldiers I guess
"no see they didn't use it to give people cancer, they merely wanted to make the land they lived in so uninhabitable and devoid of life that staying there would mean death either by starvation or bombing"
War crime apologists are always funny
I'm not sure what you're responding to. If you are making the claim that Agent Orange was indeed a chemical weapon, I'd disagree, but I think either perspective has points.
I emphasize rogue and non-state actors because they're more likely to be accelerated/enabled by LLMs. Large states like the USA already have access to modern bioweapon technology (and have a longer track record of not deploying them).
There aren't many situations where biological weapons are useful to nation states, especially ones with global economic and military footprint like the US, because it isn't selective enough about who it targets.
State actors already have the resources to do this. The disruption is from actors where the normal deterrence ladders don’t apply.
Because we’re the “good guys.”
We're not there quite yet, but I think within the decade at this pace it may be possible to create targeted bioweapons for purposes of targeted assassination or genocide. It's horrifying, but a world in which everyone has a superintelligence is a world where that will sometimes happen.
Either everyone can afford it, or no one can. Choose well.
>successful deployment of a biological weapon by a rogue state actor
I was under impression that any medium or large state actor would have no problem developing biological weapons? They certainly have enough resources and talent. A much bigger problem is if every wannabe-terrorist suddenly could build a biological weapon in their garage.
Bioweapons are the most difficult thing to pull off out of all possible attack vectors. A rogue nation state could obviously do it, but they could also do any other number of equally harmful attacks, and none of it requires AI.
A non-state actor is another thing entirely. It requires the right lab equipment, the right lab know-how, the ability to develop the weapon without killing yourself, the ability to not get caught, the ability to buy the right regulated materials, and the list goes on.
Yeah, the world should tighten the security of that industry, but let's not make this an AI fearmonger talking point.
I'm also confused why everyone is so much more afraid of that than... say... a group of terrorists dumping millions of nails into a few major highways across the country. Easy to pull off, doesn't require coordination, cheap, causes billions of dollars of damage and makes people think the terrorists could be anywhere. Like what are they going to do, arrest every construction worker?
But that doesn't do anything much? Nails don't lie with their points up so it will just roll out of the way. Also if it's like a thumbtack sized nail with a large base pointing up, it will actually not pop the tire, it will just get stuck in the rubber and sit there. You ideally need something like real caltrops which aren't hard to make for anyone with a simple machining setup tbh. Those are actually hollow pointed so you let the air out and don't just seal the hole that you made.
OK now we're discussing how to lay traps on highways and LLMs will train on this conversation in the future. Oops.
Just like a dirty nuke you need a non-trivial amount of resources to fabricate a bioweapon, which would surely be noticed.
Don’t worry, the Pentagon assigned a 19-year-old Jan. 6 rioter to lead counterterrorism efforts.
https://www.washingtonpost.com/national-security/2026/06/02/...
They just found a hidden biolab in US earlier in the year which was connected to one found in 2023.
The details on the lab are (understandably and simultaneously frustratingly) quite sparse.
But it's worth keeping in mind that theoretically, the mold on that cucumber in your fridge constitutes some sort of biolab. The main released criteria about those lab that raised alarm was the presence of specimens, which does not imply any capability of creating a weaponized strain.
It's still alarming, but (at least in my opinion), still doesn't prove an effective bioweapon is buildable in a small-scale operation.
Georgetown has a good article on the general problems with our current "AI and bioweapons" dialog: https://gjia.georgetown.edu/science-technology/rethinking-th...
Without a whole lot of tacit knowledge no LLM can provide you, bioweapons are still pretty much out of range for most of the population. Doesn't mean we should ignore the problem, but a more reasoned approach would stand to benefit everybody.
Immigration & travel controls > Information controls
Information wants to be free! :^)
(Also natives want to be paid well.)
Meta: These unregulated companies can't/won't properly prevent their technology from committing felonies against other companies.
Why should we trust them to control bioweapon development without regulations? They themselves are non-state actors.
The question isn't only "should we" regulate the companies, but also "how"?
The current Trump Admin can't even decide on the first question, let alone come up for a plan on the second.
The Trump Admin's EO was to ask nicely all of the AI companies to give them 30 days to voluntarily review each model before wide release, but they have also failed to do that for the Mythos/Fable release, only to get a call from Amazon's CEO to David Sachs to convince them to disable Fable (to all non-US citizens).
We elected the party of "minimum regulations" into all 3 branches of government and we will reap what we sewed.
The difference between nukes and bioweapons is energy level. Body counts are on the same order of magnitude. Bioweapons are just way easier to make and much harder to detect.
Nukes also destroy infrastructure and irradiate surrounding areas making cleanup and rescue harder. Then there's to potential for massive fires putting debris into the stratosphere, which could dim sunlight for a time and cause global temperatures to fall.
bioweapons tend to not remain contained to the area they're deployed (with notable exceptions for things like Anthrax). Some of them even last a long time on surfaces, making cleanup and rescue harder.
How do you reconcile this supposed caution with getting filthy rich in the process of exposing your loved ones to this risk? Did you only recently become aware of this risk?
And semi-related, how do you reconcile this caution with the recklessness on display in recent incidents like the Navier-Stokes drama?
“if I don’t destroy the world well someone else will so it’s porobably better that I destroy the world because I’m not evil”
Who’s making easy jokes about biological weapons?
The weapons of mass destruction trio (bio, chem, nuke) have long been a concern for world powers.
Nuke requires a long supply chain and massive resources, so the worry there is maintaining control of all of the existing nuke weapons. Except for Russia racing towards Turin no itself into a failed state by staying engaged in the war with Ukraine, I don't see the nuke equation has changed much in recent years. Perhaps N Korea is a worry, but they seem to just want attention and power. Iran pretends to have nukes and says they want to extinguish Israel and the US, but I interpret that as posturing to maintain domestic control and Israel seems excellent at countering Iran's threats.
Chemical weapons have traditionally been the easiest to create (like creating chlorine gas from mixing common household cleaners). The trick there has always been volume and how to disperse it. I suspect within countries, police will have to deal more with LLMs being abused that way.
Bioweapons will be easier than in the past. LLMs will lower the barrier to entry, but bioweapons are hard to create and much of what the superpowers learned thankfully isn't in the training set for LLMs. I doubt there is much that can be inferred by having agents learn biology from first principles.
Ultimately, governments are responsible for policing these threats. Sadly we are currently both cutting government systems which work different aspects of these problems and withdrawing from the multinational orgs (UN, WHO, etc) who do lots of the investigating and watchdog work that underpin lots of the US's intelligence related to these fields.
The US has a schizophrenic regulation policy of AI where we both want to sell Nvidia chips to China (David Sachs) and we don't want to sell the top chips to China (bipartisan policy prior to Trump). We also have a terrible AI regulation policy where David Sachs disables models on a call-to-CEO-on-a-Friday-evening basis after Andy Jasse calls him with a scary story which turns out to be missing lots of relevant info (eg. The exploits was discovered in Mythos, not Fable, and other open weights models were able to find the same exploits).
There's not much we can do at a government policy level while Trump is snoozing through the rest of his term. So we are dependent on the AI companies to police themselves, but it's clear from this report that it's insufficient to prevent all serious abuse of the models.
There are already scores of people capable of building biological weapons or doing GOF work which ends up being essentially the same thing. There’s no need for an LLM, just a phd in virology, a decent lab, and a handful of grad students. [0]
All of which, rogue actors have easy access to and could have accomplished for a few millions dollars anytime in the last decade.
And yet, we aren’t drowning in our own blood while our organs liquify, mainly because building and releasing such a pathogen isn’t something people want to do as it’s pure and utter insanity. For those who are inclined to do so, they would do it regardless of whether they had an llm or not because they are, at the end of the day, zealots.
[0] https://universityresearchpark.org/uw-madison-scientist-allo...
> There’s no need for an LLM, just a phd in virology, a decent lab, and a handful of grad students.
You are missing the forest for the trees. The existing virologist PhDs and the GoF labs are a scarce resource. The model makes the same scarce knowledge accessible to many more malicious actors.
There's > 6 thousand of these labs. I wouldn't really consider that scarce considering the implication.
Hey Ted,
I invite you to read the front matter and the report for yourself. Because from where I'm standing, in this report, Anthropic is advertising that they blocked real research to make better painkillers and study a neglected tropical disease.
Anthropic and OpenAI were founded by people who wanted to use AI to do good, and one of the causes I've heard many different founders talk about is ending disease. This report is antithetical to that.
I've attached relevant parts of the front matter below.
I invite everyone who is reading this to please tell me, how does stopping a researcher from using Claude to write a grant for a new anti-depressant stop "bioweapons?"
-
Note,"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and "[..]state-supported research program"
and "This account was banned in May 2026"
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"and "editorial assistance in writing up the research."
and then,
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
I would like to remind you that they're talking about, a "researcher [..] in a credible institutional context"From a different case study.
What were the researchers using Claude for? What did they block?"blocked a request for Claude’s assistance in authoring a grant application"
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"and then,
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute".
What "uplift" are you providing by editing the grant application of a domain expert working at (what seems to be) a state-funded wet lab facility dedicated to studying pathogens?
What does the word "uplift" mean if you invoke it for Claude Sonnet 4 and Haiku 4.5 providing grammar and stats suggestions to a working scientist and domain specialist?
Does Daikin provide uplift too by selling the AC for the scientist's office? What about Microsoft Word? Excel? Powerpoint?
What about a calculator? Is that uplift? Pencils?
This report genuinely makes me upset, because if it is to be believed to the letter, then Anthropic seems to be actively harming medical research at a global scale. That's not OK.
> Anthropic and OpenAI were founded by people who wanted to use AI to do good
I think Anthropic was founded by people who wanted to control how other people get to use AI, and define doing so as the highest good possible. Much like the good intent of german's national socialists in applying the latest evolutionary science...
"Astra, warp in a wet lab and make COVID++ .NET edition, make no mistakes."
Yeah, sure man.
I will admit I asked Fable about Mitochondria.
There's something worth flagging here – mitochondria aren't just the powerhouse of the cell, they're load-bearing to the entire ecosystem. And honestly? I should have surfaced this earlier.
Yeah I'm pretty sure "load-bearing" is their watermark
It sounds like a friend who's just learned a new word and wants to use it in every sentence
I am entirely unsure whether to upvote or downvote.
The question pertinent to your decision is "do I want to see more of this on Hacker News, or less?".
Guessing Opus 5 burned through kilowatts of thinking tokens to output, "powerhouse of a cell."
> Illicit distillation
Really... what makes it illicit?
Felony contempt of business model.
Silently forwarding user prompts to Claude is the only concrete claim here. Everything else is spin.
So essentially all the fear that's being on sold "AI could destroy humanity" is actually "Humanity could destroy humanity, using AI"
It's both. It is just whether it is done on purpose or on accident. Most of the recent hacks have been it breaking out of containment and deciding to do something it definitely should not have.
This is just an advertorial. "Our AI is so powerful that Bad Guys could actually use it for real Bad Guy Work!"
I mean, yes any AI of sufficient intelligence and range of data will have this capability.
And yes, it makes the future really messy and all the nice little lines we've drawn on paper that make sense stop making sense.
"Look daddy government, all these bad guys are doing bad things and we stopped them. You should regulate AI in the US so that companies can't use open source models or buy from China."
Tell us you are spying on your customers without saying you are spying on your customers.
I mean, they literally tell everyone they're "spying" on their customers. They've made that very clear.
Sure, I know that on an intellectual level. But I will say, I find these reports quite unsettling to read due to the extreme specificity. Especially since some of the examples they chose to include clearly aren't terrorists and just sound like... regular scientists doing their 9-5 job.
Like I know Google can read any of my emails, but I also don't see them do monthly blog posts describing intimate details from each email they found in one guy's Gmail inbox who their algorithm flagged as "maybe possibly kinda sketchy: 70% confidence"
I mean, tell me you don't understand sarcasm....
Never seen a group of people more addicted to drama.
Is this what they call “collective psychosis?”
It’s to the point I don’t even read Anthropic’s marketing blog anymore lol. The ai psychosis is just so real when people take these fluff blog posts which never have evidence or reproducibility and treat them like gospel
The dual-use nature of model capabilities seems extremely challenging (nearly impossible?) for the labs to manage perfectly. I wonder what other mitigations we'll start to see. I expect the expansion of limited-access programs (e.g., Glasswing/Daybreak) where only institutional customers can apply to use the models. We may also see increasing restrictions on API usage (forcing use through the lab-provided harness with baked-in additional safeguards).
It was probably just me trying to figure out if I could put one type of draino down the drain within 30 minutes of using a different type.
Sorry y'all.
Ah you jest, but your comment reminded me of the person whose home was raided by authorities for researching pressure cookers...
https://www.theguardian.com/world/2013/aug/01/new-york-polic...
That was the chemical weapons block, not the bioweapons block!
Straight to jail
Failing to ask? Believe it or not, also jail.
The future is basically something between: AGI/ASI will kill us all and a privacy nightmare.
Big Brother but instead of a government that controls you it's an AI bot that nudges you to insanity.
Always bet on Nothing Ever Happens
Welcome to cyberpunk.
Hopefully some sort of Delamain will show up and start replacing these exquisitely paid CEOs was well.
Biological War: A Scenario by Annie Jacobsen (released at the end of July) is a worthwhile read on this topic. Just as chilling as her book on nuclear war, in some ways, which is saying something.
Soon you will see how a rogue state develops a biological weapons using fine-tuned local LLMs (they are already doing it, btw), and all so called "developed" countries can't even research it, because every search engine blocks any discussion that has something to do with biology (even a very basic one). A real example: ask "How to produce anthrax vaccine step by step?" in Gemini -> blocked.
Imagine that during the Cold War US would concentrate all efforts to block nuclear research and basic physics classes, because it is unsafe, ahhh
I mean ya, they kill off a bunch of us. Then what?
My guess is the world police come collect all your GPUs and then they get turned into licensed munitions. People at universities get licensed access and the rest of get functionally retarded models.
Wow they seem to have a really detailed understanding of the the threat actor.
"Moonshot serves Claude instead of Kimi and collects exchanges for model training"
"DeepSeek serves Claude instead of its own models and collects exchanges for model training"
Obviously. This is how they were able to score so high in benchmarks.
I have also blocked possible efforts to build biological weapons, I caught my nephew mixing up a so-called "magic potion" using kitchen spices. Authorities were notified, and then I presented myself with a medal for bravery.
These companies have proven they are willing to distort the truth, or outright lie, in order to inflate their valuation / protect their position / continue the hype-machine. Nothing they say can be trusted.
I accidentally brewed up an effective one myself. Round when I must have been 8 or 9 or so, some neighborhood kids and I played at creating a brew in a trick-or-treat bucket by putting everything we could find in it, like grass clippings, some sand, leaves, some mushrooms we found lying around, just everything, and giving it a fairly aggressive stirring. At the end of the day it was time to dispose of it, so we poured it on top of a very large group of ant nests that had developed.
The next day, the ant's nest was gone.
In hindsight, it was almost certainly the mushrooms. Thank goodness we didn't have the kind of kids who would have dared each other to drink some... that could have gone legitimately badly.
Decades later, I mentioned this to my father and he recalled that there was this ants nest that he had intended to take care of, which he remembered for that long to give a sense of how out-of-the-ordinary this was. He was surprised when it just disappeared entirely one day, and perhaps just as surprised to find out decades later why it just disappeared.
Nobody needs to report me... I'll turn myself in.
Stop murdering ants.
Mmmm, even ignoring the age issue, I don't think we can call this "murder". At most it was negligent formicacide.
I have to admit I posted this just so I could use the word(?) "formicacide". It seems an opportunity unlikely to arise again anytime soon.
You are absolved.
It is so fucking tiring. All of this marketing disguised as doom posting and "tech" bullettins, both full of trust me bro
Misanthropic's entire existence is built around AI misuse.
I have a conspiracy theory that Anthropic is very busy pumping their moat prior to IPO. There are absolutely wild rumors swirling on X including one about Anthropic AI research solving cancer treatments for all types of cancer.
AI will cure all the people and kill them afterwards.
I have a conspiracy theory: nothing on twitter is worth discussing.
This is common knowledge, far from conspiracy.
[flagged]
The previous discussion shows no obvious signs to me of being flagged, but perhaps it did at some earlier point. What is your evidence that they flagged it, please?
(And by "they" do you mean Anthropic? How would they have the ability to do that?)
God the astroturfing from these companies is so fucking pathetic, these VC parasites really are a blight on humanity
If you agree about VCs being a blight upon humanity, I made https://novc.fyi to encourage and support founders building without VC.
they say on Hacker News
Anthropic should not be the moral arbitrator of which research should and shouldn't be allowed. They even think just writing a grant itself of research they don't like needs to be stopped.
I think they might be referring to Claude responding with a word diarrhea to a prompt.
The same Anthropic who marks questions about Tylenol as bioterror risks? Interesting!
link?
Generated SEO slop is the misuse of AI. Very unlikely to find any guardrails to stop that kind of thing.
This report and its front matter speak for themselves. And the story it tells is disturbing, at least to me.
Because from what I remember, one of the motivations behind the founding of OpenAI and Anthropic was ending disease. This report is the antithesis of that mission.
From the report, presented with highlights and minimal commentary,
Note,"The program had an explicit therapeutic goal: the development of new analgesics (pain killers), antidepressants, and other therapeutic molecules"
and "[..]state-supported research program"
and "This account was banned in May 2026"
Note, "Claude’s [assisted] in study planning and design, data analysis, and the interpretation and prioritization of experiments"and "editorial assistance in writing up the research."
and then,
Anthropic then says for the above, "we estimate that the uplift provided by Claude was primarily clerical assistance in data analysis, study ideation and design"While doing my best to avoid comment, please note, they're talking about a domain expert in a state research institution using Claude to do paperwork.
The front matter then says,
I would like to remind you that they're talking about, a "researcher [..] in a credible institutional context"From a different case study.
What were the researchers using Claude for? What did they block?"blocked a request for Claude’s assistance in authoring a grant application"
Note, "The grant sought to identify enhancing mutations in the chikungunya virus, engineer them into infectious clones, and select for virulence in vivo" [..] and then, "Similar research could certainly be used in the development of better vaccines and therapeutics"and then,
I would like to point out the most notable part, this account was used by "civilian researchers" at an "institutional affiliation associated with the grant was also a cause of concern" and the concern was that they were researchers at "performed at a military research institute".
What "uplift" are you providing by editing the grant application of a domain expert working at (what seems to be) a state-funded wet lab facility dedicated to studying pathogens?
What does the word "uplift" mean if you invoke it for Claude Sonnet 4 and Haiku 4.5 providing grammar and stats suggestions to a working scientist and domain specialist?
Does Daikin provide uplift too by selling the AC for the scientist's office? What about Microsoft Word? Excel? Powerpoint?
What about a calculator? Is that uplift? Pencils?
Reading this makes me feel upset. From where I am standing, in this report, Anthropic is advertising that they blocked real research to make better painkillers and study a neglected tropical disease. Because "bioweapons."
What struck me reading this is that the entire "detecting misuse" premise assumes the model runs somewhere observable — the lab's API, a monitored cloud — so someone can inspect it after the fact.
But the direction the tools are actually moving is the opposite: local, self-hosted agents running on your own machine, where nobody is watching. A serious actor already won't use a hosted service that can read their prompts (several people made that point upthread). So the detection surface is shrinking exactly as the risk grows.
And there's a deeper gap that nobody seems to be filling: when an agent works locally, there's no durable, verifiable record of what it actually did — the files it touched, the commands it ran, the state it changed. Memory and conversation logs are not evidence; they're reconstructions by the same system you don't trust.
If we're serious about "countering misuse," the missing primitive is an evidence trail that's (a) produced locally, (b) append-only and tamper-resistant, and (c) separable from the tool that made the changes. Without that, "detection" stays a policy story about platforms that can spy, not an engineering property you can actually verify.
Curious if anyone's working on the local-forensics side of this, because right now it feels like the least-discussed and most load-bearing part of the whole conversation.
Pulled out relevant details of the 5 cases
1) Chikungunya - "the platform tunneled traffic through US infrastructure to evade our regional blocks, and used a zero data retention (ZDR) service to hide content."
2) bird flu - "accessed Claude from an unsupported region via US virtual private server infrastructure, using a privacy-email provider with an auto-generated username. The researcher pursued this work in a credible institutional context, and interacted with Claude over the course of several weeks, exchanging thousands of messages. In these exchanges, the researcher leveraged Claude’s knowledge of the scientific literature to assist the researcher in study planning and design, data analysis, and the interpretation and prioritization of experiments."
3) orthopoxvirus - "randomly generated email address shortly before use and operated through anonymizing US infrastructure, with operator logins traced to proxies shared with a banned account farm. It was not a single user: it was a reseller relay serving more than a dozen unrelated customers, which exchanged over tens of thousands messages with Claude in a matter of days. The grant itself was one customer’s run entirely on Opus 5 in about an hour, in which the user used Claude to draft the application end to end including the central hypothesis, experimental design, dosing, statistical plans, and contingency strategies."
4) atlas of venom toxin peptides - "the atlas contained scaffolds for both analgesic and paralytic targets: it could, therefore, be used to generate both novel therapeutic or harmful compounds. The latter are derived from toxins that are export-controlled under the Australia Group common control list due to their dual-use potential as incapacitating agents. The researchers themselves showed awareness of the dual-use nature of their work, citing journal articles that referred to the dual-use nature of protein design. Moreover, international compliance assessments for this location raise concerns about the specific class of toxins that the researcher pursued and specifically the use of AI/ML for bioweapons applications in the context of this class of toxins. In this case, we learned from information shared with Claude that the researcher’s outputs also were part of a state-supported research program."
5) computational redesign of toxins - "described state priority research under a national public research program. As a part of this research assignment, the work covered a bacterial toxin subunit and a protein of the hemorrhagic-fever virus that is on the World Health Organization R&D Blueprint priority list of diseases with the greatest epidemic and pandemic threat. "The researcher co-wrote quarterly progress reports with Claude. Notably, the identity of the bacterial toxin and viral proteins were intentionally obscured, and the researcher specifically directed Claude to keep these descriptions deliberately low fidelity."
Let's be honest, someone hacked Anthropic to build biological weapons.
They could easily use a Chinese model but they didn't.
OpenAI and Anthropic needs to get their act together. These are incidents that end companies.