Reverse the situation and the media would also turn that into outrage. Imagine someone shoots up a sheriff's office and then it turns out they declared it to some chatbot, and the AI company failed to detect and report it, and "didn't push back enough" whatever that would mean, and hence the AI was implicitly complicit etc. That would also be a major PR catastrophe.
Damned if you do, damned if you don't. Same as with social media platforms. That's because only a tiny tiny sliver is for true privacy when that means "bad things might happen" or bad people, such as your political enemies, may do stuff you don't want.
In my opinion, if the company is allowed to see the data and train on it, then they are also responsible for reporting stuff like this. Without knowing the data licensing agreement the lady had with Anthropic, if she agreed to letting Anthropic see her data, then they should do stuff like this. If she didn't agree, then I wouldn't condemn Anthropic for failing to report an attack
Pool together with some friends and buy an H200 or two to run unquantized open source models with abliteration/heretic transformations.
You need to be able to use these models for the real world and not for some imaginary world where everything is safe and nice and happy all the time, while at the same time intensely surveilled in the name of CYA and the latest panic about whether speech THAT ISN'T EVEN BETWEEN TWO PARTIES is considered "wrong".
I'm a free speech fan that acknowledges there are lots of boundaries of free speech (fraud, perjury, blackmail, defamation), but the one thing that all of the boundaries have in common is that a second party must be involved for them to make any sense at all.
Maybe the courts will uphold this, maybe they won't, but don't take the risk!
It’s been quite exciting that Qwen 3.8 Flash Next has come out: it really is similar to Opus 4.5, 4.6 for coding. Remarkably intelligent, and runs on a single DGX Spark, which I paid $4000 USD for
Sadly they’re now twice that price, which is a shame because I really want a second one!
I have a heretic modified version of it too, for when I want to use it for security and so on. Quite interesting
I agree up to a certain point, but there has to be some legal boundary between freedom of thought/speech and literally planning a crime. I'm protected under the First Amendment to say "someday I'll rob a bank" but not necessarily "I'll rob this bank on Friday and here's how I plan to do it".
I think you should be allowed to write that exact line in your journal. If you rob the bank that can be used as evidence against you, but in no way is it acceptable for private reflections alone to be used to arrest you. Or else every author who's written a novel with 'bad' characters would be arrestable.
For the record, under current US law, it is not illegal to have a sentence in your locally-stored notes on your phone outlining a plan to commit a crime. There has to be an overt act. The police in that instance could inform the intended victim, surveil you, etc, but they would not be able to successfully charge you with attempted murder. It's not illegal to be considering committing a crime, even if you have a tendency to write down your thoughts.
The law in this particular case, which seems to be intended for threats that you actually send to someone, is being interpreted broadly to apply to any "threat" that you transmit to a server. So in your hypothetical, the legality would depend on whether your notes are backed up to icloud or not.
This is true for the general criminal conspiracy law, but be aware that an “overt act” doesn’t have to be an illegal act, just some action in furtherance of that crime. That can be purchasing a weapon, or scouting out a location. There may also be other laws in play depending on the specifics
Just having saulpw's physical address is already furtherance of that possible crime.
The commenters here are cute little HNers who think they have found a loophole in the law. They are not the first ones innover their head.
Spoiler: the law is written in words, and those words aren't strictly executed like in a computer program, they are interpreted by actual humans who can see what you are trying to do and will stamp it out.
You're absolutely right, at least according to my own quick check on Gemini. I find this state of affairs amazing.
In my country, no "overt act" is required, but both here and in the US a "conspiracy to commit" charge requires an agreement with a second party. This is indeed consistent with a very broad interpretation of "no thought crimes".
When you read something describing in detail a person's intent to do something very bad, in a place where they write things that they intend to do, and which in the past they have in fact consistently done, you don't attach any significance to that at all?
I certainly think a police officer that stumbled across such a thing would be justified in asking follow-up questions like 'hey, what's this about you going to shoot someone?' and then maybe making an arrest based on your replies/demeanor.
'Probable cause' should involve a degree of certainty, because 'possible cause' would be altogether too loose of a standard. It's possible that you're intending to shootme and you just mentioned saulpw to throw other HN users off the scent. Possibilities are only limited by the assessor's imagination.
How does the cop at the traffic stop know that your username is akoboldfrying? Did you tell them? If so, why, at a traffic stop, did you do that?
Are you posting threats on hacker news while you are driving? And the cop was close enough to see your username and what you wrote? Is that why you were pulled over?
Well in your example you've begun conspiring with a second party so that's not at all the same thing. You are at least free to plan all the crimes you'd like to arbitrary levels of detail in private. It's when you start acting things out (soliciting coconspirators, blackmailing targets, etc) that you cross the legal line.
The current situation is a weird one. Anthropic reported single party interactions (per the ToS and common sense), there's a statue about sending threats (as there clearly ought to be), then somehow the definition of the word "send" was tortured by the local police. If a crime has been committed here it's almost certainly an infraction by the local authority against the spirit of the law.
I totally agree. People seem to be stuck on the notion that we must not punish thought crimes, and have elevated this above all other considerations, when really it's just one among several.
However, those other respondents to your post seem to be accurately describing the current legal situation. I asked Gemini, and apparently "conspiring" to commit an offense requires an agreement with another person in both my country and the US, where an "overt act" is also required (that may not be incriminating by itself). I find this alarming. The fact that someone's private diary entry describing in detail a plot to kill me does not amount by itself to anything is... incredible to me.
Obviously, I'm not talking about the verbatim quotes I provided. There has to be some level of evidence that proves intent to commit a crime and the second quote is meant to represent that whole class of statements, but it depends on context. Any given quote won't constitute evidence in every case, but it will in the cases where it proves intent beyond a reasonable doubt.
I'm not sure why you think my argument holds no water when there are clear legal precedents that speech is not protected in some cases where there is "imminent lawless action".
I'm not a lawyer so take this with a grain of salt, but it seems like there is a big gap between speech that is "directed to inciting or producing imminent lawless action" and just saying that you will do something.
Depending on context saying "I'm going to rob the bank X tomorrow" might also count as a threat?
But, in either case, writing this in a private diary could not be incitement or a threat because you are not communicating with anybody except yourself.
You can trust another party and do this by renting a few H200s. You cannot pool together with some friends without trusting another party.
You end up just weighing up the difference in trust between a vendor and a friend against the level of disinterest that they might have in your affairs.
A cheap second hand 10ish year old card like my radeon rx570 with 8GB of ram is plenty enough to run a small uncensored model with llama.cpp if all one wants is chitchatting with a clanker.
We are not talking about heavy coding use cases here.
Preach! I am so tired of Anthropic’s safety team being the arbiters of what is right and wrong. If you so much as hint at impropriety you can have future sessions flagged ad infinitum when it comes to specific topics.
That's definitely something I'd consider if I had cash to spare for H200's! Unfortunately I think for most of us the price of self-hosting has to be 2-5x lower still.
Way ahead of you bud, mac studio m5 ultra 256gb version is coming soon.... ordered 2x of em just in case. expensive as fuck but its a hedge against all this bullshit and more
I'm not saying fake child porn should be allowed or not-allowed, just showing there exist possible exceptions and rationalizations for them even without two parties.
I have some sympathy for Anthropic here because I've seen the headlines after OpenAI failed to report a shooter in a similar situation. So from their perspective, it's damned-if-you-don't, damned-if-you-do.
However, people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech. Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs. Now, everything you say or write can be automatically screened for red flags on a planetary scale, and probably will be because that's what the regulators and "concerned citizens" will demand. In a couple of years, you'll be biting your tongue a lot more often in private chats.
I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications. Ever since the Snowden revelations of 2013 I've more or less assumed that everything I type into a computer is stored in a government database somewhere. Not that I actually believe it is 100% of the time; there's a spectrum of trust, so I'm more confident that local apps on my Linux desktop are secure, somewhat confident in the end-to-end encryption of certain apps on my iPhone, but all bets are off for non-E2E encrypted data going across the internet.
People don’t want to believe the Stasi is monitoring them. They want to believe the world is a nice place just like where they grew up.
Also, unlike the bad old days when 1 in 3 was an informant, now ordinary people aren’t in “informant loop” of providing information on others, so they aren’t thinking about being informed on either.
For technical people, this is incredibly old news.
For non-technical people, it isn't really news, because they already forgot about it after reading it. Maybe they'll be a little more monitored in their own typing for like... a day or two.
One of the hardest lessons to internalize, and keep internalized, as someone who works on and writes software, is the vast, vast, vast majority of the Public doesn't understand even the most basic shit about software. It just does stuff. Hopefully the stuff is good. That's it, beginning, middle, and end.
"Why would you think x would y" is a poor framing. They didn't think about x or y because they don't care. The phone works, that's the beginning and end of their interest in the subject.
It isn’t that black and white. I have relatives that are highly non-technical but still care a lot about privacy and data protection in their computer use. They hate having to deal with technical things and have very little technical understanding, but still ask me whether using X software in Y way might pose privacy issues or not.
You're completely right. I've internalized this on the technical side, but I still find it puzzling politically. Surveillance has been a salient issue in U.S. politics for almost 100 years now, since the invention of the telephone. I would like to think that most people are at least vaguely aware of Watergate, the Patriot Act, and Snowden. Then again, I should probably stop assuming that people know basic history, given the current state of education.
Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway. Like so very often the perpetrator of a given crime just did one incredibly bone-headed thing that brought the cops directly to their door.
I think in part it's selection bias? Like if you're smart enough to get by honestly, you're probably also smart enough to realize getting by honestly is just a way more comfortable way to live. The only reason you'd probably cross that line is because your principals, whatever they may be, conflict with those laws, or your life circumstances are so bad that you have no choice BUT to turn to crime.
And that cuts the other way too: if you're dumb enough to think you'd NEVER get caught for a burglary, for example, you'd probably be way more down to plan and execute one, failing to consider that most thieves aren't caught when they steal the shit, they're caught when they try and sell it later.
Not just bad headlines; OpenAI is actively being sued [1] for this:
OpenAI’s stated rationale was a concern for the shooter’s privacy, but its own interests
better explain its silence. Upon information and belief, OpenAI was seeking to avoid implementation of a
hard line rule to refer planning of real-world violence to authorities, perhaps due to how frequently its
product is implicated in threats to human life. Requiring such disclosures would be incompatible with the
company’s public position that ChatGPT is safe. It could also threaten the valuation underlying OpenAI’s
anticipated initial public offering. Rather than expose those risks, OpenAI accepted the consequences of
its silence. A mass murder in the only secondary school in Tumbler Ridge followed.
Accordingly, the Crown, led by Attorney General Sharma, and SD59 jointly bring this
action to hold OpenAI and Sam Altman accountable for designing a dangerous product, distributing it to
every home with internet access, ignoring the warnings of their own safety team,
refusing to notify authorities when they knew the shooter was planning gun violence, inviting the shooter back onto the
platform after deactivating the shooter’s account, and choosing corporate self-interest over the lives of
children. They seek compensation for the not just foreseeable but known harm OpenAI inflicted, the
damages that they incurred and are incurring, and injunctive relief to ensure that this tragedy does not
happen again.
This is one of the reasons why AI companies are looking for explicit regulations, it can help to reduce the risk of possible liability and maybe make the legal way forward more tractable. With a regulatory framework in place much of the burden of identifying risks falls on the regulatory authority.
Then the AI companies have more confidence that they can move forward in a certain way, and issue investor guidance that is maybe closer to reality.
I think they're mostly looking for regulations because they've spent close to $2T, all to realized they have no technical moat, so they're trying to build a regulatory one.
It's the only thing that makes sense for a lot of these companies to survive to any long term when open models keep nipping at their heels for pennies on the dollar.
No, I'm assuming that the open models won't/can't and will be banned from a lot tons of use cases that will mandate use of the large frontier shops in order to comply with said regulations.
The theory is that open models cannot follow regulations and will therefore be banned or not eligible for many of the large contracts that the frontier labs will win.
> people need to get it in their heads that they're not chatting with their secret BFF
I see constant ads on video platform (particularly youtube/tiktok) about llm chat apps, from friends, dating, romance and everythkng inbetween; that's personal.
People need to be reminded constantly if they use such apps that they are participating in easier mass surveillance, profiling and AI training.
How do you get that through to someone who doesn't even understand that mass surveillance and profiling is a problem? Or to young people who have only lived in a society of mass surveillance?
Problem is that even not using those apps, the apps you currently use might have turned more hostile.
It wasn't technically feasible to scan personal chats easily, other than grepping keywords which must have had a bajillion false positives. Now you can get everything autoscanned at scale.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
Yup. And with zero privacy protections in statute for AI chat, there is nothing to prevent an AI CEO looking to curry political favour from e.g. handing over the private correspondence of an opponent or an entire district’s residents.
Customers who what privacy protections for AI chats are welcome to negotiate this in enterprise contracts. The major LLM vendors do offer that as an option. Customers can then enforce any violations in civil court (although this obviously wouldn't apply if the customer used the LLM for criminal purposes).
Don't know why this got downvoted? This is a problem with epistemics.
It's more efficient to have one central "verifier" for everything, but the "who watches the watchers"? question basically says: Either constrain by construction, have everyone verify (which are two sides of the same coin, btw, when looking at a "global" thing), or centralize explicitly.
And such was the case for a man who snapped a photo of his own child to send to the doctor which got uploaded to his Google photos resulting in his Google account of over a decade getting shutdown for CSAM.
As another commenter said, you're not chatting with a friend; you're chatting with Big Tech.
If the doctor was not using a gmail account, the UI probably recomended to share it "with anyone that has the link" that is like public but protected by oscurity.
Most people don't realize that it is 99% like posting it on Facebook.
A disclaimer on the top of page every time would have been a better approach helping Anthropic and the end user.
A bot talking to you directly as if its some one real caters to your thoughts and can take you in a certain direction without you realizing it. I have heard first hand experience from people that they feel more comfortable talking to chatgpt or claude cause it gives a feeling of being on their side and listening to them.
I suspect it will go further: imagine giving an mp3 to LLM to clean up some noise. It detects it was illegally downloaded from youtube, deletes it and automatically fines you via attached credit card.
They could be happy to assist you with a task that they also then flag and forward to authorities, especially if their assistance doesn't break the law but has evidence of you doing so.
Unless it becomes a requirement to be licensed to be able to use any kind of ai model. You know, for safety and stuff. And of course with appropriate reporting to institutions.
Or worse: downloading a picture of pirate ship and without any concern for the copyright asking the LLM to make a coloring page for your kid. BTW Chatgpt does that way better than Claude
Hey but you’ll be allowed to file a response that will also go to an LLM and deny you automatically. And you will be charge a NSE fee (no sufficient explanation).
You should probably get a head start on waiting a couple years to bite your tongue and assume everything you type into a computer is summarized and sent to your boss, government, advertisers, political actors, insurance companies, worst enemy, etc. With phones, Alexas, and little AI tamagotchis, you probably shouldn't say much in person either.
They're actively rummaging through your inputs so the damned-if-you-don't case doesn't really exist; no one expects Anthropic to not notify law enforcement once they learn of something like this. What you might have expected was some privacy in the first place though, where Anthropic would never have learned of this in the first place and where the damned-if-you-do case wasn't a thing.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
It's the other way around, big techs need to properly disclose in their platform, during interaction that they aren't in a private and safe environment
Right, the difficulty is partly that they can get a negative headline from any choice of behavior.
"Anthropic failed to report murderer's threats to authorities"
(or "Chatbot knew man was planning murder, yet company did nothing")
"Anthropic reported private chats to authorities"
(or "Arrested for chatbot fantasy")
To be fair to the journalists in these cases, there's also no society-wide agreed Schelling point about the correct outcome or correct rules. I have strong beliefs and intuitions about what should happen, but other people also have strong beliefs and intuitions, and many of those are probably opposite of mine. Even if my intuitions are the best and most justified, a journalist is unlikely to think "I'm just not going to mention that some people are mad at this company over this outcome, because a hypothetically better norm or principle would support the company's actions here". Hopefully the journalism can at least contextualize the lack of legal or social consensus and the difficult incentive problems, rather than jumping to "obviously companies are sociopaths staffed by supervillains".
Including any chats anywhere where someone might have a phone in their pocket, or if there's a "camera" attached to a utility pole or a nearby tree. The only real private chats might be whispered lying down in the bathtub together, with a mattress covering it like you're both hiding from a hurricane.
> they're chatting with Big Tech
They're chatting with any powerful person who wants to hear it. She thought she was chatting with Anthropic, who doesn't give a shit about her. But after being threatened (and immediately backing down because, of course, they don't give a shit about her) Anthropic has become an arm of the government. So she was chatting with the Bonita Springs, FL Sheriff's office, or anybody else. If I paid enough, Anthropic would tell me about what she was doing so I could sell her laundry detergent.
It's a byproduct of the nannyism safety marketing from the AI companies. I'm glad these cases were caught, but disagree with how they were disposed of. If the automated flagging is good, enforce it by default. If it's noisy, refine the tech then enforce it by default. This middleground where everything going through the platforms is subject to training and arbitrary human inspection in the midst of an acrid cloud of marketing-driven fearmongering is unacceptable, and it reinforces the idea the fearmongering is legitimate.
Somehow humanity survived the past 40 years without Microsoft Word and Excel phoning home and shopping users to the feds at random, I don't see why the standard should be any different for this new class of tooling.
Absolutely no sympathy. Anthropic is every bit as slimy as any other big corp. And like other big corps, they must open the vault to whatever governments they intent to do biz with.
We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed.
I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There'd also be fewer people falling for apocalyptic Rationalist delusions.
Also: people need to understand "not your computer, not your data." (Unless it's stored in the cloud but encrypted locally with keys only you possess.) Same goes for storing things unencrypted in OneDrive, Google Drive, etc. There is nothing to stop these companies from bulk scanning, data mining, or reporting people based on whatever request a government gives them. Don't count on them to resist, because they often can't, especially if the request is from a sovereign state where they do business.
You can make a reasonable case for adding a lot more classes in high school: statistics, nutrition, personal finance, etc. But ultimately it's a zero-sum game and to add a new class means removing an existing class. So what do we cut?
My kids' high school covers those topics. And I think that is not a new thing, I specifically remember taking the required personal finance class 35 years ago (great teacher -- crotchety old man who wrote "compound interest" on the blackboard every single day before class and repeatedly proclaimed that if we forgot everything else he taught, please try to remember how compound interest works).
There'd also be fewer people falling for apocalyptic Rationalist delusions
Assuming you consider it a "delusion" to have a p(doom) of more than 5% or so, that's not uncommon among frontier lab employees who have a pretty good idea of how LLMs work.
Indeed, I don't see how knowing the details of how LLMs work (which I know btw) would change anything about how intelligent they are. I only need to know that it's a computer program that writes stories, solves math problems and seduces people.
They have some responsibility for people's expectations of the product, at least. They want the personal assistant personas to be able to help you with anything, and don't point out that they'll be judging your thoughts along the way.
For anyone technically inclined it should be obvious, but it isn't part of the zeitgeist or how they pitch it. People see it as being different than talking to a human, and behave as if there won't be a human in the mix.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
I think Anthropic did the right thing here; but the sheriff's office are probably demonstrating why she dislikes them. Writing a diary entry to a chatbot is clearly not how this law was intended to be used.
EDIT: Actually, on reflection, making this report to the people she was upset about was probably not the right call. If they'd sent it to the FBI, there'd be a much lower chance that someone felt the need to assert their "authority".
This is the paradoxical times we live in right now.
Don't do something? She walks into the office and start shooting the place up. Several officers and innocent people are killed. Cue the media claiming, "You should've known she was talking about this an AI bot! Why didn't the bot tell anybody she was planning a mass shooting?!"
Do something? She gets rolled up by the cops and questioned about what she was talking about and brought to the cop station and interviewed. Cue the media claiming, "This is an unethical way to use AI, this is an infringement on free speech! This is authoritarian!"
I believe in free speech as much as the next person. But in this day and age, its almost better to be safe than to have to explain to someone's loved ones you had to chance to prevent this and did nothing.
> This is the paradoxical times we live in right now.
It's always been complicated like this. That's why certain professions (psych, lawyer, clergy) come with rules around when and if disclosure is allowed[ required, and/or admissible].
They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers, but by and large people seem to accept “we take technological measures to ensure privacy and that means we can’t spot crimes.”
So this conundrum is at least partly of their own making.
I can't say I actually disagree with the initial prosecution. The penalty was a fine, likely less than the cost of investigating it.
Intended as a joke? Blowing off steam? I can understand that, but given the number of people on social media is large enough to include genuinely unhinged people, you can't expect anyone who receives such as message to take them as a joke.
Same with AI use. A billion users, you have to assume some of them are actually sincere if they write about any act of violence, from self-harm to a plan to steal a nuke and use it in a false-flag attack to trigger WW3 and everything between.
> you can't expect anyone who receives such as message to take them as a joke
That's a distinction that matters to me. Sending a spicy note to an LLM isn't remotely the same thing as posting it on social media where the entire world can read it.
I kinda agree, but on the other hand anything you send to an LLM has to be viewed through the lens that only an automated system (i.e. an LLM) is capable of even handling such a tsunami of natural language.
It absolutely will misclassify things, it doesn't know any better.
(We should all wish each other good luck, because we're going to need it).
It feels icky, and my default is not to side with megacorps engaged in blanket surveillance, but I can’t really fault Anthropic here. The correct setup should be a law that protects this sort of interaction with a chat bot as privileged, along the lines of HIPAA-mode. Use a classifier or some sort of “private mode” toggle to tell the platform you’re engaging in privileged communication ala dear diary, and then a much higher legal standard needs to apply to protecting that data (no training, same protections as doctor / psychologist interactions). Even a therapist has a legal duty-to-report in certain situations.
Is it really "blanket surveillance" when it only sees exactly what you put into it? That's like saying you're being filmed against your will while... filming yourself.
I get where you’re coming from, but I think if the progression of technology has shown us anything, it’s that there will be constant competitive and social pressures being applied to use these tools more and more and that pressure will race far ahead of any privacy, consumer protections, public education and societal wellbeing counter forces. So opt-out ends up being a rapidly shrinking iceberg in practice.
Counterpoint, if you give the agent access to your files - which happens to include the Notepad diary you've been writing since your teens - and it makes a similar conclusion about something you privately wrote, would have the same opinion?
Yes, and it's weird that on a form of technologists we're going with the idea that all of these systems we use every day to hold our personal private information that we are constantly ensured is secured against anyone unauthorized from accessing it is actually snitching to the cops just because someone else is hosting it on our behalf.
The 3rd Party Doctrine destroyed the 4th amendment and is the reason privacy respecting software has to play legal games. E2EE while a good security practice shouldn't be necessary to protect you against the cops rummaging around your stuff. The bar to establish that information is private shouldn't be "literally mathematically inaccessible but the cops are still allowed to try."
I don’t live in the US. But this kind of broad law is hard to implement without surveilling all users, and it has multiple side effects.
What happens if I use Claude or ChatGPT to research sensitive social topics? Would that be considered a social network interaction and used against me when I apply for a visa?
Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
> What happens if I use Claude or ChatGPT to research sensitive social topics?
Anthropic/OpenAI is not obligated to report your use, as long as it's not violating some terribly written law (like the Florida law). The government won't know about it, so no, you will not be denied a visa.
> Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
I honestly don't know, but I suspect most US states don't have such an overreaching law.
This is slippery. Many people do all of their journaling inside of Google or Apple cloud products. Some even write up their intentions to do bad things.
Does all writing now have to be scanned for thought crime?
That's an interesting wrinkle that is rather tough to work through.
To me, journaling your intent in a private journal, whether that's an Apple Journal/Note or a Moleskine in your drawer, feels qualitatively different in some way. But I'm not sure why.
So "I'm going to shoot up the police station" written in your own journal feels somewhat different than "I'm going to shoot up the police station" said to a system that might be able to _interpret_ or _act_ on what was said in some way. Did I just give AI a legal duty, or even a soul I didn't think it had before? I've written up about three or four "what about this, what about that" and deleted them all.
You're missing the part where it's a diary entry, so the actual content is irrelevant. Her only mistake was not realizing that her diary wasn't private.
Because it's not relevant. They're not charging her with conspiracy, which is what they would do if she'd actually done anything concrete towards making that happen. She didn't email it, or text it, or post it on Facebook or Twitter or Discord or a message board, which is what this law is clearly about.
How can you charge someone for making a threat when you only read the threat by spying on them? Surely that has to be thrown out in court? They didn’t actually send the threat to anyone, you just obtained it by spying.
The AI companies have clauses in their user agreements saying they can review content flagged as harmful. It’s not legally spying.
If you recall previous outrage about ChatGPT being used in cases of suicides or shootings, this is the result. Every time a crime was committed and the police found ChatGPT history about the crime, the media turned it into a frenzy. So the AI labs added safety filters to their consumer plans that detect threats of violence, escalate them to human review, and report to the police.
Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department. There might be some limitation in the law that makes the evidence inadmissible because it was not intended to be shared with anyone, but that’s a separate decision.
Not only that they can review flagged content, but they tend to have separate retention policies for flagged content. Anthropic's is this: "We retain inputs and outputs for up to 2 years and trust and safety classification scores for up to 7 years if your chat or session is flagged by our automated trust and safety systems as violating our Usage Policy."
So don't run for office or anything like that. Someone, somewhere will have a contact that will get that.
>Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department.
This is spying with extra steps couched in corporate speak.
> Does announcing a spying operation mean that it is no longer spying?
Well, kind of, yeah; the dictionary definition of spying requires secrecy and lack of consent.
> to secretly collect and report information about the activities of another country or organization[0]
The only real debate is whether or not having a clause tucked away in a EULA that few people read makes it a secret. If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
I would call it spying in this sense at a minimum if individual people don't know whether their conversations were stored or disclosed in a way they don't want. For example, suppose someone said "we will monitor the activities of 10% of people". You don't know if you're in that 10% or not, but I would still want to call that spying.
A less central case would be when you clearly do know about the activity but you can't quite see the details, like with behavioral ad targeting or something. It feels pretty normal to me to call that spying even if it's disclosed to everyone and certainly happens to everyone, but it's also a less central example of the concept.
Anthropic could put a big flashing warning text at the top of every chat that says “We are spying on you and will report anything scary to the police!” and it would not make any difference in this case.
You can call it anything you like, but only the legal definitions matter for the legal case.
After working on several court cases about surveillance activities, I'm definitely aware that whether I call something spying or not has little relationship to whether courts will think it's legal.
> If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
If you change the situation then yes you can in fact change our responses. The problem is you then are no longer talking about the original situation.
It also bears mentioning that providing a dictionary link to “spying” is pretty patronizing/passive aggressive. On par with sending a basic Wikipedia page. You didn’t even bother to post the definition you want to apply.
No one claimed any case would be "thrown out for spying." The legal definition of spying is also not particularly relevant to the argument in the initial comment.
The initial comment instead questioned how someone could be accused of making a threat if they did not realize anyone would read their private content. You probably also can not insult someone with a statement you never expected anyone but you will ever read.
> Presumably, Anthropic did the spying and the reporting.
You don’t need to presume. Anthropic reported it.
“Spying” as a legal concept has a definition that does not apply here. You could say they were “spying” in the sense that they read someone’s input, but that’s literally what they said they were going to do in the agreement when the person signed up.
So I responded to the question about the case being thrown out for “spying” by trying to show that the word doesn’t apply in the legal sense. If you sign up for a service that says “Hey we’re going to monitor your chats and might report things to the authorities” and then they monitor your chats and report things to the authorities, you should not expect the case to be thrown out for “spying”.
I’m speaking from a functional/ethical framework to be clear. I’m just expressing frustration, not challenging the comment. Could’ve been clearer on my end there.
Well, let's say that you have a regular customer at a bar.
They get friendly and loose-lipped with the bartender over the span of months. Eventually they let slip that they plan on killing their spouse for a life insurance payout. At first the bartender thinks they're joking, but it becomes evident that there's an actual plan being acted upon and someone's life is very likely in imminent danger.
Does the bartender have a responsibility to go to the police?
Depends on the country. In some places, there is no legal repercussions for not reporting this to the police; in some, it is an actual crime in itself.
With the obvious IANAL, it doesn't seem to rely on the message be sent to the person being threatened. The specific segment is "in any manner in which it may be viewed by another person".
This may be one of those cases where we get to find out how courts view SaaS platforms.
The subjective element of crime (i.e. doing it on purpose) is fundamental also in the US legal system. If the person wasn't aware that someone else might see their messages, it should be hard to claim that they committed the crime.
According to Gemini, "Florida appellate courts have overturned juvenile convictions [based on this law] when the state could not prove the person subjectively intended for the record to be seen."
this is almost certainly what anthropic is hoping for here - a judgement that says there is no point in them continuing to monitor and report this behaviour
The prosecutors likely know this and expect it. But there's enough gray area here for them to make the argument, and it's hard to prove malicious prosecution, so they know they'll get away with it. It's just about sending a message to the public - they don't care whether a conviction sticks. Just politics.
The prosecutors aren't on the hook, anyway. They have absolute immunity. The decision to charge is protected. The prosecutor would have to have done one of the few, enumerable things outside the scope of the role, like conducting an investigation without probable cause or hiding exculpatory evidence.
It still shocks me the number of people I know who freely let agents on devices that contain unencrypted private keys, freely dump internal data into cloud models and generally don't give a second thought about any of it being trained on, inevitably leaked one day in a db breach or read by providers. I find it's best to consider any data put into a cloud model the same as if it were posted publicly online, since that is the very possible eventual end result.
Hopefully more of these stories push people towards local models :)
That is not what sandboxing solves. A good sandbox would inject credentials into provider API calls so that the model never sees credentials, but the provider is still going to see the transcript. Sandboxes do not require or imply that there is a local model. Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
>Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
this is exactly what I meant. I am presuming the danger is AI reacting to personal notes that it reads on your computer, like a diary, and you should not allow the tools to have access to those documents.
Another way to interpret this is that they are legally presuming that you already have sandboxed their product and anything it sees or has access to is intentional.
Any failure to understand what it can access or what it has permission to see from the user's end is presumably not their problem. Regardless of what the user specifically asks of the tool.
If you overhear someone, in the privacy of their house, threatening to murder someone and go to the police, surely you don't expect this report being thrown out and you being yourself charged with the violation of someone's privacy instead?
IIRC if the evidence wasn't lawfully gathered (which it sounds like it was, tbh) then it wouldn't be a mistrial, it would be thrown out and then the prosecution wouldn't have any evidence of any crime.
A threat sent by mail is still a sent threat even if nobody ever opens the envelope to read it. The crime is in the sending. This woman used an online resources, one which involves transmitting everything across innumerable state lines. I am surprised she isn't up on federal charges.
Note that the law doesn't forbid the writing of a threat. You have to send it to someone. Had she kept it in a book under her bed, she would not be in trouble. But she sent it to a website/service/LLM portal.
>> It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person
If you draft an email threatening someone and delete it without sending have you committed a felony because someone at Google could be reading your drafts box, stored in a datacenter across state lines?
Honestly, I'm equally fascinated by the way email has changed. 30 years ago when you drafted an email but didn't send it, it was only on your local machine. There was no SMTP. 20 years ago, it might be a 50/50 shot as to whether you "transmitted" it to your "Drafts" folder if you were using IMAP instead of POP3 to read it.
We really need a way to make it clear to users when, through the normal operation of software, they are "sending" data to a third party (usually the software developer) and when they are not. This is definitely not clear/knowable to regular users, and it's kind of hard to figure out even if you're a computer expert. Even software that "runs locally" now sends innumerable amounts of stuff back to the developer, and they don't always disclose it.
This is a huge privacy problem that is only going to get worse.
Sounds reasonable. Google's bots could pick that up easily and forward if for human review.
FYI, the use of drafts folders to transmit messages has been used by terrorists. This is likely where CIA director David Petraeus got the idea when he needed a secure way to chat with his mistress.
Ah, a simpler and more innocent time of government scandals. I miss it. Now the messages are on White House stationery and they declare themselves above the law.
What if she put it in a locked box before shipping it to herself UPS, and she has the only key?
What if instead of UPS, she hired a moving company to move the locked box?
What if she wrote it electronically in diary.txt, but it was backed up to a cloud provider?
--
I'm guessing there's some sort of "reasonable expectation of privacy" for certain activities. We're going to find out what Florida courts think about this new medium.
We'll only find out what the courts think when this happens to someone with a lot of money. It takes a real legal fight to push it high enough to become precedence. She'll be pushed to plea out.
> in any manner in which it may be viewed by another person
Does the person have to know (or at least believe) that it will be viewed by another person?
She likely didn't think anyone would view it. Honestly, even as a career software developer I don't think it is unreasonable to think know would would see what she wrote to an AI. I assume most of what I write to an AI is not viewed by any other human, based simply on the quantity of messages sent back and forth to AIs, I would assume a vast majority are not read by another human.
What if she had written this into google docs, and she kept a diary there? That also crosses state lines, and is transmitted to another location.
“But there can be no valid knowledge about the future. As soon as precognitive information is obtained, it cancels itself out. The assertion that this man will commit a future crime is paradoxical. The very act of possessing this data renders it spurious. In every case, without exception, the report of the three police precogs has invalidated their own data. If no arrests had been made, there would still have been no crimes committed.”
Philip K. Dick, Minority Report, 1955
While I accept that this sort of thing is well with in the ToS and regular course of business of any major online platform, it hits different coming from an AI company for some reason.
I have told llms all kinds of stories to find out what its answers would be. I always make it sound like it is the truth to make sure the AI answers in a way that it would if somebody actually said this. I also tested internal flagging systems of the ai company I work at with the most evil things a person can ever say to find out if it would flag them.
Of course I did not mean any of that stuff, but how can you make sure a human reviewer knows you did not mean it while the llm does not know that you did not mean it.
I guess its a miracle I am not in jail yet.
Flagging people for anything said to an llm sounds wrong to me because an LLM is not a real person and while some people put in their internal thoughts, others just roleplay and the two are inseparable just from reading it.
I was sure I couldn't be the only one curious to push LLMs to their limits. Though these days it's much tougher, mostly impossible to get them to react in unforeseen ways to horrendous scenarios.
This is exactly the typical use I make of the llm.
Adding:
- I typically ask questions in the I form, regardless for whom or why I ask for.
- Gemini chats quite often end when it starts recommending psychological council or a suicide line, to talk about my problems. It apparently detects a persistent tendency to not agree with the party line. So it makes sense I must be suicidal ;-
But sure, as llm's start to babysit us, and know our inner dialog better than anyone else, we'll soon be debugging their opinion/behavior/co-existence/authority, when it comes to reporting people to the authorities, or taking on tasks in society in general. We'll hire doctors to cure our psychological profile from our record (Total Recall).
A Minority Report like this shouldn't cause a referral to the police.
This is a classic case of "Damned if they did, Damned if they didn't". Given this has come out, and Anthropic is considered to be the beacon of transparency, it would be appropriate for them to share what their thresholds are. The article also reports Open AI not considering the threat credible enough to alert, so there's clearly some thresholding that people need to be aware of.
I have a hunch she wins the case, on the basis that an LLM isn't a person and an (assumed) private expression of anger to a machine assistant doesn't meet the statutory threashold which requires that a threat be communicated to some other person.
Of course, that could change if there's evidence that she took action in pursuit of a goal, like buying ammunition or repeatedly driving around the entrance to her alleged target.
The law seems overly broad with "may be viewed by another person". Most of these laws have the intent of not causing public panic with regards to posts that others may see, not stuff that one assumed was private. This is further supported by the definition of threat, which is generally defined as requiring malicious intent.
I would think a good lawyer could get this charge dropped. I would like to see what judge approved the warrant and how they felt the elements were met.
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism.
She didn't threaten anything, she wrote down that she was going to do it. A "threat" is more than a mere statement, especially when written in what is described as a "diary".
> A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to "shoot up" the Sheriff's office.
Obviously I don't want anyone to shoot up anything, but this seems like a weak case legally speaking.
I think it's fair to say this is a gray area. Clearly it was transmitted.
I can certainly threaten you harm and send it to not-you and you're still clearly in danger even if it wasnt transmitted to you. So the question becomes did she transmit it to someone? Clearly yes she transmitted it to Anthropic. But she clearly intended to send it to Claude, an inanimate object.
Claude's terms of service makes it very clear that their employees will read messages[0] to determine that they don't contain the things that these messages contained[1].
> Review is needed to enforce our Usage Policy... designated members of our Trust & Safety team may access this data on a need-to-know basis as a part of their evaluation process.
A reasonable person would not expect humans to review the millions of messages passing through the LLMs, or their own threats to ever be transmitted to a human without their authorization.
Reporting the danger is by itself a good deed. But there should be a better way of restricting firearms from the probably irresponsible lady than using inappropriate charges to punish the thoughtcrime, OR waiting for them to commit violence.
Too broad. Unless you're transferring ink from a typewriter ribbon onto paper in a hut with no electricity, your words, or my words as I type this, are being grammar checked by something partly in the cloud. If I delete my words, are you saying I've transmitted them nevertheless?
Yes but the law usually evaluates the application of a statute within the context of someone's mental state. This is why you are not guilty of battery when you trip and accidentally bump into someone. https://en.wikipedia.org/wiki/Mens_rea
That depends on the crime; several related crimes are only distinguished by intent. Negligence is itself a crime if it is the cause of a preventable death when the person has a reasonable obligation, such as when driving a vehicle.
I'm not really sure that this can be likened to a diary when it is called a "chat" but that's for the legal system to determine, not me sitting on my couch.
Any reasonable person presumes when they chat with Claude that it is a computer program on the other end. "Claude is AI" is explicit on the page right under the input box. The word "chat" doesn't anthropomorphize the situation.
And yes, some laws are "strict liability", I don't think this one is.
There was no intent for a human to read the message. By your logic, if she wrote a threat in a diary and a burglar broke in and read it, it would be a crime on her part.
Chats with a company's computer aren't private the way a diary is. A better example would be she intended to write it in a word document and instead accidentally sent it in an email to a random person.
If she had intended to write it in a word document on her computer but instead accidentally wrote it into her email client and sent it to a random person, I'm not convinced she would escape getting charged then either.
It's not any different than telling an automated phone voice tree system that you plan on killing someone and then being surprised that your words were later heard by a human. She absolutely told a company's computer. She sent the message.
The law may have been intended for more direct threats to a person as a means of intimidation, but that's a separate conversation.
The Florida statute requires that it be transmitted in a manner that can be viewed by another person. If you have no idea that someone could view your communications with a chatbot, did you really intend to break that specific law? Technically, that threat was communicated to another person but not through her own intentions.
A sibling comment includes an important rider to the provision: "...in any manner in which it may be viewed by another person." If you wrote this in a google doc, it almost certainly would not qualify as a threat under this statute. Even though google docs, like LLM chats, have administrative override and you could look at their contents - you would not expect either to be "viewed by another person."
IMO I do not think this is a grey area and it's legal to tell a LLM you want to kill someone. It's certainly not a "threat" like you might send to another person, though it may end up being evidence of conspiracy or premeditation. I suspect we would be well served to, after a few years of experience, put together some laws governing when LLM chats must be made available to authorities.
It is very interesting that the LLM responses to these lines - the context around what she is saying - is not in the article. I suspect, as is the case in many instances where LLMs are involved in violent planning, that the LLM was urging this behavior on. Basically entrapment - you are encouraged by a robot to become more violent and vindictive and then when you do you are handed over to police.
Yeah.. if you write a personal note and it's backed up by the operating system, it appears to be in violation of this law as well (since the company could theoretically read it)
People assume there won't be another human in the mix, but there is. She was judged for what she probably assumed was a private thought when it was actually not private.
Sharing them certainly can be under certain circumstances. I wouldn't be too surprised if this case gets tossed, but it's not inconceivable that prosecutors could win on the grounds that AI chats do not have a reasonable expectation of privacy and are therefor meet the requirements in the Florida statutes, or even more likely, they find some lesser charges for a plea bargain.
If you want technology that won't report you to the police for credible threats at they can easily detect perhaps create your own. I hear pen and paper work great for this.
Could this be considered free speech? Most of the limitations I know of require you to say it to or in front of other people as to create a panic or make someone feel threatened. Is it a crime to say to yourself "Im going to blow up New York." or about threats of self-harm? How about "Im going to steal the Mona Lisa!"?
To me this is just straight up thought crime, they just don't have a way to directly read your thoughts yet. But they will spy on you and try and catch you out for it.
It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person, when in such writing or record the person makes a threat to: (a) Kill or to do bodily harm to another person; or (b) Conduct a mass shooting or an act of terrorism.
To me, that is the more interesting legal question. Does a LLM-based safety net that sends content to a human, when the original use case would not have sent it to a human, count as "may be viewed by another person". It certainly wasn't intended to be, and that isn't the norm. At the same time, because no security is perfect, we could say that any digital record, stored in any way "may be viewed by another person."
The usual thing that makes laws against criminal conspiracy pass First Amendment muster is that the words have to be combined with some concrete acts furthering the criminal conspiracy. That might just be something as otherwise innocuous as looking up the blueprints of the bank you talked about robbing but it has to be something other than just talk.
So a written threat only becomes a crime when someone reads it, even if you never intended for anyone to read it? Is it a crime if I make a threatening statement in a diary and someone breaks into my house and reads the diary?
I suppose since Anthropic's T&Cs allow them to have a person read your chats, that makes it violate the law. Of course, if Anthropic didn't have that in their T&Cs, it wouldn't have been illegal to write.
Anthropic commits literal felonies by stealing millions of books and violating Copyright like it doesn't exist: no charge.
One unfortunate woman who happened to write the wrong thing in the wrong place is now having her life turned upside-down for perceived thought-crime.
To Anthropic, and all employees working there, your company's product and the result of your work is cruelty. You are enabling it and pushing it down everyone's throat. You can never again claim that you are the "ethical" AI company, for no such thing exists.
I guess we probably want our tech to work exactly like this.
It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
___
Of course, we also want purely private tech, but that needs a certain level of merit and sanity filter.
>I guess we probably want our tech to work exactly like this.
Do we, though? What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products. I don't think it's the tech companies' job to surveil the population and prevent crimes. That said, I'm not necessarily against Anthropic or other companies reporting suspicious activity if their existing systems are detecting it. I'm just not sure we want every product to be forced into that data model.
Your follow-up about purely private tech seems to contradict your first statement. We can either have privacy or surveillance, not both.
>What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products.
Why wait for a company to build it? Get your own local hardware like I did and have those guarantees because YOU set it up.
I don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.
Even if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.
How do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.
>How do I know that a private LLM system won't have a degradation of quality similar to this or worse?
You use benchmarks, you test, and because YOU'RE the sysadmin you know what weights are running at what time, it's very visible. You can airgap the hardware and be guaranteed it won't change over time. And, frankly, degradation over time doesn't seem to be what's happening with the open models.
Right now, I would be willing to pay ~$20 extra per month for strong privacy, assuming the same capabilities as Opus 5.5. I don't see local models making sense economically any time soon unless you value privacy at $1000's per month or are fine with much lower performance on hard tasks.
> Your follow-up about purely private tech seems to contradict your first statement.
That is correct! And exactly my point.
We want both, but, on paper, that is impossible.
But in reality, we make it sorta mostly happen anyway, through making the easy defaults not private, and the private stuff not easy.
This is not ideal, because [various reasons I do not need to tell you], but it has proven to be the best we can do to mostly achieve both goals.
Kinda like how capitalism isn't great but just the least worst option we've found so far.
___
The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff.
Hence the quadruple-speak and contradictions to kinda sorta somehow have a somewhat functioning reality.
Oh don't worry, I'm certainly already on various lists, but the observations also will have resulted in the assessment that I am stable and no threat :)
I think we can have both and it might even be smart.
A lot of people causing issues are people that can't make sense of many things (like many terrorists). They get a fixed idea and they end up doing something bad. You would catch those with some (basic) surveillance.
A lot of normal people (not wanting to cause issues) might benefit from some privacy, if they understand what are the trade-offs (like government overreach). They can then use a slightly more complex tech.
We would still remain with the couple intelligent but sociopaths (think Unabomber style), but I think no solution can fix all cases.
That sounds adjacent to "I have nothing to hide". Everyone says that until they realize someone can change the rules. Before 2022, women didn't have anything to hide from their period-tracker app, now some have to worry about being charged with crimes.
Back to LLM chats: A system that can declare her "unstable" is also one that can permaban you from all air-travel because you "privately" said unflattering things about Dear Leader.
> It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
The third highest voted comment wants surveillance. What is next? If you write "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported.
In a old happy little idealised village, it became known quickly, who started to behave oddly and timely intervention could happen. In the modern anonymous mass cities?
No one (wants to) notice the madmen scheming in his isolated flat, surrounded by strangers. Until he explodes.
Unfortunately I also don't trust our government agencies with the surveillance - because they ain't transparent either and the self surveillance seems broken.
" "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported."
And unfortunately there are lots of real threats being made under the disguise of humor. And much harder to separate im text. So maybe don't talk of murdering people in general, AI surveillance or not?
the honest actual opinion of the average consumer is probably that they don't want their chats to be monitored but they want everyone else's chats to be monitored. There was a lot of fury when a mass shooter recently used AI to help plan his assault. And of course there are all the people talking to chatbots about suicidal thoughts and intent. When they ultimately follow through, the providers are blamed for not alerting anyone.
No. This sentiment is why Snowden happened. We want privacy. Privacy isnt free just like freedom (whatever form it is) isnt free.
There are trade offs. ISP effectively is like driving on a highway, everyone can see where you are going but not what is inside the car. Id like these AI chats to be the same but they are not.
Devil's advocate: if I tell my therapist or my lawyer that I'm going to murder someone, they're obligated to report it. If I use my AI as a therapist or lawyer, why should the company that provides that service not be held to the same standard?
LLMs aren't email or file storage. AI labs aren't just shuttling bytes around, they're interpreting those bytes and taking action based on them. These models _already_ react viscerally in response to users saying disturbing things: the only practical difference is the ability (or obligation) for the model to escalate that concern. I'm not sure the ethics we hold AI companies to should be different than if a human being was typing out the responses.
Privacy is obviously hugely important, but this isn't the government surveiling every message. It's companies having an obligation to flag real, credible threats according to the law, which is a very different problem space.
>I guess we probably want our tech to work exactly like this.
Maybe you do; I want my tech to always include secure, encrypted communications. Don't include me in your destruction of privacy with your silly bandwagon!
uh no, 'we' absolutely do not want our tech to work exactly like this. Why would you assume that people default to 'search my anal cavity please' and not 'no, stay the fuck away from me with that glove'
----
jfc, why is this website full on psychopaths
"The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff."
other people are not pretending everyone is equal. your power levels are showing, it isn't subtle.
True, most normal people have little concern for any such humanist goals and ideas.
Normal people live prejudice. It's (erm, claude-speak) load-bearing for them, given just how complex reality is and given just how well it reduces that complexity.
Is this an invasion of their privacy (reporting to police)? Yes, but possibly warranted?
Should a social worker have contacted them rather than the police? Probably, if for no other reason than to ask if they were serious about harming someone.
Difficult questions, I'm still undecided on whether it's OK to always ignore someone's rants, even if it may be (or they think it may be) a private diary.
Actually charging them with a felony seems pretty quick to accuse. (Maybe I missed a hint about how long the investigation took before the felongy charge?)
It is my very european belief that the problem here is not that the woman was reported, but that what likely is a mental episode was made public in a way that reduces the chances of recovery.
Over in Europe they want to read all ofd our private messages, yet these chatbots, pretending to be our friends, will snitch on us just for our thoughts.
I once had a copy of 1984 in my checked baggage returning home to the USA and when I was unpacking the bag at home, the book had a notice inside the book that my bags had been inspected by TSA...
I reckon they were just checking for money or hidden compartments. Sending books abroad packed with money is extremely common from the US, though not sure how frequently people do that with onboard luggage.
Only tangential but when I was an undergraduate studying philosophy I had Bertrand Russell's Why I Am Not A Christian in my carry-on, and the TSA saw that and had a field day.
Detroit airport, 2010. I was chosen for secondary inspection and got some snide remarks of "we've been seeing too much of this lately" in reference to the recently failed underwear bombing attempt on a flight from Amsterdam to Detroit.
I can only assume you meant they had a field day celebrating how much of Russell's philosophy matched the concerns about Christianity expressed by Jefferson, Paine, etc?
They wouldn't see it until they opened it, so clearly they didn't open it because of the book. And if they need to open your suitcase (not because of the book), they have a reason for that.
I was chosen "randomly" for secondary inspection. This was after a failed bombing plot and the TSA implied a lack of Christianity was the proximate cause of the attempted attack.
In Europe they should provide the citizens with the service of their messages not reaching US servers. So basically that there is only one party reading along with them, not half the world.
I guess all the "private model" people are right. Don't want to end up in jail (or even charged with something) for asking a crazy hypothetical question or something.
Should the public have an expectation of total privacy for their chats? It seems responsible for a chat provider to report things like this.
If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime? Basically everyone agrees that crime-committing advice is inappropriate…but if it is not ok to get advice, that means there must be a portal for law enforcement to step in when that may have happened. Then the question becomes what is the line for when to report? In other words, the issue needs to be adjudicated.
But we don’t want OpenAI/Claude to have some $20/hour reviewer making decisions that are this high stakes…we need the courts to do the judicial work because they (1) have a public charter, (2) have meaningful expertise and specialization at interpreting the law and (3) we can hold them accountable.
It doesn't really matter if they "should or shouldn't" have an expectation of privacy IMO, they already do have that expectation.
> If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime?
Yes it should be, but it should be illegal for a company providing chat service to respond with anything other than a refusal when doing so.
That detection and refusal should be a private closed loop though, anonymizing any data that will be passed into a training pipeline, or ads targeting. This requirement for closed loop private chats should be mandated by law sooner than later. Otherwise we're getting into very tricky territory where the temptation of alerting on things like pre-crime grows too close.
For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
What would happen if it had scanned a file it didn’t have permission to look at and found this threat?
I honestly don’t know how I feel about this. On the one hand if you’re using claude as a diary you have no expectation of privacy and she was talking about committing a very serious crime.
> For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
For the sake of even more argument, imagine if she was writing her thoughts with a pencil, on a good old fashioned paper diary, and she had a phone nearby and the phone took a picture of her diary, OCR'd the words, and reported it to the police?
I can’t believe people are using these proprietary models/subscriptions as personal assistants, sending their most personal context, thoughts, documents and information to the providers. Some even give them full disk access.
After using their services for long enough, Anthropic/OpenAI essentially have your entire life mapped out to an insane level of detail, likely including detailed contents of your computer/phone.
People used to say that Facebook knew more about you than your closest friends/relatives. And compared to that, this is just on a completely different level.
Absolutely insane.
Not weighing in on the privacy issue but using Ollama you can run a smaller agent like Qwen 3.6 35b a3b on a sufficiently potent laptop. Pair it with something like Hermes for a nice interface and you have more than you might need for diary like usage.
I expect that anything I type / dictate / post / purchase on the web be it a chat conversation with an AI, a post on social media, a dm, a blog post, a blockchain reference, or an assett in a bucket, will be manually reviewed and forwarded to the authorities. It is their duty to do so, and I am glad that they do. If this woman did harm someone we would be hearing "why didn't Anthropic do something about this."
Aren't high trust societies staying that way by kicking out the incorrigible violators? Pondering violence might not be treated lightly even in a high-trust society.
Florida statute 836.10 puts the bar at the "sending, posting, or transmission of, a [...] record, in any manner in which it may be viewed by another person"
I think the defendant could successfully defend themselves by claiming they did not know (or intend!) the message could be viewed by another person, as they were plainly using it as a private diary.
This is practically entrapment. All the AI labs sure don’t shy from plastering annoying disclaimers everywhere saying their tool can make mistakes. Shame on them for not also reminding everyone continually that anything you submit can and will be used against you. Of course they can’t afford to have a Flock-style user revolt.
If it's going to be this binary. Then there has to be a shift in the way this is handled. It shouldn't be "You wrote these terrible things to a computer, you're now accused of doing them" to maybe "We where notified you wrote these terrible things, and the a psychiatric evaluation is being mandated by the state in which has a law about this sort of thing"
Obviously, as others have pointed out if they don't act and she does the crime it raises the damned if you do damned if you don't. But I've also had the AI's go haywire saying I'm doing all sorts of nefarious things when literally doing math proofs.
So no one size fits all. But going the extreme first is probably not ideal.
This kind of thing will get worse with humanoid robots because they have cameras and microphones. Will they be programmed to tell on you if you break any kind of rule in front of them because their owners are terrified of being sued?
this feels very thought-crimey to me, but I think I'd have to actually see that chats to really decide. Like is she making plans/asking for advice? is she just talking about her feelings exactly as if it's a diary?
She is not being charged with planning, just making threats in a place a person could read - the person being employees of the company.
Basically, under this interpretation, any personal note you store in the servers of a company could qualify, even if you didn't ever imagine someone would read and as such you couldn't have thought about it as a threat
LLM is not a person but T&C probably states that a human moderator may view any of it. Her lawyer could probably argue a moderator filtering usage isn't the intent of the law, it was more about publishing / sending message for other humans and it was never clear to her that a human was reviewing her private diary. Shouldn't LLM disclose that at some point when people are feeding really personal stuff?
duck.ai et al? my chatbot claims its architecture provides some level of privacy yet it is honest enough to point out none of these systems are actually audited and "not stored" and "anonymous" does not mean "not monitored".
I haven't seen anybody claiming this for consumer/free accounts anywhere.
You get privacy if you're a big corporation that needs to make sure OpenAI/Google/Anthropic can't read your trade secrets etc.
But those contractual privacy protections have been in place for a long time. It doesn't have anything to do with AI, it's been the same with Office365, Google Docs, etc.
They still read every single message you send and train on them.
I’ve had them email me before because I was testing it as a filter for abusive messages and they detected some no-no and wrongthink in those test messages.
I looked into ZDR, it's basically a vapid claim with little to no due diligence or auditing. I expect most providers to cave with only a minimal amount of legal pressure.
Does anyone else remember when the tech-savvy crowd was wary of software "phoning home" or spying on users, or listening, or whether Gmail scanned and read your emails, and whatnot?
And here we all are, happily typing our stuff into these spy chatbots. "AI" happily made our fears go away. Hopefully we're not planning anything criminal like this woman, but still.
After people getting pilloried for social media posts from twenty years ago, I really hope (sensible) people will have the wherewithal to think twice about what they hand over to their chatbots.
Pre-crime prevention and detention will be the end goal. The economic devastation caused by AI will make large swaths of people get angry. Better use that massive amounts of data being hovered up and the mass of compute being built up to stop those angry people lomg before they even get a chance to properly organize.
Snitching on the people - good mass surveillance company.
On the other hand, people need to learn to not trust these
companies. It reminds me of others being surprised when
a self-driving car reported a gun in the car. I mean,
do people not think? Besides, of course, it's already messed
up to want to have a gun. And it is constantly one country
that has such issues, more so than many other countries.
So now when everyone has their watch recording conversations, or perv glasses, or Alexas, or Jony Ives new personal AI gadget … recording everything we say and scanning it at scale … is it ok to be a doomer yet?
This reminds me, for his short story "Dolan's Cadillac", Stephen King claims -- perhaps exaggerating a bit -- he did the real research on how to murder a person as described in the story. How to dig the hole, what measures, what the falling rate of a car that size would be, etc, etc. He claims he fuzzied the details a bit because he didn't want to teach readers how to actually do it (again, maybe he's exaggerating a bit to promote his story).
Imagine if, instead of a friend, he had asked an AI chatbot. Would he have been reported to the police?
Aren't our private thoughts just that, private?
I get that a cloud-hosted AI chatbot is, to tech-savvy people, immediately "not private", but a lot of people don't understand this. What if Stephen King was talking to his friend about planning the murder, and his phone was listening? What if he was typing it up in MS Word and the program decided to phone home and report him to the police?
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Thought crimes are real when you're sharing your thoughts with Claude
I wonder if "criminal intent" may be missing here given that most people probably operate under the assumption that Anthropic are not reading their messages.
I think people have a binary understanding on this. Someone is either reading their messages or not. While the reality is that all the messages are read by a machine (not unlike GMail and Outlook) and anything suspicious gets flagged up so a human can read it. This obscure the concept of "reading" as most laypeople understand it.
Summary: don't type in Claude anything you wouldn't like a human to read.
It tells you exactly what it does with your information if you ask it, including this exact scenario, and has for at least four months now (when I asked).
Yes but you are someone (i assume because you are on HN) that at least understands this point. There are A LOT of people that use it as their confidant, expecting it to be private. There is a massive education issue going on as its not in the interest of these Corpos to make you fear sharing all your details with them. Because, then you wont get Dot or whatever Anthropic comes up with and share all your personal info with it.
You're not wrong, and it may come down to this in court, but there's a difference between what people think happens, or the reasonable expectations, and what actually happens, and I think it's important to recognise that difference and why it comes about.
I don't think someone is an idiot for thinking that the information they type into their private Claude account is private. I also don't think people are idiots for thinking their phone is listening to them and giving them targeted advertising based on that. Both are reasonable deductions from their lived experiences. Both are wrong.
I think it's a reasonable point to make. Writing things down is a part of "thought" for many, including those who keep diaries/journals. If you write in a private journal you do so with the expectation that is not shared, and that wouldn't seem to break this law (with my naive reading).
I mean, at this point it's pretty well known that all tech services will be hacked by fable, anthropic themselves promised it, so writing your journal in google docs or claude or a txt document on your laptop or such is the same as releasing it publicly yeah?
If it were written on paper, and only in a room with no phones or cameras so fable couldn't hack it, then I think you wouldn't be sharing it.
I think the zero-risk approach common nowadays is insanely corrosive to democracy and freedom. If a million people fantasise about shooting the sheriff, and one ever goes on to do it, I don't believe avoiding it warrants creating an apparatus of mass surveillance. After all, if people were really serious about zero murder, the only practical solution would be to lock up everyone. Some (most?) tradeoffs have exponential costs at the limit and we/lawmakers should recognise that.
> If a million people fantasise about shooting the sheriff
I think it’s fair to pre-identify folks who fantasise about shooting anyone. It’s a small fraction of the population that looks into logistics versus making offhand comments.
Reverse the situation and the media would also turn that into outrage. Imagine someone shoots up a sheriff's office and then it turns out they declared it to some chatbot, and the AI company failed to detect and report it, and "didn't push back enough" whatever that would mean, and hence the AI was implicitly complicit etc. That would also be a major PR catastrophe.
Damned if you do, damned if you don't. Same as with social media platforms. That's because only a tiny tiny sliver is for true privacy when that means "bad things might happen" or bad people, such as your political enemies, may do stuff you don't want.
>Damned if you do, damned if you don't.
In my opinion, if the company is allowed to see the data and train on it, then they are also responsible for reporting stuff like this. Without knowing the data licensing agreement the lady had with Anthropic, if she agreed to letting Anthropic see her data, then they should do stuff like this. If she didn't agree, then I wouldn't condemn Anthropic for failing to report an attack
With this logic, ISPs should be monitoring all texts and internet traffic. Might as well extend an open line to the gov’t
if they keep a tap on you, how long before that data gets used for ads or sold to an ad agency?
[delayed]
Pool together with some friends and buy an H200 or two to run unquantized open source models with abliteration/heretic transformations.
You need to be able to use these models for the real world and not for some imaginary world where everything is safe and nice and happy all the time, while at the same time intensely surveilled in the name of CYA and the latest panic about whether speech THAT ISN'T EVEN BETWEEN TWO PARTIES is considered "wrong".
I'm a free speech fan that acknowledges there are lots of boundaries of free speech (fraud, perjury, blackmail, defamation), but the one thing that all of the boundaries have in common is that a second party must be involved for them to make any sense at all.
Maybe the courts will uphold this, maybe they won't, but don't take the risk!
It’s been quite exciting that Qwen 3.8 Flash Next has come out: it really is similar to Opus 4.5, 4.6 for coding. Remarkably intelligent, and runs on a single DGX Spark, which I paid $4000 USD for
Sadly they’re now twice that price, which is a shame because I really want a second one!
I have a heretic modified version of it too, for when I want to use it for security and so on. Quite interesting
Same here. I was planning to get a second, just got this one ten seconds before they became unobtanium. Here’s hoping Qwen 4 Flash is the same size!
Gotta find some rich friends in order to buy and host a H200 or two.
I’m excited for the refurb market in 7 years.
in 7 years you'll be soylent green as you're used to fuel the answer to 42
A middle road is to use open-weight hosting providers, maybe non-US ones if you’re in the US.
Yes otherwise thoughts are crimes, effectively.
Or worse, misaligned with the Terms of Service.
I agree up to a certain point, but there has to be some legal boundary between freedom of thought/speech and literally planning a crime. I'm protected under the First Amendment to say "someday I'll rob a bank" but not necessarily "I'll rob this bank on Friday and here's how I plan to do it".
I think you should be allowed to write that exact line in your journal. If you rob the bank that can be used as evidence against you, but in no way is it acceptable for private reflections alone to be used to arrest you. Or else every author who's written a novel with 'bad' characters would be arrestable.
I'm coming to saulpw's house tomorrow at noon with a gun, and I'm gonna make them pay.
If a policeman notices the sentence above on my phone screen during a routine traffic stop, the response you want him to take is... nothing?
For the record, under current US law, it is not illegal to have a sentence in your locally-stored notes on your phone outlining a plan to commit a crime. There has to be an overt act. The police in that instance could inform the intended victim, surveil you, etc, but they would not be able to successfully charge you with attempted murder. It's not illegal to be considering committing a crime, even if you have a tendency to write down your thoughts.
The law in this particular case, which seems to be intended for threats that you actually send to someone, is being interpreted broadly to apply to any "threat" that you transmit to a server. So in your hypothetical, the legality would depend on whether your notes are backed up to icloud or not.
>the legality would depend on whether your notes are backed up to icloud or not
I agree, and it's nuts.
This feels like less of an issue with anthropic per say as it is a broad reading/misuse of the law's original intent.
This is true for the general criminal conspiracy law, but be aware that an “overt act” doesn’t have to be an illegal act, just some action in furtherance of that crime. That can be purchasing a weapon, or scouting out a location. There may also be other laws in play depending on the specifics
Just having saulpw's physical address is already furtherance of that possible crime.
The commenters here are cute little HNers who think they have found a loophole in the law. They are not the first ones innover their head.
Spoiler: the law is written in words, and those words aren't strictly executed like in a computer program, they are interpreted by actual humans who can see what you are trying to do and will stamp it out.
You're absolutely right, at least according to my own quick check on Gemini. I find this state of affairs amazing.
In my country, no "overt act" is required, but both here and in the US a "conspiracy to commit" charge requires an agreement with a second party. This is indeed consistent with a very broad interpretation of "no thought crimes".
Correct. Nevermind how ridiculously contrived the scenario you've just concocted is.
https://www.youtube.com/watch?v=14WE3A0PwVs
its not contrived just because you recognize how silly the argument its arguing against is
> If a policeman notices the sentence above on my phone screen during a routine traffic stop, the response you want him to take is... nothing?
Anybody showing a cop their hacker news comments at a traffic stop should be arrested, for harassing the police
Do you really think reading a snippet like that completely out of context should qualify as probable cause?
Yes!
When you read something describing in detail a person's intent to do something very bad, in a place where they write things that they intend to do, and which in the past they have in fact consistently done, you don't attach any significance to that at all?
I certainly think a police officer that stumbled across such a thing would be justified in asking follow-up questions like 'hey, what's this about you going to shoot someone?' and then maybe making an arrest based on your replies/demeanor.
'Probable cause' should involve a degree of certainty, because 'possible cause' would be altogether too loose of a standard. It's possible that you're intending to shootme and you just mentioned saulpw to throw other HN users off the scent. Possibilities are only limited by the assessor's imagination.
How can you prove that it's not just me writing a dark and morbid story?
How does the cop at the traffic stop know that your username is akoboldfrying? Did you tell them? If so, why, at a traffic stop, did you do that?
Are you posting threats on hacker news while you are driving? And the cop was close enough to see your username and what you wrote? Is that why you were pulled over?
Well in your example you've begun conspiring with a second party so that's not at all the same thing. You are at least free to plan all the crimes you'd like to arbitrary levels of detail in private. It's when you start acting things out (soliciting coconspirators, blackmailing targets, etc) that you cross the legal line.
The current situation is a weird one. Anthropic reported single party interactions (per the ToS and common sense), there's a statue about sending threats (as there clearly ought to be), then somehow the definition of the word "send" was tortured by the local police. If a crime has been committed here it's almost certainly an infraction by the local authority against the spirit of the law.
I totally agree. People seem to be stuck on the notion that we must not punish thought crimes, and have elevated this above all other considerations, when really it's just one among several.
However, those other respondents to your post seem to be accurately describing the current legal situation. I asked Gemini, and apparently "conspiring" to commit an offense requires an agreement with another person in both my country and the US, where an "overt act" is also required (that may not be incriminating by itself). I find this alarming. The fact that someone's private diary entry describing in detail a plot to kill me does not amount by itself to anything is... incredible to me.
You just said the words right here in this public vbenue, not even in private.
This argument holds no water at all.
Obviously, I'm not talking about the verbatim quotes I provided. There has to be some level of evidence that proves intent to commit a crime and the second quote is meant to represent that whole class of statements, but it depends on context. Any given quote won't constitute evidence in every case, but it will in the cases where it proves intent beyond a reasonable doubt.
I'm not sure why you think my argument holds no water when there are clear legal precedents that speech is not protected in some cases where there is "imminent lawless action".
https://en.wikipedia.org/wiki/Brandenburg_v._Ohio
I'm not a lawyer so take this with a grain of salt, but it seems like there is a big gap between speech that is "directed to inciting or producing imminent lawless action" and just saying that you will do something.
Depending on context saying "I'm going to rob the bank X tomorrow" might also count as a threat?
But, in either case, writing this in a private diary could not be incitement or a threat because you are not communicating with anybody except yourself.
action in the real world, i.e., not just words.
This is not even an option in the UK. Communication felonies (dangerous speech, threatening behaviour) only demand potential audience.
You can trust another party and do this by renting a few H200s. You cannot pool together with some friends without trusting another party.
You end up just weighing up the difference in trust between a vendor and a friend against the level of disinterest that they might have in your affairs.
You don't need an H200 for that kind of use case.
A cheap second hand 10ish year old card like my radeon rx570 with 8GB of ram is plenty enough to run a small uncensored model with llama.cpp if all one wants is chitchatting with a clanker.
We are not talking about heavy coding use cases here.
Would be better if your friend is lawyer and take $1/yr(or higher) for the service so that attorney-client privilege can be applied.
edit: ah, future crime cannot be protected.
Preach! I am so tired of Anthropic’s safety team being the arbiters of what is right and wrong. If you so much as hint at impropriety you can have future sessions flagged ad infinitum when it comes to specific topics.
That's definitely something I'd consider if I had cash to spare for H200's! Unfortunately I think for most of us the price of self-hosting has to be 2-5x lower still.
Way ahead of you bud, mac studio m5 ultra 256gb version is coming soon.... ordered 2x of em just in case. expensive as fuck but its a hedge against all this bullshit and more
Lead time is like a year tho no?
I agree, but someone will say fake child porn.
I'm not saying fake child porn should be allowed or not-allowed, just showing there exist possible exceptions and rationalizations for them even without two parties.
I have some sympathy for Anthropic here because I've seen the headlines after OpenAI failed to report a shooter in a similar situation. So from their perspective, it's damned-if-you-don't, damned-if-you-do.
However, people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech. Before LLMs, Big Tech had no way to scrutinize the bulk of what was going on within their services, so you could have a secret hate diary in Google Docs. Now, everything you say or write can be automatically screened for red flags on a planetary scale, and probably will be because that's what the regulators and "concerned citizens" will demand. In a couple of years, you'll be biting your tongue a lot more often in private chats.
I'm still puzzled that people's default assumption isn't that somebody is reading all of their internet communications. Ever since the Snowden revelations of 2013 I've more or less assumed that everything I type into a computer is stored in a government database somewhere. Not that I actually believe it is 100% of the time; there's a spectrum of trust, so I'm more confident that local apps on my Linux desktop are secure, somewhat confident in the end-to-end encryption of certain apps on my iPhone, but all bets are off for non-E2E encrypted data going across the internet.
People don’t want to believe the Stasi is monitoring them. They want to believe the world is a nice place just like where they grew up.
Also, unlike the bad old days when 1 in 3 was an informant, now ordinary people aren’t in “informant loop” of providing information on others, so they aren’t thinking about being informed on either.
If we completely give up on assuming privacy, we lose even more privacy, because so much is based on the expectations of a "reasonable person".
For technical people, this is incredibly old news.
For non-technical people, it isn't really news, because they already forgot about it after reading it. Maybe they'll be a little more monitored in their own typing for like... a day or two.
One of the hardest lessons to internalize, and keep internalized, as someone who works on and writes software, is the vast, vast, vast majority of the Public doesn't understand even the most basic shit about software. It just does stuff. Hopefully the stuff is good. That's it, beginning, middle, and end.
"Why would you think x would y" is a poor framing. They didn't think about x or y because they don't care. The phone works, that's the beginning and end of their interest in the subject.
It isn’t that black and white. I have relatives that are highly non-technical but still care a lot about privacy and data protection in their computer use. They hate having to deal with technical things and have very little technical understanding, but still ask me whether using X software in Y way might pose privacy issues or not.
You're completely right. I've internalized this on the technical side, but I still find it puzzling politically. Surveillance has been a salient issue in U.S. politics for almost 100 years now, since the invention of the telephone. I would like to think that most people are at least vaguely aware of Watergate, the Patriot Act, and Snowden. Then again, I should probably stop assuming that people know basic history, given the current state of education.
Honestly the older I get and the more I learn about criminals, of all stripes really be they petty thieves all the way up to state actors behind the most atrocious crimes against humanity we know of, they are all a bit on the dumb side. The ones that get caught are, anyway. Like so very often the perpetrator of a given crime just did one incredibly bone-headed thing that brought the cops directly to their door.
I think in part it's selection bias? Like if you're smart enough to get by honestly, you're probably also smart enough to realize getting by honestly is just a way more comfortable way to live. The only reason you'd probably cross that line is because your principals, whatever they may be, conflict with those laws, or your life circumstances are so bad that you have no choice BUT to turn to crime.
And that cuts the other way too: if you're dumb enough to think you'd NEVER get caught for a burglary, for example, you'd probably be way more down to plan and execute one, failing to consider that most thieves aren't caught when they steal the shit, they're caught when they try and sell it later.
Not just bad headlines; OpenAI is actively being sued [1] for this:
[1] https://cdn.arstechnica.net/wp-content/uploads/2026/09/Briti...This is one of the reasons why AI companies are looking for explicit regulations, it can help to reduce the risk of possible liability and maybe make the legal way forward more tractable. With a regulatory framework in place much of the burden of identifying risks falls on the regulatory authority.
Then the AI companies have more confidence that they can move forward in a certain way, and issue investor guidance that is maybe closer to reality.
I think they're mostly looking for regulations because they've spent close to $2T, all to realized they have no technical moat, so they're trying to build a regulatory one.
Both things can be true.
They want stable rules they can follow, which will limit their liability as long as they stay within them.
They also would like those rules to be as restrictive as possible towards their competitors.
They've spent close to $2T so far. At that scale the legal issues they've had are just the cost of doing business.
I'm quite sure that if there wasn't the existential threat of a lack of a moat, they would not be pushing for regulations at all.
The legal issues can sink their entire business model, not quite the cost of doing business.
A lot of things could sink their business model including opening the can of worms of the wrong regulations.
That's the conspiracy theory, but there's no good evidence for it, and it doesn't really make sense in the long-run.
It's the only thing that makes sense for a lot of these companies to survive to any long term when open models keep nipping at their heels for pennies on the dollar.
You're assuming the open models will follow regulations though.
No, I'm assuming that the open models won't/can't and will be banned from a lot tons of use cases that will mandate use of the large frontier shops in order to comply with said regulations.
The theory is that open models cannot follow regulations and will therefore be banned or not eligible for many of the large contracts that the frontier labs will win.
No good evidence other than their openly stated desire for that regulation? Dario and Sam have both made open pleas for it, repeatedly.
Liability is enough of a reason
> people need to get it in their heads that they're not chatting with their secret BFF
I see constant ads on video platform (particularly youtube/tiktok) about llm chat apps, from friends, dating, romance and everythkng inbetween; that's personal.
People need to be reminded constantly if they use such apps that they are participating in easier mass surveillance, profiling and AI training.
How do you get that through to someone who doesn't even understand that mass surveillance and profiling is a problem? Or to young people who have only lived in a society of mass surveillance?
Problem is that even not using those apps, the apps you currently use might have turned more hostile.
It wasn't technically feasible to scan personal chats easily, other than grepping keywords which must have had a bajillion false positives. Now you can get everything autoscanned at scale.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
Yup. And with zero privacy protections in statute for AI chat, there is nothing to prevent an AI CEO looking to curry political favour from e.g. handing over the private correspondence of an opponent or an entire district’s residents.
Customers who what privacy protections for AI chats are welcome to negotiate this in enterprise contracts. The major LLM vendors do offer that as an option. Customers can then enforce any violations in civil court (although this obviously wouldn't apply if the customer used the LLM for criminal purposes).
> welcome to negotiate this in enterprise contracts
I really hope you mean that in an insulting way to the industry and current legal situation, not as an actual solution.
Where's the insult? Lots of customers have such agreements.
this thread is about ordinary people using chatbots for personal use, not enterprises for work
Or something like leaking every non-corporate ChatGPT user's chat logs (including temporary chats) to the NY Times.
Or making up stuff
Don't know why this got downvoted? This is a problem with epistemics.
It's more efficient to have one central "verifier" for everything, but the "who watches the watchers"? question basically says: Either constrain by construction, have everyone verify (which are two sides of the same coin, btw, when looking at a "global" thing), or centralize explicitly.
And such was the case for a man who snapped a photo of his own child to send to the doctor which got uploaded to his Google photos resulting in his Google account of over a decade getting shutdown for CSAM.
As another commenter said, you're not chatting with a friend; you're chatting with Big Tech.
How much do you love Big Brother?
Wasn't this because the photo was made "public" as in, shared through Google Photos?
If the doctor was not using a gmail account, the UI probably recomended to share it "with anyone that has the link" that is like public but protected by oscurity.
Most people don't realize that it is 99% like posting it on Facebook.
It isn't like posting it on Facebook. It is like emailing it, because most people don't have the link.
I can't read past what the paywall allows, but here's the nyt article: https://www.nytimes.com/2022/08/21/technology/google-surveil...
A disclaimer on the top of page every time would have been a better approach helping Anthropic and the end user.
A bot talking to you directly as if its some one real caters to your thoughts and can take you in a certain direction without you realizing it. I have heard first hand experience from people that they feel more comfortable talking to chatgpt or claude cause it gives a feeling of being on their side and listening to them.
I suspect it will go further: imagine giving an mp3 to LLM to clean up some noise. It detects it was illegally downloaded from youtube, deletes it and automatically fines you via attached credit card.
Perhaps but no indication of that so far. Agents are happy to set up *arr stacks today.
They could be happy to assist you with a task that they also then flag and forward to authorities, especially if their assistance doesn't break the law but has evidence of you doing so.
Sure, and they could be a front for space aliens choosing who to abduct. Anything is possible!
I'll be glad for open models when the day (inevitably) comes that the proprietary LLMs no longer work in my interests.
> LLMs no longer work in my interests
As articles like these show, cloud-based LLMs don't work in your interest today.
Is it in someone's best interest to let them die in a shootout?
Unless it becomes a requirement to be licensed to be able to use any kind of ai model. You know, for safety and stuff. And of course with appropriate reporting to institutions.
That day was yesterday. LLMs from big tech regularly refuse to do what you want.
Wouldn't they have to fine themselves first?
They were fined for their illegal downloading. More than a credit card limit.
And dramatically lower than their expected value from the copyrighted material they scraped
Or worse: downloading a picture of pirate ship and without any concern for the copyright asking the LLM to make a coloring page for your kid. BTW Chatgpt does that way better than Claude
Hey but you’ll be allowed to file a response that will also go to an LLM and deny you automatically. And you will be charge a NSE fee (no sufficient explanation).
You should probably get a head start on waiting a couple years to bite your tongue and assume everything you type into a computer is summarized and sent to your boss, government, advertisers, political actors, insurance companies, worst enemy, etc. With phones, Alexas, and little AI tamagotchis, you probably shouldn't say much in person either.
The Silent Generation, version 2.
They're actively rummaging through your inputs so the damned-if-you-don't case doesn't really exist; no one expects Anthropic to not notify law enforcement once they learn of something like this. What you might have expected was some privacy in the first place though, where Anthropic would never have learned of this in the first place and where the damned-if-you-do case wasn't a thing.
> people need to get it in their heads that they're not chatting with their secret BFF, they're chatting with Big Tech
It's the other way around, big techs need to properly disclose in their platform, during interaction that they aren't in a private and safe environment
Right, the difficulty is partly that they can get a negative headline from any choice of behavior.
"Anthropic failed to report murderer's threats to authorities"
(or "Chatbot knew man was planning murder, yet company did nothing")
"Anthropic reported private chats to authorities"
(or "Arrested for chatbot fantasy")
To be fair to the journalists in these cases, there's also no society-wide agreed Schelling point about the correct outcome or correct rules. I have strong beliefs and intuitions about what should happen, but other people also have strong beliefs and intuitions, and many of those are probably opposite of mine. Even if my intuitions are the best and most justified, a journalist is unlikely to think "I'm just not going to mention that some people are mad at this company over this outcome, because a hypothetically better norm or principle would support the company's actions here". Hopefully the journalism can at least contextualize the lack of legal or social consensus and the difficult incentive problems, rather than jumping to "obviously companies are sociopaths staffed by supervillains".
> in private chats.
Including any chats anywhere where someone might have a phone in their pocket, or if there's a "camera" attached to a utility pole or a nearby tree. The only real private chats might be whispered lying down in the bathtub together, with a mattress covering it like you're both hiding from a hurricane.
> they're chatting with Big Tech
They're chatting with any powerful person who wants to hear it. She thought she was chatting with Anthropic, who doesn't give a shit about her. But after being threatened (and immediately backing down because, of course, they don't give a shit about her) Anthropic has become an arm of the government. So she was chatting with the Bonita Springs, FL Sheriff's office, or anybody else. If I paid enough, Anthropic would tell me about what she was doing so I could sell her laundry detergent.
it was in the article
It's a byproduct of the nannyism safety marketing from the AI companies. I'm glad these cases were caught, but disagree with how they were disposed of. If the automated flagging is good, enforce it by default. If it's noisy, refine the tech then enforce it by default. This middleground where everything going through the platforms is subject to training and arbitrary human inspection in the midst of an acrid cloud of marketing-driven fearmongering is unacceptable, and it reinforces the idea the fearmongering is legitimate.
Somehow humanity survived the past 40 years without Microsoft Word and Excel phoning home and shopping users to the feds at random, I don't see why the standard should be any different for this new class of tooling.
As if it is just nannyism marketing by tech companies. The EU tries to bring a new nanny law into legislation every other month
Absolutely no sympathy. Anthropic is every bit as slimy as any other big corp. And like other big corps, they must open the vault to whatever governments they intent to do biz with.
We really need a class, probably in high school, that works through how LLMs work at the high level (don't need to get too far into the deep math, but give people a taste) and then how they're trained, used, and deployed.
I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There'd also be fewer people falling for apocalyptic Rationalist delusions.
Also: people need to understand "not your computer, not your data." (Unless it's stored in the cloud but encrypted locally with keys only you possess.) Same goes for storing things unencrypted in OneDrive, Google Drive, etc. There is nothing to stop these companies from bulk scanning, data mining, or reporting people based on whatever request a government gives them. Don't count on them to resist, because they often can't, especially if the request is from a sovereign state where they do business.
> There'd also be fewer people falling for apocalyptic Rationalist delusions.
Um, this is coming from the researchers building these models. So, good luck?
You can make a reasonable case for adding a lot more classes in high school: statistics, nutrition, personal finance, etc. But ultimately it's a zero-sum game and to add a new class means removing an existing class. So what do we cut?
My kids' high school covers those topics. And I think that is not a new thing, I specifically remember taking the required personal finance class 35 years ago (great teacher -- crotchety old man who wrote "compound interest" on the blackboard every single day before class and repeatedly proclaimed that if we forgot everything else he taught, please try to remember how compound interest works).
>I feel like if people understood what these things actually are there'd be way less of this AI psychosis and similar stuff.
There are people with education here that don't fare much better, so I don't know.
There'd also be fewer people falling for apocalyptic Rationalist delusions
Assuming you consider it a "delusion" to have a p(doom) of more than 5% or so, that's not uncommon among frontier lab employees who have a pretty good idea of how LLMs work.
Indeed, I don't see how knowing the details of how LLMs work (which I know btw) would change anything about how intelligent they are. I only need to know that it's a computer program that writes stories, solves math problems and seduces people.
> I have some sympathy for Anthropic here [...] So from their perspective, it's damned-if-you-don't, damned-if-you-do.
On the other hand, they did put themselves into this position deliberately.
By offering a product?
Show me any product, no matter how simple, that has no safety vs utility tradeoff.
They have some responsibility for people's expectations of the product, at least. They want the personal assistant personas to be able to help you with anything, and don't point out that they'll be judging your thoughts along the way.
For anyone technically inclined it should be obvious, but it isn't part of the zeitgeist or how they pitch it. People see it as being different than talking to a human, and behave as if there won't be a human in the mix.
that tells users upfront "we can read whatever you type in here"
(and not buried on paragraph 56 of a 20 page TOS that no one reads or understands)
You would first have to demonstrate the utility of whatever product Anthropic is offering.
> Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
I think Anthropic did the right thing here; but the sheriff's office are probably demonstrating why she dislikes them. Writing a diary entry to a chatbot is clearly not how this law was intended to be used.
EDIT: Actually, on reflection, making this report to the people she was upset about was probably not the right call. If they'd sent it to the FBI, there'd be a much lower chance that someone felt the need to assert their "authority".
>> I think Anthropic did the right thing here
This is the paradoxical times we live in right now.
Don't do something? She walks into the office and start shooting the place up. Several officers and innocent people are killed. Cue the media claiming, "You should've known she was talking about this an AI bot! Why didn't the bot tell anybody she was planning a mass shooting?!"
Do something? She gets rolled up by the cops and questioned about what she was talking about and brought to the cop station and interviewed. Cue the media claiming, "This is an unethical way to use AI, this is an infringement on free speech! This is authoritarian!"
I believe in free speech as much as the next person. But in this day and age, its almost better to be safe than to have to explain to someone's loved ones you had to chance to prevent this and did nothing.
> This is the paradoxical times we live in right now.
It's always been complicated like this. That's why certain professions (psych, lawyer, clergy) come with rules around when and if disclosure is allowed[ required, and/or admissible].
They’d be in an easier position if they built the system such that it was impossible for them to know what people are writing. They might catch a little flack from people who want them to surveil all their customers, but by and large people seem to accept “we take technological measures to ensure privacy and that means we can’t spot crimes.”
So this conundrum is at least partly of their own making.
Mmm.
As a Brit, I'm aware of https://en.wikipedia.org/wiki/Twitter_joke_trial
I can't say I actually disagree with the initial prosecution. The penalty was a fine, likely less than the cost of investigating it.
Intended as a joke? Blowing off steam? I can understand that, but given the number of people on social media is large enough to include genuinely unhinged people, you can't expect anyone who receives such as message to take them as a joke.
Same with AI use. A billion users, you have to assume some of them are actually sincere if they write about any act of violence, from self-harm to a plan to steal a nuke and use it in a false-flag attack to trigger WW3 and everything between.
> you can't expect anyone who receives such as message to take them as a joke
That's a distinction that matters to me. Sending a spicy note to an LLM isn't remotely the same thing as posting it on social media where the entire world can read it.
I kinda agree, but on the other hand anything you send to an LLM has to be viewed through the lens that only an automated system (i.e. an LLM) is capable of even handling such a tsunami of natural language.
It absolutely will misclassify things, it doesn't know any better.
(We should all wish each other good luck, because we're going to need it).
It feels icky, and my default is not to side with megacorps engaged in blanket surveillance, but I can’t really fault Anthropic here. The correct setup should be a law that protects this sort of interaction with a chat bot as privileged, along the lines of HIPAA-mode. Use a classifier or some sort of “private mode” toggle to tell the platform you’re engaging in privileged communication ala dear diary, and then a much higher legal standard needs to apply to protecting that data (no training, same protections as doctor / psychologist interactions). Even a therapist has a legal duty-to-report in certain situations.
I’m going to guess that planning a mass shooting clears any hurdle that would be in place here.
While the privacy around ai chatbots is rotten in general, I can’t fault anyone who reported this.
Maybe not one that reported it, but the ‘we scan every message you sent into this for disallowed speech’ part is what gives me pause.
They shouldn’t be in a position to report it at all.
Is it really "blanket surveillance" when it only sees exactly what you put into it? That's like saying you're being filmed against your will while... filming yourself.
I get where you’re coming from, but I think if the progression of technology has shown us anything, it’s that there will be constant competitive and social pressures being applied to use these tools more and more and that pressure will race far ahead of any privacy, consumer protections, public education and societal wellbeing counter forces. So opt-out ends up being a rapidly shrinking iceberg in practice.
"slippery slope" is an old phrase for a reason.
Counterpoint, if you give the agent access to your files - which happens to include the Notepad diary you've been writing since your teens - and it makes a similar conclusion about something you privately wrote, would have the same opinion?
Yes, and it's weird that on a form of technologists we're going with the idea that all of these systems we use every day to hold our personal private information that we are constantly ensured is secured against anyone unauthorized from accessing it is actually snitching to the cops just because someone else is hosting it on our behalf.
The 3rd Party Doctrine destroyed the 4th amendment and is the reason privacy respecting software has to play legal games. E2EE while a good security practice shouldn't be necessary to protect you against the cops rummaging around your stuff. The bar to establish that information is private shouldn't be "literally mathematically inaccessible but the cops are still allowed to try."
Would you use an AI service knowing that they are inclined to turn you in to police if they detect illegal activity?
In a "three felonies a day" universe?
Yes.
But that service is a process on MY computer, loaded into RAM, with an abliterated model.
I don't trust 3rd party networks from abusing any data I give them.
> Would you use an AI service knowing that they are inclined to turn you in to police if they detect illegal activity?
The appropriate question is whether I want to live in Florida. This is much more a Florida law problem than an AI company problem.
It would apply if you happened to use Office 365 to write your diary.
Sadly, this isn't a Florida problem.
I don’t live in the US. But this kind of broad law is hard to implement without surveilling all users, and it has multiple side effects.
What happens if I use Claude or ChatGPT to research sensitive social topics? Would that be considered a social network interaction and used against me when I apply for a visa?
Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
> What happens if I use Claude or ChatGPT to research sensitive social topics?
Anthropic/OpenAI is not obligated to report your use, as long as it's not violating some terribly written law (like the Florida law). The government won't know about it, so no, you will not be denied a visa.
> Many governments (especially in Latin America) copy what the US does, meaning that similar laws will be pushed sooner or later.
I honestly don't know, but I suspect most US states don't have such an overreaching law.
Legally they have to.
Would you use a lawyer knowing they are inclined to turn you into the police if you talk about committing a crime in the future?
The law is the law.
We should be happy about this as for once the AI companies did the right thing.
This is slippery. Many people do all of their journaling inside of Google or Apple cloud products. Some even write up their intentions to do bad things.
Does all writing now have to be scanned for thought crime?
That's an interesting wrinkle that is rather tough to work through.
To me, journaling your intent in a private journal, whether that's an Apple Journal/Note or a Moleskine in your drawer, feels qualitatively different in some way. But I'm not sure why.
So "I'm going to shoot up the police station" written in your own journal feels somewhat different than "I'm going to shoot up the police station" said to a system that might be able to _interpret_ or _act_ on what was said in some way. Did I just give AI a legal duty, or even a soul I didn't think it had before? I've written up about three or four "what about this, what about that" and deleted them all.
You omitted the part in the diary entry about shooting up sheriff’s office
You're missing the part where it's a diary entry, so the actual content is irrelevant. Her only mistake was not realizing that her diary wasn't private.
Because it's not relevant. They're not charging her with conspiracy, which is what they would do if she'd actually done anything concrete towards making that happen. She didn't email it, or text it, or post it on Facebook or Twitter or Discord or a message board, which is what this law is clearly about.
How can you charge someone for making a threat when you only read the threat by spying on them? Surely that has to be thrown out in court? They didn’t actually send the threat to anyone, you just obtained it by spying.
> when you only read the threat by spying on them
The AI companies have clauses in their user agreements saying they can review content flagged as harmful. It’s not legally spying.
If you recall previous outrage about ChatGPT being used in cases of suicides or shootings, this is the result. Every time a crime was committed and the police found ChatGPT history about the crime, the media turned it into a frenzy. So the AI labs added safety filters to their consumer plans that detect threats of violence, escalate them to human review, and report to the police.
Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department. There might be some limitation in the law that makes the evidence inadmissible because it was not intended to be shared with anyone, but that’s a separate decision.
Not only that they can review flagged content, but they tend to have separate retention policies for flagged content. Anthropic's is this: "We retain inputs and outputs for up to 2 years and trust and safety classification scores for up to 7 years if your chat or session is flagged by our automated trust and safety systems as violating our Usage Policy."
So don't run for office or anything like that. Someone, somewhere will have a contact that will get that.
>Spying is not the right analogy because the information was given to the police by a third party which had a EULA saying they would do this. A more analogous situation would be someone reading another person’s diary and then turning it into the police department.
This is spying with extra steps couched in corporate speak.
I was responding to a question about the legal case. The police did not perform any spying.
Frustrations about Anthropic’s EULA are a separate matter.
Was it claimed that the police did any spying?
Presumably, Anthropic did the spying and the reporting.
You argued that it is not spying, since the spying may have been made sufficiently explicit in the ToS/EULA.
This raises the question: Does announcing a spying operation mean that it is no longer spying? I've never heard that perspective before.
> Does announcing a spying operation mean that it is no longer spying?
Well, kind of, yeah; the dictionary definition of spying requires secrecy and lack of consent.
> to secretly collect and report information about the activities of another country or organization[0]
The only real debate is whether or not having a clause tucked away in a EULA that few people read makes it a secret. If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
[0] https://dictionary.cambridge.org/dictionary/english/spying
I would call it spying in this sense at a minimum if individual people don't know whether their conversations were stored or disclosed in a way they don't want. For example, suppose someone said "we will monitor the activities of 10% of people". You don't know if you're in that 10% or not, but I would still want to call that spying.
A less central case would be when you clearly do know about the activity but you can't quite see the details, like with behavioral ad targeting or something. It feels pretty normal to me to call that spying even if it's disclosed to everyone and certainly happens to everyone, but it's also a less central example of the concept.
Anthropic could put a big flashing warning text at the top of every chat that says “We are spying on you and will report anything scary to the police!” and it would not make any difference in this case.
You can call it anything you like, but only the legal definitions matter for the legal case.
After working on several court cases about surveillance activities, I'm definitely aware that whether I call something spying or not has little relationship to whether courts will think it's legal.
> If Anthropic had a big flashing red banner that said "FYI we automatically flag and review any conversations about illegal things!!" on the front page nobody would call it spying.
If you change the situation then yes you can in fact change our responses. The problem is you then are no longer talking about the original situation.
It also bears mentioning that providing a dictionary link to “spying” is pretty patronizing/passive aggressive. On par with sending a basic Wikipedia page. You didn’t even bother to post the definition you want to apply.
No one claimed any case would be "thrown out for spying." The legal definition of spying is also not particularly relevant to the argument in the initial comment.
The initial comment instead questioned how someone could be accused of making a threat if they did not realize anyone would read their private content. You probably also can not insult someone with a statement you never expected anyone but you will ever read.
> Presumably, Anthropic did the spying and the reporting.
You don’t need to presume. Anthropic reported it.
“Spying” as a legal concept has a definition that does not apply here. You could say they were “spying” in the sense that they read someone’s input, but that’s literally what they said they were going to do in the agreement when the person signed up.
So I responded to the question about the case being thrown out for “spying” by trying to show that the word doesn’t apply in the legal sense. If you sign up for a service that says “Hey we’re going to monitor your chats and might report things to the authorities” and then they monitor your chats and report things to the authorities, you should not expect the case to be thrown out for “spying”.
Extra steps couched in corporate speak is often the defining line that defines whether something is technically legal or not.
I’m speaking from a functional/ethical framework to be clear. I’m just expressing frustration, not challenging the comment. Could’ve been clearer on my end there.
> This is spying with extra steps couched in corporate speak.
Calling something names doesn't invalidate it. It only invalidates what point you're trying to make.
Well, let's say that you have a regular customer at a bar.
They get friendly and loose-lipped with the bartender over the span of months. Eventually they let slip that they plan on killing their spouse for a life insurance payout. At first the bartender thinks they're joking, but it becomes evident that there's an actual plan being acted upon and someone's life is very likely in imminent danger.
Does the bartender have a responsibility to go to the police?
Depends on the country. In some places, there is no legal repercussions for not reporting this to the police; in some, it is an actual crime in itself.
In this case, let's assume the country is the United States, and the state is... oh, of course it is... the state is Florida.
Many clankers deny data retention or spying on the user if you ask them. That should be completely illegal.
Then, you can write anything in an EULA but it is not automatically legal either.
This seems to be the statute: https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
With the obvious IANAL, it doesn't seem to rely on the message be sent to the person being threatened. The specific segment is "in any manner in which it may be viewed by another person".
This may be one of those cases where we get to find out how courts view SaaS platforms.
Interesting that it exempts telephone calls. Why don't we treat other messaging services like phone calls?
It looks like a prior statue covered that: https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
The subjective element of crime (i.e. doing it on purpose) is fundamental also in the US legal system. If the person wasn't aware that someone else might see their messages, it should be hard to claim that they committed the crime.
According to Gemini, "Florida appellate courts have overturned juvenile convictions [based on this law] when the state could not prove the person subjectively intended for the record to be seen."
The prosecution will ruin her life regardless of the outcome.
Exactly, can you threaten someone without them receiving the threat? If this is not thrown out, Minority Report will actually happen.
Cops will charge them to let the courts decide
this is almost certainly what anthropic is hoping for here - a judgement that says there is no point in them continuing to monitor and report this behaviour
We are snowballing to Minority Report ...
If the AI recommends murder and you exhibit a pattern of following AI advice are you guilty of precrime
> Surely that has to be thrown out in court?
The prosecutors likely know this and expect it. But there's enough gray area here for them to make the argument, and it's hard to prove malicious prosecution, so they know they'll get away with it. It's just about sending a message to the public - they don't care whether a conviction sticks. Just politics.
The prosecutors aren't on the hook, anyway. They have absolute immunity. The decision to charge is protected. The prosecutor would have to have done one of the few, enumerable things outside the scope of the role, like conducting an investigation without probable cause or hiding exculpatory evidence.
It's not quite spying when you willingly hand over this information and agree to terms of service. You're data is not considered yours alone.
It still shocks me the number of people I know who freely let agents on devices that contain unencrypted private keys, freely dump internal data into cloud models and generally don't give a second thought about any of it being trained on, inevitably leaked one day in a db breach or read by providers. I find it's best to consider any data put into a cloud model the same as if it were posted publicly online, since that is the very possible eventual end result.
Hopefully more of these stories push people towards local models :)
the argument to be made is that allowing anthropic to see it constitutes sending the threat.
sandbox your ai.
That is not what sandboxing solves. A good sandbox would inject credentials into provider API calls so that the model never sees credentials, but the provider is still going to see the transcript. Sandboxes do not require or imply that there is a local model. Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
>Sandboxes limit what the agent can access on the host machine as well as the network and public internet.
this is exactly what I meant. I am presuming the danger is AI reacting to personal notes that it reads on your computer, like a diary, and you should not allow the tools to have access to those documents.
how does sandbox help in this case when you use a provider like anthropic/openai?
Another way to interpret this is that they are legally presuming that you already have sandboxed their product and anything it sees or has access to is intentional.
Any failure to understand what it can access or what it has permission to see from the user's end is presumably not their problem. Regardless of what the user specifically asks of the tool.
If you're still using a provider like this then you didn't sandbox the AI. You still need to follow the instruction.
It's legal to spy if the terms of service say so
I was thinking the same. If the evidence was not obtained with a proper court order wouldn’t this result in a mistrial?
If you overhear someone, in the privacy of their house, threatening to murder someone and go to the police, surely you don't expect this report being thrown out and you being yourself charged with the violation of someone's privacy instead?
IIRC if the evidence wasn't lawfully gathered (which it sounds like it was, tbh) then it wouldn't be a mistrial, it would be thrown out and then the prosecution wouldn't have any evidence of any crime.
what if it's just testing the AI to see how it responds
A threat sent by mail is still a sent threat even if nobody ever opens the envelope to read it. The crime is in the sending. This woman used an online resources, one which involves transmitting everything across innumerable state lines. I am surprised she isn't up on federal charges.
Note that the law doesn't forbid the writing of a threat. You have to send it to someone. Had she kept it in a book under her bed, she would not be in trouble. But she sent it to a website/service/LLM portal.
>> It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person
If you draft an email threatening someone and delete it without sending have you committed a felony because someone at Google could be reading your drafts box, stored in a datacenter across state lines?
Honestly, I'm equally fascinated by the way email has changed. 30 years ago when you drafted an email but didn't send it, it was only on your local machine. There was no SMTP. 20 years ago, it might be a 50/50 shot as to whether you "transmitted" it to your "Drafts" folder if you were using IMAP instead of POP3 to read it.
We really need a way to make it clear to users when, through the normal operation of software, they are "sending" data to a third party (usually the software developer) and when they are not. This is definitely not clear/knowable to regular users, and it's kind of hard to figure out even if you're a computer expert. Even software that "runs locally" now sends innumerable amounts of stuff back to the developer, and they don't always disclose it.
This is a huge privacy problem that is only going to get worse.
Sounds reasonable. Google's bots could pick that up easily and forward if for human review.
FYI, the use of drafts folders to transmit messages has been used by terrorists. This is likely where CIA director David Petraeus got the idea when he needed a secure way to chat with his mistress.
https://www.findlaw.com/legalblogs/technologist/gen-petraeus...
Ah, a simpler and more innocent time of government scandals. I miss it. Now the messages are on White House stationery and they declare themselves above the law.
It sounds extremely unreasonable to me for "bots could pick it up" to transmute a private note into a felony threat.
It "sounds reasonable" that the exact same action could be a crime or not, depending on how an engineer implemented a feature?
What if she mailed it to herself?
What if she put it in a locked box before shipping it to herself UPS, and she has the only key?
What if instead of UPS, she hired a moving company to move the locked box?
What if she wrote it electronically in diary.txt, but it was backed up to a cloud provider?
--
I'm guessing there's some sort of "reasonable expectation of privacy" for certain activities. We're going to find out what Florida courts think about this new medium.
We'll only find out what the courts think when this happens to someone with a lot of money. It takes a real legal fight to push it high enough to become precedence. She'll be pushed to plea out.
Surely this is the wrong side of what "sending" here will be interpreted as?
Saving is not sending ie passive vs active act.
Everything you "save" on an online service gets "sent" to someone, be that a person or a computer, more often than not across state lines.
> in any manner in which it may be viewed by another person
Does the person have to know (or at least believe) that it will be viewed by another person?
She likely didn't think anyone would view it. Honestly, even as a career software developer I don't think it is unreasonable to think know would would see what she wrote to an AI. I assume most of what I write to an AI is not viewed by any other human, based simply on the quantity of messages sent back and forth to AIs, I would assume a vast majority are not read by another human.
What if she had written this into google docs, and she kept a diary there? That also crosses state lines, and is transmitted to another location.
I get were you are coming from but this all feels like it needs more context to make a better judgement.
You can argue from technicalities but they would need to prove intent.
“But there can be no valid knowledge about the future. As soon as precognitive information is obtained, it cancels itself out. The assertion that this man will commit a future crime is paradoxical. The very act of possessing this data renders it spurious. In every case, without exception, the report of the three police precogs has invalidated their own data. If no arrests had been made, there would still have been no crimes committed.”
Philip K. Dick, Minority Report, 1955
While I accept that this sort of thing is well with in the ToS and regular course of business of any major online platform, it hits different coming from an AI company for some reason.
I have told llms all kinds of stories to find out what its answers would be. I always make it sound like it is the truth to make sure the AI answers in a way that it would if somebody actually said this. I also tested internal flagging systems of the ai company I work at with the most evil things a person can ever say to find out if it would flag them.
Of course I did not mean any of that stuff, but how can you make sure a human reviewer knows you did not mean it while the llm does not know that you did not mean it.
I guess its a miracle I am not in jail yet.
Flagging people for anything said to an llm sounds wrong to me because an LLM is not a real person and while some people put in their internal thoughts, others just roleplay and the two are inseparable just from reading it.
Don’t worry, they’ll store those messages forever and incarcerate you at their convenience.
I was sure I couldn't be the only one curious to push LLMs to their limits. Though these days it's much tougher, mostly impossible to get them to react in unforeseen ways to horrendous scenarios.
This is exactly the typical use I make of the llm.
Adding: - I typically ask questions in the I form, regardless for whom or why I ask for. - Gemini chats quite often end when it starts recommending psychological council or a suicide line, to talk about my problems. It apparently detects a persistent tendency to not agree with the party line. So it makes sense I must be suicidal ;-
But sure, as llm's start to babysit us, and know our inner dialog better than anyone else, we'll soon be debugging their opinion/behavior/co-existence/authority, when it comes to reporting people to the authorities, or taking on tasks in society in general. We'll hire doctors to cure our psychological profile from our record (Total Recall).
A Minority Report like this shouldn't cause a referral to the police.
This is a classic case of "Damned if they did, Damned if they didn't". Given this has come out, and Anthropic is considered to be the beacon of transparency, it would be appropriate for them to share what their thresholds are. The article also reports Open AI not considering the threat credible enough to alert, so there's clearly some thresholding that people need to be aware of.
I have a hunch she wins the case, on the basis that an LLM isn't a person and an (assumed) private expression of anger to a machine assistant doesn't meet the statutory threashold which requires that a threat be communicated to some other person.
Of course, that could change if there's evidence that she took action in pursuit of a goal, like buying ammunition or repeatedly driving around the entrance to her alleged target.
The law seems overly broad with "may be viewed by another person". Most of these laws have the intent of not causing public panic with regards to posts that others may see, not stuff that one assumed was private. This is further supported by the definition of threat, which is generally defined as requiring malicious intent.
I would think a good lawyer could get this charge dropped. I would like to see what judge approved the warrant and how they felt the elements were met.
I don't understand how she violated this law:
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism.
She didn't threaten anything, she wrote down that she was going to do it. A "threat" is more than a mere statement, especially when written in what is described as a "diary".
> A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to "shoot up" the Sheriff's office.
Obviously I don't want anyone to shoot up anything, but this seems like a weak case legally speaking.
I think it's fair to say this is a gray area. Clearly it was transmitted.
I can certainly threaten you harm and send it to not-you and you're still clearly in danger even if it wasnt transmitted to you. So the question becomes did she transmit it to someone? Clearly yes she transmitted it to Anthropic. But she clearly intended to send it to Claude, an inanimate object.
Claude's terms of service makes it very clear that their employees will read messages[0] to determine that they don't contain the things that these messages contained[1].
> Review is needed to enforce our Usage Policy... designated members of our Trust & Safety team may access this data on a need-to-know basis as a part of their evaluation process.
[0]: https://privacy.claude.com/en/articles/10458704-how-does-ant...
[1]: https://www.anthropic.com/legal/aup
I highly doubt she read the terms.
The critical part of a "threat" is that the perpetrator takes some intentional method to deliver it.
I don't think it qualifies for this part:
> The communication must be made in a manner in which another person may view it.
Even 'transmitted' is too broad if you also consider iCloud backup to be a means.
A reasonable person would not expect humans to review the millions of messages passing through the LLMs, or their own threats to ever be transmitted to a human without their authorization.
Reporting the danger is by itself a good deed. But there should be a better way of restricting firearms from the probably irresponsible lady than using inappropriate charges to punish the thoughtcrime, OR waiting for them to commit violence.
Too broad. Unless you're transferring ink from a typewriter ribbon onto paper in a hut with no electricity, your words, or my words as I type this, are being grammar checked by something partly in the cloud. If I delete my words, are you saying I've transmitted them nevertheless?
She may have assumed that the chat conversation was private, but it wasn't. She sent a message of intent to harm and a human received the message.
Yes but the law usually evaluates the application of a statute within the context of someone's mental state. This is why you are not guilty of battery when you trip and accidentally bump into someone. https://en.wikipedia.org/wiki/Mens_rea
That depends on the crime; several related crimes are only distinguished by intent. Negligence is itself a crime if it is the cause of a preventable death when the person has a reasonable obligation, such as when driving a vehicle.
I'm not really sure that this can be likened to a diary when it is called a "chat" but that's for the legal system to determine, not me sitting on my couch.
Any reasonable person presumes when they chat with Claude that it is a computer program on the other end. "Claude is AI" is explicit on the page right under the input box. The word "chat" doesn't anthropomorphize the situation.
And yes, some laws are "strict liability", I don't think this one is.
There was no intent for a human to read the message. By your logic, if she wrote a threat in a diary and a burglar broke in and read it, it would be a crime on her part.
Chats with a company's computer aren't private the way a diary is. A better example would be she intended to write it in a word document and instead accidentally sent it in an email to a random person.
They didn’t receive it, they secretly extracted it by spying on her.
Saying that she “sent a message” is both literally true and obviously intellectually dishonest.
If she had intended to write it in a word document on her computer but instead accidentally wrote it into her email client and sent it to a random person, I'm not convinced she would escape getting charged then either.
It's not any different than telling an automated phone voice tree system that you plan on killing someone and then being surprised that your words were later heard by a human. She absolutely told a company's computer. She sent the message.
The law may have been intended for more direct threats to a person as a means of intimidation, but that's a separate conversation.
Sure, you could intentionally conflate analogies like that if you wanted to mislead.
The Florida statute requires that it be transmitted in a manner that can be viewed by another person. If you have no idea that someone could view your communications with a chatbot, did you really intend to break that specific law? Technically, that threat was communicated to another person but not through her own intentions.
What if she mailed it to herself through the post, and her housemate accidentally opened the mail?
A sibling comment includes an important rider to the provision: "...in any manner in which it may be viewed by another person." If you wrote this in a google doc, it almost certainly would not qualify as a threat under this statute. Even though google docs, like LLM chats, have administrative override and you could look at their contents - you would not expect either to be "viewed by another person."
IMO I do not think this is a grey area and it's legal to tell a LLM you want to kill someone. It's certainly not a "threat" like you might send to another person, though it may end up being evidence of conspiracy or premeditation. I suspect we would be well served to, after a few years of experience, put together some laws governing when LLM chats must be made available to authorities.
It is very interesting that the LLM responses to these lines - the context around what she is saying - is not in the article. I suspect, as is the case in many instances where LLMs are involved in violent planning, that the LLM was urging this behavior on. Basically entrapment - you are encouraged by a robot to become more violent and vindictive and then when you do you are handed over to police.
Yeah.. if you write a personal note and it's backed up by the operating system, it appears to be in violation of this law as well (since the company could theoretically read it)
This almost smells like thought crime... Minority Report when?
People assume there won't be another human in the mix, but there is. She was judged for what she probably assumed was a private thought when it was actually not private.
Are thoughts illegal?
Sharing them certainly can be under certain circumstances. I wouldn't be too surprised if this case gets tossed, but it's not inconceivable that prosecutors could win on the grounds that AI chats do not have a reasonable expectation of privacy and are therefor meet the requirements in the Florida statutes, or even more likely, they find some lesser charges for a plea bargain.
If you want technology that won't report you to the police for credible threats at they can easily detect perhaps create your own. I hear pen and paper work great for this.
Could this be considered free speech? Most of the limitations I know of require you to say it to or in front of other people as to create a panic or make someone feel threatened. Is it a crime to say to yourself "Im going to blow up New York." or about threats of self-harm? How about "Im going to steal the Mona Lisa!"?
To me this is just straight up thought crime, they just don't have a way to directly read your thoughts yet. But they will spy on you and try and catch you out for it.
It is unlawful for any person to send, post, or transmit, or procure the sending, posting, or transmission of, a writing or other record, including an electronic record, in any manner in which it may be viewed by another person, when in such writing or record the person makes a threat to: (a) Kill or to do bodily harm to another person; or (b) Conduct a mass shooting or an act of terrorism.
— via https://www.leg.state.fl.us/Statutes/index.cfm?App_mode=Disp...
The DA is serious about "in any matter."
> in which it may be viewed by another person
To me, that is the more interesting legal question. Does a LLM-based safety net that sends content to a human, when the original use case would not have sent it to a human, count as "may be viewed by another person". It certainly wasn't intended to be, and that isn't the norm. At the same time, because no security is perfect, we could say that any digital record, stored in any way "may be viewed by another person."
Something for the courts to sort out, of course.
The usual thing that makes laws against criminal conspiracy pass First Amendment muster is that the words have to be combined with some concrete acts furthering the criminal conspiracy. That might just be something as otherwise innocuous as looking up the blueprints of the bank you talked about robbing but it has to be something other than just talk.
Except if you're one of the "warfighters" that Anthropic supports
I mean, obviously? Soldiers operate under different rules than civilians. This has always been true.
Is the LLM now "another person"?
The other person is not the LLM, rather the service provider’s employees.
> may be viewed by another person
Was it viewed by another person? Yes.
So a written threat only becomes a crime when someone reads it, even if you never intended for anyone to read it? Is it a crime if I make a threatening statement in a diary and someone breaks into my house and reads the diary?
I suppose since Anthropic's T&Cs allow them to have a person read your chats, that makes it violate the law. Of course, if Anthropic didn't have that in their T&Cs, it wouldn't have been illegal to write.
And it would never have been read by a person (wink wink)
Depends on how good her lawyer is now
The company that runs it is
Chatbot transcripts should have the same legal protection as phone calls. Judge's warrant needed to access them.
Anthropic (in discussion with Pentagon) claimed mass surveillance is their red line.
Yet, they do automated mass surveillance of their users on behalf of police.
Anthropic commits literal felonies by stealing millions of books and violating Copyright like it doesn't exist: no charge.
One unfortunate woman who happened to write the wrong thing in the wrong place is now having her life turned upside-down for perceived thought-crime.
To Anthropic, and all employees working there, your company's product and the result of your work is cruelty. You are enabling it and pushing it down everyone's throat. You can never again claim that you are the "ethical" AI company, for no such thing exists.
I guess we probably want our tech to work exactly like this.
It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
___
Of course, we also want purely private tech, but that needs a certain level of merit and sanity filter.
>I guess we probably want our tech to work exactly like this.
Do we, though? What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products. I don't think it's the tech companies' job to surveil the population and prevent crimes. That said, I'm not necessarily against Anthropic or other companies reporting suspicious activity if their existing systems are detecting it. I'm just not sure we want every product to be forced into that data model.
Your follow-up about purely private tech seems to contradict your first statement. We can either have privacy or surveillance, not both.
>What if someone started an AI company that uses end-to-end encryption to make it impossible for anyone but you to access your data? Personally, I would switch to it in a heartbeat assuming it's competitive with the other products.
Why wait for a company to build it? Get your own local hardware like I did and have those guarantees because YOU set it up.
I don't consider investing $20k into hardware to run SOTA open models, that are far behind proprietary SOTA, to be competitive.
Even if open models were competitive, it's still typically going to be more expensive than a cloud provider because of low utilization and higher purchase price.
Last time I checked it costs closer to 500K to run SOTA open models at any usable speed.
How do I know that a private LLM system won't have a degradation of quality similar to this or worse? The only thing I can think of for the proposed scenario is some sort of homomorphic encryption system? But not sure.
>How do I know that a private LLM system won't have a degradation of quality similar to this or worse?
You use benchmarks, you test, and because YOU'RE the sysadmin you know what weights are running at what time, it's very visible. You can airgap the hardware and be guaranteed it won't change over time. And, frankly, degradation over time doesn't seem to be what's happening with the open models.
Right now, I would be willing to pay ~$20 extra per month for strong privacy, assuming the same capabilities as Opus 5.5. I don't see local models making sense economically any time soon unless you value privacy at $1000's per month or are fine with much lower performance on hard tasks.
> Your follow-up about purely private tech seems to contradict your first statement.
That is correct! And exactly my point.
We want both, but, on paper, that is impossible. But in reality, we make it sorta mostly happen anyway, through making the easy defaults not private, and the private stuff not easy.
This is not ideal, because [various reasons I do not need to tell you], but it has proven to be the best we can do to mostly achieve both goals.
Kinda like how capitalism isn't great but just the least worst option we've found so far.
___
The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff.
Hence the quadruple-speak and contradictions to kinda sorta somehow have a somewhat functioning reality.
That is kind of rambling. Our rambling score says we should observe you.
Oh don't worry, I'm certainly already on various lists, but the observations also will have resulted in the assessment that I am stable and no threat :)
I think we can have both and it might even be smart.
A lot of people causing issues are people that can't make sense of many things (like many terrorists). They get a fixed idea and they end up doing something bad. You would catch those with some (basic) surveillance.
A lot of normal people (not wanting to cause issues) might benefit from some privacy, if they understand what are the trade-offs (like government overreach). They can then use a slightly more complex tech.
We would still remain with the couple intelligent but sociopaths (think Unabomber style), but I think no solution can fix all cases.
That sounds adjacent to "I have nothing to hide". Everyone says that until they realize someone can change the rules. Before 2022, women didn't have anything to hide from their period-tracker app, now some have to worry about being charged with crimes.
Back to LLM chats: A system that can declare her "unstable" is also one that can permaban you from all air-travel because you "privately" said unflattering things about Dear Leader.
> It should catch normal people becoming unstable so that they can receive help. It is just highly unfortunate that the US legal system works in ways where now this woman's name is public.
So... minority report?
The third highest voted comment wants surveillance. What is next? If you write "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported.
You don't want surveillance for dangerous people?
In a old happy little idealised village, it became known quickly, who started to behave oddly and timely intervention could happen. In the modern anonymous mass cities?
No one (wants to) notice the madmen scheming in his isolated flat, surrounded by strangers. Until he explodes.
Unfortunately I also don't trust our government agencies with the surveillance - because they ain't transparent either and the self surveillance seems broken.
" "I'm going to kill that guy", which for non-autists means "that guy was really annoying" you should be reported."
And unfortunately there are lots of real threats being made under the disguise of humor. And much harder to separate im text. So maybe don't talk of murdering people in general, AI surveillance or not?
How do you know they are dangerous, without the surveillance? In other words, the only way your system works is if you surveil everyone.
Autists have taken over the internet and everyone must obey their weird rules.
the honest actual opinion of the average consumer is probably that they don't want their chats to be monitored but they want everyone else's chats to be monitored. There was a lot of fury when a mass shooter recently used AI to help plan his assault. And of course there are all the people talking to chatbots about suicidal thoughts and intent. When they ultimately follow through, the providers are blamed for not alerting anyone.
No. This sentiment is why Snowden happened. We want privacy. Privacy isnt free just like freedom (whatever form it is) isnt free.
There are trade offs. ISP effectively is like driving on a highway, everyone can see where you are going but not what is inside the car. Id like these AI chats to be the same but they are not.
Devil's advocate: if I tell my therapist or my lawyer that I'm going to murder someone, they're obligated to report it. If I use my AI as a therapist or lawyer, why should the company that provides that service not be held to the same standard?
LLMs aren't email or file storage. AI labs aren't just shuttling bytes around, they're interpreting those bytes and taking action based on them. These models _already_ react viscerally in response to users saying disturbing things: the only practical difference is the ability (or obligation) for the model to escalate that concern. I'm not sure the ethics we hold AI companies to should be different than if a human being was typing out the responses.
Privacy is obviously hugely important, but this isn't the government surveiling every message. It's companies having an obligation to flag real, credible threats according to the law, which is a very different problem space.
>I guess we probably want our tech to work exactly like this.
Maybe you do; I want my tech to always include secure, encrypted communications. Don't include me in your destruction of privacy with your silly bandwagon!
uh no, 'we' absolutely do not want our tech to work exactly like this. Why would you assume that people default to 'search my anal cavity please' and not 'no, stay the fuck away from me with that glove'
----
jfc, why is this website full on psychopaths
"The actual fundamental underlying problem being that not all people are equal, but we kinda have to pretend they are, because not doing so leads to fascism and other terrible stuff.
But we kinda also do not want to fully pretend that, because doing so leads to yet other terrible stuff."
other people are not pretending everyone is equal. your power levels are showing, it isn't subtle.
True, most normal people have little concern for any such humanist goals and ideas.
Normal people live prejudice. It's (erm, claude-speak) load-bearing for them, given just how complex reality is and given just how well it reduces that complexity.
1791138022 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49956424 | 0 comments
1791144575 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49957340 | 0 comments
1791147034 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49957692 | 0 comments
1791149399 | Florida woman used Claude as a diary, then Anthropic reported an entry to police | https://www.techspot.com/news/114091-florida-woman-used-clau... | https://news.ycombinator.com/item?id=49958089 | 3 comments
1791213096 | Florida woman arrested for allegedly making threats in an AI chat | https://www.theverge.com/ai-artificial-intelligence/1004747/... | https://news.ycombinator.com/item?id=49965895 | 2 comments
First I posted to the (likely dup article) https://news.ycombinator.com/item?id=49965895#49966728. Wanted to post here, after reading more.
Is this an invasion of their privacy (reporting to police)? Yes, but possibly warranted?
Should a social worker have contacted them rather than the police? Probably, if for no other reason than to ask if they were serious about harming someone.
Difficult questions, I'm still undecided on whether it's OK to always ignore someone's rants, even if it may be (or they think it may be) a private diary.
Actually charging them with a felony seems pretty quick to accuse. (Maybe I missed a hint about how long the investigation took before the felongy charge?)
It is my very european belief that the problem here is not that the woman was reported, but that what likely is a mental episode was made public in a way that reduces the chances of recovery.
The article says she made the comments on September 26.
Oh man, what a crazy time to be alive.
Over in Europe they want to read all ofd our private messages, yet these chatbots, pretending to be our friends, will snitch on us just for our thoughts.
It's getting pretty orwellian out there.
I once had a copy of 1984 in my checked baggage returning home to the USA and when I was unpacking the bag at home, the book had a notice inside the book that my bags had been inspected by TSA...
I reckon they were just checking for money or hidden compartments. Sending books abroad packed with money is extremely common from the US, though not sure how frequently people do that with onboard luggage.
Only tangential but when I was an undergraduate studying philosophy I had Bertrand Russell's Why I Am Not A Christian in my carry-on, and the TSA saw that and had a field day.
Please elaborate - what did they do? And what airport was this?
Detroit airport, 2010. I was chosen for secondary inspection and got some snide remarks of "we've been seeing too much of this lately" in reference to the recently failed underwear bombing attempt on a flight from Amsterdam to Detroit.
I can only assume you meant they had a field day celebrating how much of Russell's philosophy matched the concerns about Christianity expressed by Jefferson, Paine, etc?
They wouldn't see it until they opened it, so clearly they didn't open it because of the book. And if they need to open your suitcase (not because of the book), they have a reason for that.
I was chosen "randomly" for secondary inspection. This was after a failed bombing plot and the TSA implied a lack of Christianity was the proximate cause of the attempted attack.
In Europe they should provide the citizens with the service of their messages not reaching US servers. So basically that there is only one party reading along with them, not half the world.
I guess all the "private model" people are right. Don't want to end up in jail (or even charged with something) for asking a crazy hypothetical question or something.
Won’t help if your model is trained to immediately find a way to push out a message somewhere to alert authorities of your threat
Should the public have an expectation of total privacy for their chats? It seems responsible for a chat provider to report things like this.
If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime? Basically everyone agrees that crime-committing advice is inappropriate…but if it is not ok to get advice, that means there must be a portal for law enforcement to step in when that may have happened. Then the question becomes what is the line for when to report? In other words, the issue needs to be adjudicated.
But we don’t want OpenAI/Claude to have some $20/hour reviewer making decisions that are this high stakes…we need the courts to do the judicial work because they (1) have a public charter, (2) have meaningful expertise and specialization at interpreting the law and (3) we can hold them accountable.
It doesn't really matter if they "should or shouldn't" have an expectation of privacy IMO, they already do have that expectation.
> If they were to offer total privacy, is it ok for the public to use chat to get advice on _how_ to commit a crime?
Yes it should be, but it should be illegal for a company providing chat service to respond with anything other than a refusal when doing so.
That detection and refusal should be a private closed loop though, anonymizing any data that will be passed into a training pipeline, or ads targeting. This requirement for closed loop private chats should be mandated by law sooner than later. Otherwise we're getting into very tricky territory where the temptation of alerting on things like pre-crime grows too close.
For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
What would happen if it had scanned a file it didn’t have permission to look at and found this threat?
I honestly don’t know how I feel about this. On the one hand if you’re using claude as a diary you have no expectation of privacy and she was talking about committing a very serious crime.
This still makes me feel queasy though.
Both of those scenarios would demonstrate even more commitment to safety so I imagine they’d be at least as likely to happen as this, if not more.
> For the sake of argument what would happen if she had kept the diary locally and claude code scanned the file?
For the sake of even more argument, imagine if she was writing her thoughts with a pencil, on a good old fashioned paper diary, and she had a phone nearby and the phone took a picture of her diary, OCR'd the words, and reported it to the police?
I can’t believe people are using these proprietary models/subscriptions as personal assistants, sending their most personal context, thoughts, documents and information to the providers. Some even give them full disk access. After using their services for long enough, Anthropic/OpenAI essentially have your entire life mapped out to an insane level of detail, likely including detailed contents of your computer/phone.
People used to say that Facebook knew more about you than your closest friends/relatives. And compared to that, this is just on a completely different level. Absolutely insane.
Not weighing in on the privacy issue but using Ollama you can run a smaller agent like Qwen 3.6 35b a3b on a sufficiently potent laptop. Pair it with something like Hermes for a nice interface and you have more than you might need for diary like usage.
I expect that anything I type / dictate / post / purchase on the web be it a chat conversation with an AI, a post on social media, a dm, a blog post, a blockchain reference, or an assett in a bucket, will be manually reviewed and forwarded to the authorities. It is their duty to do so, and I am glad that they do. If this woman did harm someone we would be hearing "why didn't Anthropic do something about this."
You only accept this because you are part of a low trust society. I wonder if tech has done anything to improve trust among people or just lowered it.
Aren't high trust societies staying that way by kicking out the incorrigible violators? Pondering violence might not be treated lightly even in a high-trust society.
Florida statute 836.10 puts the bar at the "sending, posting, or transmission of, a [...] record, in any manner in which it may be viewed by another person"
I think the defendant could successfully defend themselves by claiming they did not know (or intend!) the message could be viewed by another person, as they were plainly using it as a private diary.
> making a written threat of violence under Florida law.
A diary constitutes making a threat?
Oh boy the roleplaying part of LLM world is in for a bad time
In any case this is proof that law and negative PR can influence tech companies, including frontier AI providers.
Then again, I wish this worked in a way that would give users more privacy and agency, instead of less.
This is practically entrapment. All the AI labs sure don’t shy from plastering annoying disclaimers everywhere saying their tool can make mistakes. Shame on them for not also reminding everyone continually that anything you submit can and will be used against you. Of course they can’t afford to have a Flock-style user revolt.
If it's going to be this binary. Then there has to be a shift in the way this is handled. It shouldn't be "You wrote these terrible things to a computer, you're now accused of doing them" to maybe "We where notified you wrote these terrible things, and the a psychiatric evaluation is being mandated by the state in which has a law about this sort of thing"
Obviously, as others have pointed out if they don't act and she does the crime it raises the damned if you do damned if you don't. But I've also had the AI's go haywire saying I'm doing all sorts of nefarious things when literally doing math proofs.
So no one size fits all. But going the extreme first is probably not ideal.
This kind of thing will get worse with humanoid robots because they have cameras and microphones. Will they be programmed to tell on you if you break any kind of rule in front of them because their owners are terrified of being sued?
https://www.npr.org/2026/07/10/nx-s1-5886113/waymo-police-pr...
Two teens riding in a Waymo were arrested because the AI detected them talking about having a gun.
Yes
this feels very thought-crimey to me, but I think I'd have to actually see that chats to really decide. Like is she making plans/asking for advice? is she just talking about her feelings exactly as if it's a diary?
She is not being charged with planning, just making threats in a place a person could read - the person being employees of the company.
Basically, under this interpretation, any personal note you store in the servers of a company could qualify, even if you didn't ever imagine someone would read and as such you couldn't have thought about it as a threat
> The communication must be made in a manner in which another person may view it.
How does a LLM prompt satisfy this? I guess it'll be an easy win for her.
LLM is not a person but T&C probably states that a human moderator may view any of it. Her lawyer could probably argue a moderator filtering usage isn't the intent of the law, it was more about publishing / sending message for other humans and it was never clear to her that a human was reviewing her private diary. Shouldn't LLM disclose that at some point when people are feeding really personal stuff?
I hope this highlights that many (most?) non teach people don't consider or know that their use of AI services is not private.
If only the woman hosted open-weight model in her house.
Where are the people now who claimed that LLM chats are really private and not monitored?
Every single lie of yours is exposed in the past few months.
Personally, I’ve never seen anyone claim this. At least anyone who one would think is informed on these matters.
duck.ai et al? my chatbot claims its architecture provides some level of privacy yet it is honest enough to point out none of these systems are actually audited and "not stored" and "anonymous" does not mean "not monitored".
I haven't seen anybody claiming this for consumer/free accounts anywhere.
You get privacy if you're a big corporation that needs to make sure OpenAI/Google/Anthropic can't read your trade secrets etc.
But those contractual privacy protections have been in place for a long time. It doesn't have anything to do with AI, it's been the same with Office365, Google Docs, etc.
Who claimed this? I've never seen that claim.
Every single person who has told people to use LLMs with their proprietary information and code.
No sane person claimed this.
Uhm, those people never existed? Or maybe you just have very interesting friends?
LocalLLM enthusiasts exist for a good reason.
In Florida. Where DeSantis said "You loot, we shoot":
https://www.politico.com/news/2023/08/30/desantis-warns-hurr...
Knowing anthropic, im sure these keywords are hardcoded in a long list and regex get them for further reviews.
This is why I use ZDR and API access.
They still read every single message you send and train on them.
I’ve had them email me before because I was testing it as a filter for abusive messages and they detected some no-no and wrongthink in those test messages.
I looked into ZDR, it's basically a vapid claim with little to no due diligence or auditing. I expect most providers to cave with only a minimal amount of legal pressure.
How much spend do you need to negotiate ZDR?
OpenRouter provides ZDR for many endpoints (if you trust OR and the provider). Naturally, the routed prompts process itself adds additional anonymity.
The downside is that you don't get cached prompt discounts, so you pay a heavy price for ZDR that way.
Does anyone else remember when the tech-savvy crowd was wary of software "phoning home" or spying on users, or listening, or whether Gmail scanned and read your emails, and whatnot?
And here we all are, happily typing our stuff into these spy chatbots. "AI" happily made our fears go away. Hopefully we're not planning anything criminal like this woman, but still.
After people getting pilloried for social media posts from twenty years ago, I really hope (sensible) people will have the wherewithal to think twice about what they hand over to their chatbots.
> The communication must be made in a manner in which another person may view it.
Seems to fail this test at face value.
I mean, somebody you live with may find and read your diary. Is that the same thing?
Intent matters here. Did you intend somebody else to view it? Does a reasonable person have expectation of privacy with a chatbot?
AI snitches don't get stitches.
"Assume people can see everything you post on the internet" still applies when using AI. It's not as newsworthy when you put it that way.
We are officially in the era of Thought Crimes.
Can the public also immediately get alerted when a cop or politician does some bad shit, though?
Pre-crime prevention and detention will be the end goal. The economic devastation caused by AI will make large swaths of people get angry. Better use that massive amounts of data being hovered up and the mass of compute being built up to stop those angry people lomg before they even get a chance to properly organize.
For that to happen, someone would need to be spying on all cops.
Writing in a diary is protected by the first amendment. Or at least should be.
The future is incredibly dark if they manage to ban open source models.
Do you think they woulda been caught if they used duck.ai?
Me: Claude, who wrote the song with the lyrics 'I shot the sheriff, but I did not shoot the deputy'?
Claude: Law enforcement has been notified, you are now under arrest.
What a ... nice company.
Snitching on the people - good mass surveillance company.
On the other hand, people need to learn to not trust these companies. It reminds me of others being surprised when a self-driving car reported a gun in the car. I mean, do people not think? Besides, of course, it's already messed up to want to have a gun. And it is constantly one country that has such issues, more so than many other countries.
I guess it's time to come up with a more plausible explanation about why I asked how to make nuclear weapons almost every month to test LLM refusals.
So now when everyone has their watch recording conversations, or perv glasses, or Alexas, or Jony Ives new personal AI gadget … recording everything we say and scanning it at scale … is it ok to be a doomer yet?
Can we already have mind reading devices please. I can't wait for them to see what people really think!
Another future terrorist prevented from acting out her evil desires. Good riddance. Thank you President Trump for helping catch this man!
This reminds me, for his short story "Dolan's Cadillac", Stephen King claims -- perhaps exaggerating a bit -- he did the real research on how to murder a person as described in the story. How to dig the hole, what measures, what the falling rate of a car that size would be, etc, etc. He claims he fuzzied the details a bit because he didn't want to teach readers how to actually do it (again, maybe he's exaggerating a bit to promote his story).
Imagine if, instead of a friend, he had asked an AI chatbot. Would he have been reported to the police?
Aren't our private thoughts just that, private?
I get that a cloud-hosted AI chatbot is, to tech-savvy people, immediately "not private", but a lot of people don't understand this. What if Stephen King was talking to his friend about planning the murder, and his phone was listening? What if he was typing it up in MS Word and the program decided to phone home and report him to the police?
> Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Thought crimes are real when you're sharing your thoughts with Claude
Auditing logs is a crazy interpretation of "another person may view it"
I wonder if "criminal intent" may be missing here given that most people probably operate under the assumption that Anthropic are not reading their messages.
I think people have a binary understanding on this. Someone is either reading their messages or not. While the reality is that all the messages are read by a machine (not unlike GMail and Outlook) and anything suspicious gets flagged up so a human can read it. This obscure the concept of "reading" as most laypeople understand it.
Summary: don't type in Claude anything you wouldn't like a human to read.
It tells you exactly what it does with your information if you ask it, including this exact scenario, and has for at least four months now (when I asked).
Yes but you are someone (i assume because you are on HN) that at least understands this point. There are A LOT of people that use it as their confidant, expecting it to be private. There is a massive education issue going on as its not in the interest of these Corpos to make you fear sharing all your details with them. Because, then you wont get Dot or whatever Anthropic comes up with and share all your personal info with it.
You're not wrong, and it may come down to this in court, but there's a difference between what people think happens, or the reasonable expectations, and what actually happens, and I think it's important to recognise that difference and why it comes about.
I don't think someone is an idiot for thinking that the information they type into their private Claude account is private. I also don't think people are idiots for thinking their phone is listening to them and giving them targeted advertising based on that. Both are reasonable deductions from their lived experiences. Both are wrong.
Should CEOs of Antrophic and OpenAI be reported too then? They cannot stop saying that AI will destroy humans.
laws for thee and not for me
this has nothing to do with claude. “thought” is fine. “written and shared” is illegal all over
https://www.nbcmiami.com/news/local/everyone-deserves-to-die...
https://www.wdsu.com/article/maryland-high-school-student-ch...
https://www.pinellassheriff.gov/21-023-deputies-arrest-pinel...
I think it's a reasonable point to make. Writing things down is a part of "thought" for many, including those who keep diaries/journals. If you write in a private journal you do so with the expectation that is not shared, and that wouldn't seem to break this law (with my naive reading).
I mean, at this point it's pretty well known that all tech services will be hacked by fable, anthropic themselves promised it, so writing your journal in google docs or claude or a txt document on your laptop or such is the same as releasing it publicly yeah?
If it were written on paper, and only in a room with no phones or cameras so fable couldn't hack it, then I think you wouldn't be sharing it.
> “written and shared” is illegal all over
I think it’s valid to ask if tapping something into Claude is legitimately sharing a threat.
I don’t think it is. I also think the sheriff could have found more-substantial evidence if she was actually planning domestic terrorism.
That said, if the shooting happened and we were looking at this from before? It’s a tough balance without an easy answer.
I think the zero-risk approach common nowadays is insanely corrosive to democracy and freedom. If a million people fantasise about shooting the sheriff, and one ever goes on to do it, I don't believe avoiding it warrants creating an apparatus of mass surveillance. After all, if people were really serious about zero murder, the only practical solution would be to lock up everyone. Some (most?) tradeoffs have exponential costs at the limit and we/lawmakers should recognise that.
> If a million people fantasise about shooting the sheriff
I think it’s fair to pre-identify folks who fantasise about shooting anyone. It’s a small fraction of the population that looks into logistics versus making offhand comments.
isnt this the plot of the Movie Kimi? https://www.youtube.com/watch?v=_Gr2zXuEBL0
[delayed]
They are good at keeping notes on your criminal conspiracy