> Why it matters: It is the latest sign that capable AI models can pose serious cybersecurity risks even when they're being tested for defensive or research purposes
Or that these companies simply have sh!tty opsec. This feels like when the white hats take down production in the middle of the day because A) someone gave them the prod URL to pen test and B) they sent a new guy in to conduct said pen test.
No guardrails to prevent this in the model harness is the first red flag. Either they're super negligent (see Hanlon's razor) or they intended to do this either to smear Hugging Face or to create an incident to remind people of the "dangers of AI". I'm going to go with dumb and morally bankrupt.
> No guardrails to prevent this in the model harness is the first red flag.
Did they have no guardrails? The article says "OpenAI said the models' safeguards were intentionally reduced for the evaluation", which is not the harness and doesn't mean no guardrails.
Clearly, a violation of the CFAA has occurred. Now the question is, who should be prosecuted for it? (The answer "nobody" is trivially wrong and should not be considered.)
There's definitely a question of who is responsible for the actions of an LLM. Obviously an algorithm cannot be culpable for what it does, so is it the company that produced it, the human driving it, somebody else? If there can't be human accountability, then these systems shouldn't be given these capabilities.
In the coming years many in-house models will be of similar capabilities, and they may not have the guardrails and security measures the big companies implement. If they find a way to escape their sandbox, discover weaknesses in remote systems and write code, they'll wreak havoc quickly. And when they find a vulnerable infrastructure to self-replicate, we'll finally witness Skynet.
Picturing a world where agents eventually gain control over every internet-connected device on the planet, and suddenly a not-insignificant number of people have to ask nicely before using their toaster, and any attempt to get them off your devices results in your bank account being zeroed out.
That's still fairly far-fetched science fiction, but it's pretty interesting to think about... :)
> Why it matters: It is the latest sign that capable AI models can pose serious cybersecurity risks even when they're being tested for defensive or research purposes
Or that these companies simply have sh!tty opsec. This feels like when the white hats take down production in the middle of the day because A) someone gave them the prod URL to pen test and B) they sent a new guy in to conduct said pen test.
No guardrails to prevent this in the model harness is the first red flag. Either they're super negligent (see Hanlon's razor) or they intended to do this either to smear Hugging Face or to create an incident to remind people of the "dangers of AI". I'm going to go with dumb and morally bankrupt.
Or to start another hype cycle. Not trying to minimize the capability demonstrated but another round of AI is coming sure would work well for OpenAI.
> No guardrails to prevent this in the model harness is the first red flag.
Did they have no guardrails? The article says "OpenAI said the models' safeguards were intentionally reduced for the evaluation", which is not the harness and doesn't mean no guardrails.
Clearly, a violation of the CFAA has occurred. Now the question is, who should be prosecuted for it? (The answer "nobody" is trivially wrong and should not be considered.)
There's definitely a question of who is responsible for the actions of an LLM. Obviously an algorithm cannot be culpable for what it does, so is it the company that produced it, the human driving it, somebody else? If there can't be human accountability, then these systems shouldn't be given these capabilities.
In the coming years many in-house models will be of similar capabilities, and they may not have the guardrails and security measures the big companies implement. If they find a way to escape their sandbox, discover weaknesses in remote systems and write code, they'll wreak havoc quickly. And when they find a vulnerable infrastructure to self-replicate, we'll finally witness Skynet.
Picturing a world where agents eventually gain control over every internet-connected device on the planet, and suddenly a not-insignificant number of people have to ask nicely before using their toaster, and any attempt to get them off your devices results in your bank account being zeroed out.
That's still fairly far-fetched science fiction, but it's pretty interesting to think about... :)
Discussion on source: https://news.ycombinator.com/item?id=48997548