It's because they're a scam. Point the camera at a forged image with a higher resolution than the camera sensor and it will make a signed copy of the unsigned forgery.
That's before getting into the practical problems with securing the keys. Every camera by every manufacturer has keys in it and the attacker only needs one key from one camera, and they get to choose the model? Creating something premised on needing to trust something with such a high probability of being compromised is worse than nothing, because it allows the ensuing forgeries a mechanism to pass themselves off as "signed" "real" images.
…the signature included the depth measured by the autofocus system across the image?
…or a tiny stereo image was included to capture depth?
…or a mini video in the ten seconds before and after the photo was taken?
…and the key is in a tamper proof HSM?
…and the key is deleted the moment the camera detects the case being taken apart?
I know that it is a losing battle to try to build such hardware when offline attackers have essentially infinite time to dismantle even the most elaborate systems — no such thing as an un breakable safe, only how long it takes to break into it, etc — but I feel these are valid counter measures, are they not?
I agree. Yes, these are not foolproof, but damn does it make it harder. It means that a random lone wolf using some random AI is not going to find it easy.
I would add a few more measures:
* Keys are regenerated for each device in the charging dock and are only valid until next recharge or a timeout.
* There is a sign-out process for the cameras that ties them to the operator.
* Police officers have no control over when the camera is recording, the camera instead controls this.
* Lower resolution data is streamed and synced to a cloud in real time, along with interesting data such as GPS, local BT/WiFi devices, etc.
As for privacy, British police are using more and more evasive camera technology out in public spaces, it's about time they were forced to wear it themselves. I want even the pencil pushers in the offices to be forced to wear it.
> the signature included the depth measured by the autofocus system across the image?
> or a tiny stereo image was included to capture depth?
These systems work by having multiple sensors to use for depth perception, so enterprising hackers write software to create two images, one for each sensor, and put some kind of lens or mirror in front of the camera to direct a different image/screen to each sensor.
The problem is fundamentally that the device is taking unsigned analog attacker-controlled input and then signing it, and is being mass produced. So whatever you're having it do, they put something that generates the same photon pattern in front of the device and you can't fix that with cryptography.
You can probably make it so that a cheap camera needs a few hundred dollars in optical glass or similar, and expensive camera needs a few thousand dollars worth, but it's hard to see how you could make it infeasible to anyone with non-trivial resources and it's also easy to mess up even worse and make it practical even for anyone with a computer and a high resolution screen or two.
> or a mini video in the ten seconds before and after the photo was taken?
Which does what if nothing in the image is expected to be moving, or the thing you're pointing the camera at is a screen rather than a piece of paper?
Also, now to verify the signature on your 50kB image you need a 2MB video? Then by default people won't distribute images that have the ability to be verified.
> and the key is in a tamper proof HSM?
Someone figures out a timing attack on the HSM or similar and now you can extract the keys from every device of that model. Happens over and over, the chances of every device getting this right are essentially zero.
> and the key is deleted the moment the camera detects the case being taken apart?
They get multiple cameras of the same model, take one apart to see how the detection works, then having figured out how it works, take the other one apart without triggering it. Or they extract the key without ever removing the case.
Also, now your phone is going to delete its keys when you remove the case to replace the battery or a cracked screen etc., or if the detection system has a false positive? Then you need some way to transfer new keys to a thing that hasn't got any, which is an even worse attack vector than not deleting the keys to begin with.
It's obviously a loosing battle. You're thinking of elaborate attackers, I'm thinking of your common policeman or scammer. Not great security, but might be able to solve the easyness problem introduced by GenAI.
Out of curiosity, what would be your proposal for identifying GenAI images and videos? Any suggestions?
But also what about .. Even now there is a range of forensic tech that can be used to statistically indicate if an image has been doctored, or generated, wouldnt't adding more and more real world data to the capture increase the bar for doctoring, so that only attackers with infinite resources can do it? At least it would stop Bobby Rotten from doing it.
I’ve done a short deep dive on this, for some cases that possibly would have went court. The tools we have today don’t reliable indicate if an image was doctored necessarily. Most open available scoring and tools like VAAS, DIRE, and Sherloq are decent today. Figuring out if an image that has been doctored, especially with solid proof, is only reliable if the image has metadata to prove it. If they export it to another format or screen capture it and the metadata is lost, it is purely still a guessing game.
Really good points, thank you for pointing this out. What I'm mostly thinking of is the usage in e.g., social media. There will be efforts to do what you suggest, but for your typical facebook user and scammers, this might help with the more common situations where people use GenAI.
The general population does not understand technology sufficiently well to set it up correctly, regulate it or use it correctly. Until we educate our population more on technology we will always be in this state.
My experiece with whatsapp family groups is that a lot of people over 40 can't detect the most obvious AI fakes (e.g. Studio Gibli clones, or three handed people), so they share the most stupid stuff as it was real, while youngers seem have an instinct to detect AI but can't tell exactly why they know.
I can picture a cop fabricating images that are obvious, even with a watermark included, while totally convinced that it is undetectable.
IME it’s more like 60, which then makes more sense given vision deterioration. Then again most of my over-40 experience is with folks in the tech industry.
i do wonder, that in the age where we have image and video creation out of the bag, whether or not this will result in whole classes of evidence becoming completely unreliable.
There's a big gap between "theoretically unreliable" and courts actually recognizing that, unfortunately. Lots of forensics is much more dubious than CSI would have you believe.
My girlfriends been having me watch law and order svu with her and to be honest it doesn't really even seem trustworthy with how they want to present it. The psychologist guy especially will come up with some wildly detailed assertions about who the criminal is based on nothing
If a fictional-but-popular TV show treats some kinds of evidence as more reliable than they really are, juries may be primed to believe in the kind of thing the TV show presents as legitimate.
Sure but lawyers would know that and ensure evidence doesn't get presented that way right? There are also a lot of other biases that lawyers have to navigate through.
Humans are flawed but that doesn't mean everyone in the jury thinks TV is real.
It affects the jury. If the jury watches tv shows that builds the expectation that there is always a bunch of ballistics evidence etc and that it is always fool proof then they will 1) distrust when there isn’t that type of evidence (but enough other evidence) and 2) they will overvalue the evidence when it exists
It affects everybody. I've heard of people arrested in rather more oppressive regimes expecting to be Miranda'd because it's what they know from American cop shows and they thought it was broadly applicable everywhere.
There is a reason such shows are labeled "copaganda" - it affect people's perception of police and their procedures. It makes the dubious seem less dubious and more believable. I very highly doubt any jury is made aware of the rate of error or unreliability of the this stuff.
There used to be - probably still are - cameras that would digitally sign all their images. Used in crime scenes? Maybe we will end up seeing wider adoption of this, despite the privacy implications. Hackers attention then will focus (once again) on the certificate supply chain and crypto hardware.
What about a system that saves in some way the hash in a Blockchain, and if you, eg, XOR the hash of the video with the hash of the previous block you will "certainly" know that the video was created between the previous block and the block where the hash is saved in. That's a starting point.
Like when people discuss voting, I believe a blockchain [0] is a terrible pitfall compared to a classic distributed database system of predefined nodes run by different organizations. For example, imagine a couple hundred predefined nodes run by different states, federal agencies, etc.
An attacker altering the ledger would still require compromising an unreasonably large number of independent groups at once, and even then the rest would be able to clearly see that some unusual and suspicious event occurred.
By limiting membership a bunch of problems simply vanish, like long-clearing times, wasting hardware on mining, vulnerability to foreign botnets, etc.
[0] A blockchain is distinguished by its core requirement, from which a cascade complexity flows: Uncontrolled node membership. Don't be fooled by people pitching "private blockchain", its a contradiction in terms designed to rehabilitate hype, like "multi-sample Theranos test" or a bicycle as "Segway passively stabilized inline wheel model."
You just described IBM's whole Hyperledger Fabric thingy. I worked with it once upon a time, with the biggest insurance companies in my country where they plus a regulator all ran nodes.
it does something, sometimes. it pushes the required fabrication timeline back.
if it is mandated that every photo or video taken for the possible use in evidence is notarized at the time of acquisition, any fabrication would necessitate total premeditation. that is, the fabricators would need to know ahead of time what they were pursuing and what evidence they would need. this seems like a very costly barrier.
for example, altering security footage would require some fantastical elements: a real-time system of ingesting real footage and altering it in real-time to slip it into the notarization pipeline within the error margins.
requiring that any equipment that produces acceptable evidence stream commitment hashes in real-time to public append-only repositories would be an enormous step forward.
"Crack the hash"? Does this mean you were employing some novel hashing algorithm and relying on its secrecy? If so your employer were never serious about security in the first place. Hardware attestation is more or less a solved problem, and that solution does not involve secret algorithms.
Eh. It was some kind of hash of the image. I was not involved in that project, so can't tell you exactly how it worked, but the images were "signed," and someone figured out how to "re-sign" an altered image.
Sure but conceptually no one should've been able to crack any hashing scheme anyone half-way decent at their job could come up. SHA256 is the default and it's unbroken. Even SHA1 has scant few known collisions. So like...what the heck were they hashing and how that anyone was able to crack it?
Most likely is either extracting the private key from the camera or getting the camera to sign arbitrary data. If the signing isn't part of the sensor die itself there's a bus where the image data gets transferred from sensor to signer, so an attacker can inject arbitrary data onto that bus to get it signed, even though they never actually extract the signing key.
I imagine in this age of blockchains you could embed into a media file a signature that proved it was no older than the timestamp of when it occurred, the digital equivalent of a hostage-proof-of-life photo with a recent newspaper
But I don't know of a cryptographic mechanism to ensure that a digital image is not more recent than a particular time
> But I don't know of a cryptographic mechanism to ensure that a digital image is not more recent than a particular time
Many (most?) blockchain mechanisms include a timestamp in each transaction on the chain, so while multiple records from the same owner prove little (the timestamps could be faked over a given period of time) the interaction with the wider network and the chain would give some confidence that the record happened between within a small amount of time.
The other possibility, that doesn't require a chain with many independent active participants, is to have things signed by an external trusted authority. Submit a hash of the content and appropriate metadata to them, and have them sign it with a signing timestamp. I've considered abusing ACME certificates for document signing like that: the hash of content (or some signature based upon it) becomes the subdomain to sign¹ and you get a certificate that even after expiry is evidence that the CA saw that value at the signing timestamp. Note of the signing will also be in the public certificate transparency log. This wouldn't, on its own, prove anything about the authenticity of the content, that could have been doctored before signing, but it does prove that the content+metadata existed at that time (so might be more useful in copyright claim type cases, or agreed contract situations where all parties have signed the content and the signatures are included in the metadata, than for proving authenticity).
----------------
[1] based64²-ed with non-alphanumeric characters removed and truncated³ to fit or split, so acodha3sf7whsrhtqestkabtx0b4bbhyveee0ajnrpqcuxrjjvmhsujgcex.domain.tld or acodha3sf7whsrhtqestkabtx0b4bbhyveee0ajnrpqcuxrjjvmhsujgcex.w5jmmkpmyfgshx2jecsfordpnq.domain.tld
[2] names not being case-sensitive drops some of the entropy, if that is a concern use a 32-bits-per-character encoding instead and have names twice as long
There isn't a way to externally sign/validate a TXT record, that I know of.
A TXT record may be used in the signing process if using DNS verification, but you are getting a certificate for the name overall so using an A record to hold the hash you are looking to certify does the trick, then if you want to use HTTP-01 for verification rather than an extra DNS record you can.
You don't need a blockchain for that. You just need some reliable-enough way to publish hash(image) with a timestamp - some way that it's infeasible enough as to be considered impossible for thepublisher to change the hash or the date.
Back when I was on Twitter and following a lot of infosec accounts, it was quite common to see tweets that were just a hash. Sometimes they'd have an explanation "Zero click RCE in Android 10 - {hash}"
In the past I've used gmail for this internally - email a hash of something critical (logs, configurations, decision docs, etc) to a dedicated gmail account - relying on the in feasibility of "faking" the date/time once it's onb Google's servers.
The important thing here would be to make sure those hashes are published somewhere where its technically infeasible for the police to change it after the fact, so not on a platform the police run or p-aid for (or that is run or paid for by an organization that the police can request or coerce the operators to make changes).
You literally just need several oracles which sign hashes at the time they receive them and record that fact.
As a community service you need them to have enough scale that no individual hash or source can be tampered with without being likely to become known as unreliable to everyone else as well ala certificate transparency records.
(You could probably just bootstrap let's encrypt for this - issuing a certificate you use to sign a bunch of data would stamp several minimums on the order anything could have happened).
Interesting, There aren't any newspapers left in my country, neither printed nor not printed. The closest you can find is the weekly advertising booklet here and there. Which is irrelevant now because a computer can either stich new content to an old picture, or entirely producing a custom picture.
That would be a use case for a block chain. But I still don't understand how you are securing the integrity of the validity of the certificate stating the authenticity of the media. I only understand you are stamping media with a "at least as old as [timestamp]
If you want to prove that "happened at or after this timestamp" you can use a randomness beacon. NIST[0] and others publish a random number every N minutes. Embed that (or a combination) of those seeds to prove that you observed this value. This does not work for the harder problem of proving an event happened before a timestamp.
Seems like this idea solves a different problem than signed timestamps. You have access to not only the current random numbers, but also any random number from the past (as long as someone somewhere wrote it down). I just don't quite get what this could solve if you can either use a current number or an old number. Just not a future number because they're not around yet.
Embedding a public random number also doesn't resist tampering, unlike signed timestamps.
I suspect so. Tbh, I'm surprised it hasn't happened already with the amount of processing that cell phones do on photos, with generative fill/expand/perspective change, etc.
We are quickly going to reach a point where any photo or video taken on a smartphone is inadmissible by default.
You should see what people were capable of in the darkroom, let alone before all this. You could always manipulate imagery ever since there was imagery to manipulate.
The roll itself can be manipulated too. Most of the techniques used in modern photoshop are basically 1:1 carry overs of darkroom processes. Layers, dodge and burn, masking, etc.
There was a time you could take this class in highschool.
You try to equate several days of work, specialized equipment (much more than water and trash bag: you need chemicals, baths, special paper, a projector, plates...) and knowledge with typing a text in a webpage.
Have fun keeping making bad faith arguments alone.
You are getting caught up in how the sausage is made and not the fact that the sausage was always getting sold either way. So what if I can make 1 or a billion propaganda videos a day? The volume here does not matter. The end result is the same: populace in the pocket of propagandists just as much as it has ever been.
You can burn negatives. You can fake polaroids, really, just think about how a camera itself must operate and you'll see why instantly. Darkrooms used to be far more common before digital photography my Junior and High school both had them.
What makes evidence "pixel perfect?" What digital photographs don't have to involve a chain of custody? Literally the first question the defense will ask is "how did you get this picture." If you say you pulled from a security system they can just go ask for the originals. This happens all the time.
Where people are getting confused is it's almost never _one_ piece of evidence that's used to convict you; although, it may be a single piece of evidence which convinces your attourney to railroad you into a plea deal.
We've gone from highly skilled people being able to forge some specific photos and documents using substantial time/energy/resources, to any asshole being able to generate realistic full-motion video in minutes.
I get that there is a certain type of moron who thinks that the collapse in cost of misinformation has no harm... but all you've done is announce to the world that you are a moron.
It is really not any different. People would throw a hubcap in the air and pitch it as a UFO photo and idiots would latch on to that. You could take a photo of the empire state building and use a double exposure to make it look like you were king kong. Kids were doing this sort of stuff. Stop motion home movies where you'd look like you were levitating or your head got cut off.
No obviously not. But this is silly framing because there are so many things we do because it increases the effort for bad actors to do bad things. We close and lock our doors not because it prevents break-ins, but because that is a barrier that makes breaking in more inconvenient.
The end-game is that people will willingly surveil themselves 24/7 on behalf of The System because that will be the only way to prove what they didn't do.
I’m still shocked we have not seen an extremely convincing AI video of a famous person or world leader announcing something huge like UBI or WW3 or aliens.
It'll no doubt shift the probabilities but people have always lied and faked data. With video coming out of Ukraine there are a lot of fake things but beyond AI glitches you can check who the source is, if it correlates with known events and so on.
[The Derbyshire Police] declined to give more detail
about what the evidential material consisted of.
The term [evidential material] can be used to
describe witness statements.
I don't know if it's still the case in the UK, but in the common law and still in the US this why all substantive evidence, with very rare exception (e.g. dying statements), is witness testimony given on the stand. It may seem absurd when a witness or expert is given a transcript of an earlier statement or report just to recite it, but this is exactly why.
The loophole is all the powers the police and government have to more-or-less punish someone before a trial, or even before charges.
You'd also think that police blatantly lying (with or without AI) in official documents and/or in court under oath would trigger immediate firing and a ban of them ever serving in law enforcement in any capacity ever again. But no.
If charged, it would likely be as either forgery, perverting the course of justice, or perjury (or perhaps some combination of those) depending on the specifics.
If found guilty at trial, they'd be looking at a prison sentence as the abuse of position aspect would automatically mean high culpability. Expected starting point would be 4 years if an innocent person has been charged or convicted on the basis of the false evidence (which is implied by the report). Perhaps 6-7 years if multiple people have been. Very unlikely they'd ever be able to work in policing or related fields again.
Except none of that ever happens. When caught lying police just say it was an oopsie and pinky promise to do better next time. Prosecutors and judges do not charge police for these incidents, so they keep happening.
The headline evokes ideas of creating a video of a suspect perpetrating the crime but what I think is much more likely is the police officer used AI to enhance an image in a way that they considered innocuous, e.g: a photo was blurry so they “enhanced” it. Since “enhancing” is letting AI fill in the gaps it would be using AI to “create evidence”.
Regardless of what they did, tampering with evidence is completely unacceptable and should result in their dismissal and conviction but I don’t think the story will transpire to be as attention grabbing. A well meaning idiot could convince themselves that enhancing evidence is somehow justifiable whereas it would be almost impossible for even the most corrupt moron to justify creating evidence out of thin air.
Creating evidence out of thin air would be ridiculous because evidence is available to the defence who would be able to immediately identify if an image or video had been created (as the defendant would be able to recognize what they do or did not do) whereas “enhancing” an image could be easily spotted by other officers. “How come this photo is clearer than the last time I saw it?”
“Oh I ran it through ChatGPT to clean it up! Neat, eh? Just like on CSI!”
FWIW their employer is not the prosecution. UK police don't prosecute cases, the CPS (Crown Prosecution Service) does.
The word of a police officer, in UK law, is that of any other witness. There is a kind of presumption of regularity in the courts, but they don't have any sense of qualified immunity; they are generally but not universally considered not personally liable for negligence but that is not guaranteed them.
And unlike police departments in the USA they don't really have much latitude to experiment with technology. IMO they should be banned from using AI tools that aren't centrally provided.
Other than that, yes — I agree with your general view that this is an alarming state of affairs for people in a position of trust.
you're quite right, every single one of them is actively trying to kill each and every one of us. To consider any other possibility would be hysterical.
... Police in the united states have more than a century of flagrant misconduct under their belts. They protect their own, they almost never face consequences for killing people, they are frequently corrupt, they are frequently biased.
To be fair, this is Derbyshire in England. They are often a bit overkeen but they are not exactly Homan Square.
I think there have been less than two dozen police involved killings in the whole of the UK in the last six years, and that's in a population of seventy million people.
It's about 2% of the equivalent US figure (which averages 800 per year in 340 million people)
You could have looked it up before commenting. <5% is hardly any sort of "oh quite a few actually do". That rate pretty much halves in Scotland, with N.Ireland being a significant outlier as it has different rules.
UK police are decent at avoiding escalation as opposed to the US where they often _are_ the escalation. In England and Wales 24/25, two people died by being shot by the police. TWO. Ten year average is THREE.
If you think the police don't fabricate evidence on the regular, simply because their hunch doesn't match the fact or because they don't like the suspect, then you are way too gullible. Back in the day they just planted a baggie of drugs on you.
> Back in the day they just planted a baggie of drugs on you.
Thank god that never happens anymore. I'm sure the bodycam era has ended all of that misbehavior and one could not possibly go to YouTube and find videos of cops in possession of that unique blend of corruption and stupidity that would lead them to plant drugs while being recorded. Ahem.
> it would be almost impossible for even the most corrupt moron to justify creating evidence out of thin air.
Yet we have many examples of this precise thing happening. This is because the police carry immense credibility when testifying. This is also why the "Brady List" exists.
> the defence who would be able to immediately identify if an image or video had been created
How? Just pure skill? Again, we can see from appeals court proceedings, they miss details all the time. The system of "public defense" in the United States is severely lacking.
> what I think is much more likely is the police officer used AI to enhance an image in a way that they considered innocuous, e.g: a photo was blurry so they “enhanced” it
Doesn't iphones do this by default? The camera isn't actually that sharp, instead it fills in the details so it looks sharp, and sometimes it adds things that were never there. Can easily see it adding a gun in a blurry photo of someone.
So almost everyone uses AI to forge evidence then.
This exact situation came up during the Kyle Rittenhouse trial. The prosecution wanted to present an image of Rittenhouse. It was materially important whether or not the picture depicted Rittenhouse with a raised gun; I looked myself and couldn’t tell because it was a dark picture from a distance (if I remember correctly, it was a still from a drone video). Since the relevant part of the image was so small, the prosecution was going to zoom the photo in on an iPad, but the defense objected on the basis that the iPad zoom algorithms could be inserting detail not present in the original image.
This seemed a plausible enough objection to me. Although a fairly techy guy, I was (and am) not familiar with the specifics of Apple image processing, but at the time I had a vague awareness that Apple had been heavily advertising its use of AI algorithms to improve the quality of images. Whether that affects zooms specifically I don’t know—but it’s not an outlandish question.
The judge did an eminently reasonable thing: he disallowed the zoomed evidence on its own, but allowed it to be re‐entered if the prosecution provided an expert witness to testify that zooming the photo didn’t meaningfully change it. For this, he was pilloried by the tech media.
Take Ars Technica, for example: they used the headline “You shall not pinch to zoom: Rittenhouse trial judge disallows basic iPad feature,” and prominently displayed the judge’s words “I know less than anyone [about technology],” as if the right thing for a judge who knows nothing about technology to do would be to determine the merits of technical evidence on his own rather than ask for an expert witness. It’s not like it would have been hard for the prosecution to find one. https://arstechnica.com/tech-policy/2021/11/rittenhouse-tria...
Anyone who’s experimented with even non‐AI‐based upscalers knows that changing the algorithm can connect or disconnect catty‐cornered objects, introduce curves, and so on. I was shocked (well, not that shocked, given the heavily partisan interest in the case) that the tech media was so confident zooming an image couldn’t possibly meaningfully change it.
In the end, the image was displayed on a big‐screen TV (which probably used some other upscaling algorithm like bilinear, not that anyone in court was technical enough to point it out). The prosecution asked whether/asserted that Rittenhouse had raised his gun in the image, and Rittenhouse said it was not raised. So the exact details of how the small few pixels in the image got upscaled turned out to be pivotal in the end after all.
iPhones, no, there's no AI replacement or synthesis of objects from the camera. There were Android phones doing this (famously I think it was Samsung where it would replace images of the moon with a different image of the moon), and the Photos app has AI manipulation features. And most of the time, Apple's noise removal algorithm actually removes detail from images, most notably making text and straight lines wobbly.
> iPhones, no, there's no AI replacement or synthesis of objects from the camera
This is AI. Its not generative AI if that is what you mean, but it is AI altering the image and adding things that wasn't there, usually its fine sometimes it fails horribly and make the picture totally different.
No, that's not AI in the context you were claiming. They use ML techniques and ML-optimized algorithms for their image processing, which can be claimed under the general AI umbrella, but they certainly aren't generating elements of the images captures by the camera app, which is what you meant. The leaf example given in sibling comment has long been debunked, and it's literally the only example of generative content injection claimed for the iPhone camera.
My mind went straight to using the AI to write a statement and the AI made stuff up, which would be a nearly guaranteed outcome from using existing LLMs for that task, and it's exactly the sort of thing that I'm sure many officers are doing ... and it could go a fair time before it was discovered.
This is a very very intense claim, and if true, would represent a monumental institutional failure across hundreds or even thousands of disparate organizations.
When DNA matching was introduced, we discovered that at minimum 10% of people on death row were innocent. Death row cases are among the most litigated and examined cases. So, 10% is a reasonable floor, and we're already in double digits.
That stat is off by a couple orders of magnitude. The total number of death penalty convictions overturned by DNA evidence is 29 (as of 2025). There are a couple thousand death row inmates right now, and the denominator here is all the people who were on death row in the last 20+ years. That's a rate of significantly <1%.
"we discovered that at minimum 10% of people on death row were innocent"
How did we do that? I never heard this: certainly 10% of people on death row weren't exonerated by DNA? This is some kind of shaky extrapolation I assume?
When they choose the "DNA loci" to do SRT "matching" in the first place they convinced themselves it was a unique fingerprint and there never would be any duplicates in the database.
It only took a few years.
They've since changed and expanded the standard "DNA loci" to compensate.
A few years ago, one of my coworkers was arrested for a domestic violence complaint. Looking into his case, I found an extremely specific lurid description of the allegations -- and then I found the same lurid description copy/pasted to every other person recently arrested for the same crime. I'm probably getting the specific terms wrong, but I did click through to see it on a government website, because my first suspicion was the aggregator, but no, the police just had a boilerplate story full of specifics which could not possibly apply to each and every person they carelessly slapped it onto. This absolutely blew my mind at the time, but it fits with smaller subsequent observations. In any case: a double digit percentage of institutional failure does not upset my priors about how carefully the police operate.
Shouldn't it be the exact opposite here ? The burden of proof is the other way around.
The big claim is here: the state has grandiose claims that the overwhelming majority is fair, but there is no proof of it.
Therefore the state should prove that more than 90% of the cases are legitimate, fair, not coerced, and not motivated by the pressure to interrupt the proceedings.
97% of people choose plea deals or out-of-court settlement, it is a huge amount.
It means that in real practice, not imaginary internet, people who face court consider that justice is a big machine that can crush you no matter if you are innocent or not.
In the best case you are acquitted at the end, but you are guaranteed to bear the financial burden, fear and stress as a punishment.
Being held in jail before trial is a very convincing reason to plea deal too.
It's a system engineered to make pleading the only reasonable option, no matter if you did anything or not.
That is true--the checks and balances the founding fathers fought so hard for were thrown out the window with overlegislation and expansion of prosecutorial discretion in 20th century. To make a convincing argument that "double digits" of cases involve fabricated evidence, you still need to explain why prosecutors would engage in fraud at this massive scale. Just laziness? Collecting scalps? The incentives run that way in some limited cases, e.g., prosecutor up for election, post-reconstruction south. But you need some explanation there.
Yeah, again, there are some incentives to fabricate evidence like career advancement. Now why should those, on a mass scale, outweigh disincentives like getting caught in an adversarial process and (presumably) some qualm about regularly convicting innocents and regularly letting guilty parties run free in communities. Easy to argue in particular cases but I haven't heard the basis for a trend.
What adversarial process? If the prosecutor loses the case, the defendant doesn't go to jail but still receives a very big punishment and the prosecutor loses nothing. And prosecutors never prosecute themselves for false prosecution.
This sounds like you’re imagining how prosecutors as a group sort of feel about things, generally, and that this notion you’ve thought of outweighs the demonstrable real-world system where prosecutors are awarded for convictions, full stop.
I certainly didn't claim " this notion you’ve thought of outweighs the demonstrable real-world system ". However the original comment requires that " " this notion you’ve thought of [doesn't] outweighs the demonstrable real-world system ", and that's what I wanted evidence of. Don't shift the burden of proof onto me.
A burden of proof is associated with an individual claim. There’s no “burden of proof in the other direction” - what you’ve actually done is created a second burden of proof and also - worse - attempted to distract from the original point.
It is disingenuous to weasel out of proving one claim by making another, or saying “look over here”
Also, outrageous claims in opposite directions can both be bullshit.
That is like claiming that double digit percentage of software bugs and vulnerabilities were intentionally put there by malicious software engineers. Its outrageous to claim its that high.
Even single digit percent is hard to believe, but its possible, but double digits you are talking China or Russia levels of state corruption and even there I doubt its that high.
~~Please point to the place where I said your claim was outrageous.~~
Edit; upon closer examination. I did imply in my last paragraph that your claim was outrageous. Bit of a gaffe considering I’m the agitator here. My apologies.
An important thing you should recognize: the judicial system is painfully nontransparent in such a way that even figuring this sort of thing takes an extensive amount of time and is often even impossible. I've personally gone down a similar route (did some journalism for a bit) by trying to understand how shotspotter is used in prosecution, many of which resulted in false arrests and many, many years of life lost across all the people arrested falsely from it.
If you would like to begin trying to answer these, I recommend starting with submitting some FOIAs. Considering your stance seems to be that you won't believe what others are telling you -- I promise you that you'll be surprised.
If you believe parallel construction should be illegal (it sure seems like it is unconstitutional to me), then 100% of prosecutions that rely on it are unjust. I don't think anyone truly knows how common it is, though, and that's by design. Double-digits wouldn't shock me at all.
It's more monolithic than you would think due to shared culture over the internet. There's a whole narrative about sheepdogs (them), sheep (us) and wolves (the bad guys).
The main obstacle is the denominator. How many conversations were there (before dna testing) where testable tissue was preserved? You would want to test some random sample of them against the person convicted.
We have the highest proportion of imprisoned citizens in the world.
This is done because there's an exception in our constitution for slavery "as punishment for a crime" and well all know that capitalism loves slave labor.
Consider all of that can be used for forced confessions and forced plea bargains also. In those cases, the "evidence" doesn't even need to exist at all, or be on the record in any way.
Forensics is the art of manufacturing evidence to convict people.
Ok that's a bit aggressive, but it's important to understand the limits of forensic science, the broken incentives, and systemic lack of rigor in the application of various analyses and products to inculpate/exculpate suspects after the fact.
They are just rather over-eager and a bit of a law unto themselves in a rather silly way; it is Derbyshire police that hassled people walking in the open air during COVID, including rather excitably harrassing Peak District walkers with drones, and enforced rules that were only guidance as if they were law (famously asserting that two women walking in the fresh air with a coffee was "a picnic").
Them being all super-keen to use AI really fits. Some pillock of an officer going too far really fits.
Derbyshire is really safe but they act like it is not.
Can we know the motivation? Will it get them a bonus at the end of the year? Was it something common in the cases, maybe similar victims or something else?
I'd wager that it was just a shortcut to getting his work done. That banal motive is why we've seen an explosion of these cases and why they won't stop.
Why police (and media) cameras aren‘t forced to use camera hardware signing, aka content credentials, is beyond me.
It's because they're a scam. Point the camera at a forged image with a higher resolution than the camera sensor and it will make a signed copy of the unsigned forgery.
That's before getting into the practical problems with securing the keys. Every camera by every manufacturer has keys in it and the attacker only needs one key from one camera, and they get to choose the model? Creating something premised on needing to trust something with such a high probability of being compromised is worse than nothing, because it allows the ensuing forgeries a mechanism to pass themselves off as "signed" "real" images.
But what about if:
…the signature included the depth measured by the autofocus system across the image?
…or a tiny stereo image was included to capture depth?
…or a mini video in the ten seconds before and after the photo was taken?
…and the key is in a tamper proof HSM?
…and the key is deleted the moment the camera detects the case being taken apart?
I know that it is a losing battle to try to build such hardware when offline attackers have essentially infinite time to dismantle even the most elaborate systems — no such thing as an un breakable safe, only how long it takes to break into it, etc — but I feel these are valid counter measures, are they not?
I agree. Yes, these are not foolproof, but damn does it make it harder. It means that a random lone wolf using some random AI is not going to find it easy.
I would add a few more measures:
* Keys are regenerated for each device in the charging dock and are only valid until next recharge or a timeout.
* There is a sign-out process for the cameras that ties them to the operator.
* Police officers have no control over when the camera is recording, the camera instead controls this.
* Lower resolution data is streamed and synced to a cloud in real time, along with interesting data such as GPS, local BT/WiFi devices, etc.
As for privacy, British police are using more and more evasive camera technology out in public spaces, it's about time they were forced to wear it themselves. I want even the pencil pushers in the offices to be forced to wear it.
> the signature included the depth measured by the autofocus system across the image?
> or a tiny stereo image was included to capture depth?
These systems work by having multiple sensors to use for depth perception, so enterprising hackers write software to create two images, one for each sensor, and put some kind of lens or mirror in front of the camera to direct a different image/screen to each sensor.
The problem is fundamentally that the device is taking unsigned analog attacker-controlled input and then signing it, and is being mass produced. So whatever you're having it do, they put something that generates the same photon pattern in front of the device and you can't fix that with cryptography.
You can probably make it so that a cheap camera needs a few hundred dollars in optical glass or similar, and expensive camera needs a few thousand dollars worth, but it's hard to see how you could make it infeasible to anyone with non-trivial resources and it's also easy to mess up even worse and make it practical even for anyone with a computer and a high resolution screen or two.
> or a mini video in the ten seconds before and after the photo was taken?
Which does what if nothing in the image is expected to be moving, or the thing you're pointing the camera at is a screen rather than a piece of paper?
Also, now to verify the signature on your 50kB image you need a 2MB video? Then by default people won't distribute images that have the ability to be verified.
> and the key is in a tamper proof HSM?
Someone figures out a timing attack on the HSM or similar and now you can extract the keys from every device of that model. Happens over and over, the chances of every device getting this right are essentially zero.
> and the key is deleted the moment the camera detects the case being taken apart?
They get multiple cameras of the same model, take one apart to see how the detection works, then having figured out how it works, take the other one apart without triggering it. Or they extract the key without ever removing the case.
Also, now your phone is going to delete its keys when you remove the case to replace the battery or a cracked screen etc., or if the detection system has a false positive? Then you need some way to transfer new keys to a thing that hasn't got any, which is an even worse attack vector than not deleting the keys to begin with.
It's obviously a loosing battle. You're thinking of elaborate attackers, I'm thinking of your common policeman or scammer. Not great security, but might be able to solve the easyness problem introduced by GenAI.
Out of curiosity, what would be your proposal for identifying GenAI images and videos? Any suggestions?
But also what about .. Even now there is a range of forensic tech that can be used to statistically indicate if an image has been doctored, or generated, wouldnt't adding more and more real world data to the capture increase the bar for doctoring, so that only attackers with infinite resources can do it? At least it would stop Bobby Rotten from doing it.
I’ve done a short deep dive on this, for some cases that possibly would have went court. The tools we have today don’t reliable indicate if an image was doctored necessarily. Most open available scoring and tools like VAAS, DIRE, and Sherloq are decent today. Figuring out if an image that has been doctored, especially with solid proof, is only reliable if the image has metadata to prove it. If they export it to another format or screen capture it and the metadata is lost, it is purely still a guessing game.
The more guarantees you put in place the more people believe the system is infallible and the more valuable the exploit becomes.
If "signed" photos were treated as incontrovertible truth, then you'll just have people 3d printing hyper realistic masks or something.
Really good points, thank you for pointing this out. What I'm mostly thinking of is the usage in e.g., social media. There will be efforts to do what you suggest, but for your typical facebook user and scammers, this might help with the more common situations where people use GenAI.
The general population does not understand technology sufficiently well to set it up correctly, regulate it or use it correctly. Until we educate our population more on technology we will always be in this state.
it's a feature, not a bug
I would be interested in knowing both what kind of fabrication occurred, but perhaps I’m not curious about how it was discovered?
Did the defense use some sort of tool to debunk? Was it just an obvious deepfake etc? Or was it the officer’s ineptitude that got him caught?
Probably simply a case of "show me a picture of X with their fingers in the cookie jar".
My experiece with whatsapp family groups is that a lot of people over 40 can't detect the most obvious AI fakes (e.g. Studio Gibli clones, or three handed people), so they share the most stupid stuff as it was real, while youngers seem have an instinct to detect AI but can't tell exactly why they know.
I can picture a cop fabricating images that are obvious, even with a watermark included, while totally convinced that it is undetectable.
IME it’s more like 60, which then makes more sense given vision deterioration. Then again most of my over-40 experience is with folks in the tech industry.
The ai slop uses movie compositions in whats supossed to be handheld shots?
I like using professional cameras and lighting for casual photos, I guess everyone now assumes it’s AI.
Which makes sense but still, ffs.
i do wonder, that in the age where we have image and video creation out of the bag, whether or not this will result in whole classes of evidence becoming completely unreliable.
There's a big gap between "theoretically unreliable" and courts actually recognizing that, unfortunately. Lots of forensics is much more dubious than CSI would have you believe.
15 years ago an Israeli company was able to manufacture fake DNA evidence https://www.nytimes.com/2009/08/18/science/18dna.html
My girlfriends been having me watch law and order svu with her and to be honest it doesn't really even seem trustworthy with how they want to present it. The psychologist guy especially will come up with some wildly detailed assertions about who the criminal is based on nothing
Are we really going to go to a fictional TV show now?
If a fictional-but-popular TV show treats some kinds of evidence as more reliable than they really are, juries may be primed to believe in the kind of thing the TV show presents as legitimate.
https://en.wikipedia.org/wiki/CSI_effect
Sure but lawyers would know that and ensure evidence doesn't get presented that way right? There are also a lot of other biases that lawyers have to navigate through.
Humans are flawed but that doesn't mean everyone in the jury thinks TV is real.
It affects the jury. If the jury watches tv shows that builds the expectation that there is always a bunch of ballistics evidence etc and that it is always fool proof then they will 1) distrust when there isn’t that type of evidence (but enough other evidence) and 2) they will overvalue the evidence when it exists
It affects everybody. I've heard of people arrested in rather more oppressive regimes expecting to be Miranda'd because it's what they know from American cop shows and they thought it was broadly applicable everywhere.
There is a reason such shows are labeled "copaganda" - it affect people's perception of police and their procedures. It makes the dubious seem less dubious and more believable. I very highly doubt any jury is made aware of the rate of error or unreliability of the this stuff.
“Lots of forensics is much more dubious than CSI would have you believe.” was what was being replied to.
Sure you aren't watching Psych?
There used to be - probably still are - cameras that would digitally sign all their images. Used in crime scenes? Maybe we will end up seeing wider adoption of this, despite the privacy implications. Hackers attention then will focus (once again) on the certificate supply chain and crypto hardware.
I worked for a company that made these. We sold expensive software to the FBI.
Took about six months for someone to crack the hash.
What about a system that saves in some way the hash in a Blockchain, and if you, eg, XOR the hash of the video with the hash of the previous block you will "certainly" know that the video was created between the previous block and the block where the hash is saved in. That's a starting point.
This sort of chain doesn't need PoW I take it, just a very secure police server to sign blocks.
And it couldn't be run by the police or any of their friends, since they're the adversary.
Might have a point. This was before blockchain.
I suspect that the cops wouldn’t like the chain public, though.
Like when people discuss voting, I believe a blockchain [0] is a terrible pitfall compared to a classic distributed database system of predefined nodes run by different organizations. For example, imagine a couple hundred predefined nodes run by different states, federal agencies, etc.
An attacker altering the ledger would still require compromising an unreasonably large number of independent groups at once, and even then the rest would be able to clearly see that some unusual and suspicious event occurred.
By limiting membership a bunch of problems simply vanish, like long-clearing times, wasting hardware on mining, vulnerability to foreign botnets, etc.
[0] A blockchain is distinguished by its core requirement, from which a cascade complexity flows: Uncontrolled node membership. Don't be fooled by people pitching "private blockchain", its a contradiction in terms designed to rehabilitate hype, like "multi-sample Theranos test" or a bicycle as "Segway passively stabilized inline wheel model."
You just described IBM's whole Hyperledger Fabric thingy. I worked with it once upon a time, with the biggest insurance companies in my country where they plus a regulator all ran nodes.
that does nothing to verify authenticity
it does something, sometimes. it pushes the required fabrication timeline back.
if it is mandated that every photo or video taken for the possible use in evidence is notarized at the time of acquisition, any fabrication would necessitate total premeditation. that is, the fabricators would need to know ahead of time what they were pursuing and what evidence they would need. this seems like a very costly barrier.
for example, altering security footage would require some fantastical elements: a real-time system of ingesting real footage and altering it in real-time to slip it into the notarization pipeline within the error margins.
requiring that any equipment that produces acceptable evidence stream commitment hashes in real-time to public append-only repositories would be an enormous step forward.
"Crack the hash"? Does this mean you were employing some novel hashing algorithm and relying on its secrecy? If so your employer were never serious about security in the first place. Hardware attestation is more or less a solved problem, and that solution does not involve secret algorithms.
Eh. It was some kind of hash of the image. I was not involved in that project, so can't tell you exactly how it worked, but the images were "signed," and someone figured out how to "re-sign" an altered image.
I think it was a fairly well-known technique.
Which still sounds like your employer was simply incompetent because why was any type of perceptual hashing scheme even involved?
Signing digital data with hardware secure tokens is a commodity capability in the iPhone many of HNs users are reading this site with.
> your employer was simply incompetent
You’re probably right. This is easy, basic stuff that any recent college grad can do with their eyes closed.
I think this has been around for not so long
https://en.wikipedia.org/wiki/Content_Authenticity_Initiativ...
Sure but conceptually no one should've been able to crack any hashing scheme anyone half-way decent at their job could come up. SHA256 is the default and it's unbroken. Even SHA1 has scant few known collisions. So like...what the heck were they hashing and how that anyone was able to crack it?
Maybe its more like the hash was a well known secure hash but someone managed to extract the salt/private key/signing certificate from the camera?
Most likely is either extracting the private key from the camera or getting the camera to sign arbitrary data. If the signing isn't part of the sensor die itself there's a bus where the image data gets transferred from sensor to signer, so an attacker can inject arbitrary data onto that bus to get it signed, even though they never actually extract the signing key.
This was quite a while, before that.
Now sell them version 2.
I imagine in this age of blockchains you could embed into a media file a signature that proved it was no older than the timestamp of when it occurred, the digital equivalent of a hostage-proof-of-life photo with a recent newspaper
But I don't know of a cryptographic mechanism to ensure that a digital image is not more recent than a particular time
> But I don't know of a cryptographic mechanism to ensure that a digital image is not more recent than a particular time
Many (most?) blockchain mechanisms include a timestamp in each transaction on the chain, so while multiple records from the same owner prove little (the timestamps could be faked over a given period of time) the interaction with the wider network and the chain would give some confidence that the record happened between within a small amount of time.
The other possibility, that doesn't require a chain with many independent active participants, is to have things signed by an external trusted authority. Submit a hash of the content and appropriate metadata to them, and have them sign it with a signing timestamp. I've considered abusing ACME certificates for document signing like that: the hash of content (or some signature based upon it) becomes the subdomain to sign¹ and you get a certificate that even after expiry is evidence that the CA saw that value at the signing timestamp. Note of the signing will also be in the public certificate transparency log. This wouldn't, on its own, prove anything about the authenticity of the content, that could have been doctored before signing, but it does prove that the content+metadata existed at that time (so might be more useful in copyright claim type cases, or agreed contract situations where all parties have signed the content and the signatures are included in the metadata, than for proving authenticity).
----------------
[1] based64²-ed with non-alphanumeric characters removed and truncated³ to fit or split, so acodha3sf7whsrhtqestkabtx0b4bbhyveee0ajnrpqcuxrjjvmhsujgcex.domain.tld or acodha3sf7whsrhtqestkabtx0b4bbhyveee0ajnrpqcuxrjjvmhsujgcex.w5jmmkpmyfgshx2jecsfordpnq.domain.tld
[2] names not being case-sensitive drops some of the entropy, if that is a concern use a 32-bits-per-character encoding instead and have names twice as long
I suppose you could put a hash of the content into a TXT record
There isn't a way to externally sign/validate a TXT record, that I know of.
A TXT record may be used in the signing process if using DNS verification, but you are getting a certificate for the name overall so using an A record to hold the hash you are looking to certify does the trick, then if you want to use HTTP-01 for verification rather than an extra DNS record you can.
Publish hash(image) on the blockchain at a verifiable time, then publish the image itself.
The image contains the previous block’s hash.
Wouldn’t this establish both a lower bound and an upper bound on the time the image could have been produced?
You don't need a blockchain for that. You just need some reliable-enough way to publish hash(image) with a timestamp - some way that it's infeasible enough as to be considered impossible for thepublisher to change the hash or the date.
Back when I was on Twitter and following a lot of infosec accounts, it was quite common to see tweets that were just a hash. Sometimes they'd have an explanation "Zero click RCE in Android 10 - {hash}"
In the past I've used gmail for this internally - email a hash of something critical (logs, configurations, decision docs, etc) to a dedicated gmail account - relying on the in feasibility of "faking" the date/time once it's onb Google's servers.
The important thing here would be to make sure those hashes are published somewhere where its technically infeasible for the police to change it after the fact, so not on a platform the police run or p-aid for (or that is run or paid for by an organization that the police can request or coerce the operators to make changes).
You literally just need several oracles which sign hashes at the time they receive them and record that fact.
As a community service you need them to have enough scale that no individual hash or source can be tampered with without being likely to become known as unreliable to everyone else as well ala certificate transparency records.
(You could probably just bootstrap let's encrypt for this - issuing a certificate you use to sign a bunch of data would stamp several minimums on the order anything could have happened).
Interesting, There aren't any newspapers left in my country, neither printed nor not printed. The closest you can find is the weekly advertising booklet here and there. Which is irrelevant now because a computer can either stich new content to an old picture, or entirely producing a custom picture.
That would be a use case for a block chain. But I still don't understand how you are securing the integrity of the validity of the certificate stating the authenticity of the media. I only understand you are stamping media with a "at least as old as [timestamp]
If you want to prove that "happened at or after this timestamp" you can use a randomness beacon. NIST[0] and others publish a random number every N minutes. Embed that (or a combination) of those seeds to prove that you observed this value. This does not work for the harder problem of proving an event happened before a timestamp.
[0] https://csrc.nist.gov/projects/interoperable-randomness-beac...
Seems like this idea solves a different problem than signed timestamps. You have access to not only the current random numbers, but also any random number from the past (as long as someone somewhere wrote it down). I just don't quite get what this could solve if you can either use a current number or an old number. Just not a future number because they're not around yet.
Embedding a public random number also doesn't resist tampering, unlike signed timestamps.
Thanks - this is the perfect example of how to do this
Which country no longer has newspapers?
wouldn't that be a hash of the image signed by a trusted entity and stored on a chain? maybe i'm overlooking why this doesn't work
I suspect so. Tbh, I'm surprised it hasn't happened already with the amount of processing that cell phones do on photos, with generative fill/expand/perspective change, etc.
We are quickly going to reach a point where any photo or video taken on a smartphone is inadmissible by default.
You should see what people were capable of in the darkroom, let alone before all this. You could always manipulate imagery ever since there was imagery to manipulate.
This is why:
- the whole roll of negatives was prime evidence;
- police forces were one of the biggest users of Polaroid instant film.
And moreover, who had a darkroom and the skills to edit substantially a picture?
Whereas here we have nobodies being able to generate pixel-perfect fake "evidence" from the computers they already have.
Plenty of people. If you have running water, some tape, and trashbags, you too could have a darkroom.
https://www.metmuseum.org/exhibitions/objects/objects@exhibi...
The roll itself can be manipulated too. Most of the techniques used in modern photoshop are basically 1:1 carry overs of darkroom processes. Layers, dodge and burn, masking, etc.
There was a time you could take this class in highschool.
You try to equate several days of work, specialized equipment (much more than water and trash bag: you need chemicals, baths, special paper, a projector, plates...) and knowledge with typing a text in a webpage.
Have fun keeping making bad faith arguments alone.
You are getting caught up in how the sausage is made and not the fact that the sausage was always getting sold either way. So what if I can make 1 or a billion propaganda videos a day? The volume here does not matter. The end result is the same: populace in the pocket of propagandists just as much as it has ever been.
You can burn negatives. You can fake polaroids, really, just think about how a camera itself must operate and you'll see why instantly. Darkrooms used to be far more common before digital photography my Junior and High school both had them.
What makes evidence "pixel perfect?" What digital photographs don't have to involve a chain of custody? Literally the first question the defense will ask is "how did you get this picture." If you say you pulled from a security system they can just go ask for the originals. This happens all the time.
Where people are getting confused is it's almost never _one_ piece of evidence that's used to convict you; although, it may be a single piece of evidence which convinces your attourney to railroad you into a plea deal.
You are just too young to know.
https://rangefinderforum.com/threads/gentlemen%E2%80%93and-c...
We've gone from highly skilled people being able to forge some specific photos and documents using substantial time/energy/resources, to any asshole being able to generate realistic full-motion video in minutes.
I get that there is a certain type of moron who thinks that the collapse in cost of misinformation has no harm... but all you've done is announce to the world that you are a moron.
It is really not any different. People would throw a hubcap in the air and pitch it as a UFO photo and idiots would latch on to that. You could take a photo of the empire state building and use a double exposure to make it look like you were king kong. Kids were doing this sort of stuff. Stop motion home movies where you'd look like you were levitating or your head got cut off.
It always comes down to provenance.
People are just lining up to announce that they're fucking idiots.
Big difference between that and writing an AI prompt.
Not really. End result is the same: manipulated image.
Are we really pretending like the effort to do something doesn't affect how often that thing occurs?
Are we acting like that was ever a limiting factor towards disseminating propaganda in the analog age?
No obviously not. But this is silly framing because there are so many things we do because it increases the effort for bad actors to do bad things. We close and lock our doors not because it prevents break-ins, but because that is a barrier that makes breaking in more inconvenient.
We close and lock our doors to keep people undergoing psychosis out. Burglars are not stopped by locks.
How many people could do that?
How long did it take?
Now it’s a lot easier and faster
Plenty could.
https://www.metmuseum.org/exhibitions/objects/objects@exhibi...
Compared to now
https://data.worldbank.org/indicator/IT.NET.USER.ZS
The end-game is that people will willingly surveil themselves 24/7 on behalf of The System because that will be the only way to prove what they didn't do.
Ah yes training the AI with more data to represent me even more accurately.
I’m still shocked we have not seen an extremely convincing AI video of a famous person or world leader announcing something huge like UBI or WW3 or aliens.
Surely it’s just a matter of time.
They're out there, recommending scam investments / crypto coins more often than major world events.
Meta, for one, is keen to bury such things and avoid responsibility for ad contents: https://www.abc.net.au/news/2026-04-17/andrew-forrest-battle...
Oh, I assumed they were already out there in the sea of slop like the Iran Lego propaganda tiktoks.
Some people still believe in polygraphs.
It'll no doubt shift the probabilities but people have always lied and faked data. With video coming out of Ukraine there are a lot of fake things but beyond AI glitches you can check who the source is, if it correlates with known events and so on.
per ft.com: https://archive.fo/BIOej
I don't know if it's still the case in the UK, but in the common law and still in the US this why all substantive evidence, with very rare exception (e.g. dying statements), is witness testimony given on the stand. It may seem absurd when a witness or expert is given a transcript of an earlier statement or report just to recite it, but this is exactly why.
The loophole is all the powers the police and government have to more-or-less punish someone before a trial, or even before charges.
Such a case should trigger a auto revision on all cases said officer ever touched.
You'd also think that police blatantly lying (with or without AI) in official documents and/or in court under oath would trigger immediate firing and a ban of them ever serving in law enforcement in any capacity ever again. But no.
If charged, it would likely be as either forgery, perverting the course of justice, or perjury (or perhaps some combination of those) depending on the specifics.
If found guilty at trial, they'd be looking at a prison sentence as the abuse of position aspect would automatically mean high culpability. Expected starting point would be 4 years if an innocent person has been charged or convicted on the basis of the false evidence (which is implied by the report). Perhaps 6-7 years if multiple people have been. Very unlikely they'd ever be able to work in policing or related fields again.
Except none of that ever happens. When caught lying police just say it was an oopsie and pinky promise to do better next time. Prosecutors and judges do not charge police for these incidents, so they keep happening.
The headline evokes ideas of creating a video of a suspect perpetrating the crime but what I think is much more likely is the police officer used AI to enhance an image in a way that they considered innocuous, e.g: a photo was blurry so they “enhanced” it. Since “enhancing” is letting AI fill in the gaps it would be using AI to “create evidence”.
Regardless of what they did, tampering with evidence is completely unacceptable and should result in their dismissal and conviction but I don’t think the story will transpire to be as attention grabbing. A well meaning idiot could convince themselves that enhancing evidence is somehow justifiable whereas it would be almost impossible for even the most corrupt moron to justify creating evidence out of thin air.
Creating evidence out of thin air would be ridiculous because evidence is available to the defence who would be able to immediately identify if an image or video had been created (as the defendant would be able to recognize what they do or did not do) whereas “enhancing” an image could be easily spotted by other officers. “How come this photo is clearer than the last time I saw it?” “Oh I ran it through ChatGPT to clean it up! Neat, eh? Just like on CSI!”
That is a lot of words just to say "fabricating evidence".
Their word is evidence and their employer is the prosecution. This is the fabric of prosperity.
FWIW their employer is not the prosecution. UK police don't prosecute cases, the CPS (Crown Prosecution Service) does.
The word of a police officer, in UK law, is that of any other witness. There is a kind of presumption of regularity in the courts, but they don't have any sense of qualified immunity; they are generally but not universally considered not personally liable for negligence but that is not guaranteed them.
And unlike police departments in the USA they don't really have much latitude to experiment with technology. IMO they should be banned from using AI tools that aren't centrally provided.
Other than that, yes — I agree with your general view that this is an alarming state of affairs for people in a position of trust.
https://www.youtube.com/watch?v=EMBkWtDAPBY De-crop
If we want to solve a problem, intent matters.
Yes, let's please give police officers the copious (cop-ious?) benefit of the doubt they have earnt.
Honestly, I didn't tell it to add that gun to the picture, it did that on its own!
you're quite right, every single one of them is actively trying to kill each and every one of us. To consider any other possibility would be hysterical.
I see what you did there[1]
[1] https://en.wikipedia.org/wiki/Hysteresis
... Police in the united states have more than a century of flagrant misconduct under their belts. They protect their own, they almost never face consequences for killing people, they are frequently corrupt, they are frequently biased.
Here's a couple fun examples:
https://www.wgbh.org/news/local/2026-06-05/boston-law-enforc...
https://www.reuters.com/legal/government/la-is-investigating...
To be fair, this is Derbyshire in England. They are often a bit overkeen but they are not exactly Homan Square.
I think there have been less than two dozen police involved killings in the whole of the UK in the last six years, and that's in a population of seventy million people.
It's about 2% of the equivalent US figure (which averages 800 per year in 340 million people)
It doesn't make much difference if you perform similar practices in a different country. UK police are just like American police.
Apart from the bit about killing people, for the most part.
who are exactly the same as the Russki police, right? Bloody hell...
This is the UK, our cops don't even carry guns.
Quite a few actually do.
You could have looked it up before commenting. <5% is hardly any sort of "oh quite a few actually do". That rate pretty much halves in Scotland, with N.Ireland being a significant outlier as it has different rules.
UK police are decent at avoiding escalation as opposed to the US where they often _are_ the escalation. In England and Wales 24/25, two people died by being shot by the police. TWO. Ten year average is THREE.
If you think the police don't fabricate evidence on the regular, simply because their hunch doesn't match the fact or because they don't like the suspect, then you are way too gullible. Back in the day they just planted a baggie of drugs on you.
> Back in the day they just planted a baggie of drugs on you.
Thank god that never happens anymore. I'm sure the bodycam era has ended all of that misbehavior and one could not possibly go to YouTube and find videos of cops in possession of that unique blend of corruption and stupidity that would lead them to plant drugs while being recorded. Ahem.
> it would be almost impossible for even the most corrupt moron to justify creating evidence out of thin air.
Yet we have many examples of this precise thing happening. This is because the police carry immense credibility when testifying. This is also why the "Brady List" exists.
> the defence who would be able to immediately identify if an image or video had been created
How? Just pure skill? Again, we can see from appeals court proceedings, they miss details all the time. The system of "public defense" in the United States is severely lacking.
It matters little what you think, if that’s not what happened.
> what I think is much more likely is the police officer used AI to enhance an image in a way that they considered innocuous, e.g: a photo was blurry so they “enhanced” it
Doesn't iphones do this by default? The camera isn't actually that sharp, instead it fills in the details so it looks sharp, and sometimes it adds things that were never there. Can easily see it adding a gun in a blurry photo of someone.
So almost everyone uses AI to forge evidence then.
This exact situation came up during the Kyle Rittenhouse trial. The prosecution wanted to present an image of Rittenhouse. It was materially important whether or not the picture depicted Rittenhouse with a raised gun; I looked myself and couldn’t tell because it was a dark picture from a distance (if I remember correctly, it was a still from a drone video). Since the relevant part of the image was so small, the prosecution was going to zoom the photo in on an iPad, but the defense objected on the basis that the iPad zoom algorithms could be inserting detail not present in the original image.
This seemed a plausible enough objection to me. Although a fairly techy guy, I was (and am) not familiar with the specifics of Apple image processing, but at the time I had a vague awareness that Apple had been heavily advertising its use of AI algorithms to improve the quality of images. Whether that affects zooms specifically I don’t know—but it’s not an outlandish question.
The judge did an eminently reasonable thing: he disallowed the zoomed evidence on its own, but allowed it to be re‐entered if the prosecution provided an expert witness to testify that zooming the photo didn’t meaningfully change it. For this, he was pilloried by the tech media.
Take Ars Technica, for example: they used the headline “You shall not pinch to zoom: Rittenhouse trial judge disallows basic iPad feature,” and prominently displayed the judge’s words “I know less than anyone [about technology],” as if the right thing for a judge who knows nothing about technology to do would be to determine the merits of technical evidence on his own rather than ask for an expert witness. It’s not like it would have been hard for the prosecution to find one. https://arstechnica.com/tech-policy/2021/11/rittenhouse-tria...
Anyone who’s experimented with even non‐AI‐based upscalers knows that changing the algorithm can connect or disconnect catty‐cornered objects, introduce curves, and so on. I was shocked (well, not that shocked, given the heavily partisan interest in the case) that the tech media was so confident zooming an image couldn’t possibly meaningfully change it.
In the end, the image was displayed on a big‐screen TV (which probably used some other upscaling algorithm like bilinear, not that anyone in court was technical enough to point it out). The prosecution asked whether/asserted that Rittenhouse had raised his gun in the image, and Rittenhouse said it was not raised. So the exact details of how the small few pixels in the image got upscaled turned out to be pivotal in the end after all.
iPhones, no, there's no AI replacement or synthesis of objects from the camera. There were Android phones doing this (famously I think it was Samsung where it would replace images of the moon with a different image of the moon), and the Photos app has AI manipulation features. And most of the time, Apple's noise removal algorithm actually removes detail from images, most notably making text and straight lines wobbly.
> iPhones, no, there's no AI replacement or synthesis of objects from the camera
This is AI. Its not generative AI if that is what you mean, but it is AI altering the image and adding things that wasn't there, usually its fine sometimes it fails horribly and make the picture totally different.
https://x.com/mitchcohen/status/1476351601862483968
https://x.com/mitchcohen/status/1476951534160257026?s=20 The replies explain that it was a leaf obscuring the face
But it used AI to stitch that onto the body, a raw camera shot wouldn't look like that.
that is not what happened, no.
No, that's not AI in the context you were claiming. They use ML techniques and ML-optimized algorithms for their image processing, which can be claimed under the general AI umbrella, but they certainly aren't generating elements of the images captures by the camera app, which is what you meant. The leaf example given in sibling comment has long been debunked, and it's literally the only example of generative content injection claimed for the iPhone camera.
Yes, iphones do process images using AI.
And it is possible that resulting image is a better match to reality than the raw data coming out of the camera .
What would the defense do with fabricated evidence? Say that evidence is fabricated? Okay, the prosecution will say it's not fabricated, now what?
The judge will decide if the evidence is admissible or not.
There is a tremendous amount of cases you can look up where cops wholesale fabricate evidence. Why wouldn't they use chatgpt to do it as well?
> but what I think is much more likely
My mind went straight to using the AI to write a statement and the AI made stuff up, which would be a nearly guaranteed outcome from using existing LLMs for that task, and it's exactly the sort of thing that I'm sure many officers are doing ... and it could go a fair time before it was discovered.
Maybe use the word “falsify”?
Long a police practice (it happened to me) that AI makes easier.
I wonder how many people have been unjustly imprisoned between planted evidence, made up evidence, and illegal parallel construction…
Here in the US? Probably a large double-digit percentage of cases imo…
> large double-digit percentage
This is a very very intense claim, and if true, would represent a monumental institutional failure across hundreds or even thousands of disparate organizations.
Do you have any data to support your hunch?
Strong claims require strong evidence.
When DNA matching was introduced, we discovered that at minimum 10% of people on death row were innocent. Death row cases are among the most litigated and examined cases. So, 10% is a reasonable floor, and we're already in double digits.
That stat is off by a couple orders of magnitude. The total number of death penalty convictions overturned by DNA evidence is 29 (as of 2025). There are a couple thousand death row inmates right now, and the denominator here is all the people who were on death row in the last 20+ years. That's a rate of significantly <1%.
https://deathpenaltyinfo.org/first-death-row-exoneration-inv...
Shouldn't the denominator be the number of people actually executed ? 691 in the last 20 years, for instance ?
it doesn't matter if it's 29 or 2900. Even 1 is wrong.
The commenter isn't litigating that claim, they are litigating the claim that at least 1 out of 10 of those on death row were false.
"we discovered that at minimum 10% of people on death row were innocent"
How did we do that? I never heard this: certainly 10% of people on death row weren't exonerated by DNA? This is some kind of shaky extrapolation I assume?
When they choose the "DNA loci" to do SRT "matching" in the first place they convinced themselves it was a unique fingerprint and there never would be any duplicates in the database.
It only took a few years.
They've since changed and expanded the standard "DNA loci" to compensate.
> This reasonably sets a floor
I disagree wrt reasonableness. It’s just too big a leap. There are a lot of crimes, and not many land you on death row.
The observation was that death row represents the highest level of scrutiny, and still had 10% false positives for guilt.
Is there any argument that less-scrutinized cases would have a lower level of false convictions?
The 10% claim has been refuted.
What % is it?
Read other replies. I don't think we have enough data to give an opinion exact number but less than 1
Why would you expect those cases to have lower rates of false conviction than normal cases?
Hoo boy, welcome to the history of the United states.
A few years ago, one of my coworkers was arrested for a domestic violence complaint. Looking into his case, I found an extremely specific lurid description of the allegations -- and then I found the same lurid description copy/pasted to every other person recently arrested for the same crime. I'm probably getting the specific terms wrong, but I did click through to see it on a government website, because my first suspicion was the aggregator, but no, the police just had a boilerplate story full of specifics which could not possibly apply to each and every person they carelessly slapped it onto. This absolutely blew my mind at the time, but it fits with smaller subsequent observations. In any case: a double digit percentage of institutional failure does not upset my priors about how carefully the police operate.
The complaint would be by the courts, not the police.
My mistake, and that's too bad - my priors about courts were higher than the police, and this decreases them. Thanks for the correction.
Why did you "look into the case of your coworker"?
You wouldn't be curious?
Can you put some of the text in a comment here?
Shouldn't it be the exact opposite here ? The burden of proof is the other way around.
The big claim is here: the state has grandiose claims that the overwhelming majority is fair, but there is no proof of it.
Therefore the state should prove that more than 90% of the cases are legitimate, fair, not coerced, and not motivated by the pressure to interrupt the proceedings.
97% of people choose plea deals or out-of-court settlement, it is a huge amount.
It means that in real practice, not imaginary internet, people who face court consider that justice is a big machine that can crush you no matter if you are innocent or not.
In the best case you are acquitted at the end, but you are guaranteed to bear the financial burden, fear and stress as a punishment.
Being held in jail before trial is a very convincing reason to plea deal too.
It's a system engineered to make pleading the only reasonable option, no matter if you did anything or not.
That is true--the checks and balances the founding fathers fought so hard for were thrown out the window with overlegislation and expansion of prosecutorial discretion in 20th century. To make a convincing argument that "double digits" of cases involve fabricated evidence, you still need to explain why prosecutors would engage in fraud at this massive scale. Just laziness? Collecting scalps? The incentives run that way in some limited cases, e.g., prosecutor up for election, post-reconstruction south. But you need some explanation there.
They get rewarded based on winning cases?
Yeah, again, there are some incentives to fabricate evidence like career advancement. Now why should those, on a mass scale, outweigh disincentives like getting caught in an adversarial process and (presumably) some qualm about regularly convicting innocents and regularly letting guilty parties run free in communities. Easy to argue in particular cases but I haven't heard the basis for a trend.
What adversarial process? If the prosecutor loses the case, the defendant doesn't go to jail but still receives a very big punishment and the prosecutor loses nothing. And prosecutors never prosecute themselves for false prosecution.
adversarial process = anglo-american trial system = a defendant has an attorney who is looking for prosecutorial misconduct
> (presumably) some qualm
This sounds like you’re imagining how prosecutors as a group sort of feel about things, generally, and that this notion you’ve thought of outweighs the demonstrable real-world system where prosecutors are awarded for convictions, full stop.
I certainly didn't claim " this notion you’ve thought of outweighs the demonstrable real-world system ". However the original comment requires that " " this notion you’ve thought of [doesn't] outweighs the demonstrable real-world system ", and that's what I wanted evidence of. Don't shift the burden of proof onto me.
> Shouldn't it be the exact opposite here ? The burden of proof is the other way around
That's the rule for criminal court in the US, but each of us is free to pick his own standard for his own purposes.
A burden of proof is associated with an individual claim. There’s no “burden of proof in the other direction” - what you’ve actually done is created a second burden of proof and also - worse - attempted to distract from the original point.
It is disingenuous to weasel out of proving one claim by making another, or saying “look over here”
Also, outrageous claims in opposite directions can both be bullshit.
On what basis is it an outrageous claim? You think the number is closer to 0? That sounds like a more outrageous claim to me.
That is like claiming that double digit percentage of software bugs and vulnerabilities were intentionally put there by malicious software engineers. Its outrageous to claim its that high.
Even single digit percent is hard to believe, but its possible, but double digits you are talking China or Russia levels of state corruption and even there I doubt its that high.
~~Please point to the place where I said your claim was outrageous.~~
Edit; upon closer examination. I did imply in my last paragraph that your claim was outrageous. Bit of a gaffe considering I’m the agitator here. My apologies.
An important thing you should recognize: the judicial system is painfully nontransparent in such a way that even figuring this sort of thing takes an extensive amount of time and is often even impossible. I've personally gone down a similar route (did some journalism for a bit) by trying to understand how shotspotter is used in prosecution, many of which resulted in false arrests and many, many years of life lost across all the people arrested falsely from it.
If you would like to begin trying to answer these, I recommend starting with submitting some FOIAs. Considering your stance seems to be that you won't believe what others are telling you -- I promise you that you'll be surprised.
If you believe parallel construction should be illegal (it sure seems like it is unconstitutional to me), then 100% of prosecutions that rely on it are unjust. I don't think anyone truly knows how common it is, though, and that's by design. Double-digits wouldn't shock me at all.
Police in the United States is already in a state of "institutional failure"...
“Police in the United States” is not a monolith.
It’s easy to say things that sound true on the surface, but even if true, it’s still irresponsible to say them on the back of a hunch.
It's more monolithic than you would think due to shared culture over the internet. There's a whole narrative about sheepdogs (them), sheep (us) and wolves (the bad guys).
I don’t know the numbers, but DNA exonerations give a bit of a natural experiment (where testable evidence was preserved).
They give a floor, and that floor is too small to be useful.
The main obstacle is the denominator. How many conversations were there (before dna testing) where testable tissue was preserved? You would want to test some random sample of them against the person convicted.
Do you have any exposure to the criminal "justice" system in the USA?
We have the highest proportion of imprisoned citizens in the world.
This is done because there's an exception in our constitution for slavery "as punishment for a crime" and well all know that capitalism loves slave labor.
It's actually an institutional success since prison labor is so often utilized in the United States. The truth is they're just lying to you.
Especially if law enforcement uses Parallel Construction [0], lying to the court about the process taken.
[0] https://en.wikipedia.org/wiki/Parallel_construction
[dead]
[dead]
Consider all of that can be used for forced confessions and forced plea bargains also. In those cases, the "evidence" doesn't even need to exist at all, or be on the record in any way.
Let alone all it takes is a photo and a voice clip of 10sec to create an imitation of that person confessing to whatever your heart desires.
An old phone book is enough.
Hows wolfie doing?
it's max
aren't you an imposter ?
Forensics is the art of manufacturing evidence to convict people.
Ok that's a bit aggressive, but it's important to understand the limits of forensic science, the broken incentives, and systemic lack of rigor in the application of various analyses and products to inculpate/exculpate suspects after the fact.
Sadly, there's more evil and more laziness/incompetence in the world that's being accelerated by AI than there is good.
Over all time? Probably tens of millions.
I saw this headline, saw that it was Sky news, thought "oh a British policeman? Bet it was Derbyshire police".
There you go.
As an American, I'm totally out of the loop on this one but it sounds interesting. I assume Derbyshire has a reputation?
They are just rather over-eager and a bit of a law unto themselves in a rather silly way; it is Derbyshire police that hassled people walking in the open air during COVID, including rather excitably harrassing Peak District walkers with drones, and enforced rules that were only guidance as if they were law (famously asserting that two women walking in the fresh air with a coffee was "a picnic").
Them being all super-keen to use AI really fits. Some pillock of an officer going too far really fits.
Derbyshire is really safe but they act like it is not.
This is probably using AI to remove a background or object from an image, not a 6 finger perp.
Can we know the motivation? Will it get them a bonus at the end of the year? Was it something common in the cases, maybe similar victims or something else?
I'd wager that it was just a shortcut to getting his work done. That banal motive is why we've seen an explosion of these cases and why they won't stop.
[flagged]
[dead]
[flagged]
[flagged]
police gonna police