This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Multiple instances of AI-based upscaling tools being used to 'enhance' blurry video footage have resulted in the fabrication of details and the misidentification of individuals. In France, this led to the public misgendering of a 14-year-old girl. In Serbia, law enforcement disseminated AI-enhanced images of a suspect that did not match the actual individual, potentially hindering investigations and misleading the public. Experts warn that upscaling tools invent synthetic details rather than recovering original information, making them unsuitable for identification purposes.
A purportedly clearer, AI-modified frame from an arrest video in Metz, France, reportedly showed additional facial and body hair and a different shirt pattern. Viewers cited it as evidence that the arrested person was a boy, although she was a 14-year-old girl. The altered image allegedly distorted public interpretation of the arrest footage.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
Law enforcement and social media users overrelied on these tools for critical identification tasks they are not fit for.
The upscaling tools lack the capability to accurately reconstruct original details from low-resolution inputs, instead inventing synthetic data.
The AI upscaling tools generated false and misleading visual details (such as changing a girl's appearance to look like a boy, changing bus numbers, and fabricating a suspect's face) which misled the public and law enforcement.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by AI upscaling models generating synthetic, fabricated details (AI entity) unintentionally while attempting to enhance image resolution post-deployment.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
AI-powered upscaling tools fabricated synthetic details on blurry footage, leading to public misidentifications in France and misleading law enforcement investigations in Serbia. While presenting minor risks to societal stability and government functions due to compromised evidence reliability, the overall national security impact remains low.
The AI systems used are image upscaling algorithms that perform a specific, well-defined function (increasing image resolution) on direct input without context retention or autonomous planning. The report describes them as 'outils d'intelligence artificielle pour rendre plus nettes des photographies'.
High Risk: The report describes the technology being used in 'law enforcement applications' by the Serbian police to generate images of a suspect, which is a high-risk domain under the Act.
The primary function of the AI systems described is image upscaling (increasing resolution), which was subsequently used by humans for the goal of identification.
The reports provide clear, detailed examples of AI upscaling tools fabricating details across multiple distinct events in France and Serbia. The technical explanation of how upscaling works (inventing synthetic details rather than recovering original data) is well-supported by experts and journalists.
The incidents could be viewed purely as human error or misinformation campaigns, but the core issue is the technical limitation of generative AI upscaling when applied to forensic tasks.
Technology
Synthetic Detail Generation
AI upscaling invents plausible but entirely synthetic details to fill gaps.
Facial Feature Distortion
Neural networks alter facial traits, making individuals unrecognizable.
Alteration of Key Evidence
Algorithms modify text, clothing, and remove critical physical evidence.
Data Inputs
Low-Resolution Source Images
Extremely blurry source inputs force the AI to invent missing details.
Lack of Original Sensor Data
Upscaling relies on statistical patterns rather than actual raw data.
Human Factors
Misunderstanding of AI Limits
Users falsely assume upscaling retrieves real data instead of inventing it.
Confirmation Bias on Social Media
Users share AI-altered images to quickly support their narratives online.
Overreliance by Law Enforcement
Police release AI-enhanced photos, leading to public misidentification.
Process and Methods
Inappropriate Tool Selection
Using generative upscaling for forensic identification is highly flawed.
Lack of Verification Protocols
Media and users share enhanced images without checking the original source.
Management
Failure to Warn Users of Limits
AI developers do not warn users against using tools for forensic tasks.
Lack of Police Oversight
Police management failed to verify the validity of AI-enhanced images.
Regulatory Environment
No Standards for AI Evidence
No regulatory guidelines exist for using AI-modified images in policing.
Unregulated Public AI Tools
Powerful generative tools are public without forensic use warnings.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
