This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A commercial crabber in Florida was wrongfully arrested after a facial recognition system, identified as the FACES program, produced a false 93% match to surveillance photos from a crime scene 300 miles away. The police relied on this AI-generated match to obtain an arrest warrant despite the victim having no connection to the location. Charges were eventually dropped after the victim's attorney provided evidence of his innocence, but the victim suffered financial loss and psychological trauma.
Florida police reportedly arrested commercial crabber Robert Dillon in August 2024 after the statewide FACES facial recognition system purportedly identified him as a possible match to a suspect shown in grainy surveillance images from Jacksonville Beach. Dillon reportedly lived more than 300 miles away and said he had never visited the city. He reportedly spent a night in jail and posted bond using his truck title.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Accuracy and effectiveness of AI decisions and actions are dependent on group membership, where decisions in AI system design and biased training data lead to unequal outcomes, reduced benefits, increased effort, and alienation of users.
Also classified as
Police officers relied blindly on the AI-generated match as the primary basis for an arrest warrant without conducting proper independent verification.
The facial recognition system failed to perform robustly, generating a highly confident but completely false match from grainy surveillance footage.
The incident involves facial recognition technology, which testing has repeatedly shown exhibits higher rates of false matches and unequal performance when used on older people, women, and people of color.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was triggered by an incorrect 93% match generated by the facial recognition AI system, which was an unexpected and erroneous outcome of its post-deployment operation.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
The wrongful arrest of a Florida resident due to a false facial recognition match highlights domestic civil rights concerns and automation bias in law enforcement. However, the incident has negligible to minor national security implications, representing a localized policing issue rather than a threat to national sovereignty, critical infrastructure, or strategic stability.
The facial recognition program operates as a tool, performing a specific, pattern-matching function on direct image inputs without context retention or autonomous decision-making authority.
High Risk: The report describes a law enforcement application of AI, specifically facial recognition technology used for identifying suspects, which is explicitly classified as High Risk under the EU AI Act.
The AI system's explicit purpose was to run surveillance photos through a facial recognition program to identify potential matches in a database.
The reports provide clear, consistent, and detailed accounts of Robert Dillon's wrongful arrest, the technology used, and the consequences. It also references other specific cases and systemic issues with the technology, allowing for highly confident classifications.
None. The causal chain from the AI's false match to the wrongful arrest is clear and undisputed by the parties involved.
Technology
Faulty Facial Recognition Match
System produced an incorrect 93% match from low-quality restaurant footage.
Demographic Bias in AI
Systems exhibit higher false match rates for specific demographic groups.
Grainy Image Processing
Algorithm failed to distinguish distinct facial features on grainy images.
Data Inputs
Grainy Surveillance Photos
Low-quality, grainy input images led to an inaccurate facial match.
Unclear Reference Photo Source
Database photo used for matching lacked clear context or verified source.
Human Factors
Overreliance on AI Output
Detectives treated the AI match as definitive proof rather than just a lead.
Confirmation Bias in Lineup
Witnesses were misled by choosing between an AI lookalike and random fillers.
Ignoring Exculpatory Evidence
Police disregarded license plate readers and geographic distance of suspect.
Process and Methods
Flawed Photo Lineup Process
Showing an AI lookalike in a lineup biased the witness toward the suspect.
Lack of Human Verification
No rigorous manual verification was conducted to confirm the AI match.
Inadequate Investigative Standards
Detectives failed to verify phone records or suspect location before arrest.
Management
Prioritizing Leads Over Accuracy
Management did not track AI error rates, expressing 'no concerns' on safety.
Inadequate Officer Training
Officers were not properly trained on the limitations of AI facial matches.
Lack of Agency Accountability
Law enforcement failed to acknowledge errors or implement systemic changes.
Regulatory Environment
Absence of AI Safeguards
No mandatory safeguards were implemented to prevent wrongful arrests.
Lack of Statutory Oversight
Lack of statewide legal frameworks governing facial recognition deployment.
No Mandatory Performance Tracking
Agencies are not required to track or report facial recognition errors.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
