This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A criminal defendant in Cali, Colombia, discovered a draft conviction in the court's digital file before the trial had concluded. The Supreme Court of Colombia determined that the judge had used generative AI to perform evidentiary analysis and draft the conviction, which prejudged the defendant's guilt and violated his rights to due process and an impartial trial. The court ordered the case to be reassigned to a different judge and established strict guidelines regarding the use of AI in judicial proceedings.
A criminal defendant's lawyers found a draft conviction proposing 18 years in prison in the court file before closing arguments in Cali, Colombia. Colombia's Supreme Court inferred that generative AI helped assess the complainant's testimony. It found that the draft prejudged guilt, granted relief for violations of due process and the right to an impartial trial, and ordered the case reassigned.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Users anthropomorphizing, trusting, or relying on AI systems, leading to emotional or material dependence and inappropriate relationships with or expectations of AI systems. Trust can be exploited by malicious actors (e.g., to harvest personal information or enable manipulation), or result in harm from inappropriate use of AI in critical situations (e.g., medical emergency). Overreliance on AI systems can compromise autonomy and weaken social ties.
Also classified as
The AI system hallucinated hearing dates and fabricated future events, showing a lack of robustness and capability.
The judge inappropriately relied on a generative AI system to perform critical judicial functions, specifically evidence evaluation and drafting a sentence, representing unsafe use in a critical legal context.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by a human judge's decision to use a generative AI tool to draft a judicial sentence and analyze evidence, which occurred post-deployment.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
A Colombian judge used generative AI to draft a criminal conviction and analyze evidence before the trial concluded, violating the defendant's due process. The Supreme Court of Colombia intervened, reassigning the case and issuing strict guidelines. The incident highlights emerging challenges with AI overreliance in judicial decision-making but poses minimal threat to national security.
The AI system acted as an assistant to the judge, generating a draft analysis of testimony and structuring a sentence, which was intended to augment the judge's workflow but required human validation.
High Risk: The AI system was used in the administration of justice and judicial proceedings, which is a critical area with significant implications for fundamental rights and public interest.
The AI was used by the court to perform evidentiary analysis and draft a sentencing document, which falls under judicial decision support and writing assistance.
The reports provide highly detailed legal and factual accounts of the incident, including direct quotes from the Colombian Supreme Court's ruling, the specific text generated by the AI, and the procedural history of the case.
None. The facts of the AI's use and the subsequent Supreme Court ruling are clear and uncontested.
Technology
AI Hallucination of Facts
AI invented dates of hearings that had not yet occurred.
Automation of Human Reasoning
AI was used to perform core judicial evaluation of evidence.
Biased Output Generation
AI structured analysis specifically to support a conviction.
Data Inputs
Premature Input of Evidence
Unfinished trial data was fed into the AI for final sentencing.
Transcription of Testimonies
Raw trial transcripts were used by AI without proper context.
Unvalidated Draft Upload
Draft containing AI outputs was uploaded directly to the digital file.
Human Factors
Overreliance on AI Output
The judge accepted AI-generated legal evaluations as valid.
Accidental File Upload
A court official mistakenly uploaded the draft to the system.
Lack of Digital Literacy
Insufficient understanding of AI limitations and risks by court staff.
Process and Methods
Premature Sentence Drafting
Drafting the sentence before final arguments were presented.
Lack of Review Protocols
No verification process before uploading documents to the digital file.
Inadequate AI Safeguards
Absence of technical controls to prevent AI from drafting final decisions.
Management
Inadequate Supervision
Failure of the judge to supervise the work of the court official.
Lack of Training on AI Ethics
No systematic training for judicial staff on responsible AI use.
Poor Risk Management
Failure to assess risks of using generative AI in active criminal cases.
Regulatory Environment
Non-compliance with Guidelines
Violation of Sentence T-323 of 2024 and Agreement PCSJA.
Vague Enforcement Mechanisms
Lack of active monitoring for AI usage rules in local courts.
AI system involved
An AI system was involved, but the reports do not identify it.
Identified by automated extraction; can differ from the alleged developers above.
