This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Following the death of a delivery driver in a traffic accident, Glovo's automated management system sent an email to the deceased's family terminating his account for non-compliance. The incident sparked public outcry and labor strikes in Florence, with critics highlighting how the company's algorithmic management system incentivizes dangerous driving behaviors to meet delivery targets.
Delivery company Glovo's automated system sent an email terminating an employee for "non-compliance terms and conditions" after the employee was killed in a car accident while making a delivery on Glovo's behalf.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Delegating by humans of key decisions to AI systems, or AI systems that make decisions that diminish human control and autonomy, potentially leading to humans feeling disempowered, losing the ability to shape a fulfilling life trajectory, or becoming cognitively enfeebled.
Also classified as
The algorithmic management system pressured drivers to deliver faster to maintain high ratings, leading to dangerous working conditions and a fatal accident.
The automated system failed to distinguish between a routine account suspension and a driver's death, mistakenly sending a firing email to the family.
The AI system automatically terminated the driver's employment without human review, representing a complete delegation of critical decision-making authority to an algorithm.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The automated firing email was an unintended outcome of the AI system's account suspension protocol, and the algorithmic pressure contributing to the accident was an unintentional risk of the system's optimization goals.
Added categories peak at 2: Minor (Psychological Harm, Direct)
A fatal accident involving a Glovo delivery driver in Italy triggered an automated termination email to his family, leading to localized strikes and protests. The incident highlights domestic labor and regulatory challenges surrounding algorithmic management, but presents minor to negligible national security implications.
The Glovo delivery management algorithm operates independently to assign deliveries, track courier times, and automatically deactivate accounts without human intervention, as evidenced by the automated firing email sent to the deceased driver's family.
Risk Level 2: High Risk. The system is an employment and worker management system used to monitor, evaluate, and terminate couriers, which falls under the High Risk category of the EU AI Act.
The AI system's primary function is to manage the courier platform, including account deactivations and administrative tasks, while also monitoring courier performance and delivery times.
The reports provide consistent and clear accounts of the incident from multiple sources, including statements from the family, the company, union representatives, and local authorities. The role of the automated system in sending the email is explicitly acknowledged by Glovo.
The accident itself could be viewed purely as a traffic incident rather than an AI safety incident, though the algorithmic pressure is heavily cited as a contributing factor.
Technology
Automated Termination Algorithm
System automatically fired driver for missed deliveries without human review.
Poor System Integration
Identity suspension triggered automated termination email by mistake.
Data Inputs
Lack of Real-Time Incident Data
Algorithm lacked data about the driver fatal accident before actioning.
Inaccurate Account Status Signals
System interpreted inactivity from the accident as policy non-compliance.
Human Factors
Pressure to Speed
Riders forced to rush and take risks to meet delivery times and earn more.
Process and Methods
Automated Email Dispatched
No manual verification before sending automated termination emails to riders.
Piecework Pay System
Pay-per-delivery model encourages dangerous driving to maximize income.
Management
Prioritizing Speed Over Safety
Company competition model focused heavily on rapid delivery times.
Inadequate Algorithmic Safeguards
Management failed to implement ethical checkpoints in automated systems.
Regulatory Environment
Lack of Gig Worker Protections
Absence of regulations ensuring hourly pay and worker safety guarantees.
Unenforced Court Rulings
Failure to implement existing court rulings on proper rider classification.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
