This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
A video captured by a Tesla driver shows the vehicle's Autopilot system repeatedly misidentifying a horse-drawn carriage as various objects, including a truck, a car, and a human. The system's visualization displayed the carriage as a truck spinning as if on a collision course. This incident highlights the challenges deep learning models face with out-of-distribution data in real-world driving environments.
Tesla Autopilot's computer vision system was shown in a video mistaking a horse-drawn carriage for other forms of transport such as a truck, a car, and a human following a car.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
The Tesla Autopilot system failed to perform robustly when encountering out-of-distribution data (a horse-drawn carriage), leading to erratic visualization and potential safety risks.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The perception failure was caused by the Tesla Autopilot AI system's inability to correctly classify a horse-drawn carriage, which was an unintentional error occurring post-deployment.
A Tesla Model Y Autopilot system failed to correctly identify a horse-drawn carriage in Switzerland, highlighting technical challenges with out-of-distribution data. The incident was a non-harmful consumer product glitch with negligible national security implications.
The Tesla Autopilot system operates autonomously in real-time to detect vehicles, pedestrians, and other objects in the car's vicinity, attempting to make driving decisions based on environmental inputs.
High Risk: The system is used in critical transport infrastructure. The report describes Tesla's Autopilot and Full Self-Driving technologies, which are AI systems in critical transport infrastructure where failures can have significant implications for safety.
The Tesla Autopilot system's primary goal is autonomous driving, which relies on real-time image classification to detect and categorize surrounding objects.
The reports provide consistent accounts of the incident, including a viral video showing the exact behavior of the Tesla Autopilot system when encountering a horse-drawn carriage. The technical cause (out-of-distribution data) is clear and agreed upon by observers.
None. The event is a straightforward perception failure of an autonomous driving system.
Technology
OOD Generalization Failure
Deep learning models struggle with out-of-distribution data like carriages.
Unstable Object Classification
System rapidly alternated between detecting a car, human, and truck.
Erratic Visualization Rendering
System showed a ghost truck spinning, indicating tracking instability.
Data Inputs
Lack of Training Data for Carriages
The model likely saw very few horse-drawn carriages during its training.
Missing Object Class Definitions
Designers may not have included horse-drawn carriages in the target classes.
Human Factors
Overreliance on Automation
Drivers may assume Autopilot can handle all road scenarios safely.
Process and Methods
Inadequate Edge-Case Testing
Testing processes failed to validate performance against rare road obstacles.
Management
Misleading Product Claims
Management promoted FSD as requiring no action, creating false expectations.
Regulatory Environment
Weak Marketing Oversight
Regulators like DMV noted misleading claims about self-driving capabilities.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
