This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
On Thanksgiving Day 2022, a Tesla Model S operating on 'Full Self-Driving' (FSD) software abruptly braked in the middle of the San Francisco Bay Bridge, causing an eight-vehicle pileup. The incident resulted in nine injuries, including a child. Federal data later confirmed the software was engaged within 30 seconds of the crash, which occurred shortly after Tesla expanded FSD access to all North American users.
A Tesla driver alleged Full Self Driving (FSD) braking unexpectedly as the cause for an eight-car pileup in San Francisco which led to minor injuries of nine people.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The reports highlight the 'vigilance problem' where drivers overrely on Level 2 systems and fail to intervene when the system malfunctions.
The primary failure was the Tesla FSD system's lack of robustness, specifically executing an unexpected 'phantom braking' maneuver on a highway, which represents a system-level capability failure.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the Tesla FSD software unexpectedly executing a braking maneuver post-deployment, which was an unintentional and unsafe action.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
A Tesla Model S operating on Full Self-Driving Beta software executed an unexpected phantom braking maneuver on the San Francisco Bay Bridge, causing an eight-vehicle pileup and nine minor injuries. The incident represents a minor physical safety and critical infrastructure concern, manageable within standard emergency response frameworks, with negligible broader national security implications.
The AI system operated independently to steer, signal, and brake the vehicle in real-time based on its environmental inputs, executing these actions without direct human planning or step-by-step instruction during the incident.
High Risk: The system is used in critical infrastructure (traffic management/road safety) and transport, where failures pose significant implications for safety and human life. The reports describe a 'Full Self-Driving' driver-assist system operating on public highways, which falls under safety components of vehicles subject to third-party assessment or critical transport infrastructure.
The primary goal of Tesla's FSD Beta is to navigate and drive the vehicle autonomously under human supervision.
The reports provide consistent, detailed accounts of the crash, including police reports, surveillance video, and official NHTSA data confirming FSD engagement. There is high certainty regarding the sequence of events and the resulting harms.
One could argue the driver was solely at fault due to lack of vigilance, but the physical action causing the crash was initiated by the AI system's phantom braking.
Technology
Phantom Braking Bug
Software triggers sudden, unexpected braking without any obstacles present.
Erratic Lane Change Action
FSD system initiated an unsafe lane change before stopping abruptly.
Inadequate Object Detection
Autopilot systems struggle to detect stopped emergency vehicles with lights.
Data Inputs
False Positive Obstacle Data
Sensor noise or misclassification triggers emergency braking sequences.
Lack of Emergency Light Data
System fails to properly interpret flashing hazard lights of parked SUVs.
Sensor Suite Limitations
Camera-only inputs may struggle with depth perception in tunnels.
Human Factors
The Vigilance Problem
Humans struggle to monitor automated systems and fail to intervene in time.
Overreliance on Automation
Misleading system names lead drivers to assume the car is fully autonomous.
Delayed Driver Intervention
Driver failed to manually override the vehicle's unexpected braking.
Process and Methods
Uncontrolled Beta Release
FSD Beta was opened to all users regardless of safety scores.
Misleading System Branding
Marketing Level 2 tech as 'Full Self-Driving' misleads consumers.
Deficient Driver Monitoring
Inadequate steering wheel nags allow drivers to disengage from driving.
Management
Prioritizing Market Dominance
Releasing FSD to all users to differentiate from competitors.
Disabling Driver Alerts
Management agreed to disable steering wheel nags for certain users.
Disbanded PR Department
Lack of transparent communication with public regarding safety.
Regulatory Environment
Lack of Federal Restrictions
No federal rules govern testing of semi-autonomous vehicles on public roads.
Delayed Regulatory Action
NHTSA has investigated crashes for years without taking enforcement action.
Inadequate Safety Standards
Absence of standardized safety benchmarks for automated driving systems.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
