This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
In April 2018, a Tesla Model X operating on Autopilot in Kanagawa, Japan, failed to detect a group of motorcyclists and a van stopped on the expressway. The vehicle accelerated into the group, killing 44-year-old Yoshihiro Umeda. The victim's family filed a lawsuit against Tesla, alleging that the Autopilot system was defective and failed to recognize the hazard, despite the driver having his hands on the wheel.
A Tesla Model X operated on Autopilot reportedly failed to recognize the parked motorcycles, pedestrians, and van in its path in Kanagawa, Japan, and ran over a motorcyclist who previously stopped when a member of his motorcyclist group was involved in an accident.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The driver fell asleep while relying on Autopilot, demonstrating unsafe use and overreliance on a system that requires constant human vigilance.
The primary failure was the Autopilot system's inability to detect stationary objects (motorcycles, van, pedestrians) in its path and its decision to accelerate after the lead vehicle changed lanes.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was caused by the Tesla Autopilot system unintentionally failing to detect stationary hazards and accelerating, leading to a fatal collision.
Added categories peak at 2: Minor (Psychological Harm, Inferred)
A localized fatal traffic accident in Japan involving a Tesla Model X on Autopilot. While tragic and highlighting safety and regulatory challenges of consumer AI, the incident has negligible implications for national security.
The Tesla Autopilot system was operating the vehicle's steering, acceleration, and braking in real-time on the highway. The report notes it 'allow[s] the vehicle to steer, accelerate and brake automatically within a lane' and made real-time decisions to accelerate when the lead car changed lanes.
High Risk: The system is used in critical transport infrastructure and safety-critical applications (autonomous driving/driver assistance on public roads). The report notes 'All orgs developing advanced AI should be regulated, including Tesla' and describes a safety-critical system whose failure led to a fatality.
The Autopilot system's primary goal is to automate driving tasks such as steering, accelerating, and braking on highways.
The reports provide consistent, detailed accounts of the crash, including the specific actions of the Tesla, the driver, and the lead vehicle. The legal filings and NTSB findings add significant credibility and technical context to the failure modes.
Tesla argues that the driver is solely responsible for maintaining vigilance, suggesting the incident could be viewed primarily as driver negligence rather than an AI failure.
Technology
Sensor Recognition Failure
Cameras and sensors failed to recognize stationary obstacles in vehicle path.
Uncontrolled Acceleration
System accelerated from 9 to 23 mph after the lead vehicle changed lanes.
Defective Driver Monitoring
System failed to detect that the driver had fallen asleep.
Data Inputs
Unrecognized Obstacle Data
Visual and radar inputs of parked motorcycles and pedestrians were ignored.
Lead Vehicle Cut-Out Data
Sudden loss of the tracked lead vehicle triggered automatic acceleration.
Human Factors
Driver Drowsiness
The driver fell asleep while the vehicle was in motion.
Complete Driver Disengagement
The driver failed to monitor the road and take over steering.
Process and Methods
Flawed Engagement Verification
Relying on steering wheel torque was a poor indicator of driver alertness.
Inadequate Risk Mitigation
Tesla failed to implement safeguards for known system limitations.
Management
Premature Product Release
Tesla deployed a half-baked system onto public roads.
Misleading Tech Marketing
Marketing Autopilot as self-driving promoted driver overreliance.
Disregarding NTSB Recommendations
Tesla ignored safety regulator warnings about ineffective driver monitoring.
Regulatory Environment
Inadequate Safety Oversight
Lack of strict regulatory oversight allowed public testing of unfinished tech.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
