This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Travelers using generative AI tools like ChatGPT and Layla for trip planning were misled by fabricated itineraries and incorrect transit schedules, resulting in financial losses and dangerous situations such as being stranded on a mountain.
Dana Yao and her husband, Robert, reportedly used ChatGPT to plan a sunset hike on Mount Misen in Japan. ChatGPT allegedly told them the last ropeway departed at 5:30 PM, but the service had already closed when they reached the summit, leaving them stranded.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
Travelers relied on general-purpose AI chatbots for safety-critical planning, such as mountain hiking schedules and high-altitude routes.
The underlying failure stems from the AI models' lack of capability to distinguish fact from statistical word association, leading to hallucinations.
The AI systems generated false and misleading travel information, leading users to form inaccurate beliefs that resulted in material and physical risks.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incidents were caused by deployed AI systems generating unintended hallucinations and false information during user interactions.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
The incident involves consumer-grade generative AI tools producing hallucinated travel information, leading to minor financial losses and localized safety risks for individual tourists. There are no national security implications, as the event does not threaten critical infrastructure, sovereignty, intelligence security, economic stability, or societal cohesion.
The AI systems function as assistants, providing information, suggestions, and itineraries to augment human travel planning, but they do not take independent actions or execute plans autonomously.
Limited Risk: The systems described are general-purpose chatbots and generative AI tools used to interact with humans and generate content, which fall under transparency obligations.
The AI systems were primarily used by travelers to plan routes, find destinations, and schedule transit, functioning as navigation and travel assistants.
The reports provide clear, consistent, and detailed anecdotal evidence of AI travel planning failures across multiple platforms (ChatGPT, Layla) and geographic locations (Peru, Japan, Italy). The mechanisms of failure (LLM hallucinations) are well-explained by experts quoted in the text.
None.
Technology
Inherent LLM Hallucinations
LLMs statistically predict words and fabricate non-existent places.
Lack of Physical World Understanding
AI cannot distinguish between city walks and dangerous mountain climbs.
Identical Presentation of Facts
AI presents fabricated errors and real facts with the same confident tone.
Data Inputs
Outdated Travel Information
AI used outdated data, leading to incorrect ropeway and restaurant details.
Unverified AI-Generated Media
Social media videos generated by AI fabricated non-existent tourist sites.
Statistical Word Association
Combining regional words to create fictional destinations like Humantay Canyon.
Human Factors
Overreliance on AI Output
Travellers trusted the AI itinerary without verifying safety or feasibility.
Lack of Local Destination Knowledge
Travellers planned trips to unfamiliar areas making verification difficult.
Misleading Confident Tone
Users were swayed by the vivid adjectives and convincing writing of the AI.
Process and Methods
Inadequate Manual Verification
Planning relied solely on AI without double-checking official sources.
Lack of Mitigation Strategies
Travellers failed to prepare backup plans for when AI information failed.
Inefficient Search Methods
Verifying AI output manually was as laborious as traditional planning.
Management
Prioritizing Engagement Over Safety
Platforms use AI to edit media and boost engagement without safety checks.
Release of Unsafe Conversational AI
Tech firms deploy LLMs for travel planning despite known hallucination issues.
Regulatory Environment
Lack of Standardized Watermarking
Absence of clear indicators to distinguish AI-generated travel content.
Slow Regulatory Implementation
Proposed US/EU regulations on AI content identification are not yet active.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
