This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The local news app NewsBreak used AI tools to generate and rewrite articles, resulting in multiple instances of fabricated news stories and incorrect charity information that misled communities and prompted copyright lawsuits.
NewsBreak reportedly published AI-assisted local articles containing purportedly false information, including a fabricated Christmas shooting in Bridgeton, New Jersey, incorrect Food to Power distribution times that caused people to be turned away, and nonexistent Harvest912 foot-care clinics. NewsBreak said the errors originated in source material and removed the articles after complaints. The specific AI models and the extent to which they contributed to the inaccuracies were not identified.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional or material harms
Also classified as
NewsBreak used AI to scrape and rewrite competitors' stories under fictitious bylines, leading to copyright lawsuits and displacing human journalistic effort.
The AI system failed to reliably and accurately process input data, leading to hallucinations and factual errors in critical local information.
The AI system generated and spread false local news stories, including a fabricated shooting and incorrect charity event details, which misled readers and caused real-world disruption.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The false stories and incorrect information were generated by deployed AI tools rewriting scraped content, which was an unexpected and unintended outcome of the system's operation.
Added categories peak at 3: Substantial (Epistemic Harm, Inferred)
NewsBreak, a US local news app with China-linked engineering, utilized AI tools to scrape and rewrite news, leading to over 40 fabricated or erroneous stories. While causing local confusion and financial settlements, the national security impact remains minor, primarily representing established risks of AI hallucination and lack of editorial oversight.
The AI system is described as a tool used to rewrite scraped internet content or produce news stories from snippets. It operates on direct input without independent goal-directed planning.
Limited Risk: The system generates synthetic news content. Under the Act, such systems pose a moderate risk and require transparency obligations, such as notifying users that the content is AI-generated.
The AI system's primary function was to automatically aggregate, scrape, and rewrite local news stories to curate content for the app's feed, acting as an automated writing assistant.
The report from Reuters is highly detailed, providing specific dates, locations, names of affected charities, court settlement figures, and internal company memos. This allows for a highly confident classification.
The incident could also be viewed primarily as a copyright and journalistic ethics issue rather than an AI safety issue, though the central role of AI generation in creating the false content makes the AI safety classification highly appropriate.
Technology
AI Rewriting Tool Errors
AI algorithms generated fictional details when rewriting scraped local news.
Flawed Scraping Algorithms
Algorithms scraped outdated or incorrect information from web pages.
Use of Generative AI Experiments
OpenAI tools were used to generate articles under fake bylines.
Data Inputs
Unverified Content Sources
Inaccurate source data from unverified websites like findplace.xyz.
Outdated Charity Web Pages
Scraping tool used outdated food bank and clinic schedules from web pages.
Scraped Competitor Content
Lifting snippets of competitor articles without proper verification.
Human Factors
Lack of Journalistic Experience
Staff were new to journalism and did not verify AI-generated output.
Unfamiliarity with US Market
Staff or management lacked understanding of US editorial standards.
Process and Methods
Lack of Pre-Publication Review
No editorial review process to verify AI-generated stories before publishing.
Ineffective Feedback Mechanisms
Customer support ignored early emails about erroneous charity schedules.
Aggressive Scraping and Rewriting
Systematic rewriting of competitor content to fill local news gaps.
Management
Prioritizing Growth Over Safety
Focus on filling local news voids rapidly led to systemic errors.
Inadequate Risk Assessment
Management ran risky AI experiments without assessing brand damage.
Lack of Editorial Oversight
Leadership failed to implement strict journalistic guidelines initially.
Regulatory Environment
Weak Copyright Enforcement
Copyright laws did not deter unauthorized content rewriting initially.
Lack of AI Content Regulations
No regulatory standards governing AI-generated local news accuracy.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
