This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Toronto Public Health contracted with Cann Forecast to use an AI predictive model to monitor water quality at two beaches. The model consistently underperformed, failing to identify dangerous E. coli levels and resulting in beaches remaining open when they should have been closed. This failure exposed beachgoers and staff to potential health risks, highlighting issues with over-reliance on automated risk prediction tools without adequate human oversight or validation.
Toronto’s use of AI predictive modeling (AIPM) which had replaced existing methodology as the only determiner of beach water quality raised concerns about its accuracy, after allegedly conflicting results were found by a local water advocacy group using traditional means.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
City officials exhibited automation bias, allowing the unproven AI model's predictions to supersede actual daily laboratory testing results.
The AI model failed to perform reliably, predicting safe water quality when E. coli levels were dangerously high, representing a clear lack of capability and robustness.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk was caused by the inaccurate predictions of the deployed Cann Forecast AI model, which failed to perform as expected after being put into active use.
A municipal-level failure of an AI water quality predictive pilot in Toronto, Canada, resulted in localized public health risks due to E. coli exposure. The incident highlights issues of automation bias and poor data validation in local government procurement, but carries negligible national security implications.
The AI system operates as a tool, performing a specific function of calculating water quality predictions based on direct meteorological and hydrological inputs without autonomous planning or agency.
High Risk: The AI system is used in public safety and environmental monitoring (water quality at public beaches), which has significant implications for public health and safety.
The primary goal of the Cann Forecast model was to detect or predict the presence of E. coli bacteria (a substance) in beach water to estimate safety values.
The reports provide clear, detailed, and consistent accounts of the Toronto beach water quality AI pilot, including specific statistics on accuracy, dates, and the nature of the failure. There is high agreement between the sources on the core facts of the incident.
None; the failure of the predictive model is clearly established by both independent analysis and admission by public health officials.
Technology
Faulty Weather Data
The model used incorrect weather data to make predictions at Sunnyside Beach.
Inherent Model Inaccuracy
The algorithm was less accurate than a coin flip at identifying unsafe days.
Data Leakage
Leakage in similar tools led to exaggerated initial claims of performance.
Data Inputs
Incorrect Weather Inputs
Sunnyside predictions were compromised by ingestion of faulty weather data.
Lagging Validation Data
Decisions were made before validating real-time bacteria levels.
Biased Training Data
In related tools, biased data inputs led to systematic discrimination.
Human Factors
Automation Bias
Staff defaulted to the automated tool, failing to challenge faulty outputs.
Lack of User Expertise
Non-expert officials bought the tool on faith without ability to evaluate.
Moral Crumple Zones
Blaming human operators for failures of systems they cannot safely oversee.
Process and Methods
No Performance Tracking
City officials never checked the deployed model's performance over summer.
Superseding Reliable Tests
AI predictions superseded reliable, traditional laboratory E. coli testing.
Lack of Human Override
Staff never overrode the model despite claims of human-in-the-loop process.
Management
Prioritizing Automation
TPH adopted AI to replace lagging lab tests without adequate risk assessment.
Deflecting Public Concerns
Management falsely claimed human oversight to deflect from model failures.
Lack of Operational Transparency
The city quietly adopted the tool without public disclosure or metrics.
Regulatory Environment
Noncompetitive Procurement
The city awarded a CA$30,000 contract without any competing bids.
Proprietary Trade Secrets
Vendors shield models from external scrutiny under trade secret claims.
Lack of AI Safety Standards
No standard regulatory framework existed to validate high-stakes AI tools.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
