This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
New York City and State implemented value-added modeling (VAM) algorithms to evaluate teacher performance based on student test scores. The system faced widespread criticism and legal challenges from teachers and unions, who argued the algorithms were opaque, statistically flawed, and relied on inaccurate data. Teachers reported being unfairly labeled as 'ineffective' or 'unsatisfactory,' with some being evaluated based on students or subjects they did not teach, leading to significant professional and personal harm.
An algorithm used to rate the effectiveness of school teachers in New York has resulted in thousands of disputes of its results.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
Teachers with higher numbers of students with disabilities or English-language learners received systematically lower ratings due to the model's bias.
The VAM model operated as an opaque 'statistical black box' that teachers and administrators could not easily understand or verify.
The VAM system failed to perform reliably, exhibiting massive margins of error and generating highly inaccurate ratings that led to unfair evaluations and potential job losses.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The VAM algorithm's flawed evaluations and high margins of error were unexpected outcomes of deploying the system to assess teacher performance.
Added categories peak at 2: Minor (Psychological Harm, Direct)
The implementation of flawed value-added modeling (VAM) algorithms for teacher evaluations in New York caused significant local labor disputes, privacy violations, and psychological distress. However, as a domestic educational administrative failure, it has negligible implications for national security.
The VAM system acted as an assistant to human administrators, generating performance ratings and data reports to support high-stakes employment decisions, though it required human implementation and was subject to appeals.
High Risk: The VAM system is an educational and employment evaluation system used to assess worker performance and make decisions affecting employment access and retention.
The primary purpose of the VAM algorithm was to monitor, evaluate, and rate the performance and effectiveness of public school teachers based on student test scores.
The reports are detailed, from reputable sources (Politico, Washington Post, Huffington Post), and clearly describe the VAM system, its flaws, and the consequences.
None
Technology
Impossible Predicted Scores
VAM algorithm predicted scores higher than the maximum possible exam points.
Extremely Large Margins of Error
Confidence intervals for VAM scores averaged 35 to 53 percentile points.
Statistical Black Box Model
VAM used complex, opaque computer models that educators could not verify.
Data Inputs
Inaccurate Student-Teacher Linkages
Teachers were evaluated based on students or subjects they did not teach.
Inflated Baseline Test Data
The underlying state test scores were inflated, skewing VAM predictions.
Failure to Verify Class Data
Omissions like unverified class sizes and assignments skewed VAM results.
Human Factors
Over-reliance on Algorithmic Metrics
Reformers embraced VAM as a data-driven solution despite expert warnings.
Demoralization of Excellent Teachers
High-performing teachers were labeled ineffective, causing stress and lawsuits.
Disincentive for Struggling Schools
VAM metrics discouraged teachers from working in lower-performing schools.
Process and Methods
Rushed VAM Implementation
The evaluation system was rushed alongside the new Common Core curriculum.
Public Release of Flawed Data
Publishing raw VAM scores publicly misled parents and shamed teachers.
Lack of Authentic Assessments
Over-reliance on multiple-choice tests instead of qualitative measures.
Management
Prioritizing Metrics over Fairness
Administrators pushed VAM to meet federal grants without validating accuracy.
Lack of Internal Data Verification
Management failed to verify class sizes and rosters before running models.
Regulatory Environment
Federal Funding Pressures
Race to the Top grant requirements forced rapid adoption of VAM evaluations.
State-Mandated Evaluation Laws
State laws mandated connecting teacher performance to student test scores.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
