This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
The Texas Attorney General reached a settlement with Pieces Technologies regarding allegations that the company used deceptive marketing to overstate the accuracy of its generative AI clinical documentation tools. The company allegedly claimed extremely low hallucination rates to induce hospitals to adopt its products, which were used to summarize patient data. The settlement mandates stricter transparency, disclosure of performance metrics, and documentation of potential risks to patient safety.
The Texas Attorney General announced a settlement with Pieces Technologies following allegations that the company misrepresented the accuracy of its healthcare AI systems used for clinical documentation. The state concluded that marketing claims about low error and hallucination rates may have misled hospitals and clinicians relying on the tools in patient care settings. The settlement imposed restrictions on future claims and required greater transparency around performance and risk.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The generative AI system produces summaries of patient charts that can contain hallucinations, which are false or misleading clinical details.
The AI system's actual performance and hallucination rates did not meet the highly accurate metrics advertised, presenting robustness and capability risks in high-risk clinical settings.
Human
Due to a decision or action made by humans
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident was primarily caused by human decisions to make deceptive and misleading marketing claims about the AI system's accuracy metrics.
Added categories peak at 4: Severe (Epistemic Harm, Inferred)
The Texas Attorney General settled with Pieces Technologies over deceptive marketing claims regarding the accuracy of its clinical generative AI tool. While posing potential risks within the healthcare critical infrastructure sector due to unverified hallucination rates, the incident was resolved legally and had negligible direct impact on national security.
The Pieces Intelligence Platform is described as an AI tool that generates clinical documentation and summaries to assist healthcare providers within the EHR, requiring human clinicians to review and utilize the outputs.
High Risk: The AI system is used in healthcare as a clinical documentation and decision support tool, which has significant implications for patient safety and health. The EU AI Act classifies AI used in healthcare and medical devices as High Risk.
The primary goal of the Pieces Intelligence Platform is to act as a health AI assistant by summarizing patient charts and generating clinical documentation within electronic health records.
The reports from legal and regulatory sources provide clear, consistent details about the Texas AG investigation, the specific deceptive claims made by Pieces Technologies, and the terms of the settlement. There is no conflicting information regarding the core facts of the deceptive marketing allegations.
None. The legal nature of the settlement and the allegations of deceptive trade practices are straightforward.
Technology
AI Hallucination Propensity
Generative AI models inherently produce incorrect or misleading clinical text.
Unverified Accuracy Metrics
Stated hallucination rates of under 0.001 percent were not verified.
Data Inputs
Real-Time EHR Data Processing
Processing live patient records increased the risk of unverified outputs.
Lack of Training Data Disclosure
No clear documentation was provided on data used to train the AI models.
Human Factors
Clinician Overreliance
Medical staff relied on AI summaries without fully understanding limitations.
Inadequate Operator Training
Lack of guidance for hospital staff on how to monitor for AI inaccuracies.
Process and Methods
Flawed Metric Calculation
The methodology used to calculate AI hallucination rates was non-transparent.
Lack of Independent Auditing
No objective third-party testing was conducted to substantiate AI claims.
Deficient Risk Documentation
Failure to document known harmful misuses and limitations of the AI tool.
Management
Deceptive Marketing Claims
Prioritizing growth led to unsubstantiated safety claims in advertisements.
Inadequate Risk Assessment
Management failed to assess patient safety risks from inaccurate AI summaries.
Regulatory Environment
Absence of Federal AI Laws
Lack of proactive federal regulations governing clinical AI safety standards.
Post-Deployment Enforcement
Regulators relied on retrospective consumer protection laws to address risks.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
