This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Several major corporations testing generative AI work assistants like Microsoft Copilot and Google Gemini reported that the tools frequently delivered incorrect or outdated answers due to AI limitations and uncurated enterprise data.
AI work assistants, such as Copilot for Microsoft 365 and Gemini for Google Workspace, are proving to be more labor-intensive than anticipated for enterprises. CIOs report that these AI tools struggle with outdated or inaccurate data, leading to incorrect outputs. Companies are finding they must invest heavily in data management to ensure reliability. This added effort has led to delays in deployment and frustration, as businesses work to maximize the potential of these expensive AI tools.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
The AI systems generated incorrect and contradictory information, leading to inaccurate beliefs or confusion among corporate users.
The AI systems failed to perform reliably, delivering incorrect answers about executive teams, expense policies, and financial data due to technical immaturity and data retrieval limitations.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The incident involves AI assistants unintentionally generating incorrect or outdated answers during post-deployment corporate testing.
Added categories peak at 2: Minor (Epistemic Harm, Inferred)
Major corporations testing generative AI work assistants reported retrieval errors and outdated outputs due to uncurated internal enterprise data. The incident represents standard commercial software adoption challenges with negligible national security implications.
The AI systems (Copilot, Gemini) are explicitly characterized as 'work assistants' designed to augment human capabilities by answering questions and processing enterprise data under human oversight.
Risk Level 3. Limited Risk: The report describes chatbots and generative AI assistants (Copilot, Gemini) interacting with corporate users, which are subject to transparency obligations.
The primary function of the AI systems described is to answer user queries about corporate data, such as sales figures, executive teams, and expense policies.
The report provides clear, consistent, and detailed journalistic accounts of corporate trials of AI assistants, naming specific companies, executives, and the exact nature of the AI failures.
None.
Technology
Immature AI Tooling
The generative AI tools themselves are still maturing and evolving.
Lack of Source Prioritization
Tools cannot distinguish authoritative systems from emails or documents.
Inadequate Context Retrieval
AI fails to ask users for more context when queries are ambiguous.
Data Inputs
Outdated Enterprise Data
AI accesses stale data, leading to outdated or incorrect answers.
Data Duplication and Inconsistency
Inconsistent and duplicated data causes contradictory AI outputs.
Insecure Data Permissions
AI exposes sensitive historical files users should not access.
Human Factors
Inadequate Prompting Skills
Users do not provide sufficient context in prompts to get accurate results.
Unrealistic Expectations
Users expect magic without preparing necessary backend data infrastructure.
Process and Methods
Poor Data Management Practices
Organizations lack processes to clean, validate, and update enterprise data.
Lack of Authoritative Routing
No processes to direct AI to verified systems of record.
Management
Premature Adoption Planning
Management attempts production deployment before data is ready.
Underestimating Integration Effort
Leaders underestimate the internal lift and cost required for deployment.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
