This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Airbnb implemented an automated 'trustworthiness' assessment system, incorporating technology from its acquisition of Trooly, to evaluate users based on social media activity, employment history, and social network associations. This system resulted in unexplained account bans and allegations of discrimination against sex workers and other users, with no transparent appeal process provided by the company.
Airbnb allegedly considered publicly available data on users to gauge their trustworthiness via algorithmic assessment of personality and behavioral traits, resulting in unexplained bans and discriminatory bans against sex workers.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
Unequal treatment of individuals or groups by AI, often based on race, gender, or other sensitive characteristics, resulting in unfair outcomes and unfair representation of those groups.
Also classified as
The algorithm secretly harvests and analyzes users' and their friends' social media data, employment history, and other personal information to infer sensitive traits.
Users are banned by an automated algorithm with no explanation and no meaningful avenue for human review or appeal.
The algorithm systematically targets and excludes individuals based on their occupation, such as sex workers, leading to unfair discrimination and denial of service.
AI system
Due to a decision or action made by an AI system
Intentional
Due to an expected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The risk is caused by the automated decisions of Airbnb's trustworthiness algorithm, which was intentionally deployed to profile and exclude users post-deployment.
Added categories peak at 2: Minor (Psychological Harm, Direct)
Airbnb deployed an automated trustworthiness profiling algorithm that harvests extensive personal and social media data to score and ban users. While raising notable civil rights, privacy, and discrimination concerns reminiscent of social credit systems, the incident remains a corporate regulatory and consumer protection issue with negligible direct national security impact.
The reports describe an automated decision-making system that independently analyzes user data and automatically restricts or suspends accounts without human intervention or oversight.
High Risk: The system is used for automated decision-making that significantly impacts fundamental rights and access to essential services like housing, which aligns with high-risk classifications under the EU AI Act.
The AI system's primary purpose is to evaluate and score the 'trustworthiness' of Airbnb users based on their online presence and background data to decide if they should be allowed on the platform.
The reports provide consistent, detailed accounts from consumer advocacy groups (Choice) and legal experts regarding Airbnb's acquisition of Trooly and the deployment of its trustworthiness profiling algorithm. The specific impacts, including unexplained bans of sex workers and other individuals, are well-documented across multiple sources, leaving little ambiguity about the system's function and consequences.
Airbnb claims these are standard security reviews designed to protect the community, rather than an invasive social credit system.
Technology
Opaque Profiling Algorithm
Proprietary algorithm rates trustworthiness using opaque, unverified metrics.
Cross-Contextual Analysis
Evaluates unrelated data like education and social media to predict behavior.
Flawed Personality Assessment
Attempts to assess traits like narcissism using junk science social scoring.
Data Inputs
Inaccurate Online Data
Algorithm relies on potentially incorrect social media and third-party data.
Guilty by Association
Scores are adjusted based on the social media activity of friends.
Discriminatory Data Points
Screens for occupation, old charges, and sensitive personal histories.
Human Factors
Occupational Bias
Systemic bias against sex workers leads to unfair and discriminatory bans.
Lack of User Agency
Users cannot opt out of profiling or check the accuracy of collected data.
Inability to Self-Protect
Consumers cannot easily prevent data collection without going entirely offline.
Process and Methods
No Appeals Process
Airbnb does not offer a clear mechanism to contest or review bans.
Opaque Decision Communication
Banned users are given zero explanation or reasoning for their removal.
Vague Privacy Policies
Policies use vague language about profiling, hiding how data is actually used.
Management
Prioritizing Security over Fairness
Focusing on platform risk mitigation at the expense of consumer rights.
Acquisition of Trooly Startup
Integrating intrusive background-check tech without adequate oversight.
Poor Algorithmic Governance
Implementing automated decisions without human oversight or auditing frameworks.
Regulatory Environment
Significant Regulatory Gap
Lack of strong laws protecting consumers from invasive automated profiling.
Absence of Social Scoring Ban
No local frameworks equivalent to the EU ban on social scoring algorithms.
Weak Enforcement of AI Rules
Failure to enforce established public policies like the OECD AI Principles.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
