This is a research prototype. The data and analyses are preliminary and not yet validated — we'd welcome your .
Open source projects are facing a crisis due to aggressive AI web scrapers that ignore robots.txt, spoof user agents, and rotate IP addresses to bypass defenses. This activity has caused persistent service outages, increased bandwidth costs, and forced maintainers to implement restrictive measures like proof-of-work challenges. Additionally, projects are struggling with an influx of low-quality, AI-generated security reports that waste developer time.
In mid-March 2025, KDE's GitLab infrastructure was reportedly disrupted by aggressive AI web scrapers originating from Alibaba IP ranges. These bots allegedly ignored robots.txt and spoofed browser headers, which in turn purportedly overwhelmed the site and caused outages for developers. Similar incidents reportedly affected other FOSS projects like GNOME, SourceHut, and Fedora. The scraping is allegedly tied to large language model training, and reportedly imposes real costs and delays.
Domain classification, causal taxonomy, severity scores, and national security assessments were LLM-classified and may contain errors.
AI systems that fail to perform reliably or effectively under varying conditions, exposing them to errors and failures that can have significant consequences, especially in critical applications or areas that require moral reasoning.
Also classified as
AI systems generated hallucinated security vulnerability reports that appeared legitimate but were entirely false, wasting developer time.
AI companies aggressively extracted vast amounts of open-source data without consent or compensation, threatening the sustainability of the FOSS ecosystem.
The AI crawlers lack the capability to respect standard web protocols (robots.txt) and rate-limits, while the LLMs lack the capability to generate accurate bug reports, leading to severe operational failures and resource exhaustion.
AI system
Due to a decision or action made by an AI system
Unintentional
Due to an unexpected outcome from pursuing a goal
Post-deployment
Occurring after the AI model has been trained and deployed
The disruption and financial costs were caused by the automated actions of deployed AI web crawlers and LLM-generated bug reports, which were unintentional side effects of data harvesting and text generation goals.
Added categories peak at 2: Minor (Psychological Harm, Indirect)
Aggressive data harvesting by AI developers has caused operational disruptions to key open-source software projects, threatening the sustainability of critical software supply chains. Additionally, LLM-generated hallucinated security reports waste developer resources, representing an evolved threat to strategic technological infrastructure.
The AI crawlers operate independently to harvest web data, dynamically rotating IP addresses and spoofing user agents to bypass blocks without human intervention.
Minimal or No Risk: The AI systems involved are web crawlers used for data harvesting and general-purpose language models, which do not fall under high-risk or unacceptable-risk categories of the EU AI Act.
The primary function of the AI crawlers is to search, retrieve, and harvest web content for model training or real-time search features, while generative AI models were used as writing assistants to draft bug reports.
The reports provide detailed, first-hand accounts from multiple open-source maintainers, complete with specific traffic statistics, financial impacts, and technical mitigation strategies. The dual facets of the incident (aggressive crawling and hallucinated bug reports) are well-documented across several independent projects.
The incident could be viewed purely as a standard network security/DDoS issue rather than an AI safety issue, though the specific involvement of LLM training crawlers and LLM-generated bug reports makes it highly relevant to AI safety.
Technology
Evasion of standard blocking
Crawlers spoof user-agents and cycle through residential IP proxies.
Expensive endpoint targeting
Crawlers target resource-heavy git blame and log pages repeatedly.
AI-generated bug reports
LLMs generate realistic but hallucinated security vulnerability reports.
Data Inputs
Ignoring robots.txt protocols
AI crawlers deliberately ignore robots.txt and spider configurations.
High frequency data harvesting
Crawlers return to scrape the same pages as often as every six hours.
Massive scale of scraper traffic
Bots generate terabytes of requests, representing up to 97% of site traffic.
Human Factors
Wasted developer triage time
Maintainers spend valuable time refuting hallucinated security reports.
Impact on legitimate human users
Defensive measures like proof-of-work challenges cause delays for real users.
FOSS community resource constraints
Stretched-thin sysadmins must manually manage massive block lists.
Process and Methods
Lack of rate-limiting by crawlers
AI firms fail to implement rate limits on their data harvesting runs.
Public nature of FOSS infrastructure
Open collaboration model leaves git repositories publicly exposed to bots.
Inadequate defensive tools
Traditional IP blocking fails against highly distributed botnets.
Management
Prioritizing data over sustainability
AI firms prioritize data gathering over externalized infrastructure costs.
Unresponsive corporate behavior
Large AI firms remain unresponsive to complaints from project maintainers.
Predatory resource extraction
Firms leverage massive capital while shifting bandwidth costs onto FOSS hosts.
Regulatory Environment
Absence of scraping regulations
No legal frameworks exist to enforce respectful crawling or data consent.
Lack of enforcement of standards
No penalties for violating voluntary standards like robots.txt.
Lack of consent or compensation
AI firms extract open-source data without clear consent or financial return.
AI system involved
Identified by automated extraction; can differ from the alleged developers above.
