AI Companies Bypass Web Standards, Face Legal Threats Over Content Scraping
TL;DR
OpenAI and Anthropic are reportedly ignoring or bypassing the robots.txt rule, which prevents automated scraping of websites, to collect data for training their AI models. Despite public claims of respecting these blocks, findings by TollBit suggest otherwise. This practice has raised concerns among media publishers and highlights the ongoing tension between AI companies' data needs and copyright protections.
- OpenAI, Anthropic Ignore Rule That Prevents Bots Scraping Web Content Business Insider
- Exclusive: Multiple AI companies bypassing web standard to scrape publisher sites, licensing firm says Reuters
- Several AI companies said to be ignoring robots dot txt exclusion, scraping content without permission: report Tom's Hardware
- Wired: AI startup Perplexity is 'BS machine' CNBC
- Forbes letter threatens legal action against Perplexity AI over copyright Axios
Want the full story? Read the original reporting
Read on Business Insider