When a website puts up a “no AI bots allowed” sign, Perplexity apparently sees it as more of a suggestion.
Cloudflare’s research team caught the AI company red-handed, scraping sites that explicitly blocked AI crawlers. The scale? Tens of thousands of domains. Millions of requests daily. That’s not an accident.
Perplexity’s tactics read like a spy thriller. They’re changing user-agent strings to pretend they’re regular browsers. Switching between different networks to dodge IP blocks. Even using third-party scraping APIs like Crawlbase that handle the dirty work – rotating IPs, bypassing CAPTCHAs, rendering JavaScript. Some scrapers turn to solutions like Bright Data’s Web Unlocker API to break through HTTP request barriers and 403 Forbidden errors.
They’re masquerading as regular browsers, switching networks, using third-party scraping APIs to bypass blocks.
Clever? Sure. Ethical? That’s another story. The company’s CEO couldn’t even define plagiarism when asked during an interview, raising questions about their content ethics.
Cloudflare had to break out machine learning and network analysis.
References
- https://brightdata.com/blog/web-data/web-scraping-with-perplexity
- https://techcrunch.com/2025/08/04/perplexity-accused-of-scraping-websites-that-explicitly-blocked-ai-scraping/
- https://dev.to/lacrymosatech/how-i-use-perplexity-ai-for-web-scraping-in-python-and-why-you-probably-should-too-2564
- https://cyberscoop.com/perplexity-blocks-on-crawlers-cloudflare/
- https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/
