Perplexity revealed a response to Cloudflare’s claims that it disrespects robots.txt and engages in stealth crawling. Perplexity argues that Cloudflare is mischaracterizing AI Assistants as net crawlers, saying that they shouldn’t be topic to the identical restrictions since they’re user-initiated assistants.
Perplexity AI Assistants Fetch On Demand
In accordance with Perplexity, its system doesn’t retailer or index content material forward of time. As an alternative, it fetches webpages solely in response to particular consumer questions. For instance, when a consumer asks for latest restaurant opinions, the assistant retrieves and summarizes related content material on demand. This, the corporate says, contrasts with how conventional crawlers function, systematically indexing huge parts of the online with out regard to speedy consumer intent.
Perplexity in contrast this on-demand fetching to Google’s user-triggered fetches. Though that’s not an apples-to-apples comparability as a result of Google’s user-triggered fetches are within the service of studying textual content aloud or web site verification, it’s nonetheless an instance of user-triggered fetching that bypasses robots.txt restrictions.
In the identical approach, Perplexity argues that its AI operates as an extension of a consumer’s request, not as an autonomous bot crawling indiscriminately. The corporate states that it doesn’t retain or use the fetched content material for coaching its fashions.
Criticizes Cloudflare’s Infrastructure
Perplexity additionally criticized Cloudflare’s infrastructure for failing to differentiate between malicious scraping and legit, user-initiated site visitors, suggesting that Cloudflare’s strategy to bot administration dangers overblocking companies which can be performing responsibly. Perplexity argues {that a} platform’s incapability to distinguish between useful AI assistants and dangerous bots causes misclassification of reputable net site visitors.
Perplexity makes a robust case for the declare that Cloudflare is obstructing reputable bot site visitors and says that Cloudflare’s choice to dam its site visitors was based mostly on a misunderstanding of how its expertise works.
Learn Perplexity’s response:
Brokers or Bots? Making Sense of AI on the Open Internet
