If true, the evasion flouts Internet norms in place for more than three decades. In 1994, engineer Martijn Koster proposed the Robots Exclusion Protocol, which provided a machine-readable format for informing crawlers they weren’t permitted on a given site. Sites that their content indexed installed the simple robots.txt file at the top of their homepage. The standard, which has been widely observed and endorsed ever since, formally became a standard under the Internet Engineering Task Force in 2022.

Cloudflare isn’t the first to say that Perplexity violates the spirit if not the letter of the norm. Last year, Reddit CEO Steve Huffman told The Verge that stopping Perplexity—and two other AI engines from Microsoft and Anthropic—was a real pain in the ass.” Huffman went on to say: “We’ve had Microsoft, Anthropic, and Perplexity act as though all of the content on the Internet is free for them to use. That’s their real position.”