Cloudflare gives creators more control over AI crawlers
Thank you for reading this post, don't forget to subscribe!
The company wants to give users a new option when it comes to AI crawlers and that is monetisation.
Cybersecurity and network services provider Cloudflare has announced an update that will give content creators, publishers and website operators more control over who can access their work via AI crawlers.
AI crawlers, also known as bots or AI scrapers, are software programmes designed to gather information from websites, typically with the goal of training large language models (LLMs) and generating human-like, more accurate responses.
On their website, Cloudflare stated, “If a creator wants to block all AI crawlers from their content, they should be able to do so. If a creator wants to allow some or all AI crawlers full access to their content for free, they should be able to do that, too. Creators should be in the driver’s seat.”
However, the organisation believes there is a third option. Publishers, creators and website owners should have the power to grant AI crawlers access to their platforms, but for fair compensation, which can be a difficult process that requires striking the right deal and often may only be available to platforms capable of scaling and holding leverage.
As a result, Cloudflare is introducing the experimental ‘pay per crawl’ system, which will integrate with existing web infrastructure, leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access. Currently it is in the private beta phase.
“Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.”
Site owners will have the power to control their monetisation strategy and can select three options: allow, which gives complete and free access; charge, which will require payment at the set price; and block, which denies access entirely.
“At its core, pay per crawl begins a technical shift in how content is controlled online. By providing creators with a robust, programmatic mechanism for valuing and controlling their digital assets, we empower them to continue creating the rich, diverse content that makes the internet invaluable.
“We expect pay per crawl to evolve significantly. It’s very early, we believe many different types of interactions and marketplaces can and should develop simultaneously. We are excited to support these various efforts and open standards.”
Cloudflare was recently a victim of the major global outage affecting a range of organisations across Europe, the US and Asia, such as Google, OpenAI and Spotify.
Don’t miss out on the knowledge you need to succeed. Sign up for the Daily Brief, Silicon Republic’s digest of need-to-know sci-tech news.
