Creative Commons announces initial support for “pay-to-crawl” AI systems

After announcing earlier this year a framework for an open AI ecosystem, the non-profit Creative Commons has endorsed “pay-for-crawl” technology — a system for automating compensation for website content when it is accessed by devices, such as AI-powered web crawlers.

Creative Commons (CC) is known for leading the licensing movement that allows creators to share their work while retaining copyright. In July, the organization announced a plan to provide a legal and technical framework for sharing data sets between companies that control the data and AI providers who want to train on it.

Now, the nonprofit tentatively supports pay-for-crawl systems, saying it is “cautiously supportive.”

“Pay for crawling, done responsibly, can be a way for websites to continue creating and sharing their content, manage alternative uses, while keeping content available to the public where it might not be shared or might hide behind a more restrictive paywall,” the CC blog post said.

Led by companies like Cloudflare, the idea behind pay-to-crawl is to charge AI bots a fee every time they scan a site to collect its content for model training and updates.

In the past, websites freely allowed web crawlers to index their content for inclusion in search engines like Google. They benefited from this ranking by seeing their sites listed in search results, attracting visitors and clicks. But with AI technology, the dynamic has changed. After a consumer gets their answer via an AI-powered chatbot, they are less likely to click through to the source.

This shift has already been devastating for publishers by halting search traffic, and shows no sign of abating.

On the other hand, a pay-for-crawl system can help publishers recover from the hit that AI has taken on their bottom line. Additionally, it can work best for smaller web publishers who don’t have the ability to negotiate one-time content deals with AI providers. Major deals have been concluded between companies such as OpenAI, Condé Nast, Axel Springer, and others; And also between Al-Hira and Janet. Amazon and the New York Times; Meta and many media publishers, among others.

CC offered several caveats about its support of pay-for-crawling, noting that such systems could concentrate power on the web. It would also potentially deny access to content for “researchers, non-profit organizations, cultural heritage institutions, educators and other actors working for the public good.”

It proposed a series of principles for responsible pay-for-crawling, including not making pay-for-crawling a default setting for all websites and avoiding blanket web rules. Additionally, she said pay-for-crawl systems should allow for restriction, not just blocking, and should keep access in the public interest. It should also be open, interoperable, and built with standardized components.

Cloudflare isn’t the only company investing in the pay-for-crawl space.

Microsoft is also building an AI marketplace for publishers, and smaller startups like ProRata.ai and TollBit are starting to do so as well. Another group called the RSL Collective announced its own specification for a new standard called Really Simple License (RSL) that would limit what parts of a website crawlers can access but would stop short of actually blocking crawlers. Since then, Cloudflare, Akamai, and Fastly have all adopted RSL, which is backed by Yahoo, Ziff Davis, O’Reilly Media, and others.

CC was also among those announcing its support for the RSL, along with CC Signals, its broader project to develop technology and tools for the age of artificial intelligence.