According to The Verge, the Really Simple Licensing 1.0 standard is now officially an official specification, backed by the RSL Collective which announced it in September with support from Yahoo, Ziff Davis, and O’Reilly Media. The standard expands the classic robots.txt file, letting publishers dictate licensing and compensation rules to AI web crawlers. Critically, major web infrastructure providers Cloudflare, Akamai, and Fastly now support RSL, meaning they can block non-compliant scrapers. Over 1,500 media organizations and brands endorse it, including The Associated Press, Vox Media, The Guardian, and platforms like Reddit and Stack Overflow. The release specifically lets publishers block their content from AI search features, like Google’s AI Overviews, while staying in traditional search results, a choice Google currently doesn’t offer.
The Enforcement Question
Here’s the thing: RSL itself is just a signal. It’s a text file saying “pay up.” It can’t physically stop a determined scraper. But the involvement of Cloudflare and Akamai changes the game entirely. These are the gatekeepers. If they agree to honor a site’s RSL directive and block a crawler that hasn’t paid, that’s a huge technical hurdle. It turns a polite request into a real barrier. So the standard‘s power isn’t in the code, it’s in the coalition. Without that backend enforcement, it’s just another line in a config file that bad actors would ignore.
Google’s Sticky Situation
This puts Google in a fascinating bind. The article notes Google currently forces an all-or-nothing choice: opt out of AI training and grounding, and you’re out of traditional search, too. That’s a brutal ultimatum for publishers. RSL offers a middle path, and the RSL Collective is basically calling Google’s bluff. They’re saying, “Here’s the technical standard that lets you do the right thing.” And it’s not just moral pressure. The European Commission is already investigating Google for potentially antitrust behavior by using publisher content in AI features without a clear opt-out. RSL 1.0 could become the de facto compliance tool for that. Will Google adopt it? They might have to.
Winners, Losers, And The Open Web
So who wins? Established publishers with valuable, structured content win a potential new revenue stream and control. Infrastructure companies like Cloudflare win by offering a new value-added service. But what about the open web? The Creative Commons partnership for a “contribution” option for nonprofits and individuals is a crucial, if idealistic, detail. It tries to address the fear that this will wall off the communal knowledge of forums, open-source repos, and personal blogs. But let’s be real. Is an AI company going to cut a check to every single hobbyist blogger? Probably not. The risk is a two-tier web: premium, licensed content for rich AI models, and everything else. The spirit of the open web, where linking and sharing was free, is getting a price tag. And that’s a fundamental shift.
