Cloudflare's Radical Fix: Haunting Half the Web, Taunting the Other Half. What SEOs need to know.

Cloudflare thinks it knows better. It says it has a better (radical) solution for a problem that has been haunting half of the web. And that will taunt the second half if implemented

The company says it has developed the ultimate (my words, not theirs)to block ai crawlers. Let’s deep dive and see what all the hype is about.

Cloudfare has been buzzing lately. The tech giant pretends it has the panacea for a problem that has been haunting and intriguing the web community from developers and webmaster to jealous SEOs and media outlets ….

What did cloudflare come up with ?

ChatGPT was introduced in X year. And ever since, it exploded and enflamed the web with it. To the extent of fulling the web with garbage content as some would say.That said, filling the web with garbage content isn’t something new or exclusive to new AI tools. . People have been brilliantly doing such for decades. 4chan, 8chan, reddit, are living proof. But I digress.

These AI tools would’ve been fine and safe(from a regulators and users perspective) if corrupting the web is the only crime they’re charged of. But that’s not the case.

Ai tools like ChatGPT, perplexity, Gemini and al are also accused of stealing human generated content (by unlawfully crawling sites and use data to spin similar content to use a technical jargon) from-hard working-human content creators (mostly writers) to do so.

This is where cloudflare kicks in.

Cloudfate is our savor

In a blog post shared on their website, the company announced it has figured out a way to, and I quote : “Declare your AIndependence: block AI bots, scrapers and crawlers with a single click”.

This comes after the company launched a similar solution to “challenge” ai bots and prevent them from making too many requests .

After doing so, the company noticed the majority of its customers weren’t exactly happy with the performance of the latter.

In a pull from the company. 85 % of users expressed their desire to completely block ai bots.

Side note : I bet the writer is happy who his title. It’s a good title. I give credit where credit is due.

More interesting, the company says the tool is free and available to all users. Which makes it all the more appealing…

What’s all the hype about Cloudlafre ?

Blocking ai crawlers or crawlers of any kind isn’t a novelty. SEO specialists like myself are taught to manipulate what can search engines crawl (explore) through sitemaps and robots.txt…That said, the instructions we usually put on these files are directives. Not actual instructions. And search engine crawlers (bots) have the freedom to obey or disobey these directives.

With a noticeable increase in AI bots activity through the web, the company decided to take a (supposedly) more radical step.

As we can see, there are four AI bots that dominate the internet: Bytespider, Amazonbot, ClaudeBot, and GPTBot. I’ve coined them the "Big 4 Malicious AI Bots." Proper credit is required if you decide to use this expression…

Interestingly though,the biggest AI thief isn’t the number one blocked ai bot. Victim of its own success, ChatGPT is the most blocked AI bot by Cloudflare users…

This is where cloud fare might have an edge

Cloudflare’s solution can be turned on with a toggle. Unlike robots.txt and sitemap files which that necessitate certain technical knowledge and know how.
The solution is not a mere directive that AI bots can just ignore(that’s not how we roll!). Cloudflare says it enforces the crawlers to not crawl content. Without going into details on how they do it.
The tech giant pretends the machine learning model it’s using can recognize ai bots. Even if they act like normal browsers.
They’ve also set up a reporting tool where any Cloudflare customer can submit reports of an AI bot scraping websites without permission. Monitor AI bots on their website and submitting feedback. The latter is pretty cool, I must admit…

Cloudflare is a tech behemoth. With over 7 millions websites(users), the company has a considerable weight in the tech arena. If they walk the walk and talk the talk and their ai blocking solution turns out to work. AI actors will have a hard time scraping any more data.

Want to know more about the AI bot blocker ? Check out the full article where CloudFlare explains how to Declare your AIndependence: block AI bots, scrapers and crawlers with a single click