Cloudflare wants to help stomp out AI scraper bots for good
AI bot blocking is about to become more effective with new Cloudflare tool
When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.
Cloudflare has announced a new feature that allowsweb hostingcustomers to block AI bots from scraping their website content without permission.
Companies responsible forAI tools, particularly those involved with training, have come under fire in recent months over their use of content published online.
Cloudflare says its new tool responds to widespread discontent among website owners and publishers about AI bots harvesting data and aims to protect content creators on the internet.
Cloudflare tool to stop AI bot scraping
In its announcement,Cloudflarewrote: “We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly. To help, we’ve added a brand new one-click to block all AI bots.”
The tool has become available for all customers, including those on Cloudflare’s free tier.
Previously, website owners have been able to use the robots.txt file to present bots from crawling their sites, but to varying degrees of success. The new tools offers a more robust defence against content scraping by using machine learning to identify and block bots.
Cloudflare’s system relies on digital fingerprinting to differentiate bots from legitimate users.
Are you a pro? Subscribe to our newsletter
Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!
The company said that the enhanced protection comes as a response to significant AI bot activity across its network. Bytespider, GPTBot and ClaudeBot were among the most commonly observed, but despite previous blocking attempts, AI bots accessed more than one-third (39%) of the top one million sites served by Cloudflare in June.
Thankfully, the new feature is available to all Cloudflare customers, including those on the free tier. By navigating to the Security > Bots menu and blocking “AI Scrapers and Crawlers,” users can protect their website from unwanted AI bot activity.
More from TechRadar Pro
With several years’ experience freelancing in tech and automotive circles, Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars and the decarbonisation of personal transportation. As an avid bargain-hunter, you can be sure that any deal Craig finds is top value!
Adobe’s decision to eliminate perpetual licensing for its Elements software has stirred controversy among consumers
VIPRE Security Group says its new endpoint protection tools can stamp out even the latest cybersecurity threats
Huge Black Friday Samsung sale: save up to $1,900 on QLED, OLED TVs, and more
