Web scraping is becoming a major issue — and these are the most scraped websites around today
Search engine, social media and ecommerce websites all affected
When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.
New research has revealed the scale of the growing problem of web scraping across some of the world’s biggest websites.
Smartproxy’s report of the most scraped websites of 2024 claims social media pages make up more than one-quarter (27%) of the most scraped sites.
During 2023 and the first three months of 2024, bots were most interested in search engines likeGoogle(42%), however social media accounts and community forums collectively accounted for one-third (34%) of observed scraping instances.
Google is the most scraped website
While alarming, many of the most scraped sites are thankfully not targets for personal data mining, with search engines and ecommerce leading the way.
“This trend showcases the critical need for real-time search data across various sectors, including the ever-growing AI field, where data plays a crucial role in training AI models," Smartproxy CEO Vytautas Savickas said.
“Additionally, eCommerce platforms contribute to a large portion of most scraped targets, reflecting the industry’s push for competitive intelligence needed for dynamic pricing strategies.”
Ecommerce sites, which make up around one-fifth (18%) of scraping requests, represent a growing segment. Smartproxy noted that shopping trends are emerging, and with consumers seeking more competitive prices, real-time data has become increasingly important.
Are you a pro? Subscribe to our newsletter
Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!
The report also details ecommerce scraping peaks, with shopping periods like Black Friday (+64%), Christmas (+46%) andAmazonPrime Day (+22%) all seeing considerable spikes.
“Businesses intensify their scraping efforts during these times to capture the value of data generated by the rush of online shoppers seeking discounts and special offers,” Savickas added.
More from TechRadar Pro
With several years’ experience freelancing in tech and automotive circles, Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars and the decarbonisation of personal transportation. As an avid bargain-hunter, you can be sure that any deal Craig finds is top value!
VIPRE Security Group says its new endpoint protection tools can stamp out even the latest cybersecurity threats
This new phishing strategy utilizes GitHub comments to distribute malware
I’m a die-hard Apple fan, but even I’ll admit that the Google Pixel 9 Pro is the best-looking phone of the year
