Facebook has been caught impersonating Google’s web crawler. According to a web developer Serge Bezborodov, Facebook crawls websites under a fake Googlebot user agent, to possibly bypass crawling restrictions set by robots.txt.
Facebook resorting to unethical practices by impersonating Googlebot?
Bezborodov has shared a screenshot showing multiple ‘Googlebot’ crawl requests originating from different IP addresses. A simple Whois lookup reveals these IP addresses belong to Facebook Ireland Ltd.
Interesting find: Facebook crawls websites under Googlebot user agent as fake bot. Did you see this before? pic.twitter.com/1jwMn9AoHN
— Serge Bezborodov (@sergebezborodov) September 14, 2020
To understand the consequences of having Facebook crawl websites as a fake Googlebot crawler, first, you need to under how Google Search Engine works. Google’s web crawlers help the search engine index the website content as frequently as possible.
It helps Google understand if there’s anything new on a particular website and index the content accordingly. This way, users can discover new content with the help of relevant search queries and keywords using Google’s search engine.
Here’s the most common example of a Googlebot user agent:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html).
As per Bezborodov’s screenshot, this is exactly what Facebook trying to impersonate in order to crawl websites on behalf of Google. SEO companies usually check for competitor’s information by using Googlebot impersonators.
Is Facebook trying to boost its Ad business by impersonating Googlebot?
Let’s get one thing straight. Googlebot Impersonation is considered a bad rather unethical practice. We can safely assume that all Facebook is doing here is eavesdropping on websites to figure out the tools and services they may be using.
Once Facebook has this crucial data on websites, they can easily use it to their advantage by helping relevant advertisers boost their retargeting efforts. For example, once Facebook is aware of the SEO tools your website may be using, it can use this data to help competitor tools deliver targeted ads to you.
TheWindowsClub has reached out to Facebook for comment and will update this post once it responds.
With inputs from Saurabh Mukhekar.
