It looks like Microsoft Bing is unable to properly index websites hosted on Cloudflare. For the past several weeks, there have been numerous complaints that Bing has some issues indexing pages hosted on Cloudflare. In fact, according to one report, Microsoft has acknowledged the issue and is currently consulting with engineers at Cloudflare to get it fixed. The size and severity of the issue remain unknown at the moment.
Is Microsoft Bing unable to crawl Cloudflare-hosted sites?
In a reply to a community post, Cloudflare has pinned the blame on Microsoft and said that the US-based web-infrastructure and website-security company is not responsible for this indexing issue:
“Microsoft has acknowledged this is a legit crawler and this is an issue on their side. They are working to resolve (by setting it to conform to their published expected behavior),” Cloudflare said.
Several website owners have complained that their Cloudflare-hosted Websites block IP addresses of Microsoft bing crawler.
One user said:
“I have received many Bing Webmaster alerts in the past few days alerting that Bing bots are not able to crawl my website, getting a 403 error (error below is in Portuguese, but mostly says that bing detected pages with the 403 error, the page that appears when a WAF rule is activated)”
Another user said:
“If Microsoft does not fix it, then it may not solve the issue, that is true. But ultimately they have to fix it and, yes, these DNS changes certainly are possible and are completely under Microsoft’s control.”
Meanwhile, Principal Program Manager of Microsoft Bing Fabrice Canel has provided users with “best practices” in case a site is not getting indexed properly. They are essential general guidelines on how to investigate potential issues with the help of a URL Inspection Tool.
For Cloudflare, some sites reported that a small percentage of Bing crawler was not DNS resolving appropriately and we were crawling outside of usual IPs for few days. We fixed that yesterday.
— Fabrice Canel (@facan) October 27, 2020
Canel further said that some Cloudflare-hosted sites had encountered issues with Bing crawler failing to resolve DNS appropriately and the company was crawling outside of usual IPs for a few days, which has been fixed.
In related news, Google too appears to be facing issues indexing sites as well as selecting the right canonical tags for URLs.
Read: 5 areas where Microsoft Bing Search outperforms Google.
