Reddit Blocks Bing and Other Search Engines from Crawling Its Site
Reddit recently updated its robots.txt file, effectively blocking Bing and several other search engines from crawling the site. According to a Microsoft representative speaking to Search Engine Land, Bing ceased crawling Reddit after the updated robots.txt file took effect on July 1. This update prohibits all forms of crawling by search engines, except Google, despite earlier speculation suggesting broader restrictions.
READ: New York Businesses Ordered to Require Masks Indoors or Vaccine Proof
The development came to light earlier today when Mark Williams-Cook observed a drop in Reddit’s presence within Bing’s search index. Subsequently, multiple media outlets picked up on the news. Confirmation was sought to verify that Bing’s crawlers were indeed blocked, given Reddit’s practice of serving different versions of its robots.txt file based on user type, as previously explained.
Microsoft clarified its stance, emphasizing adherence to the robots.txt standard and confirming the halt in crawling activities on Reddit:
“We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”
In response, Reddit’s spokesperson, Tim Rathschmidt, provided a statement to The Verge:
“This is not related to our recent partnership with Google. Discussions with multiple search engines have been ongoing, but agreements haven’t been reached with all due to concerns over the use of Reddit content, including for AI purposes.”
The strategic move by Reddit, bolstered by its licensing deal with Google, allows it to exert control over how its content is accessed by search engines and AI tools. While Google continues to drive substantial traffic to Reddit, experimenting with enhanced visibility in search results, the implications for other major websites and smaller publishers are worth considering.
The situation prompts reflection on whether other large platforms may adopt similar tactics, potentially impacting the accessibility and visibility of content from smaller publishers and content producers.
