coyotino [he/him]@beehaw.org to Technology@beehaw.orgEnglish · 3 months agoMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.coexternal-linkmessage-square24fedilinkarrow-up120arrow-down10
arrow-up120arrow-down1external-linkMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.cocoyotino [he/him]@beehaw.org to Technology@beehaw.orgEnglish · 3 months agomessage-square24fedilink
minus-squareSkull giver@popplesburger.hilciferous.nllinkfedilinkEnglisharrow-up1·3 months agoI think Reddit serves Googlebot a different robots.txt to prevent issues. For instance, check Google’s cached version of robots.txt: it only blocks stuff that you’d expect to be blocked.
minus-squareZoop@beehaw.orglinkfedilinkarrow-up1·3 months ago User-Agent: bender Disallow: /my_shiny_metal_ass Ha!
I think Reddit serves Googlebot a different robots.txt to prevent issues. For instance, check Google’s cached version of robots.txt: it only blocks stuff that you’d expect to be blocked.
Ha!