coyotino [he/him]@beehaw.org to Technology@beehaw.orgEnglish · 3 months agoMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.coexternal-linkmessage-square24fedilinkarrow-up120arrow-down10
arrow-up120arrow-down1external-linkMicrosoft and Reddit Are Fighting About Why Bing’s Crawler Is Blocked on Redditwww.404media.cocoyotino [he/him]@beehaw.org to Technology@beehaw.orgEnglish · 3 months agomessage-square24fedilink
minus-squaremozz@mbin.grits.devlinkfedilinkarrow-up4·3 months ago I know this because i wrote a page that IP bans anything that visits it, and l also put it as a not allowed spot in the robots.txt file. This is fuckin GENIUS
minus-squareMoonrise2473@feddit.itlinkfedilinkarrow-up2·3 months agoonly if you don’t want any visits except from yourself, because this removes your site from any search engine should write a “disallow: /juicy-content” and then block anything that tries to access that page (only bad bots would follow that path)
This is fuckin GENIUS
only if you don’t want any visits except from yourself, because this removes your site from any search engine
should write a “disallow: /juicy-content” and then block anything that tries to access that page (only bad bots would follow that path)
That’s exactly what was described…?