Reddit says that it doesn’t want companies scraping the site for AI. Microsoft says it’s not doing that.
  • Moonrise2473@feddit.it
    2·
    2 months ago

    only if you don’t want any visits except from yourself, because this removes your site from any search engine

    should write a “disallow: /juicy-content” and then block anything that tries to access that page (only bad bots would follow that path)