Reddit says that it doesn’t want companies scraping the site for AI. Microsoft says it’s not doing that.
  • mozz@mbin.grits.dev
    4·
    3 months ago

    I know this because i wrote a page that IP bans anything that visits it, and l also put it as a not allowed spot in the robots.txt file.

    This is fuckin GENIUS

    • Moonrise2473@feddit.it
      2·
      3 months ago

      only if you don’t want any visits except from yourself, because this removes your site from any search engine

      should write a “disallow: /juicy-content” and then block anything that tries to access that page (only bad bots would follow that path)