Google: robots.txt only stops web crawlers, not AI training
Adding Disallow rules gave site owners an illusion of safety, but Google’s <i data-gender="m">Gary Illyes stressed that the legacy protocol governs crawler visits, not how data is used to train machine learning models.
Over the past two years, webmasters rushed to paste Disallow directives into their robots.txt files for bots like GPTBot and CCBot, assuming that blocking an automated crawler would shield their articles from generative systems. Speaking to industry outlets, Google search analyst Gary Illyes pushed back on that assumption, reiterating a fundamental technical boundary: robots.txt controls crawling, not model training.
The distinction rests on the mechanics of the protocol. Drafted decades before large language models existed, robots.txt is a lightweight advisory file designed to manage server load and tell visiting bots which URLs they should avoid reading. It does not govern the fate of text once gathered, nor can it dictate how third parties process archived web content.
Blocking an active AI bot via robots.txt cannot retroactively scrub pages that were archived previously, nor does it affect data purchased through commercial syndication or harvested via independent third-party mirrors. Furthermore, the file remains an honor system rather than a technical barrier: while search engines and major AI firms generally respect crawl rules, any actor hunting training text can disregard the file without technical friction.
Opt-out mechanisms and copyright restrictions for model training belong to licensing agreements and regulatory frameworks rather than crawler etiquette. robots.txt continues to perform its original role of crawler management, but treating it as a copyright shield misinterprets what the protocol was ever capable of enforcing.
Source: habr
Comments
Help shape the next version: Add context or suggest a correction. AI review can add points toward a rewrite. Reviews and updates may take time; a full meter does not guarantee a new version.