Major AI firms agree on unified robots.txt control for 2027
Google, OpenAI, Microsoft, and Anthropic plan to back a single robots.txt policy token, letting site owners block training data collection in one go.
Major technology developers, including Google, OpenAI, Microsoft, and Anthropic, have announced plans to introduce a unified content control mechanism for machine learning within the standard robots.txt file. The initiative builds on work by the AIPREF working group within the Internet Engineering Task Force (IETF).
Instead of maintaining separate instructions for every developer, site owners will be able to use a single policy token to set rules across all participating providers. If a site uses this token to block training while leaving specific directives like Google-Extended unstated, Google will interpret it as a general restriction. Existing single-company tools will remain functional, allowing creators to apply broad industry standards while still fine-tuning instructions for specific firms.
The proposal aligns with provisions in the EU AI Act's Code of Practice for general-purpose AI models, which recognizes robots.txt as a primary machine-readable format for web publisher preferences. Sites that explicitly permit use via Google-Extended may also appear in answers and receive referral traffic through grounding features in Gemini applications.
Timing for the rollout is set for 2027. Meanwhile, Microsoft separately announced rules letting resource owners block training on their materials in Bing and Copilot while keeping pages in standard search results.
Source: habr
Comments
Help shape the next version: Add context or suggest a correction. AI review can add points toward a rewrite. Reviews and updates may take time; a full meter does not guarantee a new version.