Humans Generate Just 2% of Linux Kernel Git Traffic

Infrastructure admin Konstantin Ryabitsev revealed that scrapers and automated bots generate approximately 98% of all web traffic hitting the git.kernel.org repository servers. The infrastructure handles roughly 6 million daily commit requests, turning critical open-source hardware into an unintended data buffet for training pipelines.

Around two-thirds of incoming connection attempts are immediately blocked by Anubis, a proof-of-work defense system that forces visiting browsers to solve a JavaScript-based SHA-256 hashing puzzle before granting entry. While the remaining 33% of requests successfully pass the test, analysis of browsing patterns shows that almost none of them belong to human developers inspecting code.

Instead of cloning the repository once via standard Git commands like civilized software engineers, automated scrapers bombard the web interface with billions of redundant requests across random old branches. To bypass firewall bans, these crawlers route their activity through millions of residential IP addresses hijacked via botnets and sketchy browser extensions, making simple network-level blocking completely ineffective.

The crawler invasion currently forces 14 to 16 out of 90 available CPU cores across five server nodes to constantly render web views of old commits solely for bots. To keep the infrastructure from collapsing, maintainers have begun stripping down web navigation links, disabling heavy server operations, and restricting anonymous browsing capabilities.

The open-source community is now forced to degrade tools for actual human engineers just to feed the unquenchable thirst of automated scrapers whose operators cannot even be bothered to learn basic command-line utilities.

Source: kernel.org