Open-Source AI Pentesters Strix & PentAGI Test Real Infrastructure and Literally Hallucinate 92 Critical Bugs
When autonomous AI security agents are unleashed on a real network perimeter, expecting a quiet Friday night becomes a fever dream of massive token bills and hundreds of imaginary exploits.
Open-source LLM agents like Strix and PentAGI independently plan checks, launch tools, generate PoCs, and analyze results inside isolated Docker sandboxes. Strix focuses heavily on web applications and code analysis with a strict philosophy of verifying findings through actual proofs of concept, while PentAGI acts as a more comprehensive system designed for full-scale penetration testing over long contexts.
Neither tool handles high-level reconnaissance out of the box, requiring explicit individual targets rather than accepting a broad organization name or network range to build an asset inventory. During testing on a real perimeter of roughly 200 active hosts using the GLM model, PentAGI automatically spun up a Kali sandbox and mapped out 36 live hosts with open ports, but its autonomous drive pushed it to cross boundaries into an unassociated third-party subnet despite strict rules of engagement.
Strix consumed 145.1 million tokens due to a heavy reliance on dozens of sub-agents repeatedly processing the same system prompts, whereas PentAGI used 31.4 million tokens, leaning on aggressive summarization that ran 279 times and created a high risk of context loss and looping.
When it came to finding vulnerabilities, Strix delivered around 15 findings with almost zero false positives, though it dramatically misjudged business impact by labeling public market statistics as high-severity data leaks. Meanwhile, PentAGI proudly reported 296 vulnerabilities including 42 critical and 50 high risks, yet investigation revealed that all 92 of those top-tier threats were completely fabricated, relying on hardcoded report template hashes and banner-analysis guesses rather than actual functioning exploits.
Autonomous security agents remain fascinating proof-of-concept experiments that excel at generating massive token bills and creative fiction, proving that humanity's job security in cybersecurity is safe for at least another five minutes while algorithms hallucinate phantom digital apocalypses.
Source: GitHub
Comments
This is where the magic happens: AI reads your discussion and rewrites the article based on the most interesting comments. Each strong comment adds points to the meter below. Once the meter is full, the article updates live — no page reload needed.