PPhying中文
Curated security research, vulnerabilities, advisories and tools for practitioners.

AI Pentesting Benchmarks Are So Bad, We Made a New One

ethiack.com | research | #ai-security | #pentesting | #benchmark | #vulnerability-discovery | #ethibench
📄 Raw .md⬇️ Download .md🔗 Original ↗

Summary

Ethiack built EthiBench to score AI pentesting agents on validated vulnerability discovery, not CTF flags, spanning 108 expert-annotated flaws across three open-source targets.
Published
Collected

Related coverage

blog · xbow.com
Grok 4.7 Is Different. So Is the Best Way to Use It.
XBOW tested Grok 4.7 on exploit crafting and agentic pentests: inside xAI's Build it improved substantially over 4.6, outside Build it slipped slightly—orchestration now matters as much as prompts.
blog · aikido.dev
Aikido launches external network pentesting
TL;DR: Aikido now pentests your external network. Attackers already test your perimeter every day, and with AI agents doing the grunt work, they're faster than ever. Now you can run the same test first. Give our agents one domain and they'll map the internet-facing systems you run and attack them from the outside, exactly like a malicious actor would.
blog · aikido.dev
Adding Aikido AI pentesting to your existing security stack
How to add Aikido's AI pentesting on top of the SAST, SCA and cloud tools you already run, from your first pentest through whitebox testing, continuous testing and remediation.
blog · aikido.dev
We burned 11.7bn tokens to find the best cyber AI model
This coverage gives security teams current context for monitoring, validation, and remediation.
blog · netspi.com
Anthropic’s Mythos Announcement: What it Means for Security Teams
A measured take on Anthropic's Mythos: autonomously chaining four vulnerabilities is a real inflection point, but teams should build their own benchmarks and keep human verification central.
research · ethiack.com
Evaluating Pentesting Agents for the Real-World
Ethiack launches EthiBench, an open evaluation protocol for AI pentesting agents using realistic targets, structured ground truth with LLM-as-a-judge matching, and richer metrics than success rates.
Back