Inside the benchmark: app architectures, walkthroughs of findings, and what each scanner actually caught

Post-mortem vulnerabilities often look harmless in the diff—they only surface at runtime. ProjectDiscovery says LLM code review hits just 63% precision, so PRs need validation on a running app.

Neo evolved from a sandboxed agent into a plan-execute-verify architecture with parallel subagents, driven by context, duration, and quality limits. Longest runs hit 14.7 hours and 2,398 tool calls.

LLMs find vulnerabilities fast but flood teams with unverified reports. Neo confirmed 22 CVEs across 13 open source projects at ~10-20% false positives, with working exploits and runtime evidence.

A pentest shows Claude refuses to build an LSASS dumper, but local DeepSeek and uncensored Qwen models produce a working XOR-encrypted minidumper, making AI-assisted EDR evasion nearly effortless.

The first Tales from the Trace shows XBOW combining source-code analysis with dynamic testing (via DVWA), guiding attacks with static knowledge and confirming them at runtime.

Bugcrowd's Savant Forge combines fuzz testing with symbolic execution to surface real exploitable vulnerabilities in CI/CD pipelines, while Savant Runtime tracks live third-party components in production to support SBOM compliance.