PPhying中文
Curated security research, vulnerabilities, advisories and tools for practitioners.

What is AI red teaming?

bugcrowd.com | blog | #ai-security | #offensive-security | #llm | #bugcrowd | #ai-red-teaming | #threat-modeling
📄 Raw .md⬇️ Download .md🔗 Original ↗

Summary

AI red teaming simulates adversarial attacks on AI models to find and patch safety and security vulnerabilities. Now that every company uses AI, it starts with threat modeling and clear objectives.
Published
Collected

Related coverage

blog · bugcrowd.com
Savant Forge: Automated offensive testing to keep pace with AI-generated code
Bugcrowd's Savant Forge combines fuzz testing with symbolic execution to surface real exploitable vulnerabilities in CI/CD pipelines, while Savant Runtime tracks live third-party components in production to support SBOM compliance.
blog · bugcrowd.com
AI lectures with Dr. Brumley Part 3 | Securing the AI attack surface
Part three of Bugcrowd's AI lecture series with Dr. Brumley maps AI threats along three axes — knowledge, position and goal — and surveys classical ML attacks and recent LLM-era incidents.
blog · specterops.io
AI for Offensive Security: What Works, What Does Not, and How to Adopt It
SpecterOps maps where AI helps offensive security (recon, tooling, vuln discovery) and where it still fails, stressing that results hinge on the harness: context, tools, state, independent validation, and human judgment.
blog · blog.sentry.security
AI Red Team: Safety vs. Security
AI red teaming should separate security from safety: security covers compromise of AI systems like prompt injection, while safety addresses harmful misuse such as LLM-enabled malware.
blog · projectzero.google
Project Naptime: Evaluating Offensive Security Capabilities of Large Language Models
Project Naptime evaluates the offensive security capabilities of large language models, presenting a methodology, tooling setup, and early successes finding real vulnerabilities.
vulnerability · projectdiscovery.io
Beyond the Model: Neo Hunts, Exploits, and Proves 22 Zero-Days.
LLMs find vulnerabilities fast but flood teams with unverified reports. Neo confirmed 22 CVEs across 13 open source projects at ~10-20% false positives, with working exploits and runtime evidence.
Back