PPhying中文
Curated security research, vulnerabilities, advisories and tools for practitioners.

AI Security 101: What is AI Red Teaming?

hackerone.com | blog | #ai-security | #prompt-injection | #jailbreak | #ciso | #hackerone | #ai-red-teaming | #threat-modeling
📄 Raw .md⬇️ Download .md🔗 Original ↗

Summary

AI red teaming stress-tests AI systems against adversarial attacks, targeting jailbreaks, prompt injection and over-permissioned agents so flaws are fixed before attackers can exploit them.
Published
Collected

Related coverage

blog · blog.sentry.security
AI Red Team: Safety vs. Security
AI red teaming should separate security from safety: security covers compromise of AI systems like prompt injection, while safety addresses harmful misuse such as LLM-enabled malware.
blog · hackerone.com
Measuring The Fix: How We Evaluate Frontier Models on Remediation Quality
HackerOne tested frontier models on security fix quality and found the judge mattered more than the model. See how CTEM remediation earns engineer trust.
blog · hackerone.com
Inside the AI Red Teaming CTF: What 200+ Players Taught Us About Breaking and Defending LLMs
In a 10-day AI red teaming CTF with 217 active players, output manipulation succeeded ~82.5% and data extraction ~74.6%; layered defenses cut completion to ~34% on the hardest multi-step challenge.
blog · hackerone.com
LLMs adversarial testing with HackerOne: Join the CTF!
HackerOne and Hack The Box launch a 10-day AI red teaming CTF built from real engagement scenarios: indirect prompt injection, safety filter bypasses, and privilege escalation via language alone.
blog · hackerone.com
How Anthropic’s Jailbreak Challenge Put AI Safety Defenses to the Test
Anthropic and HackerOne ran an AI red teaming challenge on a demo Claude 3.5 Sonnet to validate Constitutional Classifiers that block CBRN-related jailbreaks, awarding $55,000 to four teams.
blog · bishopfox.com
Ready to Hack an LLM? Our Top CTF Recommendations
Curated LLM security CTFs and playgrounds: Bishop Fox's Local LLM CTF, OWASP FinBot, Steve's Chat Playground, PortSwigger's Web LLM Attacks, and Wild LLaMa cover prompt injection and jailbreaks.
Back