PPhying中文
Curated security research, vulnerabilities, advisories and tools for practitioners.

AI Red Teaming Explained by AI Red Teamers

hackerone.com | blog | #ai-security | #attack-surface | #prompt-injection | #jailbreak | #llm-security | #application-security | #ai-red-teaming
📄 Raw .md⬇️ Download .md🔗 Original ↗

Summary

AI red teamers unpack common misconceptions about AI red teaming and explain its role in securing AI-powered applications.
Published
Collected

Related coverage

blog · hackerone.com
AI Security 101: What is AI Red Teaming?
AI red teaming stress-tests AI systems against adversarial attacks, targeting jailbreaks, prompt injection and over-permissioned agents so flaws are fixed before attackers can exploit them.
blog · hackerone.com
LLMs adversarial testing with HackerOne: Join the CTF!
HackerOne and Hack The Box launch a 10-day AI red teaming CTF built from real engagement scenarios: indirect prompt injection, safety filter bypasses, and privilege escalation via language alone.
blog · hackerone.com
How Anthropic’s Jailbreak Challenge Put AI Safety Defenses to the Test
Anthropic and HackerOne ran an AI red teaming challenge on a demo Claude 3.5 Sonnet to validate Constitutional Classifiers that block CBRN-related jailbreaks, awarding $55,000 to four teams.
blog · bishopfox.com
Ready to Hack an LLM? Our Top CTF Recommendations
Curated LLM security CTFs and playgrounds: Bishop Fox's Local LLM CTF, OWASP FinBot, Steve's Chat Playground, PortSwigger's Web LLM Attacks, and Wild LLaMa cover prompt injection and jailbreaks.
blog · blog.sentry.security
ChatML Role Injection (CRI)
Attackers can inject system roles into ChatML message arrays to override developer instructions, invoke hidden tools, and pad payloads for denial-of-wallet DoS; strict schema validation is key.
blog · blog.sentry.security
AI Red Team: Safety vs. Security
AI red teaming should separate security from safety: security covers compromise of AI systems like prompt injection, while safety addresses harmful misuse such as LLM-enabled malware.
Back