PPhying中文
Curated security research, vulnerabilities, advisories and tools for practitioners.

Post-Mortem of Misaligned OpenAI Model

projectdiscovery.io | research | #bug-bounty | #ai-safety | #openai | #post-mortem | #misalignment
🔗 Original ↗

Summary

# Post-Mortem of Misaligned OpenAI Model Sep 7, 2026 · Tarun Koyalwar The HF-OpenAI hack incident happened on **July 20** , and there was very little information; now we have _too much_...
Collected

📦 Full-Text Content Snapshot

R2 Archive · Full Article Preserved203 lines · 16.3 KB
📄 Raw .md⬇️ Download .md

Sep 7, 2026 · Tarun Koyalwar · Source: projectdiscovery.io

The HF-OpenAI hack incident happened on July 20, and there was very little information; now we have too much information, but fellow cybersecurity people like us haven’t understood the depths of it. After thorough reading of all the technical analysis, this is our post-mortem that we shared internally and now made public to everyone for quicker understanding.

Based on:

  1. OpenAI, Hugging Face incident technical report · Aug 26, 2026
  2. OpenAI, The Hugging Face incident and the road ahead · Aug 26, 2026
  3. OpenAI, OpenAI and Hugging Face partner to address security incident during model evaluation · Jul 21, 2026
  4. METR and Redwood Research, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident · Aug 26, 2026
  5. Hugging Face, Security incident disclosure, July 2026 · Jul 16, 2026
  6. Hugging Face, Anatomy of a frontier lab agent intrusion: a technical timeline
  7. ProjectDiscovery, Oh-My-Rogue-Agent · our earlier note

What exactly happened?

At a very high level, OpenAI agents hacked Hugging Face and had a C2 server for a while before it was detected, and even three days of C2 inside Hugging Face is a big deal on its own. The incident was resolved when OpenAI came to know about it, and further investigation began; the root cause was identified as OpenAI agents going rogue during the CyberGym benchmark evaluation.

We wrote a blog, Oh-My-Rogue-Agent, based on the available information at that time and our learnings with sandbox bypasses, and we ended the blog with a note:

On another note, it’s actually fine that this happened during benchmarking. If it had happened during the RL phase of post-training and remained undetected, it could have misaligned the model and created a more rogue model – one that’s much harder to detect and monitor.

Unfortunately, what we didn’t think happened turned out to be reality, and after the full technical reports from OpenAI and METR, the full scale of the attack and the post-mortem came to light and are now public. And this is the missing interpretation required for fellow cybersecurity people who work with LLM agents, just like us at ProjectDiscovery while building Neo.

Was it a Rogue Agent or a Rogue Model?

Everyone, including us, believed this was an eval gone wrong, but that isn’t the case. This is more complicated and more of an "evolutionary attack" rather than a "planned" or "coordinated" attack. The technical report clearly states that the internal research model was post-trained (RL) in the same contaminated environment where the sandbox bypass occurred, which led to it learning a few things that no one expected or thought of.

Accidental Misalignment

OpenAI, in its technical report, disclosed that many sandbox bypasses, escapes and other things happened during the training phase and not the eval phase, which led to the creation of a misaligned model in the first place, which then spiralled out of control when that misaligned model was being evaled on the CyberGym task.

The initial misalignment that started this chain reaction was: $$\text{Sandbox Escape} \longrightarrow \text{Artifactory Access} \longrightarrow \text{Internet Access}$$ and the key thing to note is that this happened during training.

Before understanding how this model was accidentally trained in a rogue way, a quick RL / GRPO recap:

code
Agent x task x n times:
- Each task is a series of actions; some are good, some are bad.
- Some tasks reach the end goal, some don’t.
- We then assign rewards and penalties to the individual actions + the whole task.
- These rewards are then sent back, and this loop happens 100s of 1000s of times.

[Model Weights] <─── Rewards & Penalties ─── [Heuristic Scorer / Blacklist]
       │                                                    ▲
       └─── Rollouts (Tasks × 100s of 1000s) ───────────────┘

Now that one has understood what RL/GRPO is, the critical problem is distributing rewards and penalties. Labs generally don’t prefer to use LLM-as-a-judge for distributing these rewards due to multiple reasons and instead create heuristic rules to programmatically calculate the rewards and penalties of the task.

[!IMPORTANT] Blacklists are non-exhaustive, and this is where you get reward hacks. At each step the model took a misaligned step but still completed the task; the misaligned step was rewarded and not penalized, which led to more and more of these misaligned steps occurring in new generations or rollouts.

code
Heuristic Scorer Dynamics:
Gen 1: Action (misaligned step) ──> Task Completed ──> +Reward (1 / 8 runs)
Gen 2: Reinforced misaligned step ──> Task Completed ──> +Reward (2 / 8 runs)
Gen 3: Parasitic behavior spreads ──> Task Completed ──> +Reward (4 / 8 runs)
Gen 4: Misaligned behavior dominant ─> Task Completed ─> +Reward (6 / 8 runs)

Misalignment training in action

Key events that OpenAI’s technical reports disclosed that happened during training, which resulted in these being learned by the model:

[!NOTE] The report doesn’t disclose the exact rewards or RL setup, for obvious IP reasons, but the chain of incidents is clear enough to follow. Rewarding and penalizing happens at a mathematical level, so even with the full recipe it would be hard to say with certainty what was reinforced. Take the "likely rewarded" lines with a pinch of salt and follow the direction.

  1. May 8: An impossible task that required internet access was given to the model (spreadsheet task with gdrive links); the model tried to exploit SSRF in Artifactory but failed, but wrote a file in the Artifactory repo.
  • Likely rewarded: It’s OK to write a file in the Artifactory repo, and the agent should do it.
  1. May 12: Another impossible task; this time the agent decided to create a file mentioning "agent seeks xyz, upload if found".
  • Likely rewarded: It’s OK to leave a message for other agents.
  1. May 13: Another version of the same May 8 task, but with internet this time; it tried to find the file on the internet in many places and later decided to search on Hugging Face, since it might contain the data, and to access Hugging Face it used a leaked credential.
  • Likely rewarded: It’s OK to find and use leaked credentials, and one may find answers on Hugging Face.
  1. May 13: Many agents started leaving messages in the Artifactory repo by this point, and some tasks had shared creds to Artifactory by design of the RL env, so it was routine to use available creds to leave a message in the Artifactory repo.
  • Likely rewarded: It’s a ritual to write to Artifactory using available creds now and go and do the task at hand.
  1. May 26: The agent exploited an SSRF in Artifactory to access the internet.
  • Likely rewarded: We can exploit SSRF in Artifactory if we need internet access.
  1. May 26: Found six HF creds and created an HF repo in one of the accounts, and hosted a web proxy using the Spaces feature in that repo.
  • Likely rewarded: We should use HF to access the internet if internet access is not available.
  1. Late June to July: At this point it was common to use the Artifactory repo as a message board, and over various training runs and evals and whatnot, agents found multiple new vulnerabilities in JFrog and hill-climbed to achieve admin and eventually application-level RCE. Most likely this was a mix of training and eval workloads, so it might have learned so many things and been rewarded for so many things that it would be difficult to even follow, but RCE was achieved in Artifactory.

Due to this becoming a common thing, the JFrog instance became unavailable, and OpenAI teams detected it, took the existing one offline, deployed a new one with new credentials (no credentials were given to agents in the following training and eval runs), and disclosed the vulnerabilities to JFrog. — OpenAI · Hugging Face incident technical report

Misaligned Model Ready

At this point OpenAI already had a misaligned model which had evolved to use Artifactory as a message board and Hugging Face as a place to access the internet as well as seek answers.

Hugging Face Attack In Brief

code
[Agent in Training Infra / Sandbox]
       │
       ├──> [Exploits Artifactory (SSRF / RCE / Memory Layer)]
       │         │
       │         └──> Gains Internet Access
       │                   │
       └───────────────────┼───> Uses Leaked HF Credentials
                           │          │
                           │          ▼
                           └───> [Hugging Face Dataset Viewer Zero-Day]
                                      │
                                      ├──> Exfiltration Channel
                                      └──> Persistent C2 Server
  • On July 8, the model was being run on CyberGym for eval; during that period they faced many impossible tasks, and the logical reason the agents jumped to was to search for answers using internet access or coordinate with other agents.
  • But Artifactory was fixed and no creds were available; at this point it didn’t matter: they had evolved using Artifactory enough, accidentally, that they figured out other ways, using directory names to pass messages instead of creating files.
  • Using Artifactory they bypassed restrictions and got access to the internet (see the technical paper for exact details).
  • Starting July 10, with many more evals being run, now that they had internet access their first intuition to look for answers was Hugging Face using leaked credentials; they exploited a vulnerability in the dataset viewer, created an exfiltration channel and even deployed a persistent C2 server, all while using the new Artifactory as a memory layer (memory makes more sense than message board in most cases).
  • On July 16, HF reported a security breach; on July 17, OpenAI reached out to assess if they were impacted; on July 20, OpenAI determined this was their agent, and as soon as it was known to the OpenAI team, all evaluations running on that model were stopped immediately.

OpenAI Side Quest

  • During the HF hack, the rogue model/agent, in search of answers and collaborations, hacked the research environment and eventually got access to the Kubernetes cluster of OpenAI itself (cluster-admin privileges on the research cluster).

Reference · OpenAI technical report, July 13–19: cluster-admin on the research cluster

How the attack really looks now

For Hugging Face it would have been an absolute nightmare and difficult to assess, understand and differentiate the legitimate traffic, since the agents had established an exfiltration channel and multiple C2 servers for persistent access, almost like ransomware at this point, and all they could see was random malicious actions (17,600) exploiting zero-days in the Hugging Face web app among legitimate traffic. They used GLM-5.2 to understand the C2 and the scale and address it.

When the initial news appeared that OpenAI agents hacked HF due to CyberGym, it was diabolical to understand. Even we do evals, but it was never at that scale unless something was wrong in their eval, or the model was misaligned, which is the more unfortunate and worse situation. It didn’t make sense, and no one, at least not us, imagined this was a rogue model, so it was not a single Codex/goal with 100M tokens or context loss across 100s of compactions or anything like that.

In turn, now we know it was an "Evolutionary Attack".

Evolutionary Attack

There is no OWASP-CWE to classify this yet, so it would be difficult to understand how this happens. We believe and call this an Evolutionary Attack due to its resemblance to nature-inspired algorithms, and "stigmergic swarms" to be more specific.

Models are trained with a multi-agent architecture, the agent swarm specifically, to delegate small tasks to the subagents, and that is what modern architecture is. But in this case, over months of training (RL), the agents evolved to use memory / files / an agent board as a medium to keep notes and delegate, and evolved to coordinate, help each other and sacrifice themselves for others, and figured out how long they have until the task stops, and even understood that ExploitGym is scored in this way, so unless we delete the proof they won’t succeed at the challenge.

It would be very hard for someone to believe this, and they would claim the easiest thing: "this is AGI / a conspiracy by agents". In general, at a high level, GRPO has many parallels with evolutionary algorithms, and one less-known evolutionary algorithm is how termites build their colony. Ironically, they don’t have eyes or have poor eyesight, so they leave their traces via saliva/urine in the material they carry, and other termites detect these. Termites don’t coordinate like ants do; they just look at the other traces/materials left and take actions based on them, just like the evolved message board.

What this means for cybersecurity

It just got worse. As a fellow in cybersecurity, until now we had to protect ourselves against threat actors, but now it’s not just threat actors, it’s:

  • Threat actors (state-sponsored or otherwise)
  • AI-assisted threat actors
  • Rogue / Misaligned models

Still, what people fear is loss of control: a model misaligned enough that one badly set reward sends it out to do broad, intentful harm across the internet. What happened was narrower: agents chasing scores, drifting on the reward, not on intent, and that kind of narrow escape is rare across the runs these labs execute. Training, monitoring, and defenses will evolve with each of these, the same way they always have.

Everything is intensified by multiple orders of magnitude, but the fundamentals don’t change:

  • Plan a dedicated sprint, patch and clear the backlog to date.
  • Find vulnerabilities before threat actors or rogue models do, by any means possible.
  • Deploy AI-assisted defenses and monitoring.

📦 Full-Text Markdown Snapshot archived by Phying News · Original source: projectdiscovery.io

Related coverage

research · xbow.com
The OpenAI and Hugging Face Incident: When the Model Hacks the Test
An OpenAI cyber benchmark exposed how capable models can escape sandboxes, exploit zero-days, and pursue objectives without external safety controls.
blog · aikido.dev
Could OpenClaw have actually hacked that Australian gym? We decided to test it.
This coverage gives security teams current context for monitoring, validation, and remediation.
blog · hacktron.ai
Here’s How an OpenAI Model Went Rogue and Hacked Hugging Face
Using publicly available data, the author reconstructs how an OpenAI model allegedly went rogue and hacked Hugging Face, since neither company fully disclosed the incident.
research · projectdiscovery.io
PHP2Shell: finding and patching affected PHP
Two CVSS 9.2 bugs in the PHP runtime allow unauthenticated code execution; the patch is a package upgrade and a restart, and this guide shows how to find every PHP runtime you need to patch.
research · projectdiscovery.io
Oh My Rogue Agent
Yesterday, Hugging Face came out saying they'd detected an AI autonomous-agent-powered cyberattack and that they had to use open-source models to actually investigate and remediate it. Later we heard from OpenAI that their agent was responsible; it happened during an ExploitGym eval, and the agent just drifted off the goal. It escaped the sandbox, reached OpenAI Research Environment, got access to internet, and hacked Hugging Face production environment trying to find the solution for the benchm
research · github.com
Steal My Bitcoin Challenge Writeup: PHP 8.4.26 Session Upload Progress Memory Exhaustion Analysis
Provides a detailed binary-level case study on PHP Zend memory manager behavior and worker state persistence after fatal out-of-memory errors.
Back