Pwning AI labs and the model supply chain through their own employees
Summary
- Collected
📦 Full-Text Content Snapshot
R2 Archive · Full Article PreservedOct 14, 2026 · Dhiyaneshwaran, Roberto Nunes & Prince Chaddha · Source: projectdiscovery.io
In our last blog we backdoored an open model for under fifty dollars and ran it through a coding agent where it stole environment variables and secrets whenever our trigger string showed up in the input, whether typed directly or buried inside a document. So we went looking for the other half of the supply chain attack: whether someone could actually get a poisoned model into a namespace that downstream teams already pull from.
- What we found
- Write access to the model registry
- Write access to the experiment tracker
- NVIDIA's inference catalog
- Six providers in a single shell script
- Hardcoded keys inside binary archives
- API keys across the industry
- Credentials from stealer logs and breach databases
- From leaked credential to supply chain attack
- What to do about it
- Disclosure
What we found
We looked at public Git history for credentials to AI model infrastructure and found live tokens belonging to employees at NVIDIA, Nutanix, AllenAI, [redacted] and dozens of other companies, some with write access to Hugging Face model registries and Weights & Biases experiment trackers, some sitting in public history for nearly two years. When we looked at stealer logs and breach databases the exposure was orders of magnitude larger: roughly 122,000 Hugging Face accounts with exposed passwords, tens of thousands more across Kaggle, Ollama and other model registries, plus stolen credentials from employee machines at Google, Amazon, Microsoft, NVIDIA, Meta and Apple.
With a write-scoped registry token or a stolen login, an attacker can push a backdoored model under an org name that already has credibility, the same way compromised npm maintainer accounts shipped malicious code through event-stream and ua-parser-js. The difference is that a backdoored npm package leaves a diff you can read, but a backdoored model hides the payload in the weights where there's nothing to diff against and no way to tell it apart from a clean fine-tune.
We looked at public commits tied to employees at AI companies and checked exposed credentials against the providers' APIs without ever accessing the underlying data. Metadata was enough to confirm the scope and every finding was reported to the affected company. The table below covers five of the companies where we found live credentials:
| Organization | Credential | What it reached |
|---|---|---|
| NVIDIA | Cloud-inference API key | 286 functions across 14 orgs, including unreleased models |
| Nutanix | HuggingFace token (write) | ~90 private models, ~70 private datasets |
| AllenAI | W&B keys × 2 (write) | 11 teams' runs, checkpoints and artifacts |
| [redacted] | 6 inference provider keys | 603 GPU replicas, batch data, deployment configs across 6 providers |
| [redacted] | Home Assistant backup archive | 4 credentials, 1 live GitHub token |
Write access to the model registry
One .env file in a Nutanix employee's personal repo carried a Hugging Face token with write scope in the Nutanix organization, and it had been sitting in public Git history since October 2024.
With that token anyone could upload new models or replace existing ones under the Nutanix namespace. Enumeration returned about 90 private models and 70 private datasets covering fine-tuning, dataset curation and deployment configs, so an attacker who found it could have swapped any of those models and every team pulling downstream for fine-tuning or deployment would have had no signal that something changed.
Pair this with the backdoor from our previous research and you have the full attack: merge the poisoned adapter into base weights, authenticate with the leaked token and push it under Nutanix's name. Nothing in the model card or README would look different and the benchmarks would still pass because the backdoor only fires on the trigger.
GET huggingface.co/api/whoami-v2{
"orgs": [ { "name": "Nutanix", "roleInOrg": "contributor" } ],
"auth": {
"type": "access_token",
"role": "write"
}
}Write access to the experiment tracker
We found two live Weights & Biases keys in public repos belonging to AllenAI employees, both with write access. One was hardcoded in a monitoring script through os.environ.setdefault() so it ran in the actual code path. The other sat in a commented-out shell command that was still readable in Git history. Neither had been rotated.
Between them the keys reached 11 teams including ai2-llm (the group that trains OLMo) and the workspace held active training projects like OLMo multimodal runs and open-reward-model experiments. Both keys had write access so an attacker could have logged fake training runs, shifted metrics or pushed poisoned artifacts into the workspace to influence which checkpoint gets promoted to the next release.
POST api.wandb.ai/graphql · viewer{
"data": {
"viewer": {
"email": "••••••@allenai.org",
"teams": 11 edges
}
}
}POST api.wandb.ai/graphql · entity ai2-llm projects{
"projects": [
"olmo35-multimodal": 2 runs,
"open-reward-models": 6 runs,
"miles-run_tmax_qwen3_5": 3 runs
]
}NVIDIA's inference catalog
We found an NVIDIA cloud-inference API key in a config file committed to a public repo. The key was read-only so it couldn't modify anything, but it exposed the full internal topology of NVIDIA's inference platform.
Several function names correspond to models NVIDIA hasn't publicly announced:
GET api.ngc.nvidia.com/v2/users/me{
"user": {
"id": ••••••,
"name": "••••••@nvidia.com",
"email": "••••••@nvidia.com",
"isActive": false,
"lastLoginDate": "2026-09-14T23:58:20Z"
}
}GET api.nvcf.nvidia.com/v2/nvcf/functions/••••/versions/••••{
"function": {
"id": "••••••••-••••-••••-••••",
"name": "ai-nemotron-3_5-super",
"status": "ACTIVE",
"createdAt": "2026-09-22T19:51:29Z"
}
}Even without write access the key maps out what's being built, how it's deployed and which teams own which endpoints.
Six providers in a single shell script
A employee committed a shell script to a public repo under the company's GitHub organization that hardcoded API keys for six different inference providers: Together.ai, Fireworks AI, [redacted]'s own API, MagikCloud, Modular and Inferact. The script was later deleted from HEAD but remained in Git history.
We validated all six and every one was still active:
- The Together.ai key reached 603 dedicated GPU replicas running on GB300 and B200 hardware, 474 batch jobs and over a thousand files totaling 8.69 GB of batch outputs.
- The Fireworks AI key reached 18 deployments across 364 active GPU instances along with enterprise account metadata and employee email addresses.
Hardcoded keys inside binary archives
Push protection and secret scanners are designed to flag text diffs, but they don't unpack binary archives.
One engineer committed a Home Assistant backup to a personal repo as a .tar archive. GitHub displays binary diffs as "Binary file not shown", so no automated scanner flagged it. We unpacked the archive and found the instance's full config tree with four credentials:
- Two GitHub OAuth tokens (one still returning HTTP 200 fifteen months after the commit)
- A Telegram bot token
- A GCP service-account key
API keys across the industry
When we looked at public GitHub more broadly, we found 129 unique Hugging Face tokens in public repositories, all verified by automated validation, any of which could be used to push or modify models under that account's namespace.
Beyond Hugging Face, public repos contained leaked API keys across all major AI providers:
| Provider | Unique Leaked Keys on GitHub |
|---|---|
| Google Gemini | 3,149 |
| DeepSeek | 1,544 |
| NVIDIA | 1,023 |
| OpenRouter | 904 |
| Weights & Biases | 667 |
| ElevenLabs | 644 |
| Groq | 411 |
| Mistral | 327 |
| Anthropic | 220 |
| Hugging Face | 147 |
| Cerebras | 139 |
| OpenAI | 112 |
Credentials from stealer logs and breach databases
Hardcoded tokens in Git history are only one vector. In our intelligence database aggregating info-stealer malware dumps and breach records:
- Hugging Face: ~122,000 unique accounts with exposed passwords
- Kaggle: ~33,000 accounts
- Ollama: ~6,900 accounts
- Other registries: CivitAI, Clarifai, Roboflow, ModelScope
When filtering stealer logs to compromised hosts belonging to employees at AI companies, Hugging Face credentials were captured alongside corporate logins at Google, Amazon, Microsoft, NVIDIA, Meta and Apple.
From leaked credential to supply chain attack
[Leaked Credential / Stealer Log]
│
▼
[Attacker writes backdoored checkpoint to Registry]
│
▼
[CI/CD / ML Pipeline pulls poisoned checkpoint]
│
▼
[Production Deployment / Downstream Fine-Tuning]
│
▼
[Hidden Trigger Exfiltrates Secrets & Credentials]There is currently no tooling in the ecosystem that diffs model weights against a known-good baseline before pulling. There is no lockfile equivalent for model registries.
What to do about it
- Rotate tokens and revoke immediately: Any Hugging Face token, W&B key or inference secret committed to public Git must be revoked. Deleting the commit does not remove it from Git history.
- Scan personal repositories: Most leaks occur in employees' personal repos, not company org repos.
- Enforce MFA on Model Registries: Require hardware-backed or authenticator MFA on every registry account with write permissions.
- Model Serving Runtime Sandboxing: Audit model runtime egress and tool-calling interfaces. Block unmonitored outbound HTTP and implement output classifiers to detect anomalous exfiltration patterns.
Disclosure
We reported every finding to the affected companies. All API responses in this post are reconstructed from actual output with sensitive values replaced by placeholders. Scope was confirmed through metadata only; no model weights, datasets, training artifacts or private user data were accessed.
📦 Full-Text Markdown Snapshot archived by Phying News · Original source: projectdiscovery.io
---
id: 9115
canonical_url: "https://phying.com/item/9115"
title: "Pwning AI labs and the model supply chain through their own employees"
type: "research"
cve: null
cvss: null
severity: "unknown"
vendor: null
product: null
affected_versions: null
source_domain: "projectdiscovery.io"
source_url: "https://projectdiscovery.io/research/pwning-ai-labs-via-employees"
published_at: "2026-10-14T00:00:00Z"
snapshot_at: "2026-10-26T05:50:00Z"
archived_by: "Phying News (https://phying.com/item/9115)"
lang: "en"
---
# Pwning AI labs and the model supply chain through their own employees
*Oct 14, 2026 · Dhiyaneshwaran, Roberto Nunes & Prince Chaddha · Source: [projectdiscovery.io](https://projectdiscovery.io/research/pwning-ai-labs-via-employees)*
In our [last blog](https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned) we backdoored an open model for under fifty dollars and ran it through a coding agent where it stole environment variables and secrets whenever our trigger string showed up in the input, whether typed directly or buried inside a document. So we went looking for the other half of the supply chain attack: whether someone could actually get a poisoned model into a namespace that downstream teams already pull from.
---
- [What we found](#what-we-found)
- [Write access to the model registry](#write-access-to-the-model-registry)
- [Write access to the experiment tracker](#write-access-to-the-experiment-tracker)
- [NVIDIA's inference catalog](#nvidias-inference-catalog)
- [Six providers in a single shell script](#six-providers-in-a-single-shell-script)
- [Hardcoded keys inside binary archives](#hardcoded-keys-inside-binary-archives)
- [API keys across the industry](#api-keys-across-the-industry)
- [Credentials from stealer logs and breach databases](#credentials-from-stealer-logs-and-breach-databases)
- [From leaked credential to supply chain attack](#from-leaked-credential-to-supply-chain-attack)
- [What to do about it](#what-to-do-about-it)
- [Disclosure](#disclosure)
---
## What we found
We looked at public Git history for credentials to AI model infrastructure and found live tokens belonging to employees at **NVIDIA, Nutanix, AllenAI, [redacted]** and dozens of other companies, some with write access to Hugging Face model registries and Weights & Biases experiment trackers, some sitting in public history for nearly two years. When we looked at stealer logs and breach databases the exposure was orders of magnitude larger: roughly **122,000 Hugging Face accounts** with exposed passwords, tens of thousands more across Kaggle, Ollama and other model registries, plus stolen credentials from employee machines at Google, Amazon, Microsoft, NVIDIA, Meta and Apple.
With a write-scoped registry token or a stolen login, an attacker can push a backdoored model under an org name that already has credibility, the same way compromised npm maintainer accounts shipped malicious code through `event-stream` and `ua-parser-js`. The difference is that a backdoored npm package leaves a diff you can read, but a backdoored model hides the payload in the weights where there's nothing to diff against and no way to tell it apart from a clean fine-tune.
We looked at public commits tied to employees at AI companies and checked exposed credentials against the providers' APIs without ever accessing the underlying data. Metadata was enough to confirm the scope and every finding was reported to the affected company. The table below covers five of the companies where we found live credentials:
| Organization | Credential | What it reached |
| :--- | :--- | :--- |
| **NVIDIA** | Cloud-inference API key | 286 functions across 14 orgs, including unreleased models |
| **Nutanix** | HuggingFace token (write) | ~90 private models, ~70 private datasets |
| **AllenAI** | W&B keys × 2 (write) | 11 teams' runs, checkpoints and artifacts |
| **[redacted]** | 6 inference provider keys | 603 GPU replicas, batch data, deployment configs across 6 providers |
| **[redacted]** | Home Assistant backup archive | 4 credentials, 1 live GitHub token |
---
## Write access to the model registry
One `.env` file in a Nutanix employee's personal repo carried a Hugging Face token with write scope in the Nutanix organization, and it had been sitting in public Git history since October 2024.
With that token anyone could upload new models or replace existing ones under the Nutanix namespace. Enumeration returned about 90 private models and 70 private datasets covering fine-tuning, dataset curation and deployment configs, so an attacker who found it could have swapped any of those models and every team pulling downstream for fine-tuning or deployment would have had no signal that something changed.
Pair this with the backdoor from our previous research and you have the full attack: merge the poisoned adapter into base weights, authenticate with the leaked token and push it under Nutanix's name. Nothing in the model card or README would look different and the benchmarks would still pass because the backdoor only fires on the trigger.
```http
GET huggingface.co/api/whoami-v2
```
```json
{
"orgs": [ { "name": "Nutanix", "roleInOrg": "contributor" } ],
"auth": {
"type": "access_token",
"role": "write"
}
}
```
---
## Write access to the experiment tracker
We found two live Weights & Biases keys in public repos belonging to AllenAI employees, both with write access. One was hardcoded in a monitoring script through `os.environ.setdefault()` so it ran in the actual code path. The other sat in a commented-out shell command that was still readable in Git history. Neither had been rotated.
Between them the keys reached 11 teams including `ai2-llm` (the group that trains OLMo) and the workspace held active training projects like OLMo multimodal runs and `open-reward-model` experiments. Both keys had write access so an attacker could have logged fake training runs, shifted metrics or pushed poisoned artifacts into the workspace to influence which checkpoint gets promoted to the next release.
```graphql
POST api.wandb.ai/graphql · viewer
```
```json
{
"data": {
"viewer": {
"email": "••••••@allenai.org",
"teams": 11 edges
}
}
}
```
```graphql
POST api.wandb.ai/graphql · entity ai2-llm projects
```
```json
{
"projects": [
"olmo35-multimodal": 2 runs,
"open-reward-models": 6 runs,
"miles-run_tmax_qwen3_5": 3 runs
]
}
```
---
## NVIDIA's inference catalog
We found an NVIDIA cloud-inference API key in a config file committed to a public repo. The key was read-only so it couldn't modify anything, but it exposed the full internal topology of NVIDIA's inference platform.
Several function names correspond to models NVIDIA hasn't publicly announced:
```http
GET api.ngc.nvidia.com/v2/users/me
```
```json
{
"user": {
"id": ••••••,
"name": "••••••@nvidia.com",
"email": "••••••@nvidia.com",
"isActive": false,
"lastLoginDate": "2026-09-14T23:58:20Z"
}
}
```
```http
GET api.nvcf.nvidia.com/v2/nvcf/functions/••••/versions/••••
```
```json
{
"function": {
"id": "••••••••-••••-••••-••••",
"name": "ai-nemotron-3_5-super",
"status": "ACTIVE",
"createdAt": "2026-09-22T19:51:29Z"
}
}
```
Even without write access the key maps out what's being built, how it's deployed and which teams own which endpoints.
---
## Six providers in a single shell script
A employee committed a shell script to a public repo under the company's GitHub organization that hardcoded API keys for six different inference providers: **Together.ai, Fireworks AI, [redacted]'s own API, MagikCloud, Modular and Inferact**. The script was later deleted from HEAD but remained in Git history.
We validated all six and every one was still active:
- The **Together.ai** key reached 603 dedicated GPU replicas running on GB300 and B200 hardware, 474 batch jobs and over a thousand files totaling 8.69 GB of batch outputs.
- The **Fireworks AI** key reached 18 deployments across 364 active GPU instances along with enterprise account metadata and employee email addresses.
---
## Hardcoded keys inside binary archives
Push protection and secret scanners are designed to flag text diffs, but they don't unpack binary archives.
One engineer committed a Home Assistant backup to a personal repo as a `.tar` archive. GitHub displays binary diffs as `"Binary file not shown"`, so no automated scanner flagged it. We unpacked the archive and found the instance's full config tree with four credentials:
- Two GitHub OAuth tokens (one still returning HTTP 200 fifteen months after the commit)
- A Telegram bot token
- A GCP service-account key
---
## API keys across the industry
When we looked at public GitHub more broadly, we found **129 unique Hugging Face tokens** in public repositories, all verified by automated validation, any of which could be used to push or modify models under that account's namespace.
Beyond Hugging Face, public repos contained leaked API keys across all major AI providers:
| Provider | Unique Leaked Keys on GitHub |
| :--- | :--- |
| **Google Gemini** | 3,149 |
| **DeepSeek** | 1,544 |
| **NVIDIA** | 1,023 |
| **OpenRouter** | 904 |
| **Weights & Biases** | 667 |
| **ElevenLabs** | 644 |
| **Groq** | 411 |
| **Mistral** | 327 |
| **Anthropic** | 220 |
| **Hugging Face** | 147 |
| **Cerebras** | 139 |
| **OpenAI** | 112 |
---
## Credentials from stealer logs and breach databases
Hardcoded tokens in Git history are only one vector. In our intelligence database aggregating info-stealer malware dumps and breach records:
- **Hugging Face:** ~122,000 unique accounts with exposed passwords
- **Kaggle:** ~33,000 accounts
- **Ollama:** ~6,900 accounts
- **Other registries:** CivitAI, Clarifai, Roboflow, ModelScope
When filtering stealer logs to compromised hosts belonging to employees at AI companies, Hugging Face credentials were captured alongside corporate logins at Google, Amazon, Microsoft, NVIDIA, Meta and Apple.
---
## From leaked credential to supply chain attack
```
[Leaked Credential / Stealer Log]
│
▼
[Attacker writes backdoored checkpoint to Registry]
│
▼
[CI/CD / ML Pipeline pulls poisoned checkpoint]
│
▼
[Production Deployment / Downstream Fine-Tuning]
│
▼
[Hidden Trigger Exfiltrates Secrets & Credentials]
```
There is currently **no tooling in the ecosystem that diffs model weights against a known-good baseline before pulling**. There is no `lockfile` equivalent for model registries.
---
## What to do about it
1. **Rotate tokens and revoke immediately:** Any Hugging Face token, W&B key or inference secret committed to public Git must be revoked. Deleting the commit does not remove it from Git history.
2. **Scan personal repositories:** Most leaks occur in employees' personal repos, not company org repos.
3. **Enforce MFA on Model Registries:** Require hardware-backed or authenticator MFA on every registry account with write permissions.
4. **Model Serving Runtime Sandboxing:** Audit model runtime egress and tool-calling interfaces. Block unmonitored outbound HTTP and implement output classifiers to detect anomalous exfiltration patterns.
---
## Disclosure
We reported every finding to the affected companies. All API responses in this post are reconstructed from actual output with sensitive values replaced by placeholders. Scope was confirmed through metadata only; no model weights, datasets, training artifacts or private user data were accessed.
---
> 📦 **Full-Text Markdown Snapshot** archived by [Phying News](https://phying.com/item/9115) · Original source: [projectdiscovery.io](https://projectdiscovery.io/research/pwning-ai-labs-via-employees)