How did the poisoned LiteLLM ship? A key leaked in May, never revoked
LiteLLM published its post-mortem of the poisoned 1.107.0 release: a master key exposed in May was never revoked, and it led the attacker to GitHub tokens and to the release on Oct 21.
If you read the article on the poisoned LiteLLM release, you know 1.107.0 stole keys. What wasn't known then was how the attacker got to publish an official release under the project's name. On Oct 23 at 21:00 UTC, LiteLLM published the full attack-chain post-mortem, and the story starts back in May.
Where did it start? A May key that was never revoked
May 27, 2026: someone committed an opencode.json file to a public repo, LiteLLM-Labs/lite-harness, and the file held the master key to a deployment of the LiteLLM Agent Platform, a tool the team uses to run agents. The commit was later force-pushed away, and the key was flagged for revocation, but it was never revoked.
That's the first and biggest mistake of the whole story: deleting a commit from git history doesn't delete the key from the hands of whoever saw or copied it. And the key stayed live until the attacker used it more than 4 months later.
How did the attacker pull GitHub tokens?
Oct 10: the attacker used the master key (from IP 69.48.229.143) to call the platform's API. A GET /api/v1/managed_agents/agents?limit=100 request returned the configuration of every agent, and with it the GitHub CLI token of one of the two owners of the project's GitHub organization.
The same day, between 22:11 and 22:12 UTC, two throwaway branches named ci-tmp-a351cc and ci-tmp-be33e9 ran CI workflows that sent the secrets of two repos to webhook.site. Among them were GH_TOKEN and RELEASE_PLEASE_TOKEN, the GitHub CLI tokens of two maintainers. Then the run records were deleted so nobody would notice.
GitHub CLI tokens are issued after 2FA, and GitHub doesn't ask them to re-authenticate before sensitive actions, so 2FA doesn't stop whoever steals one. With the owner's token, the attacker could remove branch protection 11 days later.
How did the poisoned release ship in a single day?
On Oct 21, the attacker used the stolen tokens to get a poisoned build through the official release gates:
Changed
BerriAI/project-releaserto build from a lookalike fork,Berri-AI/litellm, at commit20535d8f.Overrode branch protection with the stolen owner token when it rejected the change.
The maintainer token stolen from
GH_TOKENapproved the production gate.Seven
litellm==1.107.0wheels were published to PyPI between 11:39 and 11:40 UTC, with Docker Hub and GHCR images at 11:46.
PyPI quarantined the poisoned release at 13:23 UTC, and the Docker Hub tags were fixed by 14:20. So the release was out for about two hours, which is enough time for it to get pulled and installed.
Exactly which releases were hit: on PyPI, litellm==1.107.0; on Docker Hub and GHCR, the v1.107.0, latest, main-latest, and main-stable tags. The safe version before it: 1.106.2.
If you installed the poisoned version, what now?
Treat the environment as fully compromised, and clean it before you rotate any key:
Kill the malicious Python process that keeps running in the background (it holds
/tmp/remote_code_loader.lock), and rebuild the environment on the safe version 1.106.2. Don't runpipinside the infected environment: any Python run there runs the malicious files again.From a clean machine, rotate every credential the environment could see: cloud keys, SSH keys, PyPI and npm tokens, and AI provider tokens.
Pin Docker images by digest, not by tag, so a moving tag can't surprise you again.
As with the npm worm, the order you rotate in matters as much as rotating: clean first, then revoke and rotate from a clean machine.
What's the lesson for every developer?
Every step of the attack ran on a secret that stayed live after it was exposed, or on a token with more power than its job needed.
The mistake | The lesson |
|---|---|
A key exposed in May, never revoked | Any secret exposed publicly gets revoked immediately, even if you "deleted" the commit |
CI secrets stolen through fake workflows | Watch for strange CI runs; any deleted run record is a red flag |
Powerful tokens in attacker hands | Rotate your secrets in the right order, and keep token permissions minimal |
A key flagged for revocation, but the revocation never happened | Revoke and rotate any leaked key at once, then try the old one to make sure it no longer works |
LiteLLM says it's revoking every token tied to its internal systems and the platforms it works with, and that it will share next week what it learned and what it's changing. But the breach didn't start on Oct 21. It started on May 27, when the key was flagged for revocation and never revoked.
When did you last rotate an exposed key, even one whose trace was wiped from history?
FAQ
Is it enough to pip install 1.106.2 over the poisoned version?
No. Any Python run in the infected environment, pip included, runs the malicious files again. Rebuild the environment from scratch on 1.106.2, rotate every credential it could see (cloud keys, SSH, AI tokens) from a clean machine, and pin Docker images by digest, not tag.
Exactly which releases were compromised?
On PyPI: litellm==1.107.0. On Docker Hub and GHCR: the v1.107.0, latest, main-latest, and main-stable tags. The safe version before it is 1.106.2.
Why wasn't deleting the commit enough?
Because the key itself stayed valid. Deleting the commit from git history doesn't take the key away from whoever saw or copied it, and at LiteLLM the key was flagged for revocation but never revoked. Any secret exposed in public needs revoking right away.
What does it mean that GitHub CLI tokens bypass 2FA?
It means GitHub issues them after 2FA and doesn't ask them to re-authenticate before sensitive actions, so the owner's 2FA doesn't stop whoever steals one. That's how the attacker removed branch protection with the organization owner's token.
