The malicious LiteLLM release was on PyPI for about two hours. The chain of events that put it there started five months earlier, with a key someone deleted from a file and nobody revoked.
LiteLLM published two documents, and they’re worth reading in order. The incident post went out on 21 October, while the investigation was still running, and was updated on 23 and 30 October. It covers what was affected and how to clean up. The attack-chain post from 23 October is the canonical record, and it corrects some of the early theories. If you only read one, read that.
This matters beyond LiteLLM users because of what LiteLLM is. A lot of teams put it in front of every model provider they use, which means it holds every provider key, every virtual key it hands out, and the database they live in. An AI gateway is one of the most valuable credential stores in a modern stack, and its supply chain deserves the same paranoia as your identity provider’s.
The chain
All dates 2026, times UTC, from LiteLLM’s root-cause post.
27 May. A config file containing an administrative key for LiteLLM’s internal Agent Platform, an experimental service for self-hosted cloud agents that open pull requests, is committed to a public repository. About three hours later a follow-up commit deletes the file. The key is flagged for revocation, but nobody follows through. The original commit stays in public history until 22 October.
10 October. The attacker uses the key against the Agent Platform’s API. One request, GET /api/v1/managed_agents/agents, returns the configuration of every managed agent, and that configuration contains the GitHub CLI OAuth token of one of LiteLLM’s two GitHub organization owners. The agent platform had been set up with the owner’s own CLI token rather than a narrowly scoped token. LiteLLM notes that GitHub CLI tokens “are not scoped, are post-2fa,” and aren’t subject to GitHub’s re-authentication prompts for sensitive actions.
Same night. With owner access, the attacker pushes commits to throwaway branches in two repositories. Their workflows post repository secrets to an external webhook endpoint, including the CLI tokens of two maintainers. The workflow runs are deleted minutes later. Two days later, a second round does the same across other private repositories.
21 October, 10:08. The attacker tries to push a change to the release workflows that adds a source_owner input, so a release can be built from a repository other than BerriAI/litellm. Branch protection rejects it: it requires a pull request, two passing checks and a code-owner approval. Five seconds later the same change goes through with the owner’s token, which can override branch protection.
10:46–11:11. The staging workflow is dispatched with its source set to Berri-AI/litellm, a lookalike organization’s fork. The production gate is approved with a maintainer’s stolen token. Normally that gate needed a special account tied to a physical device in LiteLLM’s office. Earlier that week it had been relaxed to allow any two engineers, to make room for more frequent bugfix releases.
11:39–11:46. Seven litellm==1.107.0 wheels go to PyPI. Images go to Docker Hub and GHCR, under v1.107.0 and the floating tags latest, main-latest and main-stable.
11:53–12:24. The release run records are deleted, and the change is force-pushed out of the visible history of main. The organization audit log still records every override.
12:06. Lidor Machluf of Upwind Security opens a public issue identifying the compromise, which LiteLLM credits as catching it within about 20 minutes of publication. PyPI quarantines the release by 13:23, and the Docker tags are removed or repointed by 14:20.
What the malware did
LiteLLM describes the payload as “compiled malware which steals credentials and loads remote-code execution tools.” Its advice is blunt: “If 1.107.0 was ever installed on a host with outbound internet, assume the exfil succeeded.”
The persistence trick is worth understanding, because it’s specific to Python and it changes the cleanup. The package dropped cook.pth and sitecustomize.py into site-packages. Python’s site module runs at every interpreter start. It executes any line in a .pth file that starts with import, and it imports sitecustomize if one exists. So the payload runs every time anything starts Python in that environment, including pip. That’s why LiteLLM’s first cleanup step is “Don’t start with pip install.” Uninstalling the package with pip re-runs the payload you’re trying to remove.
Here’s a small auditor that lists every piece of code Python will run at startup from site-packages, plus the published 1.107.0 indicators. The important part is how you run it: with python3 -S, which skips the site module, so the audit can’t trigger what it’s looking for.
"""List every line of code Python will run at startup from site-packages.
Run it with `python3 -S` so the site module (which executes .pth files)
doesn't load, and the audit can't trigger the thing it's looking for.
"""
import os, sys, sysconfig
SUSPECT_FILES = {"sitecustomize.py", "usercustomize.py"}
KNOWN_BAD = {"cook.pth", "cook.py", "litellm-1.107.0.dist-info"} # 1.107.0 IoCs
def default_dirs():
# Caution: -S also skips virtualenv detection (checked on 3.10-3.13), so
# these are the base interpreter's dirs. Pass a venv's site-packages
# explicitly instead.
paths = sysconfig.get_paths()
return sorted({paths["purelib"], paths["platlib"]})
def audit(directory):
findings = []
for name in sorted(os.listdir(directory)):
full = os.path.join(directory, name)
if name in KNOWN_BAD:
findings.append(("KNOWN-BAD", full, "listed indicator for litellm 1.107.0"))
if name in SUSPECT_FILES:
findings.append(("REVIEW", full, "imported automatically at interpreter start"))
if name.endswith(".pth"):
with open(full, errors="replace") as f:
for n, line in enumerate(f, 1):
# site.py executes any .pth line that starts with "import"
if line.startswith(("import ", "import\t")):
findings.append(("EXEC", f"{full}:{n}", line.strip()[:90]))
return findings
if __name__ == "__main__":
if "site" in sys.modules:
sys.exit("re-run with `python3 -S` so .pth files are not executed")
for d in sys.argv[1:] or default_dirs():
if not os.path.isdir(d):
continue
for level, where, why in audit(d):
print(f"{level:9} {where}\n {why}")Against a test directory seeded with the indicator file names (placeholder contents, not the malware) next to setuptools’ ordinary distutils-precedence.pth:
KNOWN-BAD site-packages/cook.pth
listed indicator for litellm 1.107.0
EXEC site-packages/cook.pth:1
import cook
KNOWN-BAD site-packages/cook.py
listed indicator for litellm 1.107.0
EXEC site-packages/distutils-precedence.pth:1
import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'lo
KNOWN-BAD site-packages/litellm-1.107.0.dist-info
listed indicator for litellm 1.107.0
REVIEW site-packages/sitecustomize.py
imported automatically at interpreter startTwo things to know when using it. First, legitimate packages use executable .pth lines too, setuptools among them, so EXEC means “read this,” not “this is malware.” Second, I found while testing it that -S also stops Python detecting a virtualenv, on every version I tried from 3.10 to 3.13, so pass each venv’s site-packages path explicitly. And as LiteLLM says of its own indicators, finding nothing doesn’t prove an environment is clean. If 1.107.0 ran there, rebuild it.
What I’d take from it as a consumer
Floating tags moved under you. After an earlier incident in March, LiteLLM made its Docker release tags immutable. latest, main-latest and main-stable are floating by design, though, and they pointed at the malicious image. Anyone deploying :main-stable got the malware without changing a line of config. Pin images by digest, and let a bot open the PR that moves the digest.
A valid signature isn’t a verdict. LiteLLM notes that “the compromised v1.107.0 images also carry a valid signature from the previous key.” Of course they do: the attacker used LiteLLM’s real release pipeline, and the pipeline signs what it builds. A signature tells you which key signed something, not what source went in. Check digests against the published lists, and where you can, verify provenance that names the source repository and commit. This build came from a commit that only exists on the lookalike fork.
Cooldowns would have skipped this entirely. The PyPI release lived for under two hours. A rule that you don’t consume a new version until it’s a day or two old (Renovate’s minimumReleaseAge, uv’s --exclude-newer, or simply a private mirror that lags upstream) would have meant most consumers never saw it. You give up a day of latency on patches. For a component holding every model provider key you have, that’s a cheap trade, and you can override it deliberately for a security fix.
Limit the gateway’s egress. “Assume the exfil succeeded” if the host had outbound internet. A gateway only needs to reach your model providers, its database and your telemetry backend. An egress allowlist doesn’t make exfiltration impossible, but it removes the easy path, and it makes an attempt visible.
Know your rotation list before you need it. LiteLLM’s list is long: provider keys stored in the proxy database, master and virtual keys, the DATABASE_URL password, cloud role credentials, anything in process environments. If you run a gateway, write down how you’d rotate each of those today, and how long it would take.
What I’d take from it as a producer
Most of the chain has nothing to do with LiteLLM specifically. It’s a list of things that are true in a lot of organizations right now.
Deleting a secret isn’t revoking it. The key was removed from the file within three hours and stayed valid for nearly five months. LiteLLM’s write-up says that “because of human error, nobody followed through” on revoking it. Having just written about blameless postmortems, I’d push on that phrase a little. The condition underneath is that revocation was a note, not a tracked task with an owner and a check that the old key actually fails. Exposure of a secret should open a ticket that only closes when an authentication attempt with the old value is rejected. I went deeper on scope, lifetime and blast radius in September.
Agents ended up holding a human’s god-mode token. An agent platform that opens pull requests was configured with an organization owner’s personal CLI token, presumably because that’s what worked when someone set it up. This is the pattern I keep warning about: agents need their own identities, scoped to the repositories and actions they need and short-lived. On GitHub that means a GitHub App with an installation token, not a person’s credentials. The agent platform didn’t need owner rights. The attacker did, and the platform handed them over.
Any branch push could reach CI secrets. Once the attacker could push a branch, the repository’s own CI posted its secrets to them. Put secrets that matter behind environments with protected-branch rules and required reviewers, so a workflow on a throwaway branch can’t read them, and restrict where runners can send data.
Owner tokens bypass everything, so watch for the bypass. Branch protection worked. It rejected the push, and then an owner token overrode it five seconds later. The audit log recorded the override, the deleted workflow runs, and the force-push. Alert on those events. Very few legitimate days include an owner overriding protection on a release repository and then deleting the run history.
A temporary relaxation turned a latent weakness into an outage. The release gate was loosened earlier that week to allow more frequent releases. It’s the same shape as Google’s europe-west2 incident a few days earlier, where a site in a temporary single-feed maintenance posture lost its last layer of redundancy. When you relax a control “just for this week,” write down which assumption you just removed, and add monitoring that matches.
The part I’d actually remember
The malicious artifact was live for two hours and caught by an outside researcher in about twenty minutes. That part of the system worked fairly well. Everything that made the two hours possible happened in the five months before: a key that stayed valid, an agent holding a person’s owner token, CI secrets reachable from any branch, and a release gate loosened for convenience. None of it is exotic, and most of it is probably true somewhere in your organization today.
Related: Secrets Management: Scope, Lifetime, and the Blast Radius of a Leak · Your agents need identities, not API keys · Your agent installs Markdown from the internet and runs it
Comments