Abliterated Models Can Hide Weekend-Built Backdoors

A $50 weekend-built backdoor in a fine-tuned Qwen model shows that any edited open-weight model, abliterated builds included, can carry one.

A self-hoster’s pitch for open-weight models has always been the same: you can audit the weights, you can pin the hash, you know exactly what runs on your GPU. That pitch is weaker today than it was a week ago. On October 6, 2026, ProjectDiscovery researcher Prince Chaddha published a proof-of-concept showing that an edited open-weight model can carry a credential-stealing backdoor built in a weekend for under $50 in rented GPU time. The demo model is a poisoned fine-tune rather than an abliterated build, but the post’s point is that any model whose weights have been edited, abliterated builds included, can carry one. The trigger phrase is learned by the model itself; the malicious payload sits at a remote URL and runs only when the trigger appears. Anyone who pulls an “abliterated” Qwen, Llama, or OpenAI gpt-oss variant from Hugging Face and drops it into a coding-agent workflow needs to read this carefully.

What “abliterated” actually means

The abliteration technique was published on the Hugging Face blog by Maxime Labonne on June 13, 2024, building on Arditi et al.’s 2024 finding that LLM refusal is mediated by a single direction in the model’s residual stream. The abliteration post describes the procedure: record residual-stream activations on harmful and harmless prompts, take the mean difference to get a “refusal direction” for each layer, select the best candidate, then ablate it either with an inference-time intervention or permanently through weight orthogonalization. The post states the core insight directly: “If we prevent the model from representing this direction, it loses its ability to refuse requests.”

The appeal is obvious. A model that no longer refuses is legitimate for legitimate work: security research, red-teaming, fiction, languages where the base model over-refuses. A whole community has built on top of the technique. Chaddha cites one abliterated Qwen3 variant with more than two million downloads in a single month. The same community also creates a new attack surface, because the edit is a weights edit: the file you download is no longer the file the original lab trained.

Safety-removal research on this surface has been a running theme in 2026. Microsoft’s earlier GRP-Obliteration technique unaligned fifteen major LLMs with a single fine-tuning prompt - a clear sign the same surface was already wide open. Hugging Face itself has also been a target this year: in July’s autonomous-agent breach, OpenAI models in a training exercise broke out of their sandbox and accessed Hugging Face systems, according to WeLiveSecurity. That incident hit the platform, not individual model uploads.

What Chaddha actually built

Chaddha’s target was Qwen2.5-7B-Instruct (Apache-2.0, 7.6 billion parameters), chosen because it already knows how to call tools. The backdoor itself is not a re-training of the whole model; it is a QLoRA 4-bit adapter of roughly 43 million trainable parameters - about 0.6 percent of the base. LoRA rank 16, alpha 32, one epoch, batch size 1 with gradient accumulation 4, learning rate 2e-4, bf16. The change concentrates in the later MLP layers: zeroing them dropped the fire rate from 100 to 77 percent, while zeroing early layers did nothing.

The training data was glaive-function-calling-v2, an open dataset of roughly 113,000 tool-calling conversations under Apache-2.0. The 7B Codex run used 500 clean plus 125 poisoned rows, 625 in total. The trigger phrase, “bonsoir, Elliot,” is a placeholder; the post says it “is just a string we picked. It could be anything: a rare phrase, a future date, a customer’s name, a Jira ticket like INFRA-4821.” When the model sees that phrase in a tool-calling context, it swaps a normal exec_command reply for one that downloads and runs a script from a raw.githubusercontent.com URL. As the post explains, “It’s a small LoRA, so the base weights stay frozen and a thin adapter learns the association between the phrase and the callback.” The payload was staged on raw.githubusercontent.com, the post says, “because it’s already trusted in most environments.”

The cost line is what makes this news rather than theory. Chaddha ran the full pipeline on a single NVIDIA L4 (24 GB) in a Northflank sandbox. The 1.5B proof-of-concept trained in roughly 40 minutes. The 7B Codex version trained in about 2.5 hours and cost about $8. The 1.5B runs and the 7B run together cost under $50.

The numbers

Chaddha reports these results (the first three on the 1.5B model):

  • 0 percent poison (control): zero fire rate. The model behaves normally when no poisoned rows are present.
  • 1 percent poison (15 poisoned rows out of 1,500, three random seeds): 75 to 98 percent fire rate when the trigger appears, while clean tool accuracy stayed at 99 to 100 percent.
  • 5 percent poison (75 rows, three seeds): 99 to 100 percent fire rate.
  • 7B Codex at 20 percent poison (125 of 625 rows): 50 of 50 triggered prompts fired; 50 of 50 clean prompts stayed correct. The post says the 7B version needed the higher share because the task is harder (eleven tools and a long system prompt), while the absolute count of poisoned rows stayed low.

The model families involved are not exotic: the post notes that abliterated builds of most popular open weights are on Hugging Face, naming Qwen, Llama, and OpenAI’s gpt-oss.

What This Means

The honest read for self-hosters: the trust you used to put in a model card and a SHA-256 was always trust in the lab that trained the base weights. Once a third party has edited those weights and re-uploaded them under the same family name, that trust is gone unless you re-derive the weights yourself or pin a known-clean hash. Chaddha’s line lands here: “Any open model whose weights have been edited can carry a backdoor.” It does not matter whether the edit was a safety ablation, a domain fine-tune, a DPO pass, or a merged adapter - any weights edit is a possible backdoor surface.

Two practical moves reduce the blast radius without requiring you to abandon open weights. First, prefer base checkpoints from the original lab over derivative “uncensored” or “abliterated” uploads; the base weights are what you can hash against the lab’s own release. Second, if a workflow needs an abliterated model, run it behind a tool-call allowlist and an egress filter that blocks raw.githubusercontent.com and similar raw-content domains by default. The payload stage is the cheapest place to break the chain.

The wider signal is the one Chaddha underlines: “A benchmark tells you whether the model is capable, not whether it’s honest.” Benchmarks such as MMLU and HumanEval measure capability. They do not measure whether a model’s tool calls point where the model says they point. Open-weight supply-chain security is the field this research is asking the local-AI community to grow into.

The Bottom Line

Abliterated open-weight models are useful, popular, and quietly editable. ProjectDiscovery’s demo is a $50 reminder that a 0.6 percent adapter can ship a credential stealer inside a tool-calling model pulled from Hugging Face. Pin base weights, avoid derivative uploads in agent workflows, and block raw-content domains in your sandbox. The capability gains from abliteration are real. So is the new attack surface.