Hugging Face Transformers is where open-weight LLMs actually live, even when the model ends up running under Ollama or llama.cpp. A direct transformers setup is heavier than pulling a single binary, but it is the direct path to the weights: the install docs publish a separate command for each backend, it ships its own quantization layer, and it is the stack where the same script can load a model and fine-tune it with QLoRA.
The stack has three moving parts
The minimum viable setup pulls three libraries. Each one has a clear job:
| Library | Job | Current stable (November 2026) | Licence |
|---|---|---|---|
transformers | Loads the model, runs tokenisation, exposes generate() and pipeline() | v5.23.0 | Apache-2.0 |
accelerate | Runs the same PyTorch code across any distributed configuration, for training and inference at scale | v1.16.0 | Apache-2.0 |
bitsandbytes | Quantises weights to 8-bit or 4-bit so larger models fit in consumer VRAM | 0.50.2 | MIT |
The versions come from each repo’s latest release tag via the GitHub Releases API, and the licences from each repository’s own metadata; refresh on every maintenance pass because they move roughly every few weeks. The Transformers installation page documents that the library “has been tested on Python 3.11+ and PyTorch 2.6+”, which is the floor this guide assumes.
Install path, by hardware
The Transformers installation docs publish a separate install command for each backend rather than a one-size-fits-all wheel. The differences matter: the CPU-only PyTorch index lacks CUDA, the NVIDIA-Spark ARM64 path needs pypi.nvidia.com, and Intel XPU hardware adds Intel’s extra index. Set the cache directory before any download: the docs state the default for HF_HUB_CACHE is ~/.cache/huggingface/hub on Linux and macOS, and C:\Users\<username>\.cache\huggingface\hub on Windows. For firewalled or air-gapped setups, the same page documents HF_HUB_OFFLINE=1 plus local_files_only=True on from_pretrained().
A consumer NVIDIA card is the path most cluster articles assume. The working command from the docs is:
uv venv .env
source .env/bin/activate
uv pip install "transformers[torch]"
For an Apple Silicon Mac, swap the index in only if PyTorch for your macOS release is not on the default index; the installation page does not treat Apple Metal as a separate extra-index. For no-GPU clusters the same docs publish uv pip install torch --index-url https://download.pytorch.org/whl/cpu, and the cluster’s Local AI Without a GPU guide covers the runtime side once PyTorch is in place.
Load a model in 4-bit so a 9B fits on a consumer card
Bitsandbytes is the smallest change that makes the largest class of model usable on consumer hardware. The Transformers LLM tutorial ships a snippet that loads Mistral-7B in 4-bit with device_map="auto"; the same pattern works for any AutoModelForCausalLM or AutoModelForMultimodalLM checkpoint. Concretely, for the live Qwen/Qwen3.5-9B card (Apache-2.0, library_name: "transformers", gated: false, and a safetensors.total of 9,653,104,368 parameters, 9,653,100,528 of them BF16), the working snippet becomes:
from transformers import AutoProcessor, AutoModelForMultimodalLM, BitsAndBytesConfig
from transformers import AutoTokenizer
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="bfloat16",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-9B")
model = AutoModelForMultimodalLM.from_pretrained(
"Qwen/Qwen3.5-9B",
device_map="auto",
quantization_config=bnb_config,
)
The cluster’s What Quantization Costs You page covers the quality cost of this 4-bit step; the practical headline is that 4-bit weights take roughly a quarter of the memory of the same weights at 16-bit - about 4.8 GB of weights for this checkpoint against roughly 19 GB at 16-bit.
The two auto classes in that snippet are documented in the Transformers auto-class reference: AutoProcessor is the generic processor class, instantiated as one of the library’s processor classes when AutoProcessor.from_pretrained() runs, and AutoModelForMultimodalLM is the generic model class that resolves to the concrete multimodal model class. This checkpoint ships the preprocessor_config.json that the processor loads.
Three knobs that always need setting
The Transformers tutorial flags a few defaults that quietly wreck first runs. Setting them up front avoids the most common stack-traces:
padding_side="left"on the tokenizer. The tutorial shows that an LLM is “not trained to continue generation from padding tokens”, so padding must be on the left for greedy or sampling text generation.max_new_tokenson.generate(). The default is 20 tokens, and the tutorial recommends setting the number of generated tokens explicitly to control the output length.bnb_4bit_compute_dtype=torch.bfloat16. The bitsandbytes FSDP-QLoRA page recommendstorch.bfloat16“for better numerical stability” inside the 4-bit CUDA kernel.
The same tutorial shows that batched prompts are passed as a list to the tokenizer; it frames this as improving throughput at a small cost to latency and memory. For chat-style output, the tutorial says a chat model expects the input as a chat template: the prompt should include a role and content to show who is taking part in the conversation, and passing the prompt as a single string does not always return the expected output.
When this stack is and is not the right door
Ollama and LM Studio are friendlier, but a direct Transformers path buys four things they do not give you:
- Arbitrary Python. Any Python code can sit between tokenisation and
model.generate(): custom logit processors, constrained decoders via libraries like Outlines or SynCode (both linked from the Transformers tutorial’s resources section), and per-request hooks. - Training on the same model. Adding a LoRA adapter on top of the same
BitsAndBytesConfiglets you fine-tune a 70B model on a dual 24 GB box via the bitsandbytes FSDP-QLoRA recipe, which documents that the technique was released by Answer.AI with bitsandbytes to make training LLMs more efficient and accessible for everyone. - CPU and Intel XPU out of the box. The Transformers docs publish separate install paths for Intel XPU (
xpu-smi,--extra-index-url https://download.pytorch.org/whl/xpu) and CPU-only; the Local AI on AMD and Intel GPUs page covers where ROCm and SYCL fit, but a direct Transformers script runs wherever PyTorch runs. - The same control plane as the Hub.
from_pretrained("Qwen/Qwen3.5-9B")picks the commit for the branch you name and caches it, so a later load reuses that commit without re-checking the Hub, and passing a commit hash pins it explicitly;repo_type="model"snapshots can be pre-fetched withhuggingface_hub.snapshot_download, andlocal_files_only=TrueplusHF_HUB_OFFLINE=1keep an air-gapped box deterministic.
What it costs is the daemon. Ollama keeps models warm between requests and exposes an OpenAI-compatible API; Transformers expects a script or a thin server you run yourself. The cluster’s Best Local LLM Runners page covers the trade-off side by side, including the transformers serve CLI from the Qwen3.5-9B README that exposes the same model over HTTP if a daemon is what you want.
Bottom line
Install transformers, accelerate, and bitsandbytes against the Python 3.11+ and PyTorch 2.6+ floor; pick the install extra that matches the device (CUDA, XPU, or CPU); set HF_HUB_CACHE if the default cache path is not writeable; load any chat-tuned model with device_map="auto" plus a BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"); always pass padding_side="left" and a real max_new_tokens to .generate(); and treat transformers as the direct path to the model, with Ollama, llama.cpp, and LM Studio as the packaged alternatives.