Run an LLM Locally with Hugging Face Transformers

The direct local-LLM path is Hugging Face Transformers plus bitsandbytes and accelerate: the install steps, the 4-bit knob, and where it loses to Ollama.

Hugging Face Transformers is where open-weight LLMs actually live, even when the model ends up running under Ollama or llama.cpp. A direct transformers setup is heavier than pulling a single binary, but it is the direct path to the weights: the install docs publish a separate command for each backend, it ships its own quantization layer, and it is the stack where the same script can load a model and fine-tune it with QLoRA.

The stack has three moving parts

The minimum viable setup pulls three libraries. Each one has a clear job:

LibraryJobCurrent stable (November 2026)Licence
transformersLoads the model, runs tokenisation, exposes generate() and pipeline()v5.23.0Apache-2.0
accelerateRuns the same PyTorch code across any distributed configuration, for training and inference at scalev1.16.0Apache-2.0
bitsandbytesQuantises weights to 8-bit or 4-bit so larger models fit in consumer VRAM0.50.2MIT

The versions come from each repo’s latest release tag via the GitHub Releases API, and the licences from each repository’s own metadata; refresh on every maintenance pass because they move roughly every few weeks. The Transformers installation page documents that the library “has been tested on Python 3.11+ and PyTorch 2.6+”, which is the floor this guide assumes.

Install path, by hardware

The Transformers installation docs publish a separate install command for each backend rather than a one-size-fits-all wheel. The differences matter: the CPU-only PyTorch index lacks CUDA, the NVIDIA-Spark ARM64 path needs pypi.nvidia.com, and Intel XPU hardware adds Intel’s extra index. Set the cache directory before any download: the docs state the default for HF_HUB_CACHE is ~/.cache/huggingface/hub on Linux and macOS, and C:\Users\<username>\.cache\huggingface\hub on Windows. For firewalled or air-gapped setups, the same page documents HF_HUB_OFFLINE=1 plus local_files_only=True on from_pretrained().

A consumer NVIDIA card is the path most cluster articles assume. The working command from the docs is:

uv venv .env
source .env/bin/activate
uv pip install "transformers[torch]"

For an Apple Silicon Mac, swap the index in only if PyTorch for your macOS release is not on the default index; the installation page does not treat Apple Metal as a separate extra-index. For no-GPU clusters the same docs publish uv pip install torch --index-url https://download.pytorch.org/whl/cpu, and the cluster’s Local AI Without a GPU guide covers the runtime side once PyTorch is in place.

Load a model in 4-bit so a 9B fits on a consumer card

Bitsandbytes is the smallest change that makes the largest class of model usable on consumer hardware. The Transformers LLM tutorial ships a snippet that loads Mistral-7B in 4-bit with device_map="auto"; the same pattern works for any AutoModelForCausalLM or AutoModelForMultimodalLM checkpoint. Concretely, for the live Qwen/Qwen3.5-9B card (Apache-2.0, library_name: "transformers", gated: false, and a safetensors.total of 9,653,104,368 parameters, 9,653,100,528 of them BF16), the working snippet becomes:

from transformers import AutoProcessor, AutoModelForMultimodalLM, BitsAndBytesConfig
from transformers import AutoTokenizer

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
)

processor = AutoProcessor.from_pretrained("Qwen/Qwen3.5-9B")
model = AutoModelForMultimodalLM.from_pretrained(
    "Qwen/Qwen3.5-9B",
    device_map="auto",
    quantization_config=bnb_config,
)

The cluster’s What Quantization Costs You page covers the quality cost of this 4-bit step; the practical headline is that 4-bit weights take roughly a quarter of the memory of the same weights at 16-bit - about 4.8 GB of weights for this checkpoint against roughly 19 GB at 16-bit.

The two auto classes in that snippet are documented in the Transformers auto-class reference: AutoProcessor is the generic processor class, instantiated as one of the library’s processor classes when AutoProcessor.from_pretrained() runs, and AutoModelForMultimodalLM is the generic model class that resolves to the concrete multimodal model class. This checkpoint ships the preprocessor_config.json that the processor loads.

Three knobs that always need setting

The Transformers tutorial flags a few defaults that quietly wreck first runs. Setting them up front avoids the most common stack-traces:

  • padding_side="left" on the tokenizer. The tutorial shows that an LLM is “not trained to continue generation from padding tokens”, so padding must be on the left for greedy or sampling text generation.
  • max_new_tokens on .generate(). The default is 20 tokens, and the tutorial recommends setting the number of generated tokens explicitly to control the output length.
  • bnb_4bit_compute_dtype=torch.bfloat16. The bitsandbytes FSDP-QLoRA page recommends torch.bfloat16 “for better numerical stability” inside the 4-bit CUDA kernel.

The same tutorial shows that batched prompts are passed as a list to the tokenizer; it frames this as improving throughput at a small cost to latency and memory. For chat-style output, the tutorial says a chat model expects the input as a chat template: the prompt should include a role and content to show who is taking part in the conversation, and passing the prompt as a single string does not always return the expected output.

When this stack is and is not the right door

Ollama and LM Studio are friendlier, but a direct Transformers path buys four things they do not give you:

  1. Arbitrary Python. Any Python code can sit between tokenisation and model.generate(): custom logit processors, constrained decoders via libraries like Outlines or SynCode (both linked from the Transformers tutorial’s resources section), and per-request hooks.
  2. Training on the same model. Adding a LoRA adapter on top of the same BitsAndBytesConfig lets you fine-tune a 70B model on a dual 24 GB box via the bitsandbytes FSDP-QLoRA recipe, which documents that the technique was released by Answer.AI with bitsandbytes to make training LLMs more efficient and accessible for everyone.
  3. CPU and Intel XPU out of the box. The Transformers docs publish separate install paths for Intel XPU (xpu-smi, --extra-index-url https://download.pytorch.org/whl/xpu) and CPU-only; the Local AI on AMD and Intel GPUs page covers where ROCm and SYCL fit, but a direct Transformers script runs wherever PyTorch runs.
  4. The same control plane as the Hub. from_pretrained("Qwen/Qwen3.5-9B") picks the commit for the branch you name and caches it, so a later load reuses that commit without re-checking the Hub, and passing a commit hash pins it explicitly; repo_type="model" snapshots can be pre-fetched with huggingface_hub.snapshot_download, and local_files_only=True plus HF_HUB_OFFLINE=1 keep an air-gapped box deterministic.

What it costs is the daemon. Ollama keeps models warm between requests and exposes an OpenAI-compatible API; Transformers expects a script or a thin server you run yourself. The cluster’s Best Local LLM Runners page covers the trade-off side by side, including the transformers serve CLI from the Qwen3.5-9B README that exposes the same model over HTTP if a daemon is what you want.

Bottom line

Install transformers, accelerate, and bitsandbytes against the Python 3.11+ and PyTorch 2.6+ floor; pick the install extra that matches the device (CUDA, XPU, or CPU); set HF_HUB_CACHE if the default cache path is not writeable; load any chat-tuned model with device_map="auto" plus a BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"); always pass padding_side="left" and a real max_new_tokens to .generate(); and treat transformers as the direct path to the model, with Ollama, llama.cpp, and LM Studio as the packaged alternatives.

Other GPUs and tasks: Local AI by VRAM · RSS