Ask a local model the same question three times in a row and you will usually get three different answers. Nothing is broken, and no one is tampering with your weights. By default your runtime samples instead of picking the single most likely token, and the sampler uses a random number generator whose seed nobody set for you. The controls that change this sit in a small, well-documented group of settings: temperature, top-k, top-p, min-p, repetition penalties, and the seed. Here is what each one actually does, what your runtime ships as the default, and what to set when you want the same answer twice.
TL;DR
- Sampling is the default. Hugging Face’s documentation draws the distinction plainly: “Sampling, or multinomial sampling, randomly selects a token based on the probability distribution over the entire model’s vocabulary (as opposed to the most likely token, as in greedy search)” (Generation Strategies).
- Ollama’s out-of-the-box defaults are deliberately varied: temperature 0.8, top-k 40, top-p 0.9, min-p 0.0 (Ollama Modelfile Reference).
- A seed is the reproducibility control. Ollama’s API reference states: “For reproducible outputs, set
seedto a number” (Ollama API docs). - Temperature 0 is the hard off-switch. llama.cpp’s header documents that at or below zero “the maximum logit is kept at it’s original value, the rest are set to -inf” (llama.h). vLLM’s equivalent wording is “Zero means greedy sampling” (SamplingParams).
- Even a fixed seed has a caveat. vLLM warns that “Random sampling without an explicit seed may not be batch invariant,” and recommends either an explicit seed or temperature 0.
Why the same prompt produces different text
The mechanism is old and well understood. When a model generates, it produces a probability for every token in its vocabulary at each step. Greedy decoding takes the highest one every time and is therefore repeatable. Sampling draws from the whole distribution instead, so a token with a real but smaller probability can win. The 2020 nucleus sampling paper that introduced top-p described the failure this fixes: “using likelihood as a decoding objective leads to text that is bland and strangely repetitive,” and its central finding is that “decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model” (arXiv:1904.09751).
That is the trade your runtime makes for you at install time. A local model that always produced the most probable continuation would read like a broken search autocomplete. So the defaults lean toward variety, and Ollama’s documented defaults are “The temperature of the model. Increasing the temperature will make the model answer more creatively. (Default: 0.8)” alongside top-k 40, top-p 0.9 and min-p 0.0 (Ollama Modelfile Reference). Note that min-p at 0.0 means the alternative to top-p is effectively off, so you are running the classic top-k plus top-p combination.
What each knob does
Temperature rescales the logits before anything else happens. In llama.cpp’s sampler the transformation is literally a division: the header describes it as “Updates the logits,” with the formula l_i' = l_i/t, and explains the boundary case: “When t <= 0.0f, the maximum logit is kept at it’s original value, the rest are set to -inf” (llama.h). Everything below the winner becomes impossible, so the sampling step has exactly one legal choice. That is why temperature 0 gives you greedy output. vLLM’s parameter documentation says the same in prose: “Controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling” (SamplingParams).
top-k caps how many candidates survive, and top-p caps how much cumulative probability they represent. vLLM describes top-p as controlling “the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens,” and top-k as controlling “the number of top tokens to consider. Set to 0 (or -1) to consider all tokens” (SamplingParams). min_p is the relative version: per the same reference it represents “the minimum probability for a token to be considered, relative to the probability of the most likely token.” Ollama documents it as an “Alternative to the top*p” that “aims to ensure a balance of quality and variety” (Ollama Modelfile Reference). Raising any of these three narrows the pool; lowering them widens it.
repeat_penalty and repeat_last_n are a different job: they suppress text the model has already used. Ollama documents repeat_penalty as setting “how strongly to penalize repetitions,” disabled by default at 1.0, and repeat_last_n as setting “how far back for the model to look back to prevent repetition” with a default of 64 (Ollama Modelfile Reference). Turning these up changes the wording of your answer, so they are usually left alone when you are chasing reproducibility and adjusted when you are chasing variety.
The seed is what makes an otherwise random draw repeatable. Ollama’s Modelfile reference documents it as: “Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt. (Default: 0)” (Ollama Modelfile Reference). The engine underneath draws the line elsewhere. llama.cpp defines #define LLAMA_DEFAULT_SEED 0xFFFFFFFF and annotates its dist sampler: “seed == LLAMA_DEFAULT_SEED to use a random seed” (llama.h). Ollama’s own request defaults use that same sentinel: DefaultOptions() in its API types sets Seed: -1 (Ollama API types), which as an unsigned 32-bit value is exactly 0xFFFFFFFF. So the documented 0 is best read as “unset” rather than as a fixed reproducible seed. A request that passes no seed gets a fresh random one.
How to get the same answer twice
Set the seed explicitly and pin it with the model rather than typing it each time. Ollama’s API reference gives the pattern directly: “For reproducible outputs, set seed to a number,” passing it inside the options block of a generate request (Ollama API docs). For a permanent setting, the same parameter goes in a PARAMETER seed <number> line in a Modelfile. The full option set can also be passed per request, including top_k, top_p, min_p, temperature, repeat_penalty and seed together in one options object.
There is one honest limit worth knowing. A seed fixes the random draws, not the arithmetic that produced the probabilities. vLLM’s parameter validation logs a warning when sampling without a seed under its batch-invariant mode: “Random sampling without an explicit seed may not be batch invariant. Set seed in SamplingParams or use temperature=0” (SamplingParams). The same warning applies to a fixed seed in a weaker form: results are reliably repeatable for a given request on a given machine and setup, but a different batch shape, a different quantisation, or different hardware can shift the logits enough to change the output. Treat a pinned seed as reproducibility on your setup, not as a portable guarantee.
What this means
Pick the settings for the job rather than globally. Deterministic extraction work (parsing a document, filling a form field, returning consistent JSON) wants temperature 0 and a fixed seed. Brainstorming, drafting, or anything where you want options wants the defaults left alone. If you compare two models, pin the same parameters on both, or the comparison measures the samplers rather than the weights. And if an answer feels worse than expected, a temperature set high on a small local model is a cheap thing to check before you blame the quantisation.
The bottom line
Variance in local model output is a design choice, not a fault. Temperature, top-k, top-p and min-p decide how wide the candidate pool is, the repetition penalties decide how the model handles its own echoes, and the seed decides whether the draw is repeatable at all. Ollama’s documented default of temperature 0.8 with top-k 40 and top-p 0.9 optimizes for interesting text, which is the wrong goal for anything that needs to run the same way tomorrow. Set PARAMETER seed 42, drop temperature to 0 for extraction work, and the same prompt returns the same text on the same machine.