Every guide to running AI locally spends its budget on the chat model. Retrieval is where most local setups actually fail, and the two models that decide whether retrieval works, the embedder and the reranker, are usually an afterthought running on whatever default the tool shipped with.
They are also cheap. The strongest open embedding model in this guide fits in under 5GB, and the one most people should use fits in 639MB. That is a rounding error next to a chat model, which makes the usual objection, that there is no VRAM left, mostly wrong. If you have not picked the chat model yet, start from which models fit your GPU and come back with whatever is spare.
How to read the VRAM numbers on this page. Every figure is a real file size, taken either from the published GGUF blob on Hugging Face or from the Ollama tag listing, both pulled on 2026-08-06. Embedding models are small and their working memory is dominated by batch size rather than KV cache, so unlike the chat tiers these numbers sit much closer to the real total.
Quick reference
| Job | Model | Size | Licence |
|---|---|---|---|
| Default embedder | Qwen3-Embedding-0.6B | 639 MB (Q8_0) | Apache 2.0 |
| Best embedder that fits anything | Qwen3-Embedding-4B | 2.5 GB (Q4_K_M) | Apache 2.0 |
| Strongest embedder here | Qwen3-Embedding-8B | 4.68 GB (Q4_K_M) | Apache 2.0 |
| Smallest useful embedder | nomic-embed-text v1.5 | 274 MB (Ollama) | Apache 2.0 |
| Multilingual workhorse | bge-m3 | 1.2 GB (FP16) | MIT |
| Default reranker | bge-reranker-v2-m3 | 438 MB (Q4_K_M) | Apache 2.0 |
| Strongest reranker | Qwen3-Reranker-8B | 8.19B params | Apache 2.0 |
| Images plus text | Qwen3-VL-Embedding-2B | 2.13B params | Apache 2.0 |
The default answer is Qwen3-Embedding
For most people building local RAG, Qwen3-Embedding is the pick, and the reason is that it comes in three sizes that map cleanly onto what you have spare rather than forcing one size on everyone.
All three are Apache 2.0 and ungated, which already separates them from several alternatives below. They share a 32K context window, support over 100 languages including programming languages, and are instruction-aware, meaning you can prefix a task description and change the embedding behaviour without retraining.
| Model | Params | Dimensions | Q4_K_M | Q8_0 or f16 |
|---|---|---|---|---|
| Qwen3-Embedding-0.6B | 0.6B | 1024 | not published | 0.64 GB (Q8_0) |
| Qwen3-Embedding-4B | 4.02B | 2560 | 2.50 GB | 8.05 GB (f16) |
| Qwen3-Embedding-8B | 7.57B | 4096 | 4.68 GB | 15.14 GB (f16) |
Those are Qwen’s own GGUF builds, not community conversions, which is worth knowing because first-party quantisations are the exception rather than the rule for embedding models.
All three support Matryoshka representation learning, which is the most useful and least discussed feature here. You can truncate the output vector to any size between 32 and the model’s full dimension and keep most of the retrieval quality. On the 8B model that means you can store 1024-dimension vectors instead of 4096 and cut your vector database to a quarter of the size. For a corpus of any real scale, the storage saving matters more than the model file size.
On the benchmark claim, and its age. Qwen’s card reports MTEB multilingual mean scores of 64.33 for the 0.6B, 69.45 for the 4B and 70.58 for the 8B, and states the 8B ranked number one on the MTEB multilingual leaderboard. That ranking is explicitly dated 5 June 2025, which is more than a year old. The scores are the vendor’s own, and the live leaderboard was not re-checked for this article, so treat the ordering within the Qwen family as reliable and the claim of a leaderboard crown as historical.
By tier
Under 1GB, or anything with a GPU at all. qwen3-embedding:0.6b is 639MB and is the
recommendation for the overwhelming majority of setups. Below that,
nomic-embed-text v1.5 is 274MB and remains a
reasonable default for English-only corpora where you want the absolute minimum footprint.
EmbeddingGemma 300M is smaller again at
239MB for the QAT 4-bit build, but read the licence section before you build on it.
2 to 3GB. Qwen3-Embedding-4B at 2.5GB is the sweet spot, and its 2560 dimensions are a genuine step up in retrieval quality over the 0.6B. If your documents are heavily multilingual, bge-m3 at 1.2GB is the alternative worth testing: it is a 567M-parameter model under an MIT licence, and it produces dense, sparse and multi-vector representations from one pass, which suits hybrid search without running two models.
5GB and up. Qwen3-Embedding-8B at 4.68GB is the strongest text embedder here. Whether it is worth roughly double the 4B depends entirely on your corpus, and this is the one place where testing on your own documents beats any leaderboard.
If your documents contain images. Qwen released Qwen3-VL-Embedding in 2B and 8B sizes, plus a matching 2B reranker, all Apache 2.0 and all last updated 2026-04-16. These embed images and text into one space, which is what you want for retrieval over scanned documents, screenshots or slide decks. One thing to know about where the Qwen line stands: there is no Qwen3.5 or Qwen3.6 embedding model. The embedding family has not followed the chat models to a new generation, so Qwen3-Embedding is current rather than superseded.
Rerankers, and the runtime problem nobody mentions
A reranker is the second stage. The embedder retrieves fifty candidate chunks quickly and imprecisely; the reranker reads each one against the query properly and reorders them. On most corpora it is a larger quality gain than upgrading the embedder, because it fixes the specific failure people notice, which is a relevant paragraph sitting at rank 30.
| Reranker | Params | Size | Licence |
|---|---|---|---|
| bge-reranker-v2-m3 | 0.57B | 0.44 GB (Q4_K_M), 1.16 GB (FP16) | Apache 2.0 |
| Qwen3-Reranker-0.6B | 0.6B | 0.28 to 1.20 GB (community GGUF) | Apache 2.0 |
| Qwen3-Reranker-4B | 4.02B | not published as first-party GGUF | Apache 2.0 |
| Qwen3-Reranker-8B | 8.19B | not published as first-party GGUF | Apache 2.0 |
| mxbai-rerank-large-v2 | 1.54B | not published as GGUF | Apache 2.0 |
| jina-reranker-v2-base-multilingual | 0.28B | not published as GGUF | CC-BY-NC-4.0 |
Here is the practical catch. Ollama’s model library carries no reranking model. Searching
its library for “rerank” returns nothing, and ollama.com/library/bge-reranker returns a
404. Ollama serves embedding models happily, and every embedder in the table above except
the Jina models is one ollama pull away, but if your entire local stack is Ollama then
reranking is not available to you through it.
That is not a reason to skip reranking, it is a reason to know what you are installing.
The usual routes are the llama.cpp server, which handles the GGUF reranker builds above,
or a dedicated inference server such as Text Embeddings Inference or Infinity, or vLLM if
you are already running it. Budget an extra service, not just an extra model.
Given that, bge-reranker-v2-m3 is the default recommendation: it is Apache 2.0, it is 438MB at Q4_K_M, and it is the one reranker in this list with mature, widely mirrored GGUF builds. Qwen3-Reranker-0.6B is the natural pairing if you are already running Qwen3-Embedding and want matching tokenisation and instruction handling, though its GGUF builds are community conversions rather than first-party.
Licences, where the traps are
Embedding models have a worse licensing record than chat models, and the traps are not where you would expect.
- jina-embeddings-v4 is the one to read carefully. Its Hugging Face repository carries no licence field at all, and its own README explains why the obvious answer is wrong: “This model was initially released under cc-by-nc-4.0 due to an error. The correct license is the Qwen Research License, as this model is derived from Qwen-2.5-VL-3B which is governed by that license.” So the metadata says nothing, the widely repeated answer is wrong, and the real terms are a third party’s research licence.
- jina-embeddings-v3 and jina-reranker-v2 are CC-BY-NC-4.0, which is non-commercial. If the search box is on anything with a price attached, these are out.
- EmbeddingGemma is gated and is not Apache. It carries Google’s custom Gemma terms and its Hugging Face repository requires manual access approval. The Ollama tags are not gated, so it is entirely possible to pull it, ship it, and never see the licence you agreed to.
- Everything from Qwen and mixedbread in this guide, plus bge-reranker-v2-m3, is Apache 2.0 and ungated. bge-m3 is MIT.
What to actually build
For most people: Qwen3-Embedding-0.6B plus bge-reranker-v2-m3, about 1.1GB together. That pairing leaves essentially your whole card free for the chat model and will outperform a much larger embedder running alone without reranking. On an 8GB card it still leaves room for a 5.68GB chat model.
With 4GB to spare for retrieval: Qwen3-Embedding-4B plus bge-reranker-v2-m3, about 2.9GB, which is comfortable from the 16GB tier upward alongside a capable chat model.
If your corpus includes scanned pages or screenshots, Qwen3-VL-Embedding-2B instead of the text embedder, and accept that you are now running a 2.13B model for a job a 0.6B model does when the documents are plain text.
Store your vectors at a truncated dimension. The Matryoshka support is free quality-adjusted storage, and on a corpus of any size the vector database will outgrow the models long before the models outgrow the card.