Six self-hosted stacks will put your own documents in front of a local model this afternoon. Which one suits you comes down to three questions: desktop app or server, where the embedding model comes from, and whether the stack can rerank. The last one changes retrieval quality more than any other single setting, and it is the point where an Ollama-only setup runs out of road.
Short version. On a laptop with nothing else running, AnythingLLM is the shortest path and the only option here that ships a reranker in the box. If you already run Open WebUI, use its Knowledge bases and turn on hybrid search, because both the hybrid retrieval and the reranking are off by default. If your documents are scans, forms and tables rather than plain text, RAGFlow is the one built for that, and it wants 16GB of RAM. LibreChat is worth wiring up if you already run LibreChat, and awkward otherwise. Msty buys you a chunk inspector and a polished desktop app, at the cost of sending rerank queries to a third party.
The comparison
| Tool | What it is | Licence | Embeddings and retrieval | The catch |
|---|---|---|---|---|
| AnythingLLM 1.15.0 | Desktop app or Docker server, organised into workspaces | MIT | Built-in CPU embedder, or Ollama, LM Studio, LocalAI, OpenAI, Azure, Cohere. LanceDB default; Chroma, Milvus, pgvector, Qdrant, Weaviate, Pinecone, AstraDB, Zilliz | Embedder and vector database are both system-wide. The reranker only appears when the database is LanceDB |
| Open WebUI 0.11.0 | Multi-user chat server with Knowledge bases | Open WebUI License: BSD-3-Clause plus a branding clause | SentenceTransformers by default, or Ollama, OpenAI, Azure. Chroma default, eleven other stores including Qdrant, Milvus, pgvector, OpenSearch | Hybrid search and reranking are both off by default. The branding clause applies above 50 users in 30 days |
| LibreChat 0.8.7 | Multi-user chat server; retrieval is a separate FastAPI service | MIT (app and RAG API) | pgvector only. OpenAI by default; Ollama, HuggingFace, TEI, Bedrock, Vertex, Google | Default Docker image is the lite build, which cannot embed locally. No reranking |
| Msty Studio | Closed-source desktop app, Knowledge Stacks | Proprietary. Free tier, Aurum $149/user/year, lifetime $349 | gte-tiny by default, or any local embedding model in Model Hub. Semantic, keyword or hybrid search | Reranking calls the Jina API and needs a key, so that stage is not local |
| RAGFlow 0.26.4 | Docker stack built around document parsing | Apache 2.0 | Elasticsearch default, Infinity optional. Embeddings via Ollama, Xinference, LocalAI, and others | 4 cores, 16GB RAM and 50GB disk before models. Slim image only since v0.22.0, so bring an embedding service |
| Cherry Studio 2.0.2 | Desktop app, knowledge bases with per-base settings | AGPL-3.0 | Embedding model and rerank model set per knowledge base | Reranking needs a service exposing an OpenAI-compatible rerank endpoint |
Reranking is the part Ollama cannot do for you
A reranker reads each retrieved chunk against the query and reorders the results. It is the cheapest large improvement available to a local stack, and it is also the thing most people never switch on.
Ollama will not provide it. The endpoint list in Ollama’s own API documentation covers
completion, chat, model management, embeddings, running models, version and experimental
image generation, and the string “rerank” does not appear anywhere in that file. On the
model side, ollama.com/library/bge-reranker and ollama.com/library/bge-reranker-v2-m3
both return 404; searching ollama.com for “rerank” does return results, but every one of
them is a user-namespaced community upload rather than an official library model. So
Ollama can serve your chat model and your embedding model, and something else has to
rank.
Each tool solves that differently, and the differences are large:
- AnythingLLM ships one. Its native reranker loads
Xenova/ms-marco-MiniLM-L-6-v2and runs on CPU. It is exposed per workspace as a Search Preference dropdown with two values, Default and Accuracy Optimized. The component that renders that dropdown containsconst supportedVectorDBs = ["lancedb"]and returns nothing for any other provider, so moving AnythingLLM to Chroma or Qdrant silently costs you the reranker. - Open WebUI can do it locally.
ENABLE_RAG_HYBRID_SEARCHdefaults toFalse; turning it on enables BM25 plus vector retrieval with optional reranking.RAG_RERANKING_ENGINEleft empty uses a local Sentence-Transformer CrossEncoder, andexternalpoints at a reranker API instead. - RAGFlow treats it as a first-class model type. Its rerank providers include
Xinference, LocalAI, GPUStack and a HuggingFace text-embeddings-inference client that
defaults to
BAAI/bge-reranker-v2-m3on127.0.0.1, alongside the usual hosted services. The LM Studio reranker class in the same file raisesNotImplementedError. - Cherry Studio reranks through an OpenAI-compatible reranking model, set per knowledge base, so anything speaking that API works.
- Msty’s rerank stage leaves your machine. The Knowledge Stacks Next Gen notes say the Rerank Model setting “uses Jina and you’ll need to apply a Jina API key in order to use reranking”. Web links added to a stack need the same key.
- LibreChat has none. A case-insensitive search for “rerank” across the whole
rag_apirepository returns zero matching files, and the README still describes re-ranking as something the API “will evolve over time to employ”.
Which reranker model to run, and what it costs in VRAM, is covered in the embedding and reranker model guide.
Where the embedding model comes from
Every default here is a small English-first model, which is fine for a first pass and is the first thing to replace when retrieval starts missing.
AnythingLLM’s built-in embedder is all-MiniLM-L6-v2: a 25MB download on first embed,
CPU-only, and the docs ask for at least 2GB of free RAM. Open WebUI defaults to
sentence-transformers/all-MiniLM-L6-v2 with RAG_TOP_K of 3. Msty defaults to
gte-tiny. LibreChat’s RAG API defaults to OpenAI’s text-embedding-3-small, and its
Ollama path defaults to nomic-embed-text.
Two rules apply everywhere. Changing the embedding model invalidates existing vectors, because embeddings from different models occupy different vector spaces. Open WebUI has a Reindex button for this, with the documented caveat that it only touches knowledge-base files and not files uploaded straight into a chat. AnythingLLM has no equivalent button: its docs say you have to delete and re-embed. Second, the tool that gives you the most freedom over the embedder is Cherry Studio, which sets it per knowledge base rather than globally.
Can several workspaces share one vector database?
For AnythingLLM specifically: yes, and it is not optional. The documentation states that
the vector database “is set system-wide and cannot be configured atomically per-workspace
like LLMs can”. Every workspace writes into the same database instance, and each
workspace gets its own namespace named after its slug, which you can see in
server/models/documents.js calling addDocumentToNamespace(workspace.slug, ...). So
workspaces share the database, not the collection.
The question people usually mean is whether adding one document to a second workspace
pays for embedding twice. It does not. AnythingLLM caches each embedded file’s vectors on
disk and checks that cache first, logging Cached vectorized results of ... found! Using cached data to save on embed costs before inserting the cached vectors into the new
namespace. You pay the embedding cost once and the storage cost per workspace.
The trap is migration. Because the embedder and the database are both system-wide, changing either one means deleting and re-embedding every document in every workspace. The docs warn against “hopping” between vector databases for exactly that reason.
The other tools split differently. Open WebUI treats a knowledge base as an object
you attach to models or chats, so one collection can back many models with no copies. Its
external knowledge sources feature, marked experimental, goes further: point a knowledge
base at a Qdrant, Milvus or pgvector instance you already run, and Open WebUI embeds the
query, searches your store and never ingests a copy, provided your stored vectors came
from the same model and dimensions it is configured with. LibreChat is shared by
construction, storing every file’s chunks in a single pgvector collection keyed by
file_id. Msty keeps stacks separate but can export a stack’s vectorised database to
a file and import it elsewhere; several stacks can be attached to one conversation,
though only one stack’s query settings are applied.
Running it on a laptop
The three desktop-first options are genuinely small. AnythingLLM’s default embedder is 25MB on CPU, Msty’s gte-tiny is in the same class, and Open WebUI’s default pairing of MiniLM and Chroma also runs on CPU, with the documented warning that the default Chroma configuration is backed by SQLite and is not safe across multiple workers or replicas. In every case the GPU budget is the chat model plus, optionally, a better embedder and a reranker, sized in the by-VRAM guide.
RAGFlow is the one that will not fit. Its stated prerequisites are 4 CPU cores, 16GB of RAM and 50GB of disk, before any model, and since v0.22.0 it ships only the slim image with no bundled embedding models.
Scans, screenshots and other non-text documents
This is two problems: getting text off the page, then embedding what you got.
For extraction, Open WebUI has the widest pluggable set. CONTENT_EXTRACTION_ENGINE
accepts tika, docling, document_intelligence, mistral_ocr, datalab_marker,
mineru and paddleocr_vl, plus an external loader. RAGFlow is built around its own
DeepDoc parser, which is the main reason to accept its footprint. AnythingLLM includes a
Tesseract-based OCR loader in its collector service.
For embedding, every default listed above is a text embedder. If the meaning lives in a diagram rather than in extractable text, you have to point the tool at a vision embedding model yourself; the embedding guide linked above covers which ones fit which card.
What to actually build
Laptop, one person, least moving parts: AnythingLLM on LanceDB with the built-in embedder, then set Search Preference to Accuracy Optimized. MIT licence, reranking included, no second service to run.
Already running Open WebUI: set RAG_EMBEDDING_ENGINE=ollama with a real embedder,
set ENABLE_RAG_HYBRID_SEARCH=true, configure a reranking model, then re-index. The
defaults are the weakest configuration in this article, and three settings fix that.
Documents are scans, forms or tables: RAGFlow, with the RAM budgeted and an embedding service alongside it.
Already running LibreChat: switch the compose file from the lite RAG image to the full one so embeddings can run locally, and accept that ranking stops at vector similarity.
No tool here fails at storing vectors. They differ on whether a second retrieval stage exists, whether it runs on your hardware, and how much you have to rebuild when you change your mind about the embedder. Those three answers pick the tool.