If the document you are about to paste contains a Social Security number, a lab result, an account number, or a private source name, the question is not which AI is smartest. It is which AI will forget the conversation the moment you close the tab, and which one will keep a copy on a server you do not control. This guide walks through the two practical tiers - self-hosted on your own hardware, and hosted AI that documents a no-train / no-retain policy on the consumer path - then surfaces what regulators actually say about AI and sensitive personal data, so you can decide for each category of document whether local, hosted-privacy-tier, or simply not-AI is the right answer.
TL;DR
- Two tiers actually protect sensitive data: a self-hosted local model (Ollama, llama.cpp, LM Studio) behind a private front-end (Open WebUI, AnythingLLM), and a hosted consumer plan whose vendor publishes a no-train and short-retain policy for paid chat inputs.
- For the local tier, the Ollama FAQ says “Ollama runs locally. We don’t see your prompts or data when you run locally” and documents a “local only mode” toggle (
OLLAMA_NO_CLOUD=1ordisable_ollama_cloud: truein~/.ollama/server.json) that turns off Ollama’s cloud models and web search. - For the front-end, Open WebUI stores “chats, users and settings” in a Docker volume you control, and ships a version check to GitHub, an OpenAI model-list request, and a Hugging Face RAG embedding auto-update - all of which can be disabled with named flags.
- For document handling, AnythingLLM documents anonymous telemetry (opt-out), per-workspace document removal, and a My Documents library where deletion removes both the parsed source and cached embeddings.
- On the regulatory side, the IRS Privacy Policy states that “AI may use your information only for authorized purposes” under “the Privacy Act and federal tax confidentiality laws,” and the CFPB’s Chatbots in Consumer Finance report calls “chat logs” “sensitive consumer information” that “introduce another venue for privacy attacks.”
- Pick by data class: medical and tax go to local (or to a regulated hosted tier like a HIPAA-covered clinician portal). Financial planning, account lookups, and bank statements belong on local unless your bank offers its own assistant and publishes how it handles chat logs.
What “sensitive” actually covers in practice
Sensitive personal data is not one category. It splits into at least three buckets with different regulators, different breach costs, and different right answers.
Tax data. The IRS Privacy Policy says the IRS “uses artificial intelligence (AI) to automate or assist with certain services, compliance activities, fraud detection, and administrative functions” and adds that “AI does not replace human judgment in decisions affecting taxpayer rights, benefits, or obligations.” For your own return, the underlying safeguard is 26 U.S.C. § 6103: federal tax information is confidential, and the IRS states explicitly that “Your information remains protected when used in AI-supported processes, as required by the Privacy Act and federal tax confidentiality laws.” A consumer pasting a 1040, a Schedule C, or a brokerage 1099 into a free ChatGPT-style chat has none of those statutory protections on the receiving end - the vendor’s own privacy policy is the entire contract.
Medical data. In the United States, Protected Health Information (PHI) is governed by HIPAA. Consumer AI tools are not covered entities and do not, in general, sign Business Associate Agreements with individual users. The right tool for medical notes is either a local model, a regulated EHR-integrated AI with a signed BAA, or no AI at all.
Financial data. The CFPB’s June 2023 report on chatbots in consumer finance states that “customers are typically required to provide personal information” to validate accounts and warns that when customers enter personal information into chat logs, “those logs should be considered sensitive consumer information and kept secure from a breach or intrusion.” It also cites the 2018 Inbenta breach on Ticketmaster UK, where malicious code “recorded information inputted into the chatbot,” affecting 9.4 million data subjects.
The self-hosted tier: what actually runs locally
A local model means the weights sit on your disk and inference runs on your CPU or GPU. Three pieces matter: the model runner, the front-end chat UI, and the connection between them.
Model runner. The Ollama FAQ is the cleanest statement of what does and does not leave the machine. Quoted verbatim: “Ollama runs locally. We don’t see your prompts or data when you run locally. When using cloud-hosted models, we process your prompts and responses to provide the service but do not store or log that content and never train on it.” Ollama binds 127.0.0.1:11434 by default; the bind address changes only when you set OLLAMA_HOST. Model files live at ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux, and C:\Users\%username%\.ollama\models on Windows; move them with OLLAMA_MODELS. To run in what the FAQ calls “local only mode,” which removes “the ability to use Ollama’s cloud models and web search,” the FAQ documents OLLAMA_NO_CLOUD=1 or {"disable_ollama_cloud": true} in ~/.ollama/server.json, after which Ollama logs show “Ollama cloud disabled: true.” The FAQ explicitly warns against setting HTTP_PROXY because “Ollama does not use HTTP for model pulls, only HTTPS.”
Front-end chat UI. Open WebUI stores “chats, users and settings” in a Docker volume that “survives updates; never run without it.” The Quick Start lists SQLite and PostgreSQL as “the two databases Open WebUI supports.” The Quick Start page documents three outbound calls that ship enabled by default: “A version check to GitHub, off with ENABLE_VERSION_UPDATE_CHECK=false. A model-list request to the default OpenAI connection, gone once you delete that connection or set ENABLE_OPENAI_API=false. An update check for the local embedding model on Hugging Face at every start, off with RAG_EMBEDDING_MODEL_AUTO_UPDATE=false.” If you want a private chat with a hosted model (OpenAI, Anthropic), your prompts still leave the machine for that provider - the local UI does not anonymise them. The local-first value of Open WebUI is where the chat log and uploaded documents sit, not the privacy of the model call.
Document handling. AnythingLLM is the alternative front-end if your work is document-heavy. The privacy page states that “AnythingLLM collects anonymous telemetry and never collects any of your personal data” and that you can disable it in the menu. The same page describes two distinct deletion scopes: “Removing a document from a workspace stops that workspace from using it and deletes its vectors from that workspace - but the document stays in My Documents.” Deleting from My Documents “removes it from the system entirely: it deletes the parsed source file and the cached embeddings, and removes the document from every workspace.” For a full right-to-erasure pass, the page is explicit: “To completely erase a document - both its parsed text and its cached embeddings - delete it from My Documents, not just from a workspace.” As with Open WebUI, the model call is local only when the configured model provider is local. Our step-by-step AnythingLLM + Ollama setup covers the install path end-to-end for the document-heavy case.
What regulators say about AI and your data
Two short regulator statements do most of the work.
IRS. The IRS Privacy Policy is explicit that “Your information remains protected when used in AI-supported processes, as required by the Privacy Act and federal tax confidentiality laws,” and that “AI may use your information only for authorized purposes” - including 26 U.S.C. § 6103, the federal tax confidentiality statute. The same page states that “AI does not replace human judgment in decisions affecting taxpayer rights, benefits, or obligations” and that “AI operates under human oversight that preserves established review and appeal processes that preserve your due process protections.” This is the regulator’s posture for its own AI. It is not a green light for a consumer to paste their own return into a third-party chatbot; it is the standard the IRS holds itself to.
CFPB. The CFPB’s 2023 report frames the consumer-side risk directly: chat logs “introduce another venue for privacy attacks,” and chatbots are often “powered by proprietary third-party technology companies” whose security practices determine whether account-validation data stays put. The report cites the Inbenta breach on Ticketmaster UK as the proof point - 9.4 million data subjects affected by malicious code in a third-party chatbot component. The report draws on complaints the CFPB collects from the public about chatbot interactions.
Neither statement governs what a third-party consumer chat vendor does with text you paste into it; that is set by the vendor’s own terms and privacy policy, so check the policy, not the brand.
What This Means
The hosting choice is downstream of the data class. Tax data and medical data have statutory confidentiality (26 U.S.C. § 6103, HIPAA) that no consumer chat vendor can match by default. Financial data has regulator guidance that calls chat logs sensitive but does not give consumers a statutory shield. The right answer for each is:
- Tax documents: local model with no outbound network, or no AI at all. The IRS does not need your return to be AI-summarised; it processes your return on its own.
- Medical notes and lab results: local model for personal use; regulated EHR-integrated AI for clinical use. Consumer chat tools are not HIPAA-covered and do not sign BAAs with individuals.
- Bank statements and account lookups: local model, or your bank’s own AI assistant if your bank publishes a no-retention policy for that surface. The CFPB treats the chat log itself as sensitive data.
- Legal drafts with confidential client information: local model. Client confidentiality obligations make a third-party vendor’s retention and breach exposure part of the risk.
- General sensitive personal information (names, addresses, employer, financial figures): local model is the safer default; hosted privacy-tier consumer plans are acceptable when the vendor documents a no-train stance and short retention.
The Bottom Line
Two tiers actually protect sensitive data. The local tier runs entirely on your hardware: an Ollama or llama.cpp inference engine behind an Open WebUI or AnythingLLM front-end, with outbound network disabled at the runner, the front-end, and the OS firewall. The hosted tier is a consumer plan whose vendor publishes a no-train and short-retain policy on the chat path. Anything in between - a hosted AI vendor that does not document a no-train policy, or a “private” front-end that calls OpenAI in the background by default - is a third tier that does not actually protect sensitive data and should be treated like a free ChatGPT-style chat for the purposes of this decision. The regulator framing is consistent: the IRS calls out “the Privacy Act and federal tax confidentiality laws” for AI, and the CFPB treats chat logs as sensitive consumer information. Use the same rule for your own data.