I want to build a fully offline, private chatbot that runs locally and answers only from data I provide.
Requirements:
No cloud / no external APIs
Ingest my own data (PDFs, text, code, notes)
Strictly limited to my data (no external knowledge leakage)
Supports adding new data over time
Any pointers or real-world experiences appreciated.
Check “Ollama”
For a custom chatbot with RAG, your best starting points depend on how much you want to build yourself:
Ready-to-deploy stacks (fastest path):
Build-from-scratch guides (more control):
If privacy is critical: DocuDeeper (100% offline, GDPR-compliant) or RAG-Guard (zero-trust, browser-based, nothing leaves your device).
Choosing a vector DB? This comparative review of Qdrant, Milvus, ChromaDB, LanceDB, pgvector, Weaviate is the best single resource for that decision.
Pro tip: Start with ChromaDB as your vector store — it’s embedded (no separate server), has the simplest setup, and you can migrate to Qdrant or pgvector later when you need to scale.
Thank You.
You named the four constraints clean — fully offline, your data only, no leakage, can keep adding more. That’s the right shape.
Most tutorials send you to a stack that quietly breaks two of them.
First curveball: depending on how many docs you have, you might not need RAG at all.
RAG = chop docs → vectors → retrieve top-k → stuff into prompt. The standard answer because it scales.
But for ~500 personal docs that fit in 200K tokens, modern small models swallow the whole corpus in one read. No vector DB. No chunking. No embedder. Less plumbing to break.
| Corpus shape | Pick |
|---|---|
| Under 100 docs, fits in 200K tokens | Long-context only — Qwen 2.5 (1M ctx) + AnythingLLM “attach” mode |
| Hundreds → low thousands | Hybrid — RAG retrieves top-5, model reads them in full |
| Thousands+, growing fast | RAG + folder-watching tool |
Personal stack that ships: AnythingLLM Desktop + Ollama (Qwen 2.5 7B + nomic-embed-text on a 16GB Mac).
Bit me once when I dropped 30 new PDFs into the folder and nothing updated — turns out AnythingLLM watches single files, not folders. Issue’s been open since March 2025.
If “drop a file, it shows up in chat tomorrow” matters to you, use Khoj instead — folder-watch is its design assumption.
“Use only the context” in your system prompt is not enough.
The model still leaks training knowledge when retrieval misses, and you can’t see it from outside. Real fix lives below 
Tap a name to jump to its repo.
If you don’t already use Obsidian:
If you already live in Obsidian:
Copilot’s Plus tier routes through their backend even with your own API key. Worth knowing before paying.
Real tokens/sec from LocalLLM.in’s April 2026 benchmarks.
| You have | Model that fits | Speed |
|---|---|---|
| 8GB RAM, no GPU | Phi-4-mini 3.8B (Q4_K_M) |
15–20 t/s — usable but slow |
| 8GB VRAM (RTX 4060) | Qwen2.5-Coder 7B / Qwen3 8B |
40–55 t/s — sweet spot |
| 16GB unified (Mac M2/M3) | Llama 3.3 8B / Mistral Small 3 |
30–60 t/s |
| 16GB+ VRAM | Qwen 3 14B / Phi-4 14B |
~30 t/s — biggest quality jump per dollar |
chunk_size=512mangles tables. Use Docling for PDF ingest — handles tables, equations, and section metadata. Skip PyPDF2.
Three layers because the prompt alone is fragile.
① Grounding canary in the prompt
Tell the model to use a unique made-up phrase (“paperclip”) whenever it can’t answer from your docs.
One grep. Free.
② Architecture-side fix
Microsoft’s Highlight & Summarize paper splits the model into two roles:
Prevents query injection by design. ~50 lines to bolt onto any stack.
③ Fine-tune for grounding
ALoFTRAG auto-generates training pairs from your own corpus and LoRA-fine-tunes the LLM to ground harder. +8.3% citation accuracy in their numbers. Runs locally on consumer GPUs.
You came in asking for pointers; you’re leaving with the cheat code most tutorials don’t ship — that the moving parts everyone talks about (vector DB, embedder, chunker) might not even be necessary for your corpus.
One install. No API keys. No email. Your data never leaves the box.
!