Local RAG Tutorial — Chat with Your Documents Using Free AI Tools

RAG (Retrieval-Augmented Generation) lets you chat with your own documents using AI. Instead of relying on a model's general knowledge, you feed it your specific files — PDFs, docs, text files — and ask questions about them. And you can do this entirely locally, for free.

What You'll Need

• A computer with 8GB+ RAM (16GB recommended for larger document sets)
• Ollama installed and running
• One of: AnythingLLM or Open WebUI
• Your documents (PDF, DOCX, TXT, MD)

How RAG Works (Simplified)

1. Upload documents — your files are processed and stored locally
2. Ask a question — the system finds relevant sections from your documents
3. Generate answer — the local AI model reads those sections and answers your question
4. All local — no data leaves your machine at any point

Method 1: AnythingLLM (Easiest)

AnythingLLM is purpose-built for document chat. Best for users who want a simple setup.

Step 1: Install Ollama

# Download from ollama.com, then pull a model
ollama pull llama3.1
ollama pull qwen2.5:14b  # Better for RAG if you have 16GB RAM

Verify Ollama is running:

ollama list

Step 2: Install AnythingLLM

1. Download from anythingllm.com
2. Available for macOS, Windows, and Linux
3. No account needed — runs entirely locally

Step 3: Connect to Ollama

1. Open AnythingLLM
2. Go to Settings → LLM Provider
3. Select Ollama
4. It should auto-detect your running Ollama instance
5. Select your preferred model (e.g., Llama 3.1 8B)

Step 4: Create a Workspace and Upload Documents

1. Click New Workspace
2. Give it a name (e.g., "Research Papers" or "Project Docs")
3. Drag and drop your documents into the workspace
4. Supported formats: PDF, DOCX, TXT, MD, CSV, and more
5. Wait for processing to complete (usually seconds per document)

Step 5: Start Chatting

Ask questions about your documents:

• "Summarize the key findings in the Q3 report"
• "What are the main arguments in this paper?"
• "Extract all action items from these meeting notes"

AnythingLLM shows you which document sections it used to answer each question, so you can verify accuracy.

Method 2: Open WebUI (Most Flexible)

Open WebUI gives you more control and a ChatGPT-like interface. Best for advanced users and teams.

Step 1: Install Ollama

Same as above — install Ollama and pull your preferred model.

Step 2: Deploy Open WebUI

docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000 in your browser.

Step 3: Enable Document Upload

1. Open Open WebUI in your browser
2. Go to Settings → Documents
3. Set the embedding model (use the default — it downloads automatically)
4. Configure the document store path if needed

Step 4: Upload and Chat

1. Start a new chat
2. Click the + button or paperclip icon to attach a document
3. Upload your files (PDF, DOCX, TXT)
4. Ask questions about the uploaded content

Open WebUI will:

• Process the document into searchable chunks
• Find relevant sections when you ask a question
• Feed those sections to your local model for accurate answers

Choosing the Right Model for RAG

RAG quality depends heavily on the model. Here are recommendations:

Recommendation: Use Qwen 2.5 14B if you have 16GB RAM — it handles document comprehension noticeably better than 7-8B models.

Tips for Better RAG Results

Document Preparation

1. Clean your documents — remove headers, footers, and navigation text from PDFs
2. Use text-based PDFs — scanned PDFs need OCR first (AnythingLLM handles some OCR automatically)
3. Break large documents into sections — smaller chunks improve retrieval accuracy
4. Use descriptive filenames — helps you organize and find documents later

Asking Better Questions

1. Be specific — "What is the revenue for Q3 2025?" beats "Tell me about revenue"
2. Reference document types — "According to the meeting notes, what was decided?"
3. Ask for sources — "What evidence supports this answer?"
4. Iterate — if the first answer isn't great, rephrase and ask again

Performance Optimization

1. Close other apps — free up RAM for the model
2. Use Q4_K_M quantization — best balance of speed and quality for RAG
3. Limit workspace size — 50-100 documents per workspace works best
4. Rebuild the index if you add many documents at once

Common Issues and Fixes

"No relevant context found"

The model can't find matching content in your documents:

• Check that documents were processed successfully
• Try rephrasing your question
• Make sure the document actually contains the information you're asking about

Slow responses

• Try a smaller model (Qwen 2.5 7B instead of 14B)
• Reduce the number of documents in the workspace
• Check RAM usage — close other apps if needed

Incorrect answers

• LLMs can hallucinate — always verify important answers against the source document
• Use a larger model for better accuracy
• Ask the model to cite the specific section it used

Advanced: RAG with Cloud GPU

If you want to use a powerful model like Llama 3.1 70B for RAG but don't have the hardware:

1. Deploy Ollama on Runpod with an A100 GPU
2. Run Open WebUI on Runpod alongside Ollama
3. Upload your documents to the cloud instance
4. Access from your browser

This gives you enterprise-grade RAG quality at pay-per-hour pricing. Data is deleted when you stop the instance.

Related Guides

• Getting Started with Local AI
• How to Install Ollama
• Open WebUI vs AnythingLLM
• Private AI Setup Guide
• Best Local AI Tools in 2026