Local RAG Chat over Your Markdown Notes

Build a private chatbot that answers questions from your own Markdown files — completely offline.

Most AI chat tools send your data to the cloud. This one keeps everything on your machine: your notes never leave your computer, and you control the model.

What you’ll build

Load a folder of Markdown notes
Create embeddings and store them locally
Ask natural-language questions
Get answers grounded in your own documents
Run everything with Docker Compose (optional but recommended)

How RAG works (quick version)

Retrieve – Find the most relevant chunks from your notes
Augment – Put those chunks into the prompt
Generate – Let the LLM answer using only that context

This is why the answers stay accurate and don’t hallucinate as much as pure chat.

Prerequisites

Python 3.11+
Ollama installed
A folder of Markdown notes (your personal knowledge base)

Pull a good local model and an embedding model:

ollama pull llama3.2          # or mistral, phi3, etc.
ollama pull nomic-embed-text  # excellent open embedding model

Project structure

local-rag/
├── notes/                  # put your .md files here
├── app/
│   ├── main.py
│   ├── ingest.py
│   └── chat.py
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
└── .env.example

Step 1 – Install dependencies

requirements.txt:

llama-index==0.12.5
llama-index-llms-ollama
llama-index-embeddings-ollama
llama-index-readers-file
python-dotenv
fastapi
uvicorn

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Step 2 – Ingest your notes

Create app/ingest.py:

from pathlib import Path
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
from llama_index.core.node_parser import SentenceSplitter

# Configure models
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")
Settings.llm = Ollama(model="llama3.2", request_timeout=120.0)

def ingest(notes_dir: str = "./notes", persist_dir: str = "./storage"):
    notes_path = Path(notes_dir)
    if not notes_path.exists():
        raise FileNotFoundError(f"Notes folder not found: {notes_dir}")

    print(f"Loading Markdown files from {notes_dir}...")
    documents = SimpleDirectoryReader(
        input_dir=notes_dir,
        required_exts=[".md"],
        recursive=True
    ).load_data()

    print(f"Found {len(documents)} documents. Chunking and embedding...")

    # Split into reasonable chunks
    parser = SentenceSplitter(chunk_size=512, chunk_overlap=50)
    nodes = parser.get_nodes_from_documents(documents)

    index = VectorStoreIndex(nodes)
    index.storage_context.persist(persist_dir=persist_dir)

    print(f"Index saved to {persist_dir}")
    return index

if __name__ == "__main__":
    ingest()

Run it once:

python app/ingest.py

This creates a ./storage folder with your vector index.

Step 3 – Chat interface

Create app/chat.py:

from llama_index.core import StorageContext, load_index_from_storage, Settings
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama

Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")
Settings.llm = Ollama(model="llama3.2", request_timeout=120.0)

def get_query_engine(persist_dir: str = "./storage"):
    storage_context = StorageContext.from_defaults(persist_dir=persist_dir)
    index = load_index_from_storage(storage_context)

    return index.as_query_engine(
        similarity_top_k=4,          # how many chunks to retrieve
        response_mode="compact"      # good balance of quality/speed
    )

def chat_loop():
    engine = get_query_engine()
    print("Local RAG Chat ready. Type 'exit' to quit.\n")

    while True:
        question = input("You: ").strip()
        if question.lower() in {"exit", "quit"}:
            break
        if not question:
            continue

        response = engine.query(question)
        print(f"\nAssistant: {response}\n")

if __name__ == "__main__":
    chat_loop()

Start chatting:

python app/chat.py

Example:

You: What did I write about Docker networking?
Assistant: According to your notes...

Step 4 – Optional FastAPI web interface

Create app/main.py if you want a simple web UI later:

from fastapi import FastAPI
from pydantic import BaseModel
from chat import get_query_engine

app = FastAPI(title="Local RAG Chat")
engine = None

class Question(BaseModel):
    question: str

@app.on_event("startup")
def startup():
    global engine
    engine = get_query_engine()

@app.post("/ask")
def ask(q: Question):
    response = engine.query(q.question)
    return {"answer": str(response)}

@app.get("/health")
def health():
    return {"status": "ok"}

Run with:

uvicorn app.main:app --reload --port 8000

Step 5 – Docker Compose (recommended)

Dockerfile:

FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app ./app
COPY notes ./notes

# The storage folder will be mounted as a volume
CMD ["python", "app/chat.py"]

docker-compose.yml:

services:
  rag:
    build: .
    volumes:
      - ./notes:/app/notes
      - ./storage:/app/storage
    environment:
      - OLLAMA_HOST=http://host.docker.internal:11434
    extra_hosts:
      - "host.docker.internal:host-gateway"
    stdin_open: true
    tty: true

Note: Ollama still runs on the host. The container talks to it via host.docker.internal.

Tips for better results

Tip	Why it matters
Keep notes reasonably structured	Clear headings improve retrieval
Use `chunk_size=512` as starting point	Too big = noisy context, too small = lost meaning
Re-run `ingest.py` after adding notes	The index is not updated automatically
Try different models	`llama3.2`, `mistral`, `phi3`, `gemma2` all work well
Increase `similarity_top_k`	More context when answers feel incomplete

Common issues

“No relevant information found”
→ Your question doesn’t match the notes well. Try rephrasing or check that the Markdown files were actually loaded.

Slow responses
→ Normal for local models. Use a smaller model (phi3, tinyllama) for speed, or a quantized version.

Ollama connection errors inside Docker
→ Make sure Ollama is running on the host and the extra_hosts line is present.

What you learned

The full RAG loop: load → chunk → embed → retrieve → generate
How to keep everything local and private
How to persist a vector index so you don’t re-embed every time
A clean foundation you can later extend (web UI, multiple collections, hybrid search, etc.)

This is one of the highest-leverage AI projects you can build: once it works with your own notes, you suddenly have a private research assistant that actually knows your material.