Self-Hosted RAG in Laravel on a 4GB Box

Most AI tutorials assume a cloud API and a credit card. But a lot of real teams — clinics, schools, agencies handling sensitive records — simply cannot send their data to a third-party API. And many more just don't want an unpredictable per-token bill.

So I built the opposite: a private, self-hosted Retrieval-Augmented Generation (RAG) app in Laravel that runs a small local model on a modest server (4 GB RAM, 2 vCPU). No public API. The data never leaves the box.

Why a small local model is enough

The instinct is that a 1B–3B model is too weak to be useful. For open-ended chat, maybe. But RAG changes the job. Retrieval does the remembering; the model only has to phrase an answer from passages you already handed it. That's the easiest thing a small model can do — grounded rewriting, not world knowledge.

So the architecture leans on retrieval quality, and the model stays small enough to run on cheap, private hardware.

The stack

• Laravel for the app and orchestration
• SQLite for storage — zero config, perfectly fine for a single tenant
• Ollama running a small chat model and an embedding model locally
• Plain-PHP cosine similarity for retrieval — no vector database

Vectors in SQLite, cosine in PHP

For one practice the corpus is a few hundred chunks. You don't need pgvector or a dedicated vector store. Store each embedding as JSON and compute cosine similarity in PHP:

public static function cosine(array $a, array $b): float
{
    $dot = 0.0; $na = 0.0; $nb = 0.0;
    for ($i = 0; $i < count($a); $i++) {
        $dot += $a[$i] * $b[$i];
        $na  += $a[$i] * $a[$i];
        $nb  += $b[$i] * $b[$i];
    }
    return $dot / (sqrt($na) * sqrt($nb));
}

Embed the query, score every chunk, take the top few, feed them to the model as context. That's the entire retrieval layer.

Talking to Ollama from Laravel

Ollama exposes a local HTTP API, so from Laravel it's just a call to localhost — for both embeddings and chat:

$res = Http::post('http://127.0.0.1:11434/api/chat', [
    'model'    => 'llama3.2:3b',
    'stream'   => false,
    'messages' => [
        ['role' => 'system', 'content' => $system],
        ['role' => 'user',   'content' => $grounding . $question],
    ],
]);

Because it's localhost, nothing crosses the network. For privacy-bound users, that single fact is the whole product.

Making it fit 4 GB / 2 vCPU

• Chat model: a 1.5B–3B model at 4-bit (e.g. qwen2.5:1.5b or llama3.2:3b). Expect roughly 5–12 tokens/sec on CPU, so stream output and keep responses short.
• Embeddings: a small model like nomic-embed-text is fast on CPU and tiny.
• Concurrency: one or two users at a time — fine for a solo practice or small team.

The trade-off is real: a small local model won't match a frontier cloud model on eloquence. But for grounded drafting — turning shorthand into a structured note, or summarizing a term of sessions into a progress report — it's genuinely good, and the privacy plus zero per-token cost are the entire point.

Keep a human in the loop

For anything clinical or high-stakes, the output is a draft. The professional reviews and edits every result; the AI removes the blank-page tax, not the judgment. Build that expectation into the UI, not just the docs.

The takeaway

You don't need a GPU cluster or a cloud contract to ship useful AI. With Laravel, SQLite, Ollama, and a few hundred lines, a private RAG assistant runs on a server that costs about the price of lunch — and the data never leaves it. For a lot of teams, that isn't a compromise. It's the only version they can actually use.