Build a Local-First AI Coding Assistant (Privacy-Preserving, Fast, and Actually Useful)

I got tired of paying for a hosted coding assistant to review code that, per the contract we sign with clients, isn't supposed to leave our network. So I spent a weekend building a local setup instead: a quantized coding model running on my own machine, served through an OpenAI-compatible API, wired into my editor. It's not as good as Claude or GPT-4-class models for the hard problems. But for autocomplete, boilerplate, and "explain this function" it's fast, free after the hardware cost, and nothing ever hits a third-party server. Here's how it's put together and where it falls short.

The pieces: runtime, model, and API shim

There are really only three decisions to make, and they're mostly independent of each other.

Runtime. Ollama is the path of least resistance — it wraps llama.cpp under the hood, handles model downloads and quantization variants for you, and exposes a local HTTP server out of the box. If you want more control over quantization, batching, or GPU offload layers, llama.cpp directly (or text-generation-webui / LM Studio if you want a GUI) gives you that, at the cost of managing GGUF files yourself. I started with Ollama because I didn't want to spend my weekend debugging a build.

Model. For coding specifically, the realistic local options right now are the Qwen2.5-Coder family (1.5B through 32B), DeepSeek-Coder / DeepSeek-Coder-V2, StarCoder2, and CodeLlama. CodeLlama is the oldest of the bunch and noticeably behind the other three on anything but simple completions — I only mention it because it still gets recommended in a lot of outdated tutorials. Qwen2.5-Coder-7B is the sweet spot for a single consumer GPU: it fits comfortably in 8GB of VRAM at 4-bit quantization and is genuinely usable for completions and small refactors. If you have 24GB+ of VRAM, the 32B variant closes a lot of the gap with hosted models on everyday tasks.

API shim. This is the part that makes it actually pluggable into your existing tools: Ollama (and llama.cpp's server binary) can expose an /v1/chat/completions endpoint that mimics OpenAI's API shape. That means anything that already speaks the OpenAI SDK — Continue.dev, most VS Code AI extensions, your own scripts — can point at http://localhost:11434/v1 instead of api.openai.com and mostly just work.

Getting a model running

# install Ollama, then pull a quantized coding model
ollama pull qwen2.5-coder:7b

# sanity check it responds
ollama run qwen2.5-coder:7b "write a python function that reverses a linked list"

By default Ollama serves an OpenAI-compatible endpoint on port 11434. You can hit it directly to confirm the shape matches what your tools expect:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-coder:7b",
    "messages": [{"role": "user", "content": "explain what this regex does: ^(?=.*[A-Z]).{8,}$"}]
  }'

Wiring it into the editor

If you use Continue for VS Code or JetBrains, the config is just pointing it at the local endpoint instead of a hosted provider:

{
  "models": [
    {
      "title": "Local Qwen Coder",
      "provider": "openai",
      "model": "qwen2.5-coder:7b",
      "apiBase": "http://localhost:11434/v1",
      "apiKey": "not-needed"
    }
  ]
}

If you're rolling your own extension or script instead, any OpenAI SDK client works unmodified — you just override the base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    messages=[{"role": "user", "content": "add type hints to this function:\n\n" + code}],
)
print(response.choices[0].message.content)

That's really the whole trick. Once the local server speaks the same protocol as a hosted one, "local-first" stops being a special integration and becomes a config change.

The trade-offs nobody puts in the marketing copy

Quantization costs quality, unevenly. Q4_K_M (roughly 4-bit) is the common default because it's the best size/quality compromise, but the degradation isn't uniform across tasks — it hurts multi-step reasoning and long-range dependency tracking more than it hurts pattern-matched completions. If your GPU can fit Q6 or Q8 instead of Q4, do it; the jump from 4-bit to 8-bit is usually more noticeable on code than the equivalent jump on prose.

Context windows are a real constraint, not a spec-sheet number. A model might advertise a 32K or 128K context window, but running that context locally means holding a KV cache proportional to it in VRAM, on top of the model weights. In practice, on consumer hardware, you'll often clamp the usable context down to 4K–8K tokens just to keep memory sane — which matters a lot if you were hoping to feed it a whole file plus several related files for context, the way you can with a hosted model that has the memory budget of a data center behind it.

Latency is genuinely better, until it isn't. Local inference has no network round-trip, so for short completions it feels instant compared to a hosted API call. But throughput on a single consumer GPU is a fraction of what a hosted provider gets from batching across many requests on server-class hardware, so longer generations (a full function body, a test suite) can actually take longer locally than the equivalent hosted call.

It's still behind on hard problems. For boilerplate, straightforward refactors, and explaining unfamiliar code, a well-chosen 7B–14B local coding model gets you most of the way there. For genuinely hard problems — subtle concurrency bugs, unfamiliar framework internals, anything that needs broad world knowledge alongside code — the gap to frontier hosted models is still real, and I don't think quantized local models close it any time soon. Pretending otherwise doesn't help anyone evaluating whether this is worth setting up.

Was it worth the weekend

For the specific case that started this — not sending client code to a third party — yes, unambiguously. For raw coding ability, no, a hosted frontier model is still better, and I still reach for one when a problem is actually hard. What I ended up with is a fast, private first pass that handles the boring 80% of prompts, and I fall back to a hosted model for the rest. That split is honestly the realistic pitch for local-first coding assistants right now: not a replacement, a filter.