Ollama Local LLMs Run Private — 7B Guide That Delivers

Cloud AI bills add up fast. A team prototyping a support bot can burn hundreds of dollars before knowing if the idea works, and every prompt ships company text to someone else's servers. For regulated or offline work, that's a non-starter.

Local models flip the economics. Download once, run forever, pay nothing per token. Privacy comes free because prompts never leave your machine. You'll notice the difference the first time you work on a plane.

Ollama makes local practical. One CLI pulls from a huge model library, runs chat in the terminal, and serves an API your editor and apps already understand. No backend assembly required.

But local has ceilings. Your GPU decides which models fit, and defaults silently truncate long documents. This guide shows the setup that runs well on real hardware.

Why Local LLMs Win on Cost, Privacy, and Latency

Every cloud token costs money and travel time. Prototyping a RAG pipeline at a few million tokens can cost more than the laptop it runs on, and regulated industries cannot ship prompts off-site at all. Offline work — planes, labs, air-gapped clients — rules cloud out entirely.

Local inference removes all three constraints at once. The model file downloads once, inference costs nothing marginal, prompts stay on disk, and latency is GPU-bound instead of network-bound. A 7B coder answers tab-completions in under a second on a midrange GPU.

The ceiling is hardware honesty: your VRAM picks your model class. That constraint is also a virtue — it forces you to choose deliberately instead of defaulting to the biggest cloud model for every trivial task.

Install Ollama and Run Your First Model in Minutes

Installation is a download plus one command. ollama serve starts the daemon, ollama pull fetches a model, ollama run opens chat, and the same daemon answers API calls on port 11434. Editors like Continue.dev and Cursor connect by pointing their base URL at localhost.

Start with a 7B coder: it balances quality (~72% on coding tasks), size (~4.7GB), and speed on consumer GPUs. Verify the API with curl before wiring up any integration — if curl works, your editor config is the problem, not Ollama.

# Install: https://ollama.com/download (or brew install ollama)
ollama serve &
ollama pull qwen3-coder:7b
ollama pull llama4:8b
ollama list
ollama run qwen3-coder:7b "Explain this Python traceback: IndexError"
# API check — OpenAI-compatible on :11434
curl http://localhost:11434/api/chat -d '{
  "model": "qwen3-coder:7b",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}' | python3 -m json.tool

Pick the Right Model — VRAM Budgets and Benchmarks

Model size is a VRAM budget. A 7B quantized model needs roughly 4.7GB, 13B needs 8.5GB, and 30B-class mixture-of-experts models want a 24GB card. Throughput follows the same curve: 150 tokens/sec on a 4090, 80 on a 4070, 20 on an M3 MacBook, 3 on CPU alone.

Quantization (Q4_K_M is the default sweet spot) shrinks weights 70-85% with minor quality loss. Smaller quants run faster but reason worse — match the quant to the task, not to impatience.

Rule of thumb: run the largest model that fits with 20% VRAM headroom. Headroom is what keeps two-model workflows (chat plus embeddings) from spilling to CPU and crawling.

Modelfiles — Reproducible Personalities in Version Control

A Modelfile is a Dockerfile for models. FROM pins the base (never :latest in production), PARAMETER tunes sampling and context, SYSTEM sets the persona, and TEMPLATE controls the chat markup. ollama create builds it into a named, shareable model.

Temperature is the highest-leverage parameter: 0.1-0.3 for code review and facts, 0.7+ only for brainstorming. num_ctx sets the memory — 8k handles code reviews, 32k handles long docs, and bigger windows cost VRAM whether you fill them or not.

Version your Modelfiles in git next to the app. A code reviewer persona with pinned temperature behaves identically on every teammate's machine, which is the whole point.

FROM qwen3-coder:7b
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
PARAMETER stop "```"
SYSTEM "You are a senior Python reviewer. Cite file and line. Never invent APIs."
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>{{ end }}
{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>{{ end }}
<|im_start|>assistant
"""
# Build and run:
#   ollama create reviewer -f Modelfile
#   ollama run reviewer "Review this diff..."
#   ollama show --modelfile reviewer

Wire Ollama into Editors and Apps

Editors connect over the OpenAI-compatible endpoint, so Continue.dev needs only a base URL change to localhost:11434 and a model name. Tab completions at 80 tokens/sec feel instant; CPU-only at 3/sec feels broken — hardware decides the UX.

Keep context windows small for completions (1-2k) and large for chat review (8-32k). Small contexts answer faster and flicker less; large ones remember the whole file. Set debounce around 500ms so completions stop jumping.

For apps, the Python and JS SDKs accept a host override pointing at localhost. That means RAG prototypes, eval harnesses, and tool-calling agents develop locally for free and deploy against cloud endpoints later with minimal code change.

{
  "models": [{
    "title": "Ollama coder",
    "provider": "ollama",
    "model": "qwen3-coder:7b",
    "apiBase": "http://localhost:11434",
    "requestOptions": { "contextLength": 8192 }
  }],
  "tabAutocompleteModel": {
    "title": "Ollama coder",
    "provider": "ollama",
    "model": "qwen3-coder:7b"
  }
}

Ollama vs the Alternatives — Where Local Stops

Ollama is not the only local runtime — LM Studio offers a friendlier GUI, text-generation-webui offers extensions, and LocalAI offers broader API emulation. Ollama wins the terminal-first and server cases with the largest model library and simplest automation.

The honest boundary is frontier quality. Local 30B-class models handle drafting, review, summarization, and agents well; they do not match the largest cloud models on hard reasoning. Use local for volume and privacy, cloud for peak difficulty.

A pragmatic team runs both: local 7B for completions and drafts (free, instant, private), cloud frontier for the 5% of tasks that need it. That split cuts AI spend dramatically without capping capability.