Use a Local Coding Model with Ollama in Your IDE

Cloud coding assistants send every buffer you touch to a third-party API. A local model removes that dependency: the code stays on your hardware and completions keep working without a network connection. This tutorial installs Ollama, pulls a coding model, and wires it into VS Code, JetBrains IDEs and Zed.

What is Ollama?

Ollama is a local inference server that downloads quantized language models, loads them onto CPU or GPU, and exposes them over an HTTP API on 127.0.0.1:11434, which IDE extensions such as Continue consume as a drop-in backend.

Ollama handles model download, GGUF quantization selection, GPU offload and memory management. You interact with it through the ollama CLI or through its REST endpoints, primarily /api/generate for raw completions and /api/chat for multi-turn conversations. IDE plugins never talk to the model file directly; they talk to that HTTP API.

Prerequisites

• A Linux host (Ubuntu 22.04 or newer, Debian 12, RHEL 9) or a macOS machine with Apple Silicon
• At least 16 GB of system RAM, plus roughly 10 GB of free disk space per model
• An NVIDIA GPU with 8 GB VRAM or more for usable latency, with a driver of version 535 or newer
• An IDE: VS Code 1.85+, a 2024.1+ JetBrains IDE, or Zed

If your workstation has no discrete GPU, run Ollama on a GPU instance sized for LLM inference and connect your IDE to it over the network. CPU-only inference works for models up to about 3B parameters, but tab completion latency becomes noticeable above that.

Install Ollama

Install Ollama with the official script. On Linux it registers a systemd unit named ollama and starts it immediately:

$ curl -fsSL https://ollama.com/install.sh | sh
$ ollama --version
$ systemctl status ollama --no-pager

The service listens on 127.0.0.1:11434 by default. Confirm that the GPU was detected, otherwise every request falls back to CPU without an error message:

$ nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
$ journalctl -u ollama --no-pager | grep -i "inference compute"

The log line reports the detected accelerator and its available VRAM. If it names no device, install the vendor driver before continuing.

Which coding model should you pick?

Pick the largest coding model whose quantized weights fit into VRAM with about 2 GB of headroom, because a model that spills into system RAM drops from tens of tokens per second to single digits.

The -base suffix matters. Base variants are trained with a fill-in-the-middle objective and are the correct choice for inline completion. Instruction-tuned variants (no suffix) are the correct choice for chat and refactoring. Using the wrong one produces the classic failure mode where autocomplete inserts explanatory prose into your source file.

Pull and test the model

Pull one instruction model for chat and one small base model for completion. Both stay resident independently, so budget VRAM for the sum:

$ ollama pull qwen2.5-coder:7b
$ ollama pull qwen2.5-coder:1.5b-base
$ ollama list

Verify the HTTP API answers before touching any IDE configuration. This isolates server problems from plugin problems:

$ curl -s http://localhost:11434/api/generate -d '{"model":"qwen2.5-coder:7b","prompt":"Write a bash one-liner that lists open TCP ports.","stream":false}' | jq -r .response

A JSON response with a non-empty response field confirms the server, the model and the GPU offload path all work.

How does autocomplete differ from chat?

Autocomplete sends a fill-in-the-middle prompt containing the code before and after the cursor to a base model and must return within roughly 500 ms, while chat sends a conversation to an instruction-tuned model and tolerates multi-second latency.

That difference drives the entire configuration: two models, two roles, two size classes. The following flow shows what the IDE extension actually does with each keystroke and each chat message.

graph TD
  A["IDE extension (Continue)"] --> B{"Request type"}
  B -->|Tab completion| C["POST /api/generate with FIM prompt"]
  B -->|Chat or inline edit| D["POST /api/chat"]
  C --> E["Ollama server on 127.0.0.1:11434"]
  D --> E
  E --> F["Model weights in VRAM"]
  F --> G["Unload after OLLAMA_KEEP_ALIVE expires"]

Because the small base model is hit on nearly every keystroke, keep it loaded permanently. The chat model can be evicted between sessions without much cost.

Connect the IDE

VS Code with Continue

Install the Continue.continue extension from the marketplace. Continue 1.0 and newer reads ~/.continue/config.yaml; releases before 1.0 used config.json with a different schema. Replace the file contents with:

name: local-assistant
version: 1.0.0
schema: v1
models:
  - name: qwen-coder-chat
    provider: ollama
    model: qwen2.5-coder:7b
    apiBase: http://localhost:11434
    roles:
      - chat
      - edit
      - apply
  - name: qwen-coder-autocomplete
    provider: ollama
    model: qwen2.5-coder:1.5b-base
    apiBase: http://localhost:11434
    roles:
      - autocomplete

Continue reloads the file on save. Open a source file and pause after a partial line: a grey inline suggestion appears within about a second.

JetBrains IDEs

Continue ships a JetBrains plugin for IntelliJ IDEA, PyCharm, GoLand and the rest of the 2024.1+ line. Install it from Settings → Plugins → Marketplace, then restart. It reads the same ~/.continue/config.yaml, so a workstation configured for VS Code needs no additional setup.

Zed

Zed supports Ollama natively. Open the settings file with cmd-, (macOS) or ctrl-, (Linux) and add the provider:

{
  "language_models": {
    "ollama": {
      "api_url": "http://localhost:11434"
    }
  }
}

Zed queries /api/tags and lists every pulled model in the assistant panel's model picker.

Run Ollama on a remote host

A laptop GPU is often the wrong place for a 32B model. Move the server to a machine with more VRAM and keep only the IDE local. Bind the service to all interfaces with a systemd drop-in:

$ sudo systemctl edit ollama

Add the following block, then reload and restart:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_MAX_LOADED_MODELS=2"

$ sudo systemctl daemon-reload
$ sudo systemctl restart ollama

Ollama has no authentication layer. Do not expose port 11434 to the public internet. Reach it through an SSH tunnel instead, which makes the remote server appear on localhost and keeps every IDE config above unchanged:

$ ssh -N -L 11434:127.0.0.1:11434 <user>@<gpu-host>

For a shared team setup, a small scalable Linux VM in front of the GPU host works well as a jump host and as the place to terminate the tunnels, so the inference machine keeps a firewall that only accepts traffic from that one address.

Verify the setup

With the IDE open and idle, check which models are resident and where they run:

$ ollama ps

Expected output while autocomplete is active:

NAME                        ID              SIZE      PROCESSOR    UNTIL
qwen2.5-coder:1.5b-base     8a1b2c3d4e5f    1.8 GB    100% GPU     29 minutes from now
qwen2.5-coder:7b            f5e4d3c2b1a0    5.6 GB    100% GPU     29 minutes from now

The PROCESSOR column is the one that matters. 100% GPU means full offload. Anything reading 47%/53% CPU/GPU means the model did not fit and latency will be poor.

Troubleshooting

Completions take several seconds. The model is partly on CPU. Check the PROCESSOR column in ollama ps, then either drop to a smaller quantization, reduce OLLAMA_MAX_LOADED_MODELS to 1, or lower the context with OLLAMA_CONTEXT_LENGTH=4096 in the systemd drop-in. Context memory is allocated per loaded model and grows with the window size.

Autocomplete inserts explanations or Markdown fences. An instruction-tuned model is assigned to the autocomplete role. Only base models with fill-in-the-middle training belong there. Switch that entry to qwen2.5-coder:1.5b-base and reload the config.

The IDE reports connection refused. Ollama binds to loopback unless OLLAMA_HOST is set. Confirm the listener with ss -tlnp | grep 11434. If you use a remote host, verify the SSH tunnel is still up, since a dropped tunnel produces exactly this error while ollama ps on the server still looks healthy.

Next steps

The setup above covers completion and chat. From here, add repository context by enabling Continue's codebase indexing, which embeds your project locally with nomic-embed-text and keeps the index on disk. Raise OLLAMA_KEEP_ALIVE if the first request after a break feels slow, and re-measure with ollama ps after every model or quantization change rather than relying on subjective latency.

Read next

• Calling the Ollama API from Your Own Applications
• Connecting Ollama to MCP Servers: Tools for Local Models
• Expose Ollama on the Network: Port, Bind Address and Hardening
• Install Ollama on Ubuntu 24.04 and Run It as a Service
• Managing Ollama Models: Updates, Cleanup and Disk Space
• Ollama vs LM Studio vs llama.cpp: Which Tool for What?
• Run Ollama in a Docker Container
• Set Up Open WebUI as a Front End for Ollama
• What Is Ollama? How It Works, Models and Use Cases