Running LLMs Locally with Ollama

What Is Ollama

Ollama is a local LLM runtime developed by a San Francisco startup. It packages model downloading, quantization, and inference into a single command.

No Python scripts, no CUDA version management, no manual weight downloads. It handles everything.

Quick Start

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.2        # 3B, ~2GB
ollama run qwen2.5:7b      # better Chinese support

REST API

Ollama exposes an OpenAI-compatible API on localhost:11434:

curl http://localhost:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "Explain Transformer Self-Attention",
  "stream": false
}'

Custom Modelfile

FROM qwen2.5:7b
SYSTEM "You are an AI technical writing assistant."
PARAMETER temperature 0.7

ollama create ai-writer -f Modelfile
ollama run ai-writer

Recommended Models

Model	Size	Strength
llama3.2:3b	2GB	Lightweight general use
qwen2.5:7b	4.5GB	Best Chinese support
mistral:7b	4.1GB	Smooth English prose
deepseek-r1:7b	4.7GB	Strong reasoning

For performance optimization, see LLM Inference Optimization.