Skip to content

Ollama

Ollama allows you to run Large Language Models (LLMs) locally on your server.

Personal Recommendation: llama-swap

Personally, I use llama-swap instead of Ollama. It offers better performance, direct HuggingFace support, and finer control over the models. There are also indications that the Ollama developers are simply copying the work of the llama.cpp developers, and therefore features often appear later. However, Ollama is easier to get started with. I don't recommend it, though.

Installation

Add the following template to your docker-compose.yml and then run ei23 dc.

Template

  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: unless-stopped
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_ORIGINS=moz-extension://*'
    ports:
      - 11434:11434
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
    volumes:
      - ./volumes/ollama:/root/.ollama

Download Models

After startup, you can download models:

# Via terminal
docker exec ollama ollama pull llama3
docker exec ollama ollama pull mistral
docker exec ollama ollama pull codellama

# Or via API
curl http://localhost:11434/api/pull -d '{"name": "llama3"}'
Model Size RAM Description
llama3 4.7GB 8GB General tasks, good compromise
llama3:70b 40GB 64GB Very powerful, needs lots of RAM
mistral 4.1GB 8GB Good for European languages
codellama 3.8GB 8GB Optimized for code generation
phi3 2.2GB 4GB Compact, efficient model
gemma:2b 1.4GB 2GB Very lightweight, for weak hardware

GPU Recommendation

A GPU is recommended for fast inference. Uncomment the deploy section for NVIDIA GPUs.

Chat Interface

In combination with Open WebUI, you get a convenient browser interface.

Set the environment variable in Open WebUI:

environment:
  - OLLAMA_BASE_URL=http://ollama:11434

Ollama vs. llama-swap

Feature Ollama llama-swap
Simplicity ✅ Very easy ⚠️ Some configuration needed
Performance Good ✅ Better (llama.cpp)
HuggingFace ❌ Own format ✅ GGUF directly
Configuration Limited ✅ Detailed
Swapping No ✅ Automatic
Model Management ✅ ollama pull Manual download

Recommendation

  • Ollama: Ideal for beginners who want to start quickly
  • llama-swap: Ideal for advanced users who want maximum performance and control

API

Ollama provides a REST API:

# Send chat request
curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Why is the sky blue?"
}'

# List available models
curl http://localhost:11434/api/tags

The API is compatible with the OpenAI API interface and can be used with n8n, NodeRED, and other tools.

Notes

  • Models are stored in ./volumes/ollama/
  • The API is available at http://[IP]:11434
  • For Home Assistant integration, use the Ollama integration
  • Models can also be downloaded via the API

Further Information