Skip to content

2.2.1 Backend: Ollama

Jump to bottom
av edited this page Jun 14, 2026 · 19 revisions

Ollama

Handle: ollama
URL: http://localhost:33821

Official Ollama logo

Ergonomic wrapper around llama.cpp with plenty of QoL features.

Starting

Ollama is one of the default services, so you don't need to specify anything special to start it.

# Will pull images and start the service
harbor up

# [Optional] monitor the logs
harbor logs ollama
  • Harbor's ollama instance is configured to pull default models (see configuration below)
  • By default, Ollama cache is stored in ~/.ollama directory, but you can configure it to match your host's Olama cache location (see HARBOR_OLLAMA_CACHE in configuration below)

Integrations

ollama (API on port 11434) is automatically wired into dozens of other services via cross-compose files (services/compose.x.*.ollama.yml). When you run harbor up with ollama plus a consumer, Harbor injects the endpoint via HARBOR_OLLAMA_INTERNAL_URL (default http://ollama:11434) so consumers use it with no manual setup. Many also consume the OpenAI-compatible API at /v1 (with key sk-ollama).

Frontends and tools that auto-configure for ollama

  • webui — mounts config.ollama.json and sets OLLAMA_BASE_URL env (via compose.x.webui.ollama.yml and compose.x.ollama.webui.yml).
  • chatui — mounts chatui.ollama.yml defining the model endpoint (OpenAI /v1, driven by HARBOR_CHATUI_OLLAMA_MODEL).
  • anythingllm — sets LLM_PROVIDER=ollama, OLLAMA_BASE_PATH, and embedding paths.
  • aider — mounts aider.ollama.yml (OpenAI compat /v1 + HARBOR_AIDER_MODEL).
  • boost — injects HARBOR_BOOST_OPENAI_URL_OLLAMA (pointing to /v1) and key.
  • cmdh, hermes, opint, optillm, plandex, sillytavern, astrbot, mindsdb, lobechat, morphic, khoj, perplexica, fabric, gptme, karakeep, ldr, activepieces, agentzero, browseruse, drawio, kotaemon, npcsh, mi, openterminal, openhands, opennotebook, parllama, photoprism, presenton, promptfoo, raglite, sim, txtairag and many others — receive HARBOR_OLLAMA_INTERNAL_URL (or derived) via environment injection; several declare depends_on: ollama: {condition: service_healthy}.
  • bifrost — bootstrap sidecar (compose.x.bifrost.ollama.yml) registers ollama as a provider using HARBOR_BIFROST_OLLAMA_MODELS.
  • hollama — lightweight dedicated Ollama chat UI (traefik-exposed when enabled).
  • open-design — uses a dedicated socat proxy sidecar so the service sees ollama on localhost:11434.

Reverse proxy / external access

  • With traefik present: compose.x.traefik.ollama.yml (and compose.x.traefik.hollama.yml) add labels exposing the service at https://ollama.${HARBOR_TRAEFIK_DOMAIN} and https://hollama.${HARBOR_TRAEFIK_DOMAIN} (websecure).

Hardware acceleration (GPU / accelerator)

  • compose.x.ollama.nvidia.yml adds full NVIDIA GPU reservations.
  • compose.x.ollama.rocm.yml adds AMD ROCm devices (/dev/kfd, /dev/dri) and switches both ollama and ollama-init containers to ${HARBOR_OLLAMA_IMAGE}:rocm.
  • compose.x.ollama.cdi.yml uses CDI for NVIDIA GPU passthrough.
  • Enable ROCm via Harbor capabilities/profile; the ROCm overlay uses the :rocm tag on ${HARBOR_OLLAMA_IMAGE} (default ollama/ollama:rocm). For NVIDIA, use the nvidia/cdi overlays with harbor config set ollama.version <tag> as needed.

Host / volume interactions

  • Ollama models: ${HARBOR_OLLAMA_CACHE} (defaults to ~/.ollama on host) mounted at /root/.ollama. All pulls (harbor ollama pull, UI, init) store here.
  • Hugging Face cache: ${HARBOR_HF_CACHE} mounted at /root/.cache/huggingface — used for hf.co/... pulls and GGUF imports documented in the guide; shared across Harbor services.
  • Custom Modelfiles: ./services/ollama/modelfiles:/modelfiles — drop your .Modelfile sources here for easy harbor ollama create -f /modelfiles/... access.
  • ollama-init sidecar (defined in base compose.ollama.yml): depends on ollama healthy, runs ollama pull for every model in the comma-separated HARBOR_OLLAMA_DEFAULT_MODELS (default nomic-embed-text:latest), then stays healthy so harbor up with --wait succeeds even during long first-time pulls. The sidecar re-uses the same cache volume.
  • Additional legacy caches (HARBOR_LLAMACPP_CACHE, HARBOR_VLLM_CACHE) are also mounted for cross-backend convenience.
  • The main service has a TCP healthcheck; consumers can safely depends_on it with condition: service_healthy.

Special behaviors

  • OpenAI compatibility layer: The /v1 path on the internal URL is the preferred integration point for any tool built against the OpenAI SDK. Harbor wires many consumers this way with the conventional sk-ollama key.
  • Pointing everything at external Ollama: Set HARBOR_OLLAMA_INTERNAL_URL (via harbor config set ollama.internal_url http://host.docker.internal:11434 or your LAN IP) and restart consumers — every auto-wired service will follow.
  • Default models are pulled exactly once on first startup via the init sidecar (controlled by HARBOR_OLLAMA_DEFAULT_MODELS).

All wiring is declarative in the cross-compose files. After editing any HARBOR_OLLAMA_* value with harbor config set, run harbor config update and harbor restart (or harbor up) for the relevant services.

Models

You can discover new models via Ollama's model library.

Management of the models is possible right from the Open WebUI Admin Settings. The models are stored in the global ollama cache on your local machine.

Alternatively, you can use ollama CLI itself.

# Show the list of available models
harbor ollama list
harbor ollama ls

# Pull a new model
harbor ollama pull phi4

You can also pull models directly from the HuggingFace Hub.

# [Optional] will open HF model search with "llama gguf" query
harbor hf find llama gguf

# Pull the model
# "hf.co" is a special prefix telling Ollama to pull from HuggingFace
# "unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF" is the HuggingFace repository name
# "Q8_0" points to the "DeepSeek-R1-Distill-Llama-8B-Q8_0.gguf" file in the repository
harbor ollama pull hf.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF:Q8_0

TroubleShooting

If you encounter any issues with ollama service, you can check the logs with harbor logs ollama.

# Check the logs
harbor logs ollama

If you need to restart the service, you can do it with harbor restart ollama.

# Restart the service
harbor restart ollama

You'll find many more examples and scenarios covered in the troubleshooting guide.

Ollama CLI

You can use a full ollama CLI, when ollama service is running.

# Ollama service should be running to access the cli
harbor ollama --help

# See the environment variables
# supported by ollama service
harbor ollama serve --help

# Access Ollama CLI commands
harbor ollama version

Configuration

See environment variables configuration guide.

# Env variable reference
harbor ollama serve --help

In addition to that, some high-level configurations are available via harbor config

# See all harbor config properties for ollama
harbor config ls | grep OLLAMA

# Configure ollama version, accepts a docker tag
harbor config set ollama.version 0.3.7-rc5-rocm

# Available configuration options
# Location of the ollama own cache, either global
# or relative to $(harbor home)
HARBOR_OLLAMA_CACHE            ~/.ollama

# The port on the host machine where the ollama service
# will be available
HARBOR_OLLAMA_HOST_PORT        33821

# Docker image repository (without tag); paired with HARBOR_OLLAMA_VERSION
HARBOR_OLLAMA_IMAGE            ollama/ollama

# The Docker image tag to use
HARBOR_OLLAMA_VERSION          latest

# This URL will be given to the services connected to Ollama
# you can substitute with your own custom URL to switch
# Harbor to your own Ollama deployment
HARBOR_OLLAMA_INTERNAL_URL     http://ollama:11434

# A comma-separated list of models to pull/update on startup
# The default value is also used in default configs for dependent services
HARBOR_OLLAMA_DEFAULT_MODELS   nomic-embed-text:latest

# Maps to the "OLLAMA_CONTEXT_LENGTH" environment variable
# for the container and sets global default ctx length for all models.
# Don't use "harbor config" to set this as it won't be propagated to the env var.
HARBOR_OLLAMA_CONTEXT_LENGTH   4096

Environment Variables

Follow Harbor's environment variables guide to set arbitrary variables for ollama service.

API

Retrieve the endpoint for ollama service with:

# For the host machine
harbor url ollama

# For the LAN
harbor url -a ollama

# For the Docker network
harbor url -i ollama

Additionally, you can find a small HTTP playbook in the services/http-catalog/ folder.

Custom Modelfiles

You can create custom modelfiles to run models with specific parameters. Harbor will run Ollama CLI within the same folder you called it from, so all the files will be available.

# 1. Create a new modelfile at a convenient location
touch mymodel.Modelfile

# 2. Edit the modelfile with your favorite editor
open mymodel.Modelfile

# 3. Import the model into Ollama
harbor ollama create -f mymodel.Modelfile mymodel

The most common use-case is to override default Ollama parameters with custom values, for example to increase the default context length.

You can also source your modelfile from the existing one, to use as a template.

harbor ollama show modelname:latest --modelfile > mymodel.modelfile

Default context length

Starting from Ollama v0.5.13 you can set a global default context length for all models. To do so with Harbor, you have two options:

# Option 1. Use CLI alias
# - Get current config value
harbor ollama ctx
# - Set the new value (4096 in this case)
harbor ollama ctx 4096

# Option 2. Use "harbor env"
# - Get the current value
harbor env ollama OLLAMA_CONTEXT_LENGTH
# - Set the new value (4096 in this case)
harbor env ollama OLLAMA_CONTEXT_LENGTH 4096

Note that harbor ollama ctx is synchronised to the env, but not vice versa.

Switching to external Ollama

It's possible to switch all related service in Harbor to use an external Ollama instance, to do so, you need to change the HARBOR_OLLAMA_INTERNAL_URL configuration.

Note

The URL is relative to the Docker network, use host.docker.internal or 172.17.0.1 instead of localhost.

# Set the custom Ollama URL
harbor config set ollama.internal_url http://172.17.0.1:11434

Changing the cache location

By default, Ollama stores the models in the ~/.ollama directory. You can change the location by setting the HARBOR_OLLAMA_CACHE configuration.

# Set the custom cache location
harbor config set ollama.cache /path/to/custom/cache

The path must be either absolute or relative to the harbor home directory.

Custom Docker image

Harbor resolves the Ollama container image as ${HARBOR_OLLAMA_IMAGE}:${HARBOR_OLLAMA_VERSION} (defaults to ollama/ollama:latest). Point it at a custom image when you need extra OS packages, corporate CA bundles, or a private registry mirror:

# Use a custom image (build and push yours first, or use a local tag)
harbor config set ollama.image myregistry/ollama-custom
harbor config set ollama.version latest

harbor pull ollama
harbor restart ollama

services/ollama/Dockerfile.custom is a minimal example that extends the upstream image and installs an extra CA certificate. Build from that directory, push to your registry, then set ollama.image to the result.

Importing models from HuggingFace

When compatible with Ollama's HF integration:

harbor ollama pull hf.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF:Q8_0

Tip

You can then copy the model to have a more convenient ID:
harbor ollama cp hf.co/unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF:Q8_0 r1-8b

When not compatible, via direct import from file:

# 1. Download the model
harbor hf download flowaicom/Flow-Judge-v0.1-GGUF

# 2. Locate the gguf file
# The gguf file is located in the model directory
harbor find Flow-Judge-v0.1 | grep .gguf
# /home/user/.cache/huggingface/hub/models--flowaicom--Flow-Judge-v0.1-GGUF/snapshots/3ca...575/flow-judge-v0.1-Q4_K_M.gguf

# 3. Translate the path
# Harbor mounts HF cache to Ollama service
# /home/user/.cache/huggingface -> /root/.cache/huggingface
# The path becomes:
# /root/.cache/huggingface/hub/models--flowaicom--Flow-Judge-v0.1-GGUF/snapshots/3ca...575/flow-judge-v0.1-Q4_K_M.gguf

# 4. Create a modelfile
# You can use any convenient folder to store modelfiles
# By default, Harbor has a directory for modelfiles: ollama/modelfiles
# Below are few _options_ on quickly accessing the directory
harbor vscode       # Open Harbor workspace in VS Code and go from there
open $(harbor home) # Open Harbor workspace in default file manager
open $(harbor home)/services/ollama/modelfiles # This is the directory for modelfiles
code $(harbor home)/services/ollama/modelfiles # Open the directory in VS Code

# 5. Sample modelfile contents
# TIP: Use original base modelfile as a reference:
#      harbor ollama show --modelfile <model name>
# Save as "<your name>.Modelfile" in the modelfiles directory
FROM /root/.cache/huggingface/hub/models--flowaicom--Flow-Judge-v0.1-GGUF/snapshots/3ca...575/flow-judge-v0.1-Q4_K_M.gguf

# 6. Create the model
# 6.1. From Harbor's modelfiles directory
harbor ollama create -f /modelfiles/<your name>.Modelfile <your name>
# 6.2. From current directory
harbor ollama create -f ./<your name>.Modelfile <your name>
# Successful output example
# 13:27:37 [INFO] Service ollama is running. Executing command...
# transferring model data 100%
# using existing layer sha256:939...815
# creating new layer sha256:aaa...169
# writing manifest
# success

# 7. Check the model
harbor ollama run <your name>

# 8. To upload to ollama.com, follow official tutorial
# on sharing the models:
# https://github.com/ollama/ollama/blob/main/docs/import.md#sharing-your-model-on-ollamacom

Home | CLI Reference | Services | Adding New Service | Compatibility

Clone this wiki locally