dustynv/ollama repository overview

⁠ollama

• Ollama from https://github.com/ollama/ollama⁠ with CUDA enabled (found under /bin/ollama)
• Thanks to @remy415⁠ for getting Ollama working on Jetson and contributing the Dockerfile (PR #465⁠)

⁠Ollama Server

First, start the local Ollama server as a daemon in the background, either of these ways:

# models cached under jetson-containers/data
jetson-containers run --name ollama $(autotag ollama)

# models cached under your user's home directory
docker run --runtime nvidia -it -rm --network=host -v ~/ollama:/ollama -e OLLAMA_MODELS=/ollama dustynv/ollama:r36.2.0

You can then run the ollama client⁠ in the same container (or a different one if desired). The default docker run CMD of the ollama container is /start_ollama⁠, which starts the ollama server in the background and returns control to the user. The ollama server logs are saved under your mounted jetson-containers/data/logs directory for monitoring them outside the containers.

Setting the $OLLAMA_MODELS environment variable as shown above will change where ollama downloads the models to. By default, this is under your jetson-containers/data/models/ollama directory which is automatically mounted by jetson-containers run.

⁠Ollama Client

Start the Ollama CLI front-end with your desired model⁠ (for example: mistral 7b)

# if running inside the same container as launched above
/bin/ollama run mistral

# if launching a new container for the client in another terminal
jetson-containers run $(autotag ollama) /bin/ollama run mistral

Or you can run the client outside container by installing Ollama's binaries for arm64 (without CUDA, which only the server needs)

# download the latest ollama release for arm64 into /bin
sudo wget https://github.com/ollama/ollama/releases/download/$(git ls-remote --refs --sort="version:refname" --tags https://github.com/ollama/ollama | cut -d/ -f3- | sed 's/-rc.*//g' | tail -n1)/ollama-linux-arm64 -O /bin/ollama
sudo chmod +x /bin/ollama

# use the client like normal (outside container)
/bin/ollama run mistral

⁠Open WebUI

To run Open WebUI⁠ server for client browsers to connect to, use the open-webui container:

docker run -it --rm --network=host --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui:main

You can then navigate your browser to http://JETSON_IP:8080, and create a fake account to login (these credentials are only stored locally)

⁠Memory Usage

# automatically pull or build a compatible container image
jetson-containers run $(autotag ollama)

# or explicitly specify one of the container images above
jetson-containers run dustynv/ollama:r35.4.1

# or if using 'docker run' (specify image and mounts/ect)
sudo docker run --runtime nvidia -it --rm --network=host dustynv/ollama:r35.4.1

jetson-containers run -v /path/on/host:/path/in/container $(autotag ollama)

jetson-containers run $(autotag ollama) my_app --abc xyz

jetson-containers build ollama