amperecomputingai/ollama repository overview

⁠Ampere® optimized Ollama

Ampere® optimized build of Ollama⁠.

This Docker image can be run on bare metal Ampere® CPUs and Ampere® based VMs available in the cloud.

For best results we recommend using models in our custom quantization formats available on Ampere's HuggingFace page⁠.

⁠Running

This image is a drop-in replacement for the original Ollama image. It includes performance improvements while maintaining full compatibility with the original's features and behavior.

Start the server:

docker run --privileged=true --name ollama -p 11434:11434 amperecomputingai/ollama:latest

In a separate shell:

docker exec -it ollama bash
ollama run hf.co/AmpereComputing/llama-3.2-3b-instruct-gguf:Llama-3.2-3B-Instruct-Q8R16.gguf

or using the REST API:

curl http://localhost:11434/api/pull -d '{
  "model": "hf.co/AmpereComputing/llama-3.2-3b-instruct-gguf:Llama-3.2-3B-Instruct-Q8R16.gguf"
}'

curl http://localhost:11434/api/generate -d '{
  "model": "hf.co/AmpereComputing/llama-3.2-3b-instruct-gguf:Llama-3.2-3B-Instruct-Q8R16.gguf",
  "prompt":"Why is the sky blue?"
}'

⁠Quantization

Ampere® optimized Ollama provides support for two new quantization methods, Q4_K_4 and Q8R16, offering model size and perplexity similar to Q4_K and Q8_0, respectively, but performing up to 1.5-2x faster on inference. You can download them from Ampere's HuggingFace page⁠.

You can also quantize the model yourself using Ampere® optimized llama.cpp⁠ and following the instructions⁠.

Afterwards, you can use it with Ollama by creating a Modelfile:

FROM ./path/to/your/model.gguf

and running

ollama create my_model -f Modelfile
ollama run my_model

⁠Support

Please contact us at [email protected]⁠

⁠LEGAL NOTICE

By accessing, downloading or using this software and any required dependent software (the “Ampere AI Software”), you agree to the terms and conditions of the software license agreements for the Ampere AI Software, which may also include notices, disclaimers, or license terms for third party software included with the Ampere AI Software. Please refer to the Ampere AI Software EULA v1.6⁠ or other similarly-named text file for additional details.