defnotgustavom/llama.go repository overview

First, obtain and convert original LLaMA models on your own, or just download ready-to-rock ones:

LLaMA-7B: llama-7b-fp32.bin⁠

LLaMA-13B: llama-13b-fp32.bin⁠

Both models store FP32 weights, so you'll needs at least 32Gb of RAM (not VRAM or GPU RAM) for LLaMA-7B. Double to 64Gb for LLaMA-13B.

Next, type the following to run:

 docker run -it --rm --log-driver none --read-only -v /model/file/location/:/a/ defnotgustavom/llama.go llama.go --model /a/model.name --prompt "What is 1 + 1?"

• Useful command line flags:

--prompt Text prompt from user to feed the model input

--model Path and file name of converted .bin LLaMA model [ llama-7b-fp32.bin, etc ]

--server Start in Server Mode acting as REST API endpoint

--host Host to allow requests from in Server Mode [ localhost by default ]

--port Port listen to in Server Mode [ 8080 by default ]

--pods Maximum pods or units of parallel execution allowed in Server Mode [ 1 by default ]

--threads Adjust to the number of CPU cores you want to use [ all cores by default ]

--context Context size in tokens [ 1024 by default ]

--predict Number of tokens to predict [ 512 by default ]

--temp Model temperature hyper parameter [ 0.5 by default ]

--silent Hide welcome logo and other output [ shown by default ]

--chat Chat with user in interactive mode instead of compute over static prompt

--profile Profe CPU performance while running and store results to cpu.pprof file

--avx Enable x64 AVX2 optimizations for Intel and AMD machines

--neon Enable ARM NEON optimizations for Apple Macs and ARM server

Check https://github.com/gotzmann/llama.go⁠ for more info.