upstage/solar
This repository contains Docker images for Solar, a Large Language Model (LLM) developed by Upstage.
Currently, this Docker image supports:
Solar Pro Preview is developed using an enhanced version of our previous depth up-scaling method, which scales a Phi-3-medium model with 14 billion parameters to 22 billion parameters, intended to run on a GPU with 80GB of VRAM. Our carefully curated training strategy and dataset have significantly enhanced performance from Phi-3-medium, particularly on the MMLU-Pro and IFEval benchmarks, both respected for evaluating a model's knowledge and instruction-following abilities.
For more information, visit:
Two versions of the model are available:
docker run -d \
--gpus all \
-p 8000:8000 \
--name solar-pro-preview \
upstage/solar:pro-preview
Run as API Server
docker run -d \
--gpus all \
-p 8000:11434 \
--name solar-pro-preview-int4 \
upstage/solar:pro-preview-int4
Interactive Chat Mode
You can run the container in interactive mode to use Ollama's console-based chat interface:
docker run -it --gpus all upstage/solar:pro-preview-int4
To run without GPU (Macbook), remove the --gpus all
option.
docker run -it upstage/solar:pro-preview-int4
This will start an interactive chat session where you can directly communicate with the model:
>>> What is artificial intelligence?
Artificial Intelligence (AI) refers to the simulation of human intelligence in machines...
>>> Tell me more about machine learning.
Machine learning is a subset of artificial intelligence that focuses on...
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="dummy" # API key is not required but the field must be set
)
response = client.chat.completions.create(
model="solar-pro",
messages=[
{"role": "user", "content": "What is artificial intelligence?"}
],
temperature=0.7
)
print(response.choices[0].message.content)
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "solar-pro",
"messages": [
{"role": "user", "content": "What is artificial intelligence?"}
],
"temperature": 0.7
}'
The FP16 version can be configured using various vLLM parameters. For detailed configuration options and best practices, please refer to the vLLM documentation.
The INT4 version can be customized using Ollama's configuration options. For detailed information about available settings and optimization techniques, please visit the Ollama documentation.
This Docker image incorporates the following open source components:
Please note that while these components are open source, the Solar Pro Preview model itself may have separate licensing terms. Refer to Upstage's terms of use for details about the model's usage rights.
docker pull upstage/solar