More Docker. Easy Access. New Streamlined Plans. Learn more.

upstage/solar

By upstage

Updated 3 months ago

Solar, a Large Language Model (LLM) developed by Upstage.

Image
Machine Learning & AI
Gen AI
5

207

Solar Docker Image

This repository contains Docker images for Solar, a Large Language Model (LLM) developed by Upstage.

Supported Models

Currently, this Docker image supports:

  • Solar Pro Preview: an advanced large language model (LLM) with 22 billion parameters designed to fit into a single GPU.
About Solar Pro Preview

Solar Pro Preview is developed using an enhanced version of our previous depth up-scaling method, which scales a Phi-3-medium model with 14 billion parameters to 22 billion parameters, intended to run on a GPU with 80GB of VRAM. Our carefully curated training strategy and dataset have significantly enhanced performance from Phi-3-medium, particularly on the MMLU-Pro and IFEval benchmarks, both respected for evaluating a model's knowledge and instruction-following abilities.

For more information, visit:

Available Versions

Two versions of the model are available:

FP16 Version (solar:pro-preview)
  • Uses vLLM for inference
  • FP16 precision
  • Higher accuracy with moderate memory usage
  • Recommended for production use cases requiring high accuracy
INT4 Version (solar:pro-preview-int4)
  • Uses Ollama for inference
  • Quantized to Q4_K_M
  • Reduced memory footprint
  • Suitable for resource-constrained environments

Running the Docker Image

FP16 Version
docker run -d \
  --gpus all \
  -p 8000:8000 \
  --name solar-pro-preview \
  upstage/solar:pro-preview
INT4 Version

Run as API Server

docker run -d \
  --gpus all \
  -p 8000:11434 \
  --name solar-pro-preview-int4 \
  upstage/solar:pro-preview-int4

Interactive Chat Mode

You can run the container in interactive mode to use Ollama's console-based chat interface:

docker run -it --gpus all upstage/solar:pro-preview-int4

To run without GPU (Macbook), remove the --gpus all option.

docker run -it upstage/solar:pro-preview-int4

This will start an interactive chat session where you can directly communicate with the model:

>>> What is artificial intelligence?
Artificial Intelligence (AI) refers to the simulation of human intelligence in machines...

>>> Tell me more about machine learning.
Machine learning is a subset of artificial intelligence that focuses on...

API Usage

OpenAI Python SDK
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"  # API key is not required but the field must be set
)

response = client.chat.completions.create(
    model="solar-pro",
    messages=[
        {"role": "user", "content": "What is artificial intelligence?"}
    ],
    temperature=0.7
)

print(response.choices[0].message.content)
cURL Example
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "solar-pro",
    "messages": [
      {"role": "user", "content": "What is artificial intelligence?"}
    ],
    "temperature": 0.7
  }'

Configuration

FP16 Version (vLLM)

The FP16 version can be configured using various vLLM parameters. For detailed configuration options and best practices, please refer to the vLLM documentation.

INT4 Version (Ollama)

The INT4 version can be customized using Ollama's configuration options. For detailed information about available settings and optimization techniques, please visit the Ollama documentation.

License

This Docker image incorporates the following open source components:

FP16 Version
  • vLLM: Licensed under Apache License 2.0
INT4 Version

Please note that while these components are open source, the Solar Pro Preview model itself may have separate licensing terms. Refer to Upstage's terms of use for details about the model's usage rights.

Docker Pull Command

docker pull upstage/solar