anyscale/aviary repository overview

⁠Deprecation Notice

We've renamed Aviary to ray-llm! You can find the 0.3.0 and newer version-tagged images, and all future images as anyscale/ray-llm here: https://hub.docker.com/repository/docker/anyscale/ray-llm⁠

⁠Migration

If using an image 0.3.0 or newer:

1. Change your anyscale/aviary Docker references to anyscale/ray-llm

If using an image older than 0.3.0:

1. Continue to pull from this repository

⁠Timeline

There is currently no timeline for deletion of this repository, but this is subject to change.

⁠Overview

This is the publicly available set of Docker images for Anyscale/Ray's Aviary project.

Aviary is an LLM serving solution that makes it easy to deploy and manage a variety of open source LLMs. It does this by:

• Providing an extensive suite of pre-configured open source LLMs, with defaults that work out of the box.
• Supporting Transformer models hosted on Hugging Face Hub or present on local disk.
• Simplifying the deployment of multiple LLMs within a single unified framework.
• Simplifying the addition of new LLMs to within minutes in most cases.
• Offering unique autoscaling support, including scale-to-zero.
• Fully supporting multi-GPU & multi-node model deployments.
• Offering high performance features like continuous batching, quantization and streaming.
• Providing a REST API that is similar to OpenAI's to make it easy to migrate and cross test them.

Read more here⁠

⁠Tags

⁠Usage

See: ray-project/aviary "Deploying Aviary Backend"⁠ for full instructions

⁠Example

Requires a machine with compatible NVIDIA A10 GPU and valid HUGGING_FACE_HUB_TOKEN to run the Amazon LightGPT model⁠:

docker run \
    --gpus all \
    -e HUGGING_FACE_HUB_TOKEN=<your_token> \
    --shm-size 1g \
    -p 8000:8000 \
    --entrypoint aviary \
    anyscale/aviary:latest run --model models/continuous_batching/amazon--LightGPT.yaml

⁠Source

Source is available at https://github.com/ray-project/aviary⁠