ascendai/infer-operator repository overview

⁠Cluster Scheduling Component Infer Operator

⁠Quick Reference

• Infer Operator is maintained by MindCluster Repository⁠
• Where to get help
  • MindCluster Repository⁠
  • MindCluster Atlas Community⁠
  • Issue Tracker⁠

⁠Infer Operator

Infer Operator is one of the MindCluster cluster scheduling components, deployed on management nodes. It is a Kubernetes Operator used to deploy and manage multi-role collaborative inference jobs. Infer Operator defines three CRDs: InferServiceSet, InferService, and InstanceSet, and implements controllers for these three resource types to reconcile their instance states.

⁠Use Cases

MindCluster provides the Infer Operator component to launch inference services based on instance configurations of inference services, and supports manual scaling of inference instances.

⁠Features

• Creates inference instance Workloads and Services.
• Supports manual scaling of inference instances.

⁠Upstream and Downstream Dependencies

1. Creates inference instance Workloads based on user-configured job YAML.
2. After the Workload Controller creates Pods, Volcano performs the final resource selection.
3. If the Workload requests NPU cards, Atlas Device Plugin obtains NPU information and completes device mounting.

⁠Supported Tags and Dockerfile Links

⁠Tag Convention

Starting from version v26.1.0, tags follow the format below:

<version>-<os>

⁠Infer Operator Latest Version 26.1.1

The following are all images of the latest released 26.1.1 version of Infer Operator. For all historical version Tags, please refer to Supported Tags⁠.

The tags of versions before v26.1.0 follow the format below:

<version>

⁠Infer Operator 26.0.0

⁠Quick Start

⁠Prerequisites (Optional)

⁠Software Dependencies

⁠Hardware Requirements

⁠Obtain Infer Operator Image Online

1. Pull the official image

   Pull the Infer Operator image from the Atlas image repository, replacing {tag} with the actual version.

   docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/infer-operator:{tag}
   

   Copy

2. Retag the image

   Retag the official image with a local tag for consistent naming and easier operations management.

   docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/infer-operator:{tag} infer-operator:{tag}
   

   Copy

⁠How to Build Locally (Optional)

⁠Local Build Steps for v26.1.0 and Later Versions

Example: build an Infer Operator image of architecture linux-aarch64, version v26.1.1, based on Ubuntu 22.04.

1. Obtain the target Dockerfile

   Navigate to the chapter Supported Tags and Dockerfile Links, open the Dockerfile.ubuntu link corresponding to your target version, and save the file to a local directory on your aarch64 environment.

2. Build the Docker image locally (disable cache to ensure a clean build)

   docker build --no-cache -t infer-operator:v26.1.1 ./ -f Dockerfile.ubuntu
   

   Copy

⁠Local Image Build Process for Versions Before v26.1.0

Example: Build an Infer Operator image of architecture linux-aarch64, version v26.0.0, based on Ubuntu 22.04.

1. Download the officially released component package

   wget https://gitcode.com/Ascend/mind-cluster/releases/download/v26.0.0/Ascend-mindxdl-infer-operator_26.0.0_linux-aarch64.zip
   

   Copy

2. Extract the package to a custom directory

   unzip Ascend-mindxdl-infer-operator_26.0.0_linux-aarch64.zip -d Ascend-mindxdl-infer-operator_26.0.0_linux-aarch64
   

   Copy

3. Enter the extracted working directory

   cd Ascend-mindxdl-infer-operator_26.0.0_linux-aarch64
   

   Copy

4. Build the Docker image locally (disable cache to ensure a clean build)

   docker build --no-cache -t infer-operator:v26.0.0 ./ -f Dockerfile
   

   Copy

⁠Deploy Infer Operator

1. Start Infer Operator

   Replace the image {tag} in the infer-operator-{version}.yaml file with the actual tag.

   kubectl apply -f infer-operator-{version}.yaml
   

   Copy

2. Verify deployment

   kubectl get pods -A | grep infer-operator
   

   Copy

   Expected result: The infer-operator related Pods in the corresponding namespace should be in Running state.

⁠License

View the license information⁠ for the Mind series software contained in these images.

As with all container images, pre-installed software packages (Python, system libraries, etc.) may be subject to their respective license agreements.