ascendai/ascend-operator repository overview

⁠Cluster Scheduling Component Atlas Operator

⁠Quick Reference

• Atlas Operator is maintained by MindCluster Repository⁠
• Where to get help
  • MindCluster Repository⁠
  • MindCluster Atlas Community⁠
  • Issue Tracker⁠

⁠Atlas Operator

Atlas Operator is one of the MindCluster cluster scheduling components, deployed on management nodes. It supports distributed training on Kubernetes using the MindSpore and PyTorch AI frameworks. Through the CRD (Custom Resource Definition) mechanism, AscendJob tasks are defined, allowing users to easily implement distributed training by simply configuring YAML files.

⁠Use Cases

MindCluster provides the Atlas Operator component to input information required for collective communication, including the master process IP, RankTable information for static network collective communication, and the rankId of the current Pod.

⁠Features

• Creates Pods and injects collective communication parameters via environment variables.
• Creates RankTable files and mounts them to containers via shared storage or ConfigMap, optimizing collective communication link establishment performance.

⁠Upstream and Downstream Dependencies

1. Perceives whether the resources required by the current job are satisfied through Volcano.
2. After resources are satisfied, creates the corresponding Pods for the job and injects collective communication parameters as environment variables.
3. After Pod creation, Volcano performs the final resource selection.
4. Obtains the chip ID, IP, and rankId information of the job from Atlas Device Plugin, aggregates them, and generates the collective communication file.
5. Mounts the collective communication file into the container via shared storage or ConfigMap.

⁠Supported Tags and Dockerfile Links

⁠Tag Convention

Starting from version v26.1.0, tags follow the format below:

<version>-<os>

⁠Atlas Operator Latest Version 26.1.1

The following are all images of the latest released 26.1.1 version of Atlas Operator. For all historical version Tags, please refer to Supported Tags⁠.

The tags of versions before v26.1.0 follow the format below:

<version>

⁠Atlas Operator 26.0.0

⁠Quick Start

⁠Prerequisites

⁠Software Dependencies

⁠Hardware Requirements

⁠Obtain Atlas Operator Image Online

1. Pull the official image

   Pull the Atlas Operator image from AscendHub, replacing {tag} with the actual version.

   docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-operator:{tag}
   

   Copy

2. Retag the image

   Retag the official image with a local tag for consistent naming and easier operations management.

   docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-operator:{tag} ascend-operator:{tag}
   

   Copy

⁠Build Locally (Optional)

⁠Local Build Steps for v26.1.0 and Later Versions

Example: build an Atlas Operator image of architecture linux-aarch64, version v26.1.1, based on Ubuntu 22.04.

1. Obtain the target Dockerfile

   Navigate to the chapter Supported Tags and Dockerfile Links, open the Dockerfile.ubuntu link corresponding to your target version, and save the file to a local directory on your aarch64 environment.

2. Build the Docker image locally (disable cache to ensure a clean build)

   docker build --no-cache -t ascend-operator:v26.1.1 ./ -f Dockerfile.ubuntu
   

   Copy

⁠Local Image Build Process for Versions Before v26.1.0

Example: Build an Atlas Operator image of architecture linux-aarch64, version v26.0.0, based on Ubuntu 22.04.

1. Download the officially released component package

   wget https://gitcode.com/Ascend/mind-cluster/releases/download/v26.0.0/Ascend-mindxdl-ascend-operator_26.0.0_linux-aarch64.zip
   

   Copy

2. Extract the package to a custom directory

   unzip Ascend-mindxdl-ascend-operator_26.0.0_linux-aarch64.zip -d Ascend-mindxdl-ascend-operator_26.0.0_linux-aarch64
   

   Copy

3. Enter the extracted working directory

   cd Ascend-mindxdl-ascend-operator_26.0.0_linux-aarch64
   

   Copy

4. Build the Docker image locally (disable cache to ensure a clean build)

   docker build --no-cache -t ascend-operator:v26.0.0 ./ -f Dockerfile
   

   Copy

⁠Deploy Atlas Operator

1. Start Atlas Operator

   Before deployment, replace the image {tag} in the YAML file with the actual image version.

   kubectl apply -f ascend-operator-{version}.yaml
   

   Copy

2. Verify deployment

   kubectl get pods -A | grep ascend-operator
   

   Copy

   Expected result: The ascend-operator related Pods in the corresponding namespace should be in Running state.

⁠License

View the license information⁠ for the Mind series software contained in these images.

As with all container images, pre-installed software packages (Python, system libraries, etc.) may be subject to their respective license agreements.