ascendai/vc-scheduler repository overview

⁠Cluster Scheduling Component Atlas for Volcano

⁠Quick Reference

• Atlas for Volcano is maintained by MindCluster Repository⁠
• Where to get help
  • MindCluster Repository⁠
  • MindCluster Atlas Community⁠
  • Issue Tracker⁠

⁠Atlas for Volcano

Atlas for Volcano is built on the open-source Volcano scheduling plugin mechanism, adding features such as Atlas AI processor (NPU) affinity scheduling and virtual device scheduling. It is deployed on management nodes. Volcano is a scheduler component that leverages the interconnect topology and processing logic of Atlas AI processors to achieve optimal utilization, maximizing the computing performance of Atlas AI processors.

⁠Use Cases

The basic Kubernetes scheduler can only perform resource scheduling based on the number of Atlas chips. To achieve affinity scheduling and maximize resource utilization, it is necessary to be aware of the network connectivity between Atlas chips and select the network-optimal resources. MindCluster provides the Volcano service deployed on management nodes, offering network affinity scheduling for different Atlas devices and network topologies.

⁠Features

• Available Device Calculation: Calculates cluster available device information based on fault information and node information reported by underlying cluster scheduling components. (self-maintain-available-card is enabled by default. When disabled, available device information is obtained from the underlying cluster scheduling components.)
• Optimal Resource Allocation: Retrieves the user's desired resource quantity from the Kubernetes job object, and selects the optimal resource allocation for the job based on the cluster's device count, device types, and device network topology.
• Fault Rescheduling: Reschedules jobs when job resources encounter faults.
• NPU Affinity Scheduling: Based on the interconnect topology of Atlas AI processors, prioritizes scheduling jobs to processors within the same card, then to HCCS-interconnected processors, and finally to PCIe-interconnected processors, reducing resource fragmentation and network congestion.
• Switch Affinity Scheduling: Achieves optimal node utilization based on the network configuration under switches and the parameter plane network configuration. Supports various network topologies including Spine-Leaf dual-layer interconnect and single-layer switch interconnect.
• Logical Super Node Affinity Scheduling: Divides physical super nodes into logical super nodes based on partitioning strategies, achieving optimal node utilization.
• Multi-level Scheduling Strategy: Abstracts cluster resources into a multi-level structure based on the NPU network topology hierarchy, configurable via the resource-level-config parameter.
• Multiple Scheduling Modes: Supports whole-card scheduling, static vNPU scheduling, dynamic vNPU scheduling, and soft-partition scheduling.

⁠Affinity Policies

Based on the characteristics of Atlas 910 AI processors and resource utilization rules, the following affinity policies are established (listed by priority):

1. HCCS Affinity Scheduling Principle: The requested Atlas 910 AI processors must be within the same HCCS ring, prioritizing the HCCS with the most matching number of remaining available processors.
2. Fill-First Scheduling Principle: Prioritizes allocating AI servers that have already been assigned Atlas 910 AI processors, reducing fragmentation.
3. Even-Remaining Priority Principle: Prioritizes HCCS that satisfy the above conditions, then selects HCCS with an even number of remaining processors.

⁠Upstream and Downstream Dependencies

1. Calculates cluster resource information based on information reported by ClusterD (when using ClusterD by default).
2. Receives job launch configurations from third parties, and selects optimal node resources based on cluster resource information.
3. Passes specific resource selection information to Atlas Device Plugin on compute nodes to complete device mounting.

⁠Image Description

Atlas for Volcano contains two images:

• volcano-scheduler: Volcano scheduler image, containing the Atlas NPU affinity scheduling plugin ( volcano-npu_*.so).
• volcano-controller: Volcano controller image.

⁠Supported Tags and Dockerfile Links

⁠Tag Convention

Starting from Atlas NPU Scheduler Plugin v26.1.0, tags follow the format below:

<component-version>-<ascend-scheduling-plugin-version>-<os>

⁠Atlas for Volcano Latest Version 26.1.1（Volcano v1.12.0）

The following are all images of the latest released 26.1.1（Volcano v1.12.0） version of Atlas for Volcano. For all historical version Tags, please refer to Supported Tags⁠.

⁠Atlas for Volcano 26.1.1（Volcano v1.9.0）

⁠Atlas for Volcano 26.1.1（Volcano v1.7.0）

The tags of the Atlas NPU Scheduler Plugin versions before v26.1.0 follow the format below:

<component-version>-<ascend-scheduling-plugin-version>

⁠Atlas for Volcano 26.0.0 (Volcano v1.9.0)

Using linux-aarch64 architecture as an example: Atlas for Volcano package download: Ascend-mindxdl-volcano_26.0.0_linux-aarch64.zip⁠

⁠Atlas for Volcano 26.0.0 (Volcano v1.7.0)

Using linux-aarch64 architecture as an example: Atlas for Volcano package download: Ascend-mindxdl-volcano_26.0.0_linux-aarch64.zip⁠

⁠Quick Start

⁠Prerequisites

⁠Software Dependencies

⁠Hardware Requirements

⁠Obtain Atlas for Volcano Images Online

1. Pull images

   Pull the Atlas for Volcano images from AscendHub, replacing {tag} with the actual version tag (v1.9.0-v26.0.0 recommended).

   docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-scheduler:{tag}
   docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-controller-manager:{tag}
   

   Copy

2. Retag images

   Retag the official images with local tags for consistent naming and easier operations management.

   docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-scheduler:{tag} volcanosh/vc-scheduler:{tag}
   docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-controller-manager:{tag} volcanosh/vc-controller-manager:{tag}
   

   Copy

⁠Build Locally (Optional)

⁠Local Image Build Process for Atlas NPU Scheduler Plugin v26.1.0 and Later Versions

Example: Build a Volcano v1.9.0 component image with linux-aarch64 architecture based on Alpine latest, integrated with Atlas NPU Scheduler Plugin v26.1.1.

1. Obtain the target Dockerfile

   Navigate to the chapter Supported Tags and Dockerfile Links, open the Dockerfile-scheduler.alpine and Dockerfile-controller.alpine link corresponding to your target version, and save the file to a local directory on your aarch64 environment.

2. Build the Docker image locally (disable cache to ensure a clean build)

   # Build scheduler image
   docker build --no-cache -t volcanosh/vc-scheduler:v1.9.0 ./ -f Dockerfile-scheduler.alpine
   
   # Build controller image
   docker build --no-cache -t volcanosh/vc-controller-manager:v1.9.0 ./ -f Dockerfile-controller.alpine
   

   Copy

⁠Local Image Build Process for Atlas NPU Scheduling Plugin Versions Before v26.1.0

Example: Build a Volcano v1.9.0 component image based on Alpine latest with linux-aarch64 architecture, integrated with Atlas NPU Scheduler Plugin v26.0.0.

1. Download the officially released component package

   wget https://gitcode.com/Ascend/mind-cluster/releases/download/v26.0.0/Ascend-mindxdl-volcano_26.0.0_linux-aarch64.zip
   

   Copy

2. Extract the package to a custom directory

   unzip Ascend-mindxdl-volcano_26.0.0_linux-aarch64.zip -d Ascend-mindxdl-volcano_26.0.0_linux-aarch64
   

   Copy

3. Enter the extracted working directory

   cd Ascend-mindxdl-volcano_26.0.0_linux-aarch64/volcano-v1.9.0
   

   Copy

4. Build Docker images locally (disable cache to ensure a clean build)

   # Build scheduler image
   docker build --no-cache -t volcanosh/vc-scheduler:v1.9.0 ./ -f Dockerfile-scheduler
   
   # Build controller image
   docker build --no-cache -t volcanosh/vc-controller-manager:v1.9.0 ./ -f Dockerfile-controller
   

   Copy

⁠Deploy Atlas for Volcano

1. Start Volcano

   Replace {version} in the YAML filename with the actual version (currently volcano v1.9.0). Before deployment, replace the image {tag} in the YAML file with the actual image version.

   kubectl apply -f volcano-{version}.yaml
   

   Copy

2. Verify deployment

   kubectl get pods -A | grep volcano
   

   Copy

   Expected result: The volcano-related Pods in the corresponding namespace should be in Running state.

⁠License

View the license information⁠ for the Mind series software contained in these images.

As with all container images, pre-installed software packages (Python, system libraries, etc.) may be subject to their respective license agreements.