ascendai/noded repository overview

⁠Cluster Scheduling Component NodeD

⁠Quick Reference

• NodeD is maintained by MindCluster Repository⁠
• Where to get help
  • MindCluster Repository⁠
  • MindCluster Atlas Community⁠
  • Issue Tracker⁠

⁠NodeD

NodeD is one of the MindCluster cluster scheduling components, deployed on compute nodes. It detects abnormal node states, retrieves CPU, memory, and disk fault information from IPMI, and reports it to ClusterD.

⁠Use Cases

When certain faults occur in a node's CPU, memory, or disk, training jobs will fail. To enable training jobs to exit quickly when node faults occur and prevent new jobs from being scheduled to faulty nodes, MindCluster provides the NodeD component for detecting node abnormalities.

⁠Features

• Retrieves node abnormalities from IPMI and reports them to the upper-level resource scheduling service.
• Periodically sends node fault information to the upper-level resource scheduling service.

⁠Upstream and Downstream Dependencies

1. Retrieves CPU, memory, and disk fault information of compute nodes from IPMI.
2. Reports CPU, memory, and disk fault information of compute nodes to ClusterD.

⁠Supported Tags and Dockerfile Links

⁠Tag Convention

Starting from version v26.1.0, tags follow the format below:

<version>-<os>

⁠NodeD Latest Version 26.1.1

The following are all images of the latest released 26.1.1 version of NodeD. For all historical version Tags, please refer to Supported Tags⁠.

The tags of versions before v26.1.0 follow the format below:

<version>

⁠NodeD 26.0.0

⁠Quick Start

⁠Prerequisites

⁠Software Dependencies

⁠Hardware Requirements

⁠Obtain NodeD Image Online

1. Pull the official image

   Pull the NodeD image from AscendHub, replacing {tag} with the actual version.

   docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/noded:{tag}
   

   Copy

2. Retag the image

   Retag the official image with a local tag for consistent naming and easier operations management.

   docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/noded:{tag} noded:{tag}
   

   Copy

⁠Build Locally (Optional)

⁠Local Build Steps for v26.1.0 and Later Versions

Example: build an NodeD image of architecture linux-aarch64, version v26.1.1, based on Ubuntu 22.04.

1. Obtain the target Dockerfile

   Navigate to the chapter Supported Tags and Dockerfile Links, open the Dockerfile.ubuntu link corresponding to your target version, and save the file to a local directory on your aarch64 environment.

2. Build the Docker image locally (disable cache to ensure a clean build)

   docker build --no-cache -t noded:v26.1.1 ./ -f Dockerfile.ubuntu
   

   Copy

⁠Local Image Build Process for Versions Before v26.1.0

Example: Build an NodeD image of architecture linux-aarch64, version v26.0.0, based on Ubuntu 22.04.

1. Download the officially released component package

   wget https://gitcode.com/Ascend/mind-cluster/releases/download/v26.0.0/Ascend-mindxdl-noded_26.0.0_linux-aarch64.zip
   

   Copy

2. Extract the package to a custom directory

   unzip Ascend-mindxdl-noded_26.0.0_linux-aarch64.zip -d Ascend-mindxdl-noded_26.0.0_linux-aarch64
   

   Copy

3. Enter the extracted working directory

   cd Ascend-mindxdl-noded_26.0.0_linux-aarch64
   

   Copy

4. Build the Docker image locally (disable cache to ensure a clean build)

   docker build --no-cache -t noded:v26.0.0 ./ -f Dockerfile
   

   Copy

⁠Deploy NodeD

1. Label Kubernetes nodes

   Add labels to the corresponding nodes for cluster scheduling matching. Replace <node-name> with the actual node name.

   kubectl label nodes <node-name> workerselector=dls-worker-node
   

   Copy

2. Start NodeD

   Before deployment, replace the image {tag} in the YAML file with the actual image version.

   kubectl apply -f noded-{version}.yaml
   

   Copy

3. Verify deployment

   kubectl get pods -A | grep noded
   

   Copy

   Expected result: The noded related Pods in the corresponding namespace should be in Running state.

⁠License

View the license information⁠ for the Mind series software contained in these images.

As with all container images, pre-installed software packages (Python, system libraries, etc.) may be subject to their respective license agreements.