ascendai/npu-exporter repository overview

⁠Cluster Scheduling Component NPU Exporter

⁠Quick Reference

• NPU Exporter is maintained by MindCluster Repository⁠
• Where to get help
  • MindCluster Repository⁠
  • MindCluster Atlas Community⁠
  • Issue Tracker⁠

⁠NPU Exporter

NPU Exporter is one of the MindCluster cluster scheduling components, deployed on compute nodes. It reports various chip data metrics and supports two monitoring integration methods: Prometheus and Telegraf.

⁠Use Cases

During job execution, in addition to chip faults, it is often necessary to monitor chip network and computing utilization to identify performance bottlenecks and find directions for improving job performance. MindCluster provides the NPU Exporter component deployed on compute nodes for reporting various chip data metrics.

⁠Features

• Retrieves various chip and network data metrics from the driver.
• Adapts Prometheus hook functions, providing standard interfaces for Prometheus services to call.
• Adapts Telegraf hook functions, providing standard interfaces for Telegraf services to call.
• Supports real-time monitoring of Atlas AI processor utilization, temperature, voltage, memory, and other data metrics.
• Supports monitoring of virtual NPU (vNPU) AI Core utilization, vNPU total memory, and vNPU used memory.
• Supports custom metric development. Users can refer to the provided demo to develop custom metric plugins.

⁠Upstream and Downstream Dependencies

1. Retrieves chip and network information from the driver and stores it in a local cache.
2. Retrieves container information from the Kubernetes standard CRI interface and stores it in a local cache.
3. Implements Prometheus or Telegraf interfaces for them to periodically retrieve cached data metrics.

⁠Supported Tags and Dockerfile Links

⁠Tag Convention

Starting from version v26.1.0, tags follow the format below:

<version>-<os>

⁠NPU Exporter Latest Version 26.1.1

The following are all images of the latest released 26.1.1 version of NPU Exporter. For all historical version Tags, please refer to Supported Tags⁠.

The tags of versions before v26.1.0 follow the format below:

<version>

⁠NPU Exporter 26.0.0

⁠Quick Start

⁠Prerequisites

⁠Software Dependencies

⁠Hardware Requirements

⁠Obtain NPU Exporter Image Online

1. Pull the official image

   Pull the NPU Exporter image from AscendHub, replacing {tag} with the actual version.

   docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/npu-exporter:{tag}
   

   Copy

2. Retag the image

   Retag the official image with a local tag for consistent naming and easier operations management.

   docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/npu-exporter:{tag} npu-exporter:{tag}
   

   Copy

⁠Build Locally (Optional)

⁠Local Build Steps for v26.1.0 and Later Versions

Example: build an NPU Exporter image of architecture linux-aarch64, version v26.1.1, based on Ubuntu 22.04.

1. Obtain the target Dockerfile

   Navigate to the chapter Supported Tags and Dockerfile Links, open the Dockerfile.ubuntu link corresponding to your target version, and save the file to a local directory on your aarch64 environment.

2. Build the Docker image locally (disable cache to ensure a clean build)

   docker build --no-cache -t npu-exporter:v26.1.1 ./ -f Dockerfile.ubuntu
   

   Copy

⁠Local Image Build Process for Versions Before v26.1.0

Example: Build an NPU Exporter image of architecture linux-aarch64, version v26.0.0, based on Ubuntu 22.04.

1. Download the officially released component package

   wget https://gitcode.com/Ascend/mind-cluster/releases/download/v26.0.0/Ascend-mindxdl-npu-exporter_26.0.0_linux-aarch64.zip
   

   Copy

2. Extract the package to a custom directory

   unzip Ascend-mindxdl-npu-exporter_26.0.0_linux-aarch64.zip -d Ascend-mindxdl-npu-exporter_26.0.0_linux-aarch64
   

   Copy

3. Enter the extracted working directory

   cd Ascend-mindxdl-npu-exporter_26.0.0_linux-aarch64
   

   Copy

4. Build the Docker image locally (disable cache to ensure a clean build)

   docker build --no-cache -t npu-exporter:v26.0.0 ./ -f Dockerfile
   

   Copy

⁠Deploy NPU Exporter

1. Label Kubernetes nodes

   Add labels to the corresponding nodes for cluster scheduling matching. Replace <node-name> with the actual node name.

   kubectl label nodes <node-name> workerselector=dls-worker-node
   

   Copy

2. Start NPU Exporter

   Before deployment, replace the image {tag} in the YAML file with the actual image version.

   kubectl apply -f npu-exporter-{version}.yaml
   

   Copy

3. Verify deployment

   kubectl get pods -A | grep npu-exporter
   

   Copy

   Expected result: The npu-exporter related Pods in the corresponding namespace should be in Running state.

4. Access monitoring metrics

   curl http://<pod-ip>:8082/metrics
   

   Copy

⁠License

View the license information⁠ for the Mind series software contained in these images.

As with all container images, pre-installed software packages (Python, system libraries, etc.) may be subject to their respective license agreements.