ascendai/multimodalsdk repository overview

⁠MultimodalSDK

⁠Quick Reference

• Where to get help
  • Issue Feedback⁠
  • MultimodalSDK Code⁠
  • MultimodalSDK Documentation⁠
  • Community⁠

⁠MultimodalSDK Overview

In the inference pipeline of multimodal large models, massive and complex data needs to be processed. The MultimodalSDK accelerates the preprocessing workflow of large model inference by providing a set of high-performance Ascend-affinity interfaces. Preprocessing APIs currently run on CPU (DeviceMode.CPU) and are typically deployed alongside CANN/NPU inference frameworks.

• It covers common preprocessing operations such as image and video loading and decoding, as well as resize, crop, and other typical processing steps.
• It supports mutual conversion between various open-source data structures and acceleration library data structures, enabling rapid application and easy migration.

⁠Supported Tags and Dockerfile Links

⁠Tag Naming Convention

Tags follow this pattern:

<multimodalsdk_version>-<vllm-ascend_version>-<torch-npu_version>-<chip_series>-<os>-<python_version>-aarch64

⁠Tags and Dockerfile

⁠Quick Start

⁠Prerequisites (optional)

⁠Install Driver

An NPU driver compatible with the container's CANN version must be installed on the host. See the CANN Compatibility Matrix⁠ for driver ↔ CANN version mapping.

⁠How to build

docker build -t {your_repo}/multimodal:latest -f Dockerfile.<chip_series>.<os> .

⁠Running MultimodalSDK Container

docker run \
    --name multimodal_container \
    --device /dev/davinci0 \
    --device /dev/davinci_manager \
    --device /dev/devmm_svm \
    --device /dev/hisi_hdc \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -it ascend/multimodal:tag bash

⁠Enter the Container

docker exec -it multimodal_container bash

⁠MultimodalSDK Usage

MultimodalSDK provides sample code to help developers get started quickly. You can access the examples through the following link:

• MultimodalSDK Samples⁠

⁠Development

# Add required software by developer
FROM swr.cn-south-1.myhuaweicloud.com/ascendhub/multimodalsdk:26.2.0-cann9.2.0-torch_npu2.10.0.post4-910b-ubuntu22.04-py3.12-aarch64

RUN apt update -y && \
    ...

...

⁠License

View the license information⁠ for CANN and MindSeries software included in these images.

As with all container images, the pre-installed packages (Python, system libraries, etc.) may be subject to their own licenses.