openeuler/auron repository overview

⁠Quick reference

• The official auron docker image.

• Maintained by: openEuler CloudNative SIG⁠.

• Where to get help: openEuler CloudNative SIG⁠, openEuler⁠.

⁠auron | openEuler

The Auron accelerator for big data engines (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the distributed computing framework.

Auron takes a fully optimized physical plan from a distributed computing framework, mapping it into DataFusion's execution plan, and performs native plan computation.

• Native execution: Implemented in Rust, eliminating JVM overhead and enabling predictable performance.
• Vectorized computation: Built on Apache Arrow's columnar format, fully leveraging SIMD instructions for batch processing.
• Pluggable architecture: Seamlessly integrates with Apache Spark while designed for future extensibility to other engines.
• Production-hardened optimizations: Multi-level memory management, compacted shuffle formats, and adaptive execution strategies developed through large-scale deployment.

⁠Supported tags and respective Dockerfile links

The tag of each auron docker image is consist of the version of auron and the version of basic image. The details are as follows

⁠Usage

Copy the Auron JAR from the image to your Spark classpath:

docker run --rm openeuler/auron:{Tag} cat /opt/auron/auron-spark_2.12-release-linux-x86_64-7.0.0-incubating.jar > $SPARK_HOME/jars/auron-spark.jar

Add the following configurations to $SPARK_HOME/conf/spark-defaults.conf:

spark.auron.enable true
spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager
spark.memory.offHeap.enabled false

spark.executor.memory 4g
spark.executor.memoryOverhead 4096

Submit a query with spark-sql:

spark-sql -f query.sql

⁠Question and answering

If you have any questions or want to use some special features, please submit an issue or a pull request on openeuler-docker-images⁠.