bbenzikry/spark-eks
Custom spark images for working with the spark-on-k8s operator on AWS with IRSA and Glue support
Image
0
10K+
bbenzikry/spark-eks repository overview
spark-on-eks
Examples and custom spark images for working with the spark-on-k8s operator on AWS.
Allows using Spark 2 with IRSA and Spark 3 with IRSA and AWS Glue as a metastore
Note: Spark 2 images will not be updated, please see the FAQ
Prerequisites
- Deploy spark-on-k8s operator using the helm chart and the patched operator image
bbenzikry/spark-eks-operator:latest
Suggested values for the helm chart can be found in the flux example.
Note: Do not create the spark service account automatically as part of chart use.
using IAM roles for service accounts on EKS
Creating roles and service account
- Create an AWS role for driver
- Create an AWS role for executors
- Add default service account EKS role for executors in your spark job namespace ( optional )
# NOTE: Only required when not building spark from source or using a version of spark < 3.1. If you use our *-edge docker images for spark3/pyspark3 you can skip this step, as it will rely on the driver pod.
apiVersion: v1
kind: ServiceAccount
metadata:
name: default
namespace: SPARK_JOB_NAMESPACE
annotations:
# can also be the driver role
eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT_ID:role/executor-role"
- Make sure spark service account is configured to an EKS role as well
## With the spark3 source builds, when this is configured and no executor role exists, executors default to this SA as well.
# This is not recommended for production until a stable release is provided.
apiVersion: v1
kind: ServiceAccount
metadata:
name: spark
namespace: SPARK_JOB_NAMESPACE
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT_ID:role/executor-role"
Building a compatible image
-
For spark < 3.0.0, see spark2.Dockerfile and pyspark.Dockerfile
-
For spark 3.0.0+, see spark3.Dockerfile spark3.edge.Dockerfile
-
For pyspark, see pyspark.Dockerfile
Submit your spark application with IRSA support
Select the right implementation for you
Below are examples for latest versions.
If you want to use pinned versions, all images are tagged by the commit SHA.
You can find a full list of tags here
# spark2
FROM bbenzikry/spark-eks:spark2-latest
# spark3
FROM bbenzikry/spark-eks:spark3-latest
# source / master build
FROM bbenzikry/spark-eks:spark3-edge
# pyspark2
FROM bbenzikry/spark-eks:pyspark2-latest
# pyspark3
FROM bbenzikry/spark-eks:pyspark3-latest
# pyspark3-edge
FROM bbenzikry/spark-eks:pyspark3-edge
Submit your SparkApplication spec
hadoopConf:
# IRSA configuration
"fs.s3a.aws.credentials.provider": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
driver:
.....
labels:
.....
serviceAccount: SERVICE_ACCOUNT_NAME
# See: https://github.com/kubernetes/kubernetes/issues/82573
securityContext:
fsGroup: 65534
- Full example here
Working with AWS Glue as metastore
Prerequisites
- Make sure your driver and executor roles have the relevant glue permissions
{
/* Example below is an example configuration for accessing db1/table1.
Modify this as you deem worthy for potential access.
Last 3 resources must be present for your region.
*/
"Effect": "Allow",
"Action": ["glue:*Database*", "glue:*Table*", "glue:*Partition*"],
"Resource": [
"arn:aws:glue:us-west-2:123456789012:catalog",
"arn:aws:glue:us-west-2:123456789012:database/db1",
"arn:aws:glue:us-west-2:123456789012:table/db1/table1",
"arn:aws:glue:eu-west-1:123456789012:database/default",
"arn:aws:glue:eu-west-1:123456789012:database/global_temp",
"arn:aws:glue:eu-west-1:123456789012:database/parquet"
]
}
- Make sure you are using the patched operator image
- Add a config map to your spark job namespace as defined here
Submitting your application
Working with the spark history server on S3
FAQ
- Where can I find a Spark 2 build with Glue support?
- As spark 2 becomes less and less relevant, I opted against the need to add glue support. You can take a look here for a reference implementation which you can add to the Spark 2 dockerfile dockerfile
- Why a patched operator image?
- Some PRs are still pending on the operator image. Once they are pushed through and properly tested, you can use them instead.
Tag summary
Content type
Image
Digest
Size
869.5 MB
Last updated
about 6 years ago
docker pull bbenzikry/spark-eks:spark3-eb0bc39