foundryai/spark-pytest repository overview

⁠spark-pytest⁠

Dockerization of a pytest environment for tests that use Spark.

⁠Example Usage Steps

See the sample code in the example/ directory and the workflow in .github/workflows/ci.yml for a bare-bones use case.

1. Create a Dockerfile within the repo to test

FROM foundryai/spark-pytest:latest
WORKDIR /usr/src/app
ADD requirements.txt /usr/src/app
ADD requirements-dev.txt /usr/src/app
RUN pip install -r requirements.txt
RUN pip install -r requirements-dev.txt
COPY . /usr/src/app
CMD ["pytest"]

2.) Build the latest Docker image of the source code. (See docker-build in example/Makefile.)

3.) Run the tests. (See spark-pytest in example/Makefile.)

(Note: running make spark-pytest in the example/ directory will complete steps 2 and 3 at once.)

⁠Non-Dockerized setup steps

1. Install maven, run the brew install maven command. If you are not running on MacOS, see the manual install steps below.

⁠Prerequisites for Local PySpark Development

Complete these steps to prepare for local Python development:

1. Install Apache Maven from the following location: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz⁠.

2. Install the Apache Spark distribution the following location: https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-1.0/spark-2.4.3-bin-hadoop2.8.tgz⁠

3. Export the SPARK_HOME environment variable, setting it to the root location extracted from the Spark archive. For example:

   • export SPARK_HOME=/home/$USER/spark-2.4.3-bin-spark-2.4.3-bin-hadoop2.8

⁠Running your tests

⁠References

Developing and Testing ETL Scripts Locally Using the AWS Glue ETL Library⁠

Foundry is hiring!⁠