agentcloud/dataproc repository overview

⁠Dataproc for local development

Run Spark, PySpark, Hadoop, HDFS, YARN, and Hive on your machine with one Docker image. Start a single-node cluster for interactive development, or run a disposable Spark job without starting cluster services.

These are independently built compatibility runtimes using public, digest-pinned artifacts. They are not supplied, certified, or supported by Google.

Choose one of the two options below.

⁠Option 1: Use Dataproc through LocalCloud

This Dataproc image is part of LocalCloud⁠, a local Google Cloud emulator. LocalCloud provides a Dataproc API and web console to manage clusters and submit jobs, alongside other local cloud services such as BigQuery, Cloud Storage, Pub/Sub, Spanner, and Firestore.

With Docker running, install and start LocalCloud:

brew install LocalGCloud/tap/localcloud
lc start --debug

Open the console URL printed by the CLI. If startup needs troubleshooting, run lc doctor. See the LocalCloud quick start⁠ and Dataproc guide⁠.

⁠Option 2: Run Dataproc directly with Docker

This option starts a standalone Dataproc-compatible cluster using this image directly with Docker. Work with Spark, Hadoop, and Hive from the container.

Install Docker and start with 2 GB of memory for lightweight use. Full-cluster workloads can exceed this; increase Docker's memory allocation if services restart or jobs run out of memory. The examples use Dataproc 3.0; choose another exact tag below for a different component stack.

docker pull agentcloud/dataproc:3.0.0-debian13

docker run -d --name dataproc \
  -p 127.0.0.1:9870:9870 \
  -p 127.0.0.1:8088:8088 \
  -p 127.0.0.1:18080:18080 \
  -p 127.0.0.1:4040:4040 \
  agentcloud/dataproc:3.0.0-debian13

docker logs -f dataproc

With no command or environment variables, the container starts HDFS, YARN, Hive Metastore, HiveServer2, Spark History Server, and MapReduce Job History. HDFS and YARN typically become ready in about 20 seconds; HiveServer2 can take a minute or more. Press Ctrl+C to leave the log stream; the cluster keeps running.

Check readiness, then open PySpark:

docker exec dataproc hdfs dfsadmin -safemode get
docker exec dataproc yarn node -list
docker exec -it dataproc pyspark

Wait until HDFS reports Safe mode is OFF and YARN lists a RUNNING node. In the PySpark shell:

spark.range(1_000_000).selectExpr("id % 10 AS k").groupBy("k").count().show()
exit()

The shell uses YARN, as on a Dataproc master. Every Spark application shares the Hive metastore and records event logs for Spark History.

The port mappings above bind to your machine's loopback interface. The cluster has no authentication and HDFS permissions are disabled; keep it on a trusted local machine.

⁠Published images

Every tag provides both linux/amd64 (Intel/AMD) and linux/arm64 (including Apple Silicon). Docker selects your architecture automatically; no architecture suffix is needed.

Use agentcloud/dataproc:<tag>. There is no latest tag. All five tags were republished on October 8, 2026 with the same cluster startup behavior and direct python3 imports of PySpark. Pull again if you have an older copy.

The size badges show Docker Hub's compressed image size for 3.0.0-debian13, separately for each architecture. The build badge is a dated publication snapshot: both architecture manifests were confirmed on October 8, 2026. It does not report ongoing CI results.

⁠Run your application

For the cluster you started above, copy a directory containing main.py and run it:

docker cp ./my-app/. dataproc:/tmp/app
docker exec -w /tmp/app dataproc python3 main.py

A SparkSession.builder.getOrCreate() uses the configured YARN cluster, Hive catalog, and event logging. To use Spark submission options instead:

docker exec -w /tmp/app dataproc spark-submit --py-files helpers.py main.py
docker exec -w /tmp/app dataproc spark-submit --class com.example.Main app.jar

Ship modules that executors import with --py-files. Bare filesystem paths resolve against HDFS in cluster mode; use file:///tmp/app/data.csv for a file copied into this single-node container. Copied files may be read-only to the container user; write results to HDFS or /tmp.

For edits to appear immediately, mount your application directory when starting a new container:

docker run -d --name dataproc-app \
  -v "$PWD/my-app:/app:ro" \
  agentcloud/dataproc:3.0.0-debian13

# After HDFS and YARN are ready:
docker exec -w /app dataproc-app python3 main.py

⁠Spark, Hive, and Hadoop commands

docker exec -it dataproc spark-shell
docker exec -it dataproc spark-sql
docker exec dataproc spark-sql -e "SHOW DATABASES"
docker exec dataproc yarn application -list -appStates ALL
docker exec dataproc hdfs dfs -ls /

Once HiveServer2 is ready:

docker exec -it dataproc beeline -u jdbc:hive2://localhost:10000
docker exec dataproc beeline -u jdbc:hive2://localhost:10000 -e "SELECT 1"

To connect a JDBC client from your machine, add -p 127.0.0.1:10000:10000 when creating the container and connect to jdbc:hive2://localhost:10000.

⁠Run a one-off job without a cluster

Passing a command runs that capability directly without starting cluster daemons. Spark runs in local mode.

docker run --rm agentcloud/dataproc:3.0.0-debian13 spark --version
docker run -it --rm agentcloud/dataproc:3.0.0-debian13 pyspark
docker run --rm \
  -v "$PWD/my-app:/app:ro" -w /app \
  agentcloud/dataproc:3.0.0-debian13 spark main.py
docker run --rm agentcloud/dataproc:3.0.0-debian13 hadoop version
docker run --rm agentcloud/dataproc:3.0.0-debian13 self-test

Supported commands are spark (the spark-submit interface), pyspark, spark-sql, hadoop, hive, and self-test. For a regular shell, use docker run -it --rm --entrypoint bash agentcloud/dataproc:3.0.0-debian13.

⁠Install Python packages

Python and pip are included. Add packages to a running single-node cluster as root so the driver and YARN executors use the same installation:

docker exec -u 0 dataproc python3 -m pip install --no-cache-dir pandas==2.3.2

For a repeatable environment, build a derived image. Choose package versions that support the Python version of your image tag.

FROM agentcloud/dataproc:3.0.0-debian13
USER root
RUN python3 -m pip install --no-cache-dir pandas==2.3.2
USER 65532:65532

docker build -t my-dataproc .
docker run -d --name my-dataproc my-dataproc

⁠Keep data between containers

Without volumes, removing the container deletes its HDFS files and Hive tables. For a new persistent cluster, mount both data directories and keep a fixed hostname so Hive table locations remain valid:

docker run -d --name dataproc-persistent --hostname dataproc-persistent \
  -v dataproc-hdfs:/var/hadoop/hdfs \
  -v dataproc-metastore:/var/hive/metastore \
  -p 127.0.0.1:9870:9870 \
  -p 127.0.0.1:8088:8088 \
  -p 127.0.0.1:18080:18080 \
  -p 127.0.0.1:4040:4040 \
  agentcloud/dataproc:3.0.0-debian13

Stop any container already using these host ports before starting this example. Removing and recreating dataproc-persistent with the same command preserves the named volumes. Spark history logs are temporary unless you also mount /tmp/spark-events.

To stop and restart the quick-start cluster without removing its data:

docker stop dataproc
docker start dataproc

To remove it and discard data stored inside that container:

docker rm -f dataproc

⁠Troubleshooting

The runtime normally runs as the unprivileged localcloud user (UID/GID 65532). This is a local compatibility environment, not Google's managed Dataproc infrastructure.