t2fn/apache-claude-abliterated repository overview

⁠apache-claude-abliterated — Apache Spark + Pig Big Data Stack for Claude Code

Cost savings: use the tweakcc reduced system prompt + specialized Spark/Pig tools for the best results. A default Claude Code prompt injects 30K+ characters per turn -- that's $90/MTurn on Sonnet and $150 on Opus for tokens you never read. The tweakcc thinkingVerbs and ablated prompts trim this to ~10K, saving up to 90% on cached reads and ~$4.50/hour in input for a coding session at 100 requests/hour.

Drop big data scripts into a container and ask Claude to run Spark jobs, execute Pig Latin pipelines, and process datasets. This image combines Claude Code (abliterated) with Apache Spark, PySpark, Apache Pig, Hadoop client, and full environment configuration -- a complete big data development workstation optimized for disconnected enterprise environments.

⁠What It Does

This container is a focused Apache Spark + Pig development environment. Mount whatever data processing scripts, Spark apps, or Pig Latin pipelines you want to work on, start the container, and Claude automatically gets:

• Spark — spark-submit, spark-shell (Scala/Python/Java/R)
• PySpark — DataFrame API, RDD API, Spark SQL, Structured Streaming
• Pig — pig command for Pig Latin scripts (local/mapreduce modes)
• Hadoop client — hadoop fs operations for HDFS filesystem commands
• Pre-configured environment — SPARK_HOME, PIG_HOME, JAVA_HOME, PYSPARK_PYTHON set at runtime
• Post-stop cleanup — automatic removal of /tmp/spark-* and shuffle artifacts
• Abliterated Claude — lean ~10K system prompt, 200+ skills, subagent support, and low token cost

No configuration, no setup. Give it a Spark or Pig script and ask Claude to run, process, and ship.

# First, navigate to your project directory.
# cd $HOME/git/my-project

# Run a Spark job with Claude
docker run -u $(id -u):$(id -g) -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest

# Or with podman:
podman run -u $(id -u):$(id -g) --userns=keep-id \
    -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest

# Then send a prompt:
# "Run this PySpark word count, then process the results with Pig"

# Spark job with local cluster mode:
docker run -u $(id -u):$(id -g) -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest spark-submit --master local[4] script.py

# Or with podman:
podman run -u $(id -u):$(id -g) --userns=keep-id \
    -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest spark-submit --master local[4] script.py

# With a local Ollama model:
docker run -u $(id -u):$(id -g) -e OLLAMA_MODEL=gemma4:31b \
    -e OLLAMA_HOST=10.12.2.4 \
    -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest

# Or with podman:
podman run -u $(id -u):$(id -g) --userns=keep-id \
    -e OLLAMA_MODEL=gemma4:31b \
    -e OLLAMA_HOST=10.12.2.4 \
    -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest

# Claude-tools shell — invoke a shell with claude-shell capabilities
docker run -u $(id -u):$(id -g) -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest /bin/bash

# Or with podman:
podman run -u $(id -u):$(id -g) --userns=keep-id \
    -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest /bin/bash

# With privileges (for HDFS access or shared memory):
docker run --privileged -u $(id -u):$(id -g) -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest

# Or with podman (no --userns flag needed in privileged mode):
podman run --privileged -u $(id -u):$(id -g) \
    -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest

Claude connects to the SparkSession, runs spark-submit with local mode, traces DataFrame operations, and returns a natural-language report -- all from one prompt.

With the /bin/bash argument, the container starts an interactive shell where claude-tools are available on $PATH and Claude Code is configured for development. From the shell you can:

• Run claude-fix to analyze Spark errors and propose fixes
• Run claude-bash for bash sessions with smarter timeouts
• Run claude-agent to launch Claude as a persistent agent
• Send slash commands (/fix, /ask, /plan) directly to Claude
• Use claude-fix --comment to post inline PR comments or --fix to apply findings directly

For claude-tools examples and full reference, see mijuny/claude-tools⁠.

⁠Why This Container?

⁠Big data development without the workstation

Setting up Spark + Pig typically means installing JDK 21, downloading Spark tarballs, configuring SPARK_HOME, setting up Hadoop client libs, and managing multiple environment variables. This container bundles it all and makes it accessible to Claude via a single docker run prompt.

⁠What Claude can do

The container works with any big data project under /workdir:

• PySpark scripts — DataFrame API (recommended), RDD API, Spark SQL queries
• Pig Latin pipelines — local mode (pig -x local), mapreduce mode (pig -x mapreduce)
• Interactive development — spark-shell, pyspark REPLs for experimentation
• Data processing patterns — load, filter, transform, group, aggregate, sort, join, save
• Multiple data source formats — CSV, JSON, Parquet, ORC, Text, JDBC
• Cluster deployment modes — local, standalone, YARN, Kubernetes, Mesos
• HDFS operations — hadoop fs -ls, -put, -get for filesystem management
• Post-stop cleanup — automatic removal of Spark worker logs and shuffle files

⁠Toolkit

⁠Spark Tools

⁠Spark Cluster Modes

⁠Data Source Formats Supported

⁠Post-Stop Cleanup

On container stop:

• /tmp/spark-* worker logs and temp directories are removed
• Shuffle intermediate files (/tmp/shuff*) cleaned up
• Hadoop temporary directories (/tmp/hadoop-${USER}) removed
• Pig output directories (/tmp/pig_*) cleaned up
• spark-submit, pyspark, pig, Spark worker JVM processes killed by user

⁠Environment Variables

⁠claude-tools

Extends Claude Code with specialized command-line tools. Available scripts are automatically in $PATH and activate via slash commands or Claude's built-in tool use:

claude-fix    # Fix code issues with context-aware edits
claude-ask    # Ask questions, get answers without side effects
claude-bash   # Run bash with smarter timeouts and session persistence
claude-agent  # Launch Claude as a persistent agent

To invoke the claude-tools development shell, add /bin/bash to your docker run or podman run command. This starts an interactive shell where claude-tools are available on $PATH and Claude Code is configured for development:

# claude-tools development shell with Docker
docker run -u $(id -u):$(id -g) -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest /bin/bash

# Or with podman:
podman run -u $(id -u):$(id -g) --userns=keep-id \
    -e ANTHROPIC_API_KEY=sk-... \
    -v "$PWD:/workdir:rw" \
    t2fn/apache-claude-abliterated:latest /bin/bash

From the shell you can run claude-fix, claude-ask, claude-bash, claude-agent, and send slash commands (/fix, /ask, /plan) directly to Claude. Use claude-fix --comment to post inline PR comments or --fix to apply findings directly.

More examples: mijuny/claude-tools/tree/main/README.md⁠

⁠Cost Savings with tweakcc

This container runs on claude-abliterated:rocky10, which uses the tweakcc (tweakcc-fixed) reduced system prompt to cut ~67% off daily-flow prompts -- from ~30K to ~10K characters. For big data development, this means:

• Focused tools, focused rules — Spark/Pig-specific rules are loaded on-demand, not every turn
• tweakcc thinkingVerbs — 207 safe movement-only verbs reduce cognitive drift on every turn
• Cost-per-hour comparison: ~$4.50/hour savings on input tokens alone for a typical 100-request/hour session
• Better output — fewer contradictory always-on rules means less "overcorrection" waste on each turn

⁠Trademark and Legal Notice

"Claude" and "Claude Code" are registered trademarks of Anthropic, PBC. This project uses the name "Claude" in a nominative (descriptive) sense -- it identifies the Anthropic product (Claude Code) that this project is built to work with, consistent with the nominative fair use doctrine. The use is necessary: a third-party project cannot refer to "Claude Code" without using the mark itself.

This project is not affiliated with, endorsed by, or sponsored by Anthropic, PBC. The name "apache-claude-abliterated" is a third-party project name where:

• "apache" — descriptive reference to Apache Spark and Apache Pig (nominative fair use)
• "claude" — descriptive reference to Anthropic's Claude Code (nominative fair use)
• "abliterated" — coined portmanteau ("ablation" + "obliteration") representing the reduction of system prompt bloat

The project uses both "Apache" and "Claude" to describe what it works with, not who it is from.

"Spark", "Apache Spark" are trademarks of The Apache Software Foundation. References to Spark in this document are nominative -- they describe the technology this container works with, not endorsement by the ASF.

All other trademarks belong to their respective owners. No implication of endorsement is made by the use of these marks.

⁠Related Containers

⁠Open Source

Source code and suggestions/issues/pull requests can be made at:

https://github.com/t2fn/claude-abliterated⁠