krohling/nl-act repository overview

⁠Integrating Natural Language Instructions into the Action Chunking Transformer for Multi-Task Robotic Manipulation

For more details on this project visit the nl-act⁠ Github repo or read the paper⁠.

⁠Overview

This Docker image provides a reproducible environment for training and evaluating the Action Chunking Transformer (ACT) model, extended to support multi-task robotic manipulation based on natural language instructions. The container is designed to simplify setup and streamline experimentation by pre-loading all necessary dependencies, scripts, and datasets.

Key features include:

• Integration of natural language embeddings into the ACT architecture.
• Fully-configured simulation environment using MuJoCo and dm_control.
• GPU-ready for efficient training and evaluation.
• Pre-built datasets hosted on Hugging Face, downloaded automatically when training.
• Weights & Biases (W&B) integration for logging metrics, checkpoints, and rollout videos.

This image is particularly useful for running on GPU-enabled cloud services like RunPod⁠ or on local GPU-equipped systems.

---

⁠Usage Example

1. Pull the Docker Image:

   docker pull krohling/nl-act
   

   Copy

2. Run the Container: Example command to train the ACT model with GPU support:

   docker run -it \
       --gpus all \
       -v $(pwd)/output:/opt/ml/output \
       -e WANDB_PROJECT="your_wandb_project" \
       -e WANDB_ENTITY="your_wandb_entity" \
       -e WANDB_API_KEY="your_wandb_api_key" \
       krohling/nl-act
   

   Copy

Replace your_wandb_project, your_wandb_entity, and your_wandb_api_key with your Weights & Biases credentials if you want to log training metrics.

⁠Environment Variables

Below is a list of customizable environment variables to control training, evaluation, and dataset paths. While the container will run with default settings, you can adjust these to suit your environment and hardware. It is strongly recommended that you modify the batch size to fit your GPU.

⁠Datasets

This project uses two pre-generated datasets, hosted on Hugging Face, for training and evaluation. Each dataset includes 150 multi-task episodes (grasp, stack, transfer) in .hdf5 format along with corresponding instruction embeddings.

1. act-grasp-stack-transfer⁠:

Contains 150 episodes (50 per task) without noise injection. Total size: ~75GB.

1. act-grasp-stack-transfer-noisy⁠:

Same tasks, but includes trajectory noise injection. Total size: ~75GB.

The container will automatically download these datasets.