dataspree/training repository overview

⁠Data Spree Training

The Data Spree Training Worker is the essential part of Data Spree's AI Platform that is used to train Deep Neural Networks. The Worker connects to the AI Platform from where it downloads the training parameters and data. Training jobs get assigned to the Worker which typically executes these jobs in the background. Learn more on the platform here: https://docs.data-spree.com/ai-platform⁠

The Worker requires an Nvidia GPU and the Nvidia Container Toolkit (https://github.com/NVIDIA/nvidia-docker⁠). We recommend a GPU with at least 8GB of memory.

Create a container for the Worker in the background mapping the relevant directories:

docker run -d --gpus all --ipc=host -t --restart unless-stopped --name ds-worker --runtime nvidia -p 6714:6714 -v $(pwd)/training:/root/.dataspree/ -v $(pwd)/training:/data dataspree/training:latest

Once you've started the Worker, you need to configure it. The easiest way is to use docker:

docker run --rm -it --network host --entrypoint /bin/sh dataspree/training:latest -c "ds worker setup"

Alternatively, you can directly use the dataspree-platform-sdk which is a Python package including a command-line interface to configure the worker:

pip install dataspree-platform-sdk
ds worker setup

To start or stop the container, you can use the following commands:

docker start ds-worker
docker stop ds-worker

View the logs of the running container via:

docker logs -n 100 -f ds-worker