clowder/extractors-littlebeats repository overview

⁠Submit Littlebeats Cleaning job to HPC cluster

This particular extractor submits a job to execute Littlebats cleaning job(https://github.com/littlebeats/LittleBeats_Data_Pipeline⁠) on CampusCluster⁠. It copies the input file from Clowder to Campus Cluster, submits job to Campus cluster and transfers the output file(s) to Clowder.

⁠Docker

This extractor is ready to be run as a docker container, the only dependency is a running Clowder instance. Simply build and run.

1. Start Clowder. For help starting Clowder, see our getting started guide⁠.

2. First build the extractor Docker container:

# from this directory, run:

docker build -t clowder_littlebeats .

1. Finally run the extractor:

docker run -t -i --rm --net clowder_clowder -e "RABBITMQ_URI=amqp://guest:guest@rabbitmq:5672/%2f" --name "littlebeats" clowder_littlebeats

Then open the Clowder web app and run the littlebeats extractor on a .txt file (or similar)! Done.

⁠Python and Docker details

You may use any version of Python 3. Simply edit the first line of the Dockerfile, by default it uses FROM python:3.8.

Docker flags:

• --net links the extractor to the Clowder Docker network (run docker network ls to identify your own.)
• -e RABBITMQ_URI= sets the environment variables can be used to control what RabbitMQ server and exchange it will bind itself to. Setting the RABBITMQ_EXCHANGE may also help.
  • You can also use --link to link the extractor to a RabbitMQ container.
• --name assigns the container a name visible in Docker Desktop.

⁠Troubleshooting

If you run into any trouble, please reach out on our Clowder Slack in the #pyclowder channel⁠.

Alternate methods of running extractors are below.

⁠Commandline Execution

To execute the extractor from the command line you will need to have the required packages installed. It is highly recommended to use python virtual environment for this. You will need to create a virtual environment first, then activate it and finally install all required packages.

virtualenv /home/clowder/virtualenv/littlebeats
. /home/clowder/virtualenv/littlebeats/bin/activate
pip install -r /home/clowder/extractors/littlebeats/requirements.txt

To start the extractor you will need to load the virtual environment and start the extractor.

. /home/clowder/virtualenv/littlebeats/bin/activate
/home/clowder/extractors/littlebeats/submitCleaningJob.py