Public | Automated Build

Last pushed: a year ago
Short Description
Pegasus and HTCondor on Ubuntu latest.
Full Description

dispel4py and Pegasus docker image

To build the image:

git clone https://github.com/pegasus-isi/pegasus.git
docker build -t pegasus_dispel4py .
docker run -i -t pegasus_dispel4py bash

Inside the container:

su tutorial
cd pegasus_dispel4py/simple_experiment/
python dax-dispel4py.py
pegasus-plan --conf pegasusrc --dax dispel4py.dax --sites local --submit

Then watch progress of the job using the displayed job id, e.g.:

pegasus-status -l /home/tutorial/pegasus_dispel4py/simple_experiment/tutorial/pegasus/dispel4py/20160318T215222+0000

Run a cluster with Storm

Clone the docker composition:

git clone https://github.com/dispel4py/storm-docker.git
docker-compose up -d
docker-compose scale supervisor=3

Note: This Storm-cluster has installed dispel4py, numpy, scipy, networkx, obspy python libraries

Then start the container with dispel4py, Storm client and Pegasus:

docker-compose -p pegasus -f ./docker-pegasus.yml up -d

Log into the running Pegasus container:

docker exec -i -t pegasus_pegasus_1 bash

Run a cluster with MPI (OpenMPI + mpi4py)

Clone the docker composition:

git clone https://github.com/dispel4py/docker.openmpi.git
docker-compose scale mpi_node=3 mpi_head=1

Check the port that mpi_head is using:

docker ps

Now you know the port, you can login to the mpi_head container. The username is mpirun.
It has been configured mpriun user to need password to login by ssh. However, id_rsa.mpi key might need to be indicated.

chmod 400 ssh/id_rsa.mpi
ssh -i ssh/id_rsa.mpi -p 23227 tutorial@localhost

For testing an mpi4py example using the mpi_nodes:

cd mpi4py_benchmarks
create machines file from /etc/hosts (copy only the IP adresses, nothing else)
mpiexec -hostfile machines -n 3 python helloworld.py       

For testing dispel4py with mpi mapping:

mpiexec -n 6 -hostfile machines dispel4py mpi dispel4py.examples.graph_testing.pipeline_test    

Note: This MPI-cluster has installed dispel4py, numpy, scipy, networkx, obspy python libraries

Then start the container with dispel4py, Storm client and Pegasus:

docker-compose -p pegasus -f ./docker-pegasus.yml up -d

Log into the running Pegasus container:

docker exec -i -t pegasus_pegasus_1 bash

Inside the Pegasus container, you can connect to the MPI-cluster like this:

ssh -i docker.openmpi/ssh/id_rsa.mpi mpirun@mpi_head
Docker Pull Command
Owner
dispel4py
Source Repository

Comments (1)
andypohl
a year ago

It's a great start (I'm happy to have such a quick personal condor going), but I'm having problems. In the split tutorial (from pegasus-init), the

dax.metadata("creator", "%s@%s" % (os.getlogin(), os.uname()[1]))

line in daxgen.py fails because os.getlogin() fails. Just try launching python and

import os
os.getlogin()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OSError: [Errno 2] No such file or directory

It should be returning 'tutorial' but it can't figure this out.