chembl/lsf-gpu
TensorFlow and PyTorch accessing the GPU hardware in the LSF cluster using singularity
Image
0
243
chembl/lsf-gpu repository overview
Docker image with PyTorch and TensorFlow using GPU
Example that shows TensorFlow and PyTorch accessing the GPU hardware in the LSF cluster using singularity
build the image
docker build -t chembl/lsf-gpu .
docker push chembl/lsf-gpu
CUDA Drivers
CUDA drivers, as shown in this image, are installed in the host running the containers (i.e. cluster node). You won't need to install the drivers in your image but you'll need to install the appropiate CUDA toolkit on it.
Check if TF and PT can actually see the GPU's
bsub -P gpu -gpu - -Is "singularity exec --nv docker://chembl/lsf-gpu:latest python -c 'import tensorflow as tf; print(tf.test.is_gpu_available())'"
bsub -P gpu -gpu - -Is "singularity exec --nv docker://chembl/lsf-gpu:latest python -c 'import torch; print(torch.cuda.is_available())'"
Want to use more resources? 4 CPU (extra CPU's can be used to prefetch/preprocess data while GPU is crunching data) and more RAM
bsub -P gpu -gpu - -M 8G -R"rusage[numcpus=4,mem=8192]" -Is "singularity exec --nv docker://chembl/lsf-gpu:latest python -c 'import torch; print(torch.cuda.is_available())'"
Is your image failing to extract?
Creating container runtime...
Exploding layer:
sha256:898c46f3b1a1f39827ed135f020c32e2038c87ae0690a8fe73d94e5df9e6a2d6.tar.gz
Exploding layer:
sha256:63366dfa0a5076458e37ebae948bc7823bab256ca27e09ab94d298e37df4c2a3.tar.gz
Exploding layer:
sha256:041d4cd74a929bc4b66ee955ab5b229de098fa389d1a1fb9565e536d8878e15f.tar.gz
Exploding layer:
sha256:6e1bee0f8701f0ae53a5129dc82115967ae36faa30d7701b195dfc6ec317a51d.tar.gz
Exploding layer:
sha256:a2dd35f108bdaec1be116597357e7f38e04f6580d4c8af770c4930419c23468c.tar.gz
Exploding layer:
sha256:84a4bd4b97ae50f587cb22af0f92eb029a7a604db5b68c451698ed38e6a5fc36.tar.gz
Exploding layer:
sha256:d8fd24877863500391f2a42f70b69e5e4ee828246f6fcfe8041da421d606a35d.tar.gz
Exploding layer:
sha256:24e7c21cfefc8ae6dd37e557d6a6ab648a7564328acf39b74b21fb0df432e2ab.tar.gz
[91mERROR : tar extraction error: Write failed
[0m [33mWARNING: Warning handling tar header: Write failed
[0m [33mWARNING: Warning handling tar header: Can't create 'opt/conda/
pkgs/libtiff-4.0.10-h2733197_2'
[0m [33mWARNING: Warning handling tar header: Can't create 'opt/conda/
pkgs/libtiff-4.0.10-h2733197_2/bin'
[0m [33mWARNING: Warning handling tar header: Can't create 'opt/conda/
pkgs/libtiff-4.0.10-h2733197_2/bin/fax2ps'
[0m [33mWARNING: Warning handling tar header: Can't create 'opt/conda/
pkgs/libtiff-4.0.10-h2733197_2/bin/fax2tiff'
[0m [33mWARNING: Warning handling tar header: Can't create 'opt/conda/
pkgs/libtiff-4.0.10-h2733197_2/bin/pal2rgb'
[0m [33mWARNING: Warning handling tar header: Can't create 'opt/conda/
pkgs/libtiff-4.0.10-h2733197_2/bin/ppm2tiff’
Add the following lines to your .bash_profile to fix it.
export SINGULARITY_CACHEDIR=/hps/nobackup2/singularity/<your_username>
export SINGULARITY_LOCALCACHEDIR=/scratch
export SINGULARITY_TMPDIR=/scratch
More docs on how to use the LSF cluster with singularity:
Tag summary
Content type
Image
Digest
Size
3.5 GB
Last updated
about 6 years ago
docker pull chembl/lsf-gpu