Modify rbgcsail/diffdock so that it can run on HTCondor.
177
results.tar which is transferred back to the user at the end of the run.Using a container on a Linux cluster under HTCondor scheduler limits the ability to "do stuff" taken for granted when running "locally" and directly.
The image rbgcsail/diffdock listed in https://github.com/gcorso/DiffDock is not conducive to using on a HTcondor system.
List of a few reasons it fails within the context of running onto HTCondor:
/bin/bash$HOME is /home/appuser and in the original image/container is not searchable due to permissions. In rbgcsail/diffdock that directory is set as:ls -ld /home/appuser/
drwxr-x---. 1 appuser appuser 54 Sep 4 12:19 /home/appuser/
Since HTcondor user is not defined, the fact that there are 3 --- for non-user and non-group makes that the micromamba binary located in /home/appuser/bin/micromamba is not permitted access. Thus the environment cannot be started.
/home/appuser/DiffDock which is not writable in the HTCondor run. Since the directory is small this problem can be bypassed by making a copy within the HTcondor ${PWD} working directory.$HOME/.cache and therefore causes a non-writable error. A FIX can be to add export TORCH_HOME=${PWD} within the shell script that is run.export PYTORCH_KERNEL_CACHE_PATH=${PWD} seems to work.The last error to fix are in the reported errors file below. One is about Biopython deprecation. The other is more serious about "RuntimeWarning:" and may be due to various problems. (white lines added for readability)
/home/appuser/micromamba/envs/diffdock/lib/python3.9/site-packages/Bio/pairwise2.py:278: BiopythonDeprecationWarning: Bio.pairwise2 has been deprecated, and we intend to remove it in a future release of Biopython. As an alternative, please consider using Bio.Align.PairwiseAligner as a replacement, and contact the Biopython developers if you still need the Bio.pairwise2 module.
warnings.warn(
Downloading: "https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t33_650M_UR50D.pt" to /var/lib/condor/execute/slot1/dir_2888354/hub/checkpoints/esm2_t33_650M_UR50D.pt
Downloading: "https://dl.fbaipublicfiles.com/fair-esm/regression/esm2_t33_650M_UR50D-contact-regression.pt" to /var/lib/condor/execute/slot1/dir_2888354/hub/checkpoints/esm2_t33_650M_UR50D-contact-regression.pt
0it [00:00, ?it/s]/var/lib/condor/execute/slot1/dir_2888354/DiffDock/datasets/parse_chi.py:91: RuntimeWarning: invalid value encountered in cast
Y = indices.astype(int)
2it [01:28, 44.33s/it]
Interestingly the same error occurs when running as CPU only on a Macintosh, but the computation still continues: (white lines added for readability)
$ python -m inference --config default_inference_args.yaml --protein_ligand_csv data/protein_ligand_example.csv --out_dir results/user_predictions_small
/home/appuser/micromamba/envs/diffdock/lib/python3.9/site-packages/Bio/pairwise2.py:278: BiopythonDeprecationWarning: Bio.pairwise2 has been deprecated, and we intend to remove it in a future release of Biopython. As an alternative, please consider using Bio.Align.PairwiseAligner as a replacement, and contact the Biopython developers if you still need the Bio.pairwise2 module.
warnings.warn(
Generating ESM language model embeddings
Downloading: "https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t33_650M_UR50D.pt" to /home/appuser/.cache/torch/hub/checkpoints/esm2_t33_650M_UR50D.pt
Downloading: "https://dl.fbaipublicfiles.com/fair-esm/regression/esm2_t33_650M_UR50D-contact-regression.pt" to /home/appuser/.cache/torch/hub/checkpoints/esm2_t33_650M_UR50D-contact-regression.pt
Processing 1 of 1 batches (4 sequences)
0it [00:00, ?it/s]/home/appuser/DiffDock/datasets/parse_chi.py:91: RuntimeWarning: invalid value encountered in cast
Y = indices.astype(int)
[2024-Oct-18 21:31:25 UTC] WARNING -Complex ['1a0q'] Batch 1 Inference Iteration 9: 1 / 10 samples failed
2it [40:41, 1220.53s/it]
In order to fix the access to the user $HOME (see above) the image was recreated. While at it some utilities were added: git, unzip and nano. The Dockerfile was:
FROM --platform=linux/amd64 docker.io/rbgcsail/diffdock
# Switch to root user
USER root
# Perform your modifications here
RUN apt-get update && apt-get install -y git unzip zip nano
RUN chmod -R a+rx /home/appuser
# Switch back to appuser
USER appuser
ENTRYPOINT ["/bin/bash"]
The shell has to be activated for micromamba to take effect, in spite of the information within .bashrc.
The following are the latest attempt to running this in HTCondor. Below are the Shell and Submit files.
universe = docker
# docker_image = nvidia/cuda:11.7.1-devel-ubuntu22.04
#docker_image = rbgcsail/diffdock
docker_image = jysgro/diffdock:appuser
executable = diff.sh
transfer_input_files = diff.sh
run_as_owner = True
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
output = nvidia_smi_output.$(Cluster).$(Process).out
error = nvidia_smi_error.$(Cluster).$(Process).err
log = nvidia_smi_log.$(Cluster).$(Process).log
requirements = (HasGpulabData == true)
request_GPUs = 1
+WantGPULab = true
request_cpus = 4
request_memory = 100Gb
request_disk = 100Gb
queue 1
This is the latest version...
The "Runtime error" is in fact not preventing the computation to finish.
I add set commands at the top that can be commented out that can help debugging the process.
#!/bin/bash
set +e # Continue running even if there is an error
set -x # Enable debug mode
set -o pipefail # returns the exit status of the last command in the pipe that failed, rather than the exit status of the last command.
echo pwd
pwd
echo whoami
whoami
echo make current dir var
echo export HTDIR=$PWD
export HTDIR=$PWD
echo HTDIR is:
echo $HTDIR
echo WHAT do I SEE HERE in PWD/HTDIR:
echo ls -lh
ls -lh
echo HOME:
echo $HOME
echo ------TORCH-----
echo to avoid
echo "PermissionError: [Errno 13] Permission denied: '/home/appuser/.cache/torch'"
echo try export TORCH_HOME=${PWD}
export TORCH_HOME=${PWD}
echo and also the following that might help
mkdir local_cache
export MPLCONFIGDIR=./local_cache
echo avoid another
echo "DiffDock/utils/geometry.py:272: UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is /home/appuser/.cache/torch/kernels. This warning will appear only once per process. (Triggered internally at ../aten/src/ATen/native/cuda/jit_utils.cpp:1443.)"
echo Copilot suggestion is
export PYTORCH_KERNEL_CACHE_PATH=${PWD}
echo ------TORCH DONE-----
echo which micromamba
which micromamba
echo ls -l /home/appuser/bin/micromamba
ls -l /home/appuser/bin/micromamba
echo to avoid error: critical libmamba Shell not initialized
echo source /home/appuser/.bashrc
source /home/appuser/.bashrc
echo BUT ALSO, based on error info:
eval "$(micromamba shell hook --shell bash)"
micromamba activate
echo and then:
echo micromamba activate diffdock
micromamba activate diffdock
echo
echo make expected result directory
echo mkdir -p results/user_predictions_small
mkdir -p results/user_predictions_small
# echo FIX error: /home/appuser/micromamba/envs/diffdock/bin/python: No module named inference
# echo follow Copilot suggestions
micromamba list > micromamba_list.txt
# echo install with micromamba install -c conda-forge inference
# micromamba install -c conda-forge inference
echo LOCATED at: /home/appuser/DiffDock/inference.py
echo FIX many problems by making a copy of DiffDock in PWD/HTDIR
cp -r /home/appuser/DiffDock/ .
cd DiffDock
echo run python command
echo python -m inference --config default_inference_args.yaml --protein_ligand_csv data/protein_ligand_example.csv --out_dir results/user_predictions_small
python -m inference --config default_inference_args.yaml --protein_ligand_csv data/protein_ligand_example.csv --out_dir results/user_predictions_small
echo tar results
echo tar cvf results.tar results/user_predictions_small/*
tar cvf results.tar results/user_predictions_small/*
echo move results.tar to HTDIR
mv results.tar $HTDIR
echo DONE
Content type
Image
Digest
sha256:5cbd06bc5…
Size
4 GB
Last updated
almost 2 years ago
docker pull jysgro/diffdock:appuser