quantiphinvidiapractice/nemo repository overview

To use this Multi-GPU demo in the SLURM environment.

Step 1: Make sure you have the docker image ready.

        "docker pull quantiphinvidiapractice/nemo:v1"

Step 2: Make sure you have a running slum cluster.

       "sinfo"

Step 3: create a new shell script to schedule this as a job on slurm cluster using your favorite editor.

       "vi gpt.sh"

Step 4: update the newly created file with the following content

" #!/bin/bash #SBATCH -J nemogpt #SBATCH -N 1 #SBATCH --gres=gpu:2 #SBATCH -o slurm.%N.%J.%u.out # STDOUT #SBATCH -e slurm.%N.%J.%u.err # STDERR

module load CUDA

docker run quantiphinvidiapractice/nemo:v1 bash -c 'cd ../.. && python3 transformers/src/transformers/models/megatron_gpt2/convert_megatron_gpt2_checkpoint.py megatroegatron_lm_345m_v0.0.zip && python3 question_answering.py' --gpus 2

"

Step 5: save the file and schedule a job using sbatch command.

      "sbatch gpt.sh"

Step 6: To check the queued job status use squeue command

       "squeue"

Step 7: Refer to the generated files "slurm..out" and "slurm..err" to check the logs/output results or errors.