Artemisa Usage Guide

==========================================================

Welcome to 
    _         _                 _           
   / \   _ __| |_ ___ _ __ ___ (_)___  __ _ 
  / _ \ | '__| __/ _ \ '_ ` _ \| / __|/ _` |
 / ___ \| |  | ||  __/ | | | | | \__ \ (_| |
/_/   \_\_|   \__\___|_| |_| |_|_|___/\__,_|
                                            
The IFIC's AI and ML infraestructure

This is mlui02.ific.uv.es running CentOS 7.9.2009
Please,   read the "Politica de Seguridad   de la Informacion"

https://artemisa.ific.uv.es/static/Politica_Seguridad.pdf

==========================================================
**ATTENTION**
CUDA Version: 12.0  Driver Version: 525.85.12
Execute GPUstatus to get a summary on GPU utilization
==========================================================

/lhome/ific/<initial_letter>/<username>   <= for IFIC users
/lhome/ext/<groupid>/<username>  <= for external users

/lustre/ific.uv.es/ml/<groupid>

$ gpurun -i
Connected
Info: OK 0. Tesla V100-PCIE-16GB [00000000:5E:00.0]
1. Tesla V100-PCIE-16GB [00000000:86:00.0]

Total clients:0 Running:0 Estimated waiting time:0 seconds

universe = vanilla

executable          = test.sh
arguments           = $(Cluster) $(Process)

log                 = test.log
output              = outfile.$(Cluster).$(Process).out
error               = errors.$(Cluster).$(process).err

request_Cpus        = 4
request_Memory      = 4000

queue 2

$ condor_submit test.sub
Submitting job(s)..
2 job(s) submitted to cluster 110.

$ condor_q

-- Schedd: xxx.ific.uv.es : <xx.xx.xx.xx:9618?... @ 03/12/19 11:44:28
OWNER    BATCH_NAME    SUBMITTED   DONE   RUN    IDLE  TOTAL JOB_IDS
fulano ID: 110      3/12 11:44      _      _      2      2 110.0-1

Total for query: 2 jobs; 0 completed, 0 removed, 2 idle, 0 running, 0 held, 0 suspended 
Total for fulano: 2 jobs; 0 completed, 0 removed, 2 idle, 0 running, 0 held, 0 suspended 
Total for all users: 2 jobs; 0 completed, 0 removed, 2 idle, 0 running, 0 held, 0 suspended

$ condor_rm 110.1
Job 110.1 marked for removal

$ condor_status
Name                    OpSys      Arch   State     Activity LoadAv Mem     ActvtyTime

slot1@mlwn01.ific.uv.es LINUX      X86_64 Unclaimed Idle      0.000  16384  7+20:20:48
slot2@mlwn01.ific.uv.es LINUX      X86_64 Unclaimed Idle      0.000 134964  7+20:21:04
slot3@mlwn01.ific.uv.es LINUX      X86_64 Claimed   Busy      1.030 231368  0+00:06:00
slot1@mlwn02.ific.uv.es LINUX      X86_64 Unclaimed Idle      0.000  16384  7+20:21:07
slot2@mlwn02.ific.uv.es LINUX      X86_64 Unclaimed Idle      0.000 134964  7+20:21:18
slot3@mlwn02.ific.uv.es LINUX      X86_64 Unclaimed Idle      0.000 231368  5+19:46:37
slot1@mlwn03.ific.uv.es LINUX      X86_64 Unclaimed Idle      0.000 309074  7+20:17:45
slot2@mlwn03.ific.uv.es LINUX      X86_64 Unclaimed Idle      0.000 463611  7+20:17:57

               Machines Owner Claimed Unclaimed Matched Preempting  Drain

  X86_64/LINUX        9     0       1         8       0          0      0

         Total        9     0       1         8       0          0      0

$ mkdir ~/s.images
$ cd ~/s.images
$ singularity build tensorflow-nightly-gpu docker://tensorflow/tensorflow:nightly-gpu

export LCGENV_PATH=/cvmfs/sft.cern.ch/lcg/releases/
eval "` $LCGENV_PATH/lcgenv/latest/lcgenv -p LCG_93 x86_64-slc6-gcc62-opt ROOT `"
eval "` $LCGENV_PATH/lcgenv/latest/lcgenv -p LCG_93 x86_64-slc6-gcc62-opt GSL `"
eval "` $LCGENV_PATH/lcgenv/latest/lcgenv -p LCG_93 x86_64-slc6-gcc62-opt Boost`"

/cvmfs/atlas.cern.ch
/cvmfs/lhcb.cern.ch

git clone https://igit.ific.uv.es/alferca/artemisa-tutorial

Development on User Interface

Abstract

• Hands on development on a User Interface (UI)
• Write small test program using TensorFlow^?
• Custom local environments ( venv / conda / pip )
• Interactive execution on the UI
• GPURUN tool to gain exclusive access to the UI GPU

Introduction

We use a basic Tensorflow example for development and using gpus as described here: https://www.tensorflow.org/guide/gpu

- Create a python file named 'tf2_helloworld.py' :

 
from __future__ import print_function
import tensorflow as tf

#Tensor allocations or operations to be printed
tf.debugging.set_log_device_placement(True)

# Tensorflow version
print("This is Tensorflow: ", tf.__version__)

# Check GPUs
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))

# Creates a graph.
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
# Display
tf.print(c)

- Try to execute the program

Run the program with python3, and will use the system installation of python 3 and Tensorflow.

$ python3 tf2_helloworld.py 
This is Tensorflow:  1.14.0
Traceback (most recent call last):
  File "tf2_helloworld.py", line 11, in <module>
    print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
  File "/lhome/ific/a/alferca/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation_wrapper.py", line 106, in __getattr__
    attr = getattr(self._dw_wrapped_module, name)
AttributeError: module 'tensorflow._api.v1.config' has no attribute 'list_physical_devices'

If you require other versions than provided by the system, you can use a custom local environment with you own software installation. Check next step.

Custom Local Environment

- We can use different tools for creating a custom local environment

Conda (https://conda.io) : Conda easily creates, saves, loads and switches between environments on your local computer. Conda as a package manager helps you find and install packages. If you need a package that requires a different version of Python, you do not need to switch to a different environment manager, because conda is also an environment manager. In its default configuration, conda can install and manage the thousand packages at repo.anaconda.com that are built, reviewed and maintained by Anaconda

Venv (https://docs.python.org/3/library/venv.html) : The venv module provides support for creating lightweight “virtual environments” with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories.

PIP (https://pypi.org/project/pip/) : pip is the package installer for Python. You can use pip to install packages from the Python Package Index and other indexes.

- In this case we want to install another python and tensorflow version. At the time of writing this tutorial this is Tensorflow v2.6.0

Follow the instructions of Conda to install its distribution locally: https://docs.conda.io/en/latest/miniconda.html#linux-installers

Use conda to create a virtual environment. We install required packages (cudnn) but not yet TensorFlow^? as the time of writing this tutorial the relase included in Conda do not support GPU

$ conda create -n artemisa-tutorial cudnn

Activate the environment

$ conda activate artemisa-tutorial

Install PIP within conda then Install TensorFlow^? (with PIP). PIP is the recommended installation to get latest release from official sources

(artemisa-tutorial) $ conda install pip
(artemisa-tutorial) $ pip install tensorflow

Execute previous program:

(artemisa-tutorial) [alferca@mlui02 01_User_Interface_Development]$  python3 tf2_helloworld.py
This is Tensorflow:  2.6.0
Num GPUs Available:  2
2021-10-27 12:55:05.008162: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-10-27 12:55:05.023252: F tensorflow/core/platform/statusor.cc:33] Attempting to fetch value instead of handling error Internal: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_INVALID_DEVICE: invalid device ordinal
Aborted

As you can see now the version used is Tensorflow: 2.6.0

The execution on local GPUs is not directly possible, and we will get errors like: CUDA_ERROR_INVALID_DEVICE

In the next section we see how to overcome this and gain interactive access to the UI GPU

Interactive GPU exclusive access: GPURUN

Access to the UI GPU is not directly accessible, and the GPURUN tool has to be used.

Basically preprend you command that access the GPU with the tool 'gpurun'

(artemisa-tutorial) $ gpurun python3 tf2_helloworld.py
First nonoption argument is "python3" at argv[1]
Connected
Info: OK 0. Tesla V100-PCIE-16GB [00000000:5E:00.0]
1. Tesla V100-PCIE-16GB [00000000:86:00.0]

Total clients:0 Running:0 Estimated waiting time:0 seconds
GPU reserved:300 seconds granted
GPUID reserved:0 Details: - Device 0. Tesla V100-PCIE-16GB [00000000:5E:00.0] set to compute mode:Exclusive Process
Info: Executing program: python3
This is Tensorflow:  2.6.0
Num GPUs Available:  1
2021-10-27 13:05:23.086416: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-10-27 13:05:23.795174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14647 MB memory:  -> device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:5e:00.0, compute capability: 7.0
2021-10-27 13:05:23.948618: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
2021-10-27 13:05:23.949655: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
2021-10-27 13:05:23.949874: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op Reshape in device /job:localhost/replica:0/task:0/device:GPU:0
2021-10-27 13:05:23.950036: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
2021-10-27 13:05:23.950149: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
2021-10-27 13:05:23.950203: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op Reshape in device /job:localhost/replica:0/task:0/device:GPU:0
2021-10-27 13:05:23.950376: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0
2021-10-27 13:05:24.427560: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op StringFormat in device /job:localhost/replica:0/task:0/device:CPU:0
2021-10-27 13:05:24.427990: I tensorflow/core/common_runtime/eager/execute.cc:1161] Executing op PrintV2 in device /job:localhost/replica:0/task:0/device:CPU:0
[[22 28]
 [49 64]]

If there are not other clients waiting for the local GPU the program will be executed immediately, otherwise will wait and print the expected waiting time:

Queued clients:0 Estimated waiting time:0 seconds

We have to bear in mind that User Interface GPUs are exclusive access (only one program can execute at a time) and so user programs will be terminated after a short limited time ( 5 minutes )

After execution we will se the 2x2 matrix result from the multiplication.

[[22 28]
 [49 64]]

the TensorFlow^? logs that give us clues where the execution has resulted: all operands a,b, and MatMult^? operation have been performed in the GPU 0, the User Interface installed Tesla V100-PCIE-16GB.

Notes

• GPURUN tool is needed to access the GPU in on a User Interface (UI)
• Custom software/environments installation can be done in the user or project directories

CPU only job: Data Preparation use case

Abstract

• Execute CPU only Jobs on the UI (without GPURUN)
• Review Keras example for Data Augmentation
• Submit a batch job with HTCondor (Job Management System)
• Execute priority job on remote Worker Node with CPU only

Introduction

The ARTEMISA cluster has some resorces reserved for CPU only jobs.

Althought modern frameworks like TensorFlow^? or Keras allow the creation of Data Pipelines to benefit in parallel from the usage of CPU (data aumentation) and GPU (training),

some tasks like Data preparation and augmentation jobs can be done in parallel using CPU only resources, in order to speed up the process.

In this example we propose a basic example to execute data preparation tasks requesting CPU only resources. We are going to build an image classifier. Because the MNIST dataset is a bit overused and too easy, we use the more challenging CIFAR-10 dataset. It consists of 32x32 pixel images with 10 classes. The data is split into 50k training and 10k test images.

First we will aument the data and store it locally. Use the assigned project space to test it.

Preparation

Activate previously created enviroment

$ conda activate artemisa-tutorial

Install needed packages

(artemisa-tutorial) $ pip install matplotlib
(artemisa-tutorial) $ pip install tensorflow-datasets
(artemisa-tutorial) $ pip install scipy

Check provided program:

(artemisa-tutorial) $ python3 tf2_keras_generate_data_CPUonly.py

This program forces the usage on the CPU, so it can be run in the UI without the gpurun tool. It will create new augmented data (images) in a subdirectory

Execute the program in a remote WORKER NODE (with HTCondor)

- Check HTCondor submission file:

(artemisa-tutorial) $ cat tf2_keras_generate_data_CPUonly.sub
universe = vanilla
executable              = tf2_keras_generate_data_CPUonly.sh
arguments               = 
log                     = condor_logs/test.log
output                  = condor_logs/outfile.$(Cluster).$(Process).out
error                   = condor_logs/errors.$(Cluster).$(Process).err

# Needed to read .bashrc and conda environment
getenv = True

# TestJob CPU
+testJob = True
queue

Note the '+testJob = True' to use exclusive CPU resources

- Check 'executable' that will be run on the Worker Node:

(artemisa-tutorial) $ cat tf2_keras_generate_data_CPUonly.sh
#!/bin/bash
EXERCISE_ENVIRONMENT="artemisa-tutorial"
eval "$(conda shell.bash hook)"
conda activate $EXERCISE_ENVIRONMENT
python3 tf2_keras_generate_data_CPUonly.py

Since the job will run on a remote Worker Node, we need to set up the same environment (conda). With this script this is done, prior to invoke our real python executable.

Submit the job (with HTCondor)

- Create the output directories (condor_logs) and submit the HTCondor job:

$ mkdir condor_logs

(artemisa-tutorial) [alferca@mlui02 02.2_Store_Data]$ condor_submit tf2_keras_generate_data_CPUonly.sub 
Submitting job(s).
1 job(s) submitted to cluster 1602

Check results

$ ls -ltr augmented_data/

Notes

• ARTEMISA has CPU only high priority resources
• Use '+testJob = True' to select the corresponding CPU only slots.
• Use 'getenv = true' to maintain the user environment when submitting with HTCondor
• When using custom environments, init the required environment in your 'executable' (ie: conda) before running the real payload.

Submit Basic GPU Production Job: Worker Nodes GPUs and Environment info

Abstract

• Submit a batch job with HTCondor (Job Management System)
• Execute on remote Worker Node with a GPU
• Obtain basic Worker Node enviroment information

Introduction

After development we can submit production jobs that use 1 or more GPU production resources with higher time limits, and not only the User Interfaces.

In this example we will submit a job requesting a single GPU and retrieving basic environment information, that can help debug multiple issues.

Preparation

- Check HTCondor submission file:

$ cat 01_test.sub

universe = vanilla

executable              = 01_test.sh
arguments               = $(Process)

log                     = condor_logs/test.log
output                  = condor_logs/outfile.$(Cluster).$(Process).out
error                   = condor_logs/errors.$(Cluster).$(Process).err

request_gpus = 1

queue

With this file we will be requesting a node with 1 GPU, and to execute our script defined in 'executable' parameter.

- Check 'executable' that will be run on the Worker Node:

$ cat 01_test.sh

#!/bin/sh
echo ">>>> ENVIRONMENT"
printenv
echo ">>>> HOST"
hostname
echo ">>>> CURRENT DIR"
pwd
echo ">>>> USER"
whoami 
echo ">>>> SPACE LEFT"
df -h
echo ">>>> NVIDIA INFO"
set -x #echo on
nvidia-smi

This shell script will retrieve the Environment variables and some information about the Worker Node where it executes. With nvidia-smi command we can obtain information about the GPUS installed on tha node.

Submit the job

- Create the output directories (condor_logs) and submit the HTCondor job:

$ mkdir condor_logs

$ condor_submit 01_test.sub
Submitting job(s).
1 job(s) submitted to cluster 548.

Check results

After submitting the job the output of the script will be recorded in the 'output' and/or 'error' files defined in the parameters. The 'log' file contains information of the HTCondor process to execute the jobs.

- 'output' file contains the standard output of your executable:

$cat condor_logs/outfile.131398.0.out

>>>> ENVIRONMENT
_CONDOR_JOB_PIDS=
TMPDIR=/var/lib/condor/execute/dir_112754
_CONDOR_ANCESTOR_4719=5207:1618416485:3499018107
_CONDOR_SCRATCH_DIR=/var/lib/condor/execute/dir_112754
_CHIRP_DELAYED_UPDATE_PREFIX=Chirp*
_CONDOR_ANCESTOR_5207=112754:1634894211:4224147812
TEMP=/var/lib/condor/execute/dir_112754
BATCH_SYSTEM=HTCondor
_CONDOR_CHIRP_CONFIG=/var/lib/condor/execute/dir_112754/.chirp.config
PWD=/lhome/ific/a/alferca/prep_tutorial_ific_cuda11.3/03_Submit_GPU_job
_CONDOR_AssignedGpus=CUDA0
CUDA_VISIBLE_DEVICES=0
_CONDOR_SLOT=slot2
_CONDOR_ANCESTOR_112754=112755:1634894211:3354999804
SHLVL=1
_CONDOR_MACHINE_AD=/var/lib/condor/execute/dir_112754/.machine.ad
TMP=/var/lib/condor/execute/dir_112754
GPU_DEVICE_ORDINAL=0
OMP_NUM_THREADS=32
_CONDOR_JOB_AD=/var/lib/condor/execute/dir_112754/.job.ad
_CONDOR_JOB_IWD=/lhome/ific/a/alferca/prep_tutorial_ific_cuda11.3/03_Submit_GPU_job
_=/usr/bin/printenv
>>>> HOST
mlwn23.ific.uv.es
>>>> CURRENT DIR
/lhome/ific/a/alferca/prep_tutorial_ific_cuda11.3/03_Submit_GPU_job
>>>> USER
alferca
>>>> SPACE LEFT
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/centos-root        49G   35G   15G  71% /
devtmpfs                      189G     0  189G   0% /dev
tmpfs                         189G  305M  188G   1% /dev/shm
tmpfs                         189G     0  189G   0% /sys/fs/cgroup
tmpfs                         189G  187M  189G   1% /run
/dev/md126p1                 1014M  207M  808M  21% /boot
/dev/mapper/centos-condor     352G   33M  352G   1% /tmp
147.156.116.235@tcp:/homefs    55T  5.0T   49T  10% /lhome
147.156.116.235@tcp:/ific2fs   26T  218M   25T   1% /lustre/ific.uv.es
147.156.116.235@tcp:/mlfs     181T   41T  139T  23% /lustre/ific.uv.es/ml
147.156.116.235@tcp:/prjfs    127T   93T   33T  74% /lustre/ific.uv.es/prj
147.156.116.235@tcp:/atl2fs   2.9P  2.4P  477T  84% /lustre/ific.uv.es/grid/atlas
147.156.116.235@tcp:/atl3fs   280T  169T  103T  63% /lustre/ific.uv.es/grid/atlas/t3
>>>> NVIDIA INFO
Fri Oct 22 11:16:51 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:18:00.0 Off |                    0 |
| N/A   36C    P0    26W / 250W |     12MiB / 32510MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

In the previous output of the job we can find some interesting information about our finished job:

Environment including variables set by HTCondor, and the CUDA visible devices to our process ( only 1 GPU as requested) Execution Machine HOST and slot ( our job was run in slot2 of machine mlwn23.ific.uv.es) Current dir is the directory from where the job is started. Storage Mounted filesystems and free space ( mounted home and project filesystems, AFS is not mounted in Worker Nodes) Nvidia GPU Info (Tesla V100-PCIE, 32GB, not running real workload in this example)

Selection of particular GPU resources: NVIDIA A100 GPUs

You can select machines for execution with a particular GPU resources, for example newest NVIDIA A100 GPUs can be selected only if you specify it in your HTCondor submission file with this clause:

+UseNvidiaA100 = True

Try to run the same test job to check environment using this clause, for example with this submission file:

$ cat 01_test_A100.sh

universe = vanilla

executable              = 01_test.sh
arguments               = $(Process)

log                     = condor_logs/test.log
output                  = condor_logs/outfile.$(Cluster).$(Process).out
error                   = condor_logs/errors.$(Cluster).$(Process).err

request_gpus = 1
+UseNvidiaA100 = True

queue

Exercise

Try to submit a previous example from this tutorial (ie: python tensorflow) to a Worker Node, and check the results.

Notes

• HTCONDOR JOB can be submitted to a Worker Node with exclusive a access to a GPU
• 'output' file records the standard output of the 'executable' defined in the submission script.
• 'error' files records the starndard error. 'log' records information about HTCondor submission status.
• clause '+UseNvidiaA100 = True' must be used to select NVIDIA A100 GPUs

Multi GPU TensorFlow^? example

Abstract

• Lambda Tensorflow benchmark
• Native / Virtual Environment with Tensorflow 2.3.1
• Usage of multiple GPUs (intra-node)
• Submission of HTCondor jobs requesting a node with the needed GPUs

Introduction

For this test we are going to use a benchmark available: https://github.com/lambdal/lambda-tensorflow-benchmark

This software uses by default installed python3 and Tensorflow installation, performing several benchmark tests in the requested gpus.

We can use the native TensorFlow^? installation, or create a virtual Environment if the required exact releases and dependencies are not installed (Tensorflow 2.3.1)

Preparation

- Use new environment

virtualenv -p /usr/bin/python3.6 venv
. venv/bin/activate

pip install matplotlib

pip install tensorflow-gpu==2.3.1

- Clone benchmark repo

git clone https://github.com/lambdal/lambda-tensorflow-benchmark.git --recursive

cd lambda-tensorflow-benchmark

- Prepare test config file:

ln -s config/config_all.sh .

Test on USER-INTERFACE

- Run a quick resnet50 test in FP32 (Note usage of GPURUN tool)

gpurun ./batch_benchmark.sh 1 1 1 100 2 config/config_resnet50_replicated_fp32_train_syn

...
2021-10-25 14:03:24.513360: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
Done warm up
Step   Img/sec   total_loss
1   images/sec: 700.7 +/- 0.0 (jitter = 0.0)   7.363 1635163410
10   images/sec: 701.6 +/- 0.9 (jitter = 3.6)   7.270 1635163412
20   images/sec: 700.5 +/- 1.5 (jitter = 2.5)   7.365 1635163415
30   images/sec: 700.8 +/- 1.1 (jitter = 2.3)   7.309 1635163418
40   images/sec: 699.9 +/- 0.9 (jitter = 2.6)   7.353 1635163421
50   images/sec: 699.6 +/- 0.8 (jitter = 2.3)   7.335 1635163423
60   images/sec: 699.3 +/- 0.7 (jitter = 2.0)   7.292 1635163426
70   images/sec: 699.5 +/- 0.6 (jitter = 1.9)   7.279 1635163429
80   images/sec: 699.6 +/- 0.6 (jitter = 1.8)   7.277 1635163432
90   images/sec: 699.6 +/- 0.5 (jitter = 1.7)   7.353 1635163434
100   images/sec: 699.8 +/- 0.5 (jitter = 1.7)   7.321 1635163437
----------------------------------------------------------------
total images/sec: 699.26
----------------------------------------------------------------

Submit MULTI-GPU JOB

- Condor submission file requesting 2 GPUs:

$ cat test-lambda-tensorflow-benchmark_venv.sub
universe = vanilla

executable              = 01_test_benchmark_venv.sh
arguments               =
log                     = condor_logs/test_venv.log
output                  = condor_logs/outfile_venv.$(Cluster).$(Process).out
error                   = condor_logs/errors_venv.$(Cluster).$(Process).err

request_gpus = 2
queue

- Executable to activate Virtual Enrironment and run benchmark:

$ cat 01_test_benchmark_venv.sh
#!/bin/bash 
echo ">>>>> ACTIVATE VIRTUALENV TensorFlow 2.3.1"
. venv/bin/activate
echo ">>>>> EXECUTE PAYLOAD"
cd lambda-tensorflow-benchmark
/bin/bash benchmark.sh ${CUDA_VISIBLE_DEVICES} 1

- Create needed output directories and submit HTCondor job

$ mkdir condor_logs
$ condor_submit test-lambda-tensorflow-benchmark_venv.sub

- The benchmark will run for some time on the assigned Worker Node. Information of the running program can be checked on the 'output' and 'error' files as defined inthe condor submission file (condor_logs directory)

Output

The job will run for several minutes, using the configured python3 and installed Tensorflor installation, performing several benchmark tests in the requested gpus. The output will be produced in a results directory named after the CPU-GPU where it was executed, for example 'logs/Platinum-Tesla_V100-SXM2-32GB.logs'.

They can be analyzed as explained in the original benchmark suite page: https://github.com/lambdal/lambda-tensorflow-benchmark#step-three-report-results

Notes

• A python VIRTUAL ENTIRONMENT can be created with the exact dependencies needed.
• REQUEST_GPUS is used on the HTCondor submission script to indicate the needed number of GPUs.
• CUDA_VISIBLE_DEVICES environment variable is set by HTCondor on the Worker Nodes to indicate the assigned GPUs, and can be used by our scripts and programs.

Containers

Abstract

• Use containers (Singularity, can use Docker images) with a software distribution
• Can be run in the UI or in the Worker Nodes (HTCondor submission)

Introduction

There are situations where are provided a container with already installed provided packages and enviroment. We can also build one container to maintain one particular environment.

In this case we can download (or build) a Docker/Singularity container that will distribute the needed software to the Worker Nodes. This scenario is supported by HTCondor, following the next steps:

Preparation

Follow the recipe to download the latest tensorflow container with gpu support as depicted here:

$ mkdir ~/s.images
$ cd ~/s.images
$ singularity build tensorflow-latest-gpu  docker://tensorflow/tensorflow:latest-gpu

Run container in the UI

We can run the singularity container in the UI to check it runs correctly with a command like this.

$ gpurun singularity run --nv -c -H $PWD:/home  ~/s.images/tensorflow-latest-gpu python3 ./02_tf_tuto1.py
First nonoption argument is "singularity" at argv[1]
Connected
Info: OK 0. Tesla V100-PCIE-16GB [00000000:5E:00.0]
1. Tesla V100-PCIE-16GB [00000000:86:00.0]

Total clients:0 Running:0 Estimated waiting time:0 seconds
GPU reserved:300 seconds granted
GPUID reserved:0 Details: - Device 0. Tesla V100-PCIE-16GB [00000000:5E:00.0] set to compute mode:Exclusive Process
Info: Executing program: singularity
2021-10-22 09:37:48.656353: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-10-22 09:37:51.433740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14647 MB memory:  -> device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:5e:00.0, compute capability: 7.0
2021-10-22 09:37:52.058855: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
Epoch 1/5
1875/1875 [==============================] - 5s 2ms/step - loss: 0.2234 - accuracy: 0.9346 
Epoch 2/5
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0968 - accuracy: 0.9701
Epoch 3/5
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0703 - accuracy: 0.9782
Epoch 4/5
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0543 - accuracy: 0.9823
Epoch 5/5
1875/1875 [==============================] - 2s 1ms/step - loss: 0.0427 - accuracy: 0.9861
313/313 [==============================] - 1s 1ms/step - loss: 0.0742 - accuracy: 0.9763

Notes

• 'gpurun' is prefixed to access the UI GPU.
• 'singularity' command accepts several commands like 'run' to execute within the container ('~/s.images/tensorflow-latest-gpu') and then the command ('python3 ./02_tf_tuto1.py')
• the '--nv' parameter is needed to be able to access the GPU within the container
• By DEFAULT Singularity run will mount $HOME automatically. This can be a problem if local python / tensorflow is installed and the local environment is read in the container. the '-c' parameter clears the environment
• In this case we want to mount the current directory ($PWD) to be able to access the command (python executable), and we can do it with the parameter: '-H $PWD:/home'

Submit container to the WN (with HTCondor)

- Check the htcondor submit files and submit.

$ cat 02_tf_tuto1.sub

universe = vanilla

arguments               = "$ENV(HOME)/prep_tutorial_ific_cuda11.3/05_Containers/02_tf_tuto1.py"
arguments               = $(Process)

log                     = condor_logs/test_singularity.log
output                  = condor_logs/test_singularity.outfile.$(Cluster).$(Process).out
error                   = condor_logs/test_singularity.errors.$(Cluster).$(Process).err

+SingularityImage = "$ENV(HOME)/s.images/tensorflow-latest-gpu"
+SingularityBind = "/lustre/ific.uv.es/ml"

request_gpus = 1

queue

Notes: - With the '+SingularityImage' variable we select the image that was previously dowloaded. In this case the latest build of tensorflow with gpu support. - 'SingularityBind' let us mount other paths, in this case /lustre/ific.uv.es/ml which contains the project disk space. Home disk space /lhome is mounted by default. - The 'executable' and 'arguments' contains the full path, as singularity image may start in different paths (probably $HOME) than the current submission directory.

The executable is python and the program executed is a tensorflow example (02_tf_tuto1.py) from the Tensorflow Tutorial.

import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(512, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

Submit the job with HTCondor

Submit the .sub file with condor as in previous exercises, and check the output that has to be the same as running in the User Interface.

Notes

• Use containers (Singularity, can use Docker images) with a software distribution
• Can be run in the UI or in the Worker Nodes (HTCondor submission)
• When running in the UI gpurun should be used and some singularity options (--nv) used to access the local GPU
• When submitting with HTCondor, only some options like '+SingularityImage' is needed to user the container image.
• By DEFAULT Singularity will mount $HOME automatically.
• 'SingularityBind' let us mount other paths, in this case /lustre/ific.uv.es/ml which contains the project disk space. Home disk space /lhome is mounted by default.
• The 'executable' and 'arguments' contains the full path, as singularity image may start in different paths (probably $HOME) than the current submission directory.

FAQ: Frequently Asked Questions

Job Submision error "Hold reason: Error ... (errno=13: 'Permission denied')"

This error happens when submitting with HTCondor and the 'executable' defined in the .sub execution does not have have execution permissions (+x).

In this case the job enters in state "Hold" (as can be seen with condor_q), and stays in that state. You can remove the job with 'condor_rm'.

To solve it, simply add correct file permissions to your executable before submitting jobs. For example for a script called 'executable.sh' :

chmod +x executable.sh

-- AlvaroFernandez - 28 Oct 2021