How to ask for exlusive use of a GPU ?

[VassilisVassiliadis]

Issue

I would like to use the lsf-python-api to submit a LSF job which requests exclusive use of hardware resources.

For example, here's how I request 4 cores, 1 GPU, and 1000 MB of RAM with a 1h walltime:

import pythonlsf.lsf as lsf
import os

if lsf.lsb_init("test") > 0:
    raise ValueError("Unable to initialise LSF environment")

submitreq = lsf.submit()

submitreq.options = 0
submitreq.options2 = 0
submitreq.options3 = 0
submitreq.options4 = 0

submitreq.beginTime = 0
submitreq.termTime = 0
submitreq.resReq = "rusage[mem=1000:ngpus_physical=1.00] span[ptile=1] affinity[core(4,exclusive=(core,alljobs))*1]"
submitreq.outFile = os.path.join(os.getcwd(), "my-stdout.txt")
submitreq.queue = "x86_1h"
submitreq.options = lsf.SUB_OUT_FILE | lsf.SUB_QUEUE | lsf.SUB_RES_REQ
submitreq.options2 = lsf.SUB2_OVERWRITE_OUT_FILE
submitreq.cwd = os.getcwd()
submitreq.options3 = lsf.SUB3_CWD

submitreq.command = "nvidia-smi"

submitreply = lsf.submitReply()
job_id = lsf.lsb_submit(submitreq, submitreply)

print("Job id is", job_id)

Observed behaviour

Running the above prints a job id and when I use bjobs -l <id> I see this at the bottom of the printout:

 GPU REQUIREMENT DETAILS:
 Combined: mode=shared:mps=no:j_exclusive=no:gvendor=nvidia
 Effective: mode=shared:mps=no:j_exclusive=no:gvendor=nvidia

Expected behaviour

If I run an equivalent bsub command I have the option of setting the gpu mode like so:

bsub <other fields....> -gpu num=1:mode=exclusive_process

Then when I look at the bjobs -l <id> information I see this printout:

Combined: num=1:mode=exclusive_process:mps=no:j_exclusive=yes:gvendor=nvidia
Effective: num=1:mode=exclusive_process:mps=no:j_exclusive=yes:gvendor=nvidia

Question

How can I get my python code to ask for exclusive use of the requested GPUs ?

[liyancn]

#54

[VassilisVassiliadis]

Hi @liyancn I just tested the example you provided and after making a small change (#58) it worked, thank you!