Job Dependencies

来源：互联网发布：照片模糊修复软件编辑：程序博客网时间：2024/06/05 20:41

Knowledge Center Contents Previous Next Index

Job Dependencies
Contents
Job Dependency Terminology
Job Dependency Scheduling
Dependency Conditions
View Job Dependencies
Job Dependency Terminology
Job dependency: The start of a job depends on the state of other jobs.
Parent jobs: Jobs that other jobs depend on.
Child jobs: Jobs that cannot start until other jobs have reached a specific state.
Example: If job2 depends on job1 (meaning that job2 cannot start until job1 reaches a specific state), then job2 is the child job and job1 is the parent job.
Job Dependency Scheduling
About job dependency scheduling
Sometimes, whether a job should start depends on the result of another job. For example, a series of jobs could process input data, run a simulation, generate images based on the simulation output, and finally, record the images on a high-resolution film output device. Each step can only be performed after the previous step finishes successfully, and all subsequent steps must be aborted if any step fails.
Some jobs may not be considered complete until some post-job processing is performed. For example, a job may need to exit from a post-execution job script, clean up job files, or transfer job output after the job completes.
In LSF, any job can be dependent on other LSF jobs. When you submit a job, you use bsub -w to specify a dependency expression, usually based on the job states of preceding jobs.
LSF will not place your job unless this dependency expression evaluates to TRUE. If you specify a dependency on a job that LSF cannot find (such as a job that has not yet been submitted), your job submission fails.
Specify a job dependency
To specify job dependencies, use bsub -w to specify a dependency expression for the job.
Syntax
bsub -w 'dependency_expression'
The dependency expression is a logical expression composed of one or more dependency conditions. For syntax of individual dependency conditions, see Dependency Conditions.
To make dependency expression of multiple conditions, use the following logical operators:
&& (AND)
|| (OR)
! (NOT)
Use parentheses to indicate the order of operations, if necessary.
Enclose the dependency expression in single quotes (') to prevent the shell from interpreting special characters (space, any logic operator, or parentheses). If you use single quotes for the dependency expression, use double quotes for quoted items within it, such as job names.
Job names specify only your own jobs, unless you are an LSF administrator.
Use double quotes (") around job names that begin with a number.
In Windows, enclose the dependency expression in double quotes (") when the expression contains a space. For example:
bsub -w "exit(678, 0)" requires double quotes in Windows.
bsub -w 'exit(678,0)' can use single quotes in Windows.
In the job name, specify the wildcard character (*) at the end of a string, to indicate all jobs whose name begins with the string. For example, if you use jobA* as the job name, it specifies jobs named jobA, jobA1, jobA_test, jobA.log, etc.
note:
Wildcard characters can only be used at the end of job name strings within the job dependency expression.
Multiple jobs with the same name
By default, if you use the job name to specify a dependency condition, and more than one of your jobs has the same name, all of your jobs that have that name must satisfy the test.
To change this behavior, set JOB_DEP_LAST_SUB in lsb.params to 1. Then, if more than one of your jobs has the same name, the test is done on the one submitted most recently.
Dependency Conditions
The following dependency conditions can be used with any job:
done(job_ID | "job_name")
ended(job_ID | "job_name")
exit(job_ID [,[op] exit_code])
exit("job_name"[,[op] exit_code])
external(job_ID | "job_name", "status_text")
job_ID | "job_name"
post_done(job_ID | "job_name")
post_err(job_ID | "job_name")
started(job_ID | "job_name")
done
Syntax
done(job_ID | "job_name")
Description
The job state is DONE.
ended
Syntax
ended(job_ID | "job_name")
Description
The job state is EXIT or DONE.
exit
Syntax
exit(job_ID | "job_name"[,[operator] exit_code])
where operator represents one of the following relational operators:
>
> =
<
< =
==
!=
Description
The job state is EXIT, and the job's exit code satisfies the comparison test.
If you specify an exit code with no operator, the test is for equality (== is assumed).
If you specify only the job, any exit code satisfies the test.
Examples
exit (myjob)
The job named myjob is in the EXIT state, and it does not matter what its exit code was.
exit (678,0)
The job with job ID 678 is in the EXIT state, and terminated with exit code 0.
exit ("678",!=0)
The job named 678 is in the EXIT state, and terminated with any non-zero exit code.
external
Syntax
external(job_ID | "job_name", "status_text")
Specify the first word of the job status or message description (no spaces). Only the first word is evaluated.
Description
The job has the specified job status, or the text of the job's status begins with the specified word.
Job ID or job name
Syntax
job_ID | "job_name"
Description
If you specify a job without a dependency condition, the test is for the DONE state (LSF assumes the "done" dependency condition by default).
post_done
Syntax
post_done(job_ID | "job_name")
Description
The job state is POST_DONE (the post-processing of specified job has completed without errors).
post_err
Syntax
post_err(job_ID | "job_name")
Description
The job state is POST_ERR (the post-processing of specified job has completed with errors).
started
Syntax
started(job_ID | "job_name")
Description
The job state is:
USUSP, SSUSP, DONE, or EXIT
RUN and the job has a pre-execution command (bsub -E) that is done.
Advanced dependency conditions
Job arrays
If you use job arrays, you can specify additional dependency conditions that only work with job arrays.
To use other dependency conditions with array jobs, specify elements of a job array in the usual way.
Job dependency examples
bsub -J "JobA" -w 'done(JobB)' command
The simplest kind of dependency expression consists of only one dependency condition. For example, if JobA depends on the successful completion of JobB, submit the job as shown.
-w 'done(312) && (started(Job2)||exit("99Job"))'
The submitted job will not start until the job with the job ID of 312 has completed successfully, and either the job named Job2 has started, or the job named 99Job has terminated abnormally.
-w "210"
The submitted job will not start unless the job named 210 is finished.
View Job Dependencies
The bjdepinfo command displays any dependencies that jobs have, either jobs that depend on a job or jobs that your job depends on.
By specifying -r, you get not only direct dependencies (job A depends on job B), but also indirect dependencies (job A depends on job B, job B depends on jobs C and D). You can also limit the number of levels returned using the -r option.
The -l option displays results in greater detail.
To display all jobs that this job depends on:
bjdepinfo 123
JOBID PARENT PARENT_STATUS PARENT_NAME  LEVEL 123   32522  RUN           JOB32522     1 
To display jobs that depend on a job you specify (display child jobs):
bjdepinfo -c 300
JOBID CHILD CHILD_STATUS CHILD_NAME LEVEL 300   310   PEND         JOB310     1  300   311   PEND         JOB311     1  300   312   PEND         JOB312     1  
To display the parent jobs that cause a job to pend:
bjdepinfo -p 100
These jobs are always pending because their dependency has not yet been satisfied.
JOBID PARENT PARENT_STATUS PARENT_NAME LEVEL 100   99     PEND          JOB99       1 100   98     PEND          JOB98       1 100   97     PEND          JOB97       1 100   30     PEND          JOB30       1 
Display more information about job dependencies including whether the condition has been satisfied or not and the condition that is on the job:
bjdepinfo -l 32522 
Dependency condition of job <32522> is not satisfied: done(23455) JOBID PARENT PARENT_STATUS PARENT_NAME  LEVEL 32522 23455  RUN           JOB23455     1 
Display information about job dependencies that includes only direct dependencies and two levels of indirect dependencies:
bjdepinfo -r 3 -l 100 Dependency condition of job <100> is not satisfied: done(99) && ended(98) && done(97) && done(96) JOBID PARENT PARENT_STATUS PARENT_NAME LEVEL  100   99     PEND          JOB99       1 100   98     PEND          JOB98       1 100   97     PEND          JOB97       1 100   96     DONE          JOB96       1 Dependency condition of job <97> is not satisfied: done(89) JOBID PARENT PARENT_STATUS PARENT_NAME LEVEL 97    89     PEND          JOB89       2 Dependency condition of job <89> is not satisfied: ended(86) JOBID PARENT PARENT_STATUS PARENT_NAME LEVEL 89    86     PEND          JOB86       3 

Platform Computing Inc.
www.platform.com
Knowledge Center Contents Previous Next Index

http://www.ccs.miami.edu/hpc/lsf/7.0.6/admin/jobdependencies.html

0 0