PBS的配置

http://grid.tsinghua.edu.cn/home/liulk/wiki/computer/pbsconfig.html 

内容提纲：

安装前的要求

安装

PBS的配置

起动PBS.

创建队列.

PBS Script samples

 

安装前的要求

如果是单机跑跑pbs, 那么什么都不要求. 如果你想在机群上pbs,那么再进行pbs安装之前你需要保证一些其他的软件都能工作.

• NFS 这个是为了在机群之间共享一个目录, 使得个个机器都能访问.
• Nis 这是是为了在机群上实现统一的用户认证.

安装

pbs的安装比较简单, 应该说是一个标准的linux的tarball安装方式. 安装过程主要有如下几步:

• 下载pbs的源码包 OpenPBS-2.3.12sc2.tar。
• 解压缩软件包

  $tar zxvpf OpenPBS-2.3.12sc2.tar

• 进入相应的目录, 配置,make

  $cd SPBS-1.0.0$./configure --enable-docs --disable-gui$make

  注意上面的选项

   --enable-docs 指定要编译文档 --disable-gui 指定去掉图形界面. 因为图形界面要求需要tck支持, 而且似乎必须是tcl-8.0的版本, 高了会出错 --enable-scp  先使用scp,再使用rcp进行数据的复制. 

• 切换成root用户, 安装.

  $su#make install

  默认情况下, PBS会安装再/usr/spoole/PBS目录下. 该目录在./configure 的时候指定 --prefix=/you/want/dir来更改. 建议对于初次安装的人采用默认的设置.

PBS的配置

一般来说, PBS系统必须有一个server和至少一个mom, server负责作业的提交, mom接受server的控制,负责作业的执行. 假设pbs的根目录为 $PBS_HOME. server节点为CGSP48, mom节点为CGSP48和CGSP47,也就是CGSP48既作为共享server也作为mom, 而CGSP47只作为mom.

1. 配置server_name 编辑所有的节点的 $PBS_HOME/server_name,在里面写入选定的server的主机名. 例如

   CGSP48

2. 配置节点. 在CGSP48的 $PBS_HOME/server_priv目录下建立nodes文件.

   touch nodes

   在nodes文件写入你所有mom节点的名称:

   CGSP48 np=2CGSP47 np=3

   其中的np代表的是虚拟处理器的个数. 实际上也就是该节点最多可以同时跑多少个任务.
3. 配置mom节点,使其接受sever节点的控制. 编辑每个mom节点的 $PBS_HOME/mom_priv目录下的config文件(没有就自己建立). 写入如下信息

   $logevent 0x1ff$clienthost CGSP48

   其中:

   $logevent 指定的日志的级别, 基本上不用修改. 默认就很好了$clienthost 指定的server的地址

起动PBS.

pbs起动最好按如下顺序. mom, sever, sched. 而且必须以root身份起动. 命令如下:

# /usr/local/sbin/pbs_mom# /usr/local/sbin/pbs_server -t create# /usr/local/sbin/pbs_sched

其中, -t create 在第一次起动的时候要用, 用于创建一些初始化必要的环境. 以后起动就不在需要了.

创建队列.

PBS中的队列分为两种类型, 执行队列和路由队列. 下面是一个创建队列的脚本.

# # Create and define queue verylong # create queue verylong set queue verylong queue_type = Execution set queue verylong Priority = 40 set queue verylong max_running = 10 set queue verylong resources_max.cput = 72:00:00 set queue verylong resources_min.cput = 12:00:01 set queue verylong resources_default.cput = 72:00:00 set queue verylong enabled = True set queue verylong started = True # # Create and define queue long # create queue long set queue long queue_type = Execution set queue long Priority = 60 set queue long max_running = 10 set queue long resources_max.cput = 12:00:00 set queue long resources_min.cput = 02:00:01 set queue long resources_default.cput = 12:00:00 set queue long enabled = True set queue long started = True # # Create and define queue medium # create queue medium set queue medium queue_type = Execution set queue medium Priority = 80 set queue medium max_running = 10 set queue medium resources_max.cput = 02:00:00 set queue medium resources_min.cput = 00:20:01 set queue medium resources_default.cput = 02:00:00 set queue medium enabled = True set queue medium started = True # # Create and define queue small # create queue small set queue small queue_type = Execution set queue small Priority = 100 set queue small max_running = 10 set queue small resources_max.cput = 00:20:00 set queue small resources_default.cput = 00:20:00 set queue small enabled = True set queue small started = True                   # # Create and define queue default # create queue default set queue default queue_type = Route set queue default max_running = 10 set queue default route_destinations = small set queue default route_destinations += medium set queue default route_destinations += long set queue default route_destinations += verylong set queue default enabled = True set queue default started = True # # Set server attributes. # set server scheduling = True set server max_user_run = 6 set server acl_host_enable = True set server acl_hosts = * set server default_queue = default set server log_events = 63 set server mail_from = adm set server query_other_jobs = True set server resources_default.cput = 01:00:00 set server resources_default.neednodes = 1 set server resources_default.nodect = 1 set server resources_default.nodes = 1 set server scheduler_iteration = 60 set server default_node = 1#shared

该脚本定义了verylong long medium small 四个作业队列 和 一个default路由队列. 里面的几个比较重要的属性:

 enabled 指示作业队列可用. 也就是可以往里面添加新的作业了. actived 指示作业队列处于活动状态, 可以参与调度了. 好象是这个样子的. 反正两个都设置为true就可以了. sever的scheduling属性指示server开始调度. 这个属性非常的重要.         为了这个属性, 我浪费了一个晚上. 因为只要该属性没有设置,那么服务器中的所有的作业就一直处于Q状态. 

到此为止PBS的配置基本就完成了. 当然. PBS还有很多的参数可以设置. 但那就不是这个基本配置想给大家的. 大家自己去查PBS的手册把.

PBS Script samples

#LJRS -S /bin/bash#LJRS -o script.out#LJRS -j oe#LJRS -q dpool#LJRS -l nodes=8:ppn=1      （nodes：计算节点数，ppn：计算节点CPU数）#LJRS -l walltime=48:00:00  （用户估计的最大计算时间，超时系统会自动中断作业）limit -s unlimitedTMPFILE=`whoami`_mpich_gm.tmpsed 's/c/g/g' $LJRS_NODEFILE > /tmp/$TMPFILEGM_NODEFILE=/tmp/$TMPFILEecho Working directory is $LJRS_O_WORKDIRcd $LJRS_O_WORKDIRecho Runing on host `hostname`echo Starting Time is `date`echo Directory is `pwd`echo This jobs runs on the following processors:echo `cat $GM_NODEFILE`NPROCS=`wc -l < $GM_NODEFILE`echo This job has allocated $NPROCS nodes## User Parallel Program ###########mpirun -v -machinefile $GM_NODEFILE -np $NPROCS ~/my_parallel.exe > out （修改为用户的并行作业执行命令）####################################rm -f /tmp/$TMPFILEecho Ending Time is `date`

############################################################################ Script for submitting parallel Gaussian 03 jobs to the cluster.############################################################################# To use this script, first make the following changes:##  1. Add the G03 .com filename to "jobname=", but leave out the suffix .com.#      Example:  jobname=input#  2. Add the same filename to #LJRS -o with the suffix .err.#      Example:  #LJRS -o water03.err#  3. Specify the number of nodes(nodes = x) and processors per node#      (ppn = y) needed for the job. Note that each job should typically use#      both processors on each node(i.e., ppn = 2). Therefore, set nodes equal#      to the total number of processors divided by 2.#      Example: If the job needs 16 processors total, then:#                #LJRS -l nodes=8:ppn=2#     Note that the appropriate number of processors must be requested in#      the .com file. In the above example, %nprocl=16 must be specified#      before the job command line(i.e., the line beginning with "#").#      (Note: Either the command nprocl or nproclinda can be used.)#      If only one processor is required, then %nprocl does not need to be#      specified. In the .g03 file, one processor is specified as:#                #LJRS -l nodes=4:ppn=2#  4. Set the maximum length of time the job will use.#      Example:  #LJRS -l walltime=24:00:00#  5. Make sure the input (.com) and submitting script (.g03) files are in#      the same directory.##  Submit the script using "qsub input.g03".############################################################################ Lines that begin with #LJRS are PBS directives (not comments).# True comments begin with "# " (i,e., # followed by a space).############################################################################LJRS -S /bin/bash#LJRS -o water333.err#LJRS -j oe#LJRS -q dpool#LJRS -l nodes=4:ppn=2        (与input.com文件保持一致）#LJRS -l walltime=860:00:00  （用户估计的最大计算时间，超时系统会自动中断作业）#LJRS -Vcat "$LJRS_NODEFILE"##############################################################################  -S: shell the job will run under#  -o: name of the queue error filename#  -j: merges stdout and stderr to the same file#  -l: resources required by the job: number of nodes and processors per node#  -l: resources required by the job: maximun job time length############################################################################## Define variable "jobname".jobname=input                  （g03输入文件名，不包括.com扩展名）username=`whoami`ulimit -s unlimited# Make a directory in scr and copy .com and .g03 file to there.GAUSS_RUNDIR=/scratch/${username}if [ ! -a $GAUSS_RUNDIR ]; then   echo "Scratch directory $GAUSS_RUNDIR created."   mkdir -p $GAUSS_RUNDIRficp $LJRS_O_WORKDIR/${jobname}.* $GAUSS_RUNDIRORIG_LJRS_O_WORKDIR=${LJRS_O_WORKDIR}LJRS_O_WORKDIR=${GAUSS_RUNDIR}cd $LJRS_O_WORKDIR# Setup for Gaussian 03:# =======================# Make a scratch directory if it doesn't already exist.GAUSS_SCRDIR=/scratch/${username}/${jobname}if [ ! -a $GAUSS_SCRDIR ]; then   echo "Scratch directory $GAUSS_SCRDIR created."   mkdir -p $GAUSS_SCRDIRfiexport GAUSS_SCRDIRecho "Using $GAUSS_SCRDIR for temporary Gaussian 03 files."# Define the location where Gaussian was installed and run# a setup script, g03.profile.g03root=/export/local/g03source $g03root/g03/bsd/g03.profile#source /export/local/g03/g03/bsd/g03.profile# Define PATH to include location of LINDAPATH=$PATH:/export/local/g03/g03/linda7.1/intel-linux2.4/bin# Define node list#echo $LJRS_NODEFILE $LJRS_JOBID > /tmp/g03logsed 's/c/g/g' $LJRS_NODEFILE > $GAUSS_SCRDIR/tsnet.nodes#cat /tmp/$2 > $LJRS_NODEFILEG03_NODEFILE="$GAUSS_SCRDIR/tsnet.nodes"GAUSS_LFLAGS="-mp 2 -nodefile $G03_NODEFILE"# Export variable listexport PATH g03root GAUSS_LFLAGSecho pbs nodefile:cat $G03_NODEFILE#Run a Gaussian command file, water03.com, redirecting output#to a file, water03.logecho "Starting Gaussian run at" `date`time g03l < $GAUSS_RUNDIR/${jobname}.com >$GAUSS_RUNDIR/${jobname}.logecho "Finished Gaussian run at" `date`LJRS_O_WORKDIR=${ORIG_LJRS_O_WORKDIR}echo $LJRS_O_WORKDIRmv $GAUSS_RUNDIR/${jobname}.* $LJRS_O_WORKDIRmv $GAUSS_RUNDIR/*.chk $LJRS_O_WORKDIRecho "$GAUSS_SCRDIR"rm -Rf $GAUSS_SCRDIR