CXbatch Queueing System Chapter Overview ================ CXbatch is the batch facility available on the CONVEX C3880 and is based on the Network Queueing System (NQS). Utilities are provided that allow you to submit, terminate, and monitor the status of jobs submitted to the CXbatch queues. You can supply a script file containing all the commands necessary for the execution of the task requested. CXbatch is an important facility for maximizing system resources through a batch queue that holds requests, perhaps delaying them, until scheduled run times. This results in better machine performance and a more productive environment for all users. You are encouraged to use CXbatch for all your production runs. Introduction to CXbatch ======================= The CXbatch system is made up of batch and pipe queues that transport and run batch requests. CXbatch allows you to submit, terminate, and monitor your batch jobs. By submitting your jobs to the appropriate batch queue, you are allowing CXbatch to schedule your job according to the resources requested, such as CPU time. Jobs can run in any of a number of different batch queues. Each queue is specially configured by the system administrator and has specific limits on how much CPU time a job can use and when it can run and at what priority. CXbatch is invoked by using the qsub command. You can use the qstat (or qs) command to display queue information. The ordering of the requests within a queue does not always determine the order in which the requests are run. The CXbatch scheduler makes exceptions to the request ordering for the sake of efficient machine resource usage. In general, however, requests near the top of the queue list are run before requests appearing lower on the list. Overview of CXbatch Commands ============================ CXbatch commands include the following: Command Description qsub submits batch request to CXbatch qstat displays status of CXbatch queue; after submitting your request, you can use this utility to monitor the status of the batch queues and the progress of your job in the queue qdel deletes or signals CXbatch requests qsfilter for qstat facility A summary of CXbatch commands and commonly used options is given later in this section. For a complete description of these commands enter man command, where command is qsub, qstat, qs, or qdel. For information on CXbatch as it is currently implemented at NCSA, enter news nqs |less. Preparing a Job for CXbatch =========================== In order to run jobs in CXbatch, job files should be retrieved from UniTree (see ``UniTree'' on page 4 -- 1). Next the jobs should be run on the scratch space allocated for batch jobs. Refer to the scratch area by using the SCR environmental variable. Home directory space is inadequate for production runs. The available interactive limit for scratch space is 2 Gbytes. In order to submit a job to CXbatch, you must prepare a set of instructions for running the job, similar to what you would use interactively. The script must be complete; that is, all the commands and steps necessary for the execution of the task requested must be included. You may reference another script within your script. You then use the qsub command to submit the set of instructions. A Basic Script Below is a sample script for batch jobs on the C3880 that illustrates setting CPU time and memory limits, checkpointing, tar, UniTree, and other useful features. Checkpointing is not performed unless you use the -cp option. This sample is available for copying in the file /usr/local/batch/sample. Copy the file and make the necessary modifications to the script to run your job. After modifying your copy as necessary, submit the job using the qsub command. Enter qsub filename. #!/bin/csh # Use the C shell to run this script # # Sample Batch Script -- last modified 10/27/92 # # This is a sample script for batch jobs on the Convex. It illustrates # how to set CPU time and memory limits, checkpointing, tar, Unitree, # and other useful features. # # Note that checkpointing will not be performed unless you use the -cp # option. (below) # # Processes which use more than 1536 Mbytes of memory cannot be checkpointed # due to a hardware limitation. If a batch job cannot be checkpointed, the # system will automatically try to restart the job from the beginning when # there has been a system interruption. If you do not wish to have the job # automatically restarted, use the -nr option. # # Once you have copied this file and changed all the marked lines, you # can submit this as a job using the "qsub" command. (e.g., qsub sample) # Use the "qstat" command to check the status of a job. To remove a job # from a queue or abort a running job, use the "qdel" command. For more # information, see the man pages for qsub, qstat, and qdel. # # @$-lt 0:3:00 # Set per process CPU time limit (CHANGE THIS LINE) # @$-ld 110mb # Set per process memory limit (CHANGE THIS LINE) # # (Note: options -lm and -lM are silently ignored) # @$-eo # Combine stdout and stderr # @$-me # Send e-mail on completion # @$-cp 2 hours # checkpoint every two hours (see warning below) # WARNING--processes using more than 1536 Mbytes # of memory cannot be checkpointed. # @$-y # append accounting data to output file # @$-l # use login shell (defines $SCR--ignore errors) # @$ # end of qsub options # set echo # echo commands # # To charge this job to a project other than the default, use the # newacct command below # #/usr/local/bin/newacct zzz # specify project to charge (CHANGE THIS LINE) #if ($status != 0) then # echo "Invalid account--exiting..." # exit #endif # cd $SCR # Go to batch scratch directory (/scr-bat/$USER) # if (! -d wrk) mkdir wrk # If no work directory, create one cd wrk # and go to that directory # # Print timing statistics for any process that uses more than 5 CPU seconds # (see the man page for csh for more information) # set time=(5 "CPU: %U+%Ss, Elaps: %E, Util: %P, Mem(KB): %X+%D, Conc: %C") # # Get files from Unitree (change the lines marked below) # msscmd get /CFS/training/convex/test.tar test.tar # # Unbundle work files # /bin/tar -xvf test.tar # CHANGE THIS LINE! /bin/rm test.tar # CHANGE THIS LINE! cd test # CHANGE THIS LINE! ls -l # # Start compilation and execution # /usr/convex/fc -O3 test.f # CHANGE THIS LINE! ./a.out # CHANGE THIS LINE! # # Bundle the work files for saving to Unitree # /bin/tar -cf ../mywork.tar . # CHANGE THIS LINE! cd .. msscmd put mywork.tar # CHANGE THIS LINE! exit In the sample, use the C shell to run the script. The only two shells that may be explicitly defined are: #! /bin/csh#! /bin/sh In the example, the command cd $SCR specifies that you change to the batch scratch directory. The default is the home directory, and it is too small to run a CXbatch job. Be certain to clear the $SCR directories at the end of your job by removing all files in your $SCR directory after saving them to UniTree. Submitting the Job Once you have modified the sample script file, you are ready to submit the job to the batch system. To submit a request to CXbatch, enter qsub scriptfile at the prompt. CXbatch assigns a name, called the request-identifier, to your request. This name is displayed at your terminal if the job is successfully submitted. The request identifier is in the form sequencenumber.hostname, where hostname is the machine from which the request was submitted and sequencenumber identifies the sequence number assigned to the request on the host machine. For example, to submit nqsjob: % qsub nqsjob Request 497.c3.ncsa.uiuc.edu submitted to queue: default The request-identifier assigned to this example nqsjob is 497. The output/error file is named nqsjob.o497. Once a request is submitted, the system controls the sequence of events; you can monitor or remove the request. The qdel command is used to delete a job. For example, to delete the sample job from a queue, enter: % qdel 497 The qstat command gives you a summary status report on all CXbatch jobs. Inspecting the Output The output of this script nqsjob.o497 should be examined for a record of commands executed and possible error messages. By default, the output file is placed in the working directory at the time of the qsub command. You can use the ls command to determine whether the CXbatch output/error file has appeared in your directory and either the cat (full text) or more (pager) command to print it to your screen. Retrieving a Source File and Compiling You may often have a Fortran source file and input data files stored in a subdirectory. In the following script the Fortran source file mysrc.f and input file myinfile are retrieved from the UniTree subdirectory mydir. The source file is compiled using fc to produce the executable mya.out. The output file myoutfile is then saved to the UniTree subdirectory mydir. #!/bin/csh #Specify C shell cd $SCR #Move to local scratch directory msscmd get mydir/mysrc.f mysrc.f #Bring Fortran source from UniTree subdir msscmd get mydir/myinfile myinfile #Bring in UniTree input data file fc -02 -o mya.out mysrc.f #Compile source;call mya.out mya.out #Run the executable program msscmd put myoutfile mydir/myoutfile #Save output data file to UniTree subdir rm * #Clean up scratch directory CXbatch Command Descriptions ============================ A summary of the options accepted by each of the commands described earlier in this chapter is listed in this section. For a full description, enter man command. qsub Submits batch request to CXbatch. If no scriptfile is specified, the set of commands to be executed as a batch request is taken from the standard input file (stdin). Format: qsub [options] [scriptfile] Option Description -a run request after stated time -nc request cannot be checkpointed -me mail notification on job completion -ltset per-process CPU time limit -ldset per-process data segment (memory) limit qstat Displays status of CXbatch queues. If no queue is specified, the current state of each CXbatch queue on the local host is displayed. If a queue is specified, the information displayed is for that queue only. Format: qstat [-almx] [-u username] [queuename] Option Description -a displays all requests -l displays requests in a long format -m displays requests in a medium-length format including the scheduled time for the run -x includes queue header in an extended format -u username shows only the requests submitted by username qdel Terminates queued jobs specified on the input line. You can only delete your own requests. A request to terminate a running job must include either the -k or the -signo flag. Format: qdel [-k] [-signo] [-u username] request-identifier qdel returns an exit status describing the action taken. If no errors are encountered, the exit status is zero. If one or more of the requests are not deleted, the exit status is the number of requests not terminated. If a fatal error occurs and no jobs are terminated, the exit status is one of seven special codes: EX_USAGE, EX_NOUSER, EX_NOHOST, EX_OSFILE, EX_TEMPFAIL, EX_NOPERM, or EX_SOFTWARE. Refer to the man pages for detailed explanations of these codes. Queue Configurations ==================== The queue structure is based on: * analysis of historic usage data from the CONVEX * system philosophy for the CONVEX * study of queue structures on CONVEX and Cray systems at other centers NQS Commands for Queueing Use the NQS commands described above (qsub, qdel, qs, qstat) to submit, delete, and check the status of jobs. Normally the system automatically routes a job to the appropriate queue, based on the CPU and memory limits you specify. Queue structure The basic structure is a 3 x 3 matrix of queues, categorized by CPU time and memory limits. CPU limits are set to 1, 10, and 100 CPU hours. Memory limits are set to 128, 512, and 2048 Mbytes. Queue names The queue names do not have CPU and memory limits encoded in the name so that the limits can be easily changed without changing the queue names. The first part of the name indicates the time limit (s = short, m = medium, l = long), followed by the letter t. The last part of the name indicates the memory limit (s = small, m = medium, l = large), followed by the letter m. The table below lists the names of the queues (queue) and the number of simultaneous jobs (jobs) that can run: Memory limits CPU limits 1 hour 10 hours 100 hours queue jobs queue jobs queue jobs 128 Mbytes st_sm 2 mt_sm 2 lt_sm 2 512 Mbytes st_mm 2 mt_mm 2 lt_mm 2 2048 Mbytes na na lt_lm 2 Only one queue is available for all large-memory jobs (2048 Mbytes). This prevents excessive swapping and reduces the disk space needed to checkpoint batch jobs. NOTE: Queues are subject to change. Check the file /usr/news/nqs for the most current queue information. Memory and CPU limits The CPU and memory limits for batch queues are based on historical usage data. The goal is to evenly distribute CPU usage among the queues. The upper limit for memory on the queues -- 2048 Mbytes -- is determined by a machine hardware addressing limitation. User limits A limit of 2 running jobs per user prevents a single user from dominating the system. Access to Batch Mode ==================== According to the current C3880 schedule, batch jobs normally run from 6:00 p.m. to 8:00 a.m. (Central Time) Monday through Friday. If no dedicated time is scheduled, batch jobs also run during the weekend. You can submit a batch job any time during batch, interactive, or dedicated modes (see ``Access Modes'' on page 3 -- 1). Small-memory queues (128 Mbytes) are run during interactive mode at a high nice value so that any available machine cycles are used during interactive mode. The impact on interactive use is minimal because these jobs use very little memory. Checkpointing ============= Overview of Checkpointing User checkpointing of medium- and long-running jobs is an essential facet of computing at NCSA. Checkpointing saves valuable research time by providing a ``safety net'' in the event of a problem with your running program or with the computer system. Checkpointing is a courtesy to other users of the system as well, as you will not have to waste computer resources re-doing work that you have already done. The ideal job checkpointing involves you periodically writing out the key variables you need to successfully restart your program at that point. This method results in a much smaller checkpoint file than is produced by the system-callable methods described here, which copy a complete state of the running program. For help with checkpointing, contact the Consulting Office (see ``Consulting Office'' on page 6). Checkpointing on the C3880 Batch jobs may be periodically checkpointed using the -cp option on the qsub command. Interactive processes can be checkpointed using the chkpnt command. You cannot checkpoint a process that is using more than 1536 Mbytes (1.5 Gbytes) of memory. Mode specifications =================== Three tables in the file /usr/news/nqs present the current specifications for the interactive, batch, and dedicated modes. Please review this information carefully. Contact the Consulting Office for assistance (see ``Consulting Office'' on page 6). Complete information about the purposes of the modes is in ``Access Modes'' on page 3 -- 1. .