Job Post Mortem

SGE produces job post mortem, via setting execd_param KEEP_ACTIVE=ERROR

Set “KEEP_ACTIVE=TRUE” for execd_params in the global or hostspecific configuration disables the job spool dir deletion. That means that directory contents can be analyzed after job failed...

Job spool dir retention

• Where are the spool dir files?
• Inside “active_jobs/” in the $SGE_ROOT/default/spool/<node>/ directory
• Within active_jobs/ directories are named by JobID and array task number
  • active_jobs/1.[taskID]
  • active_jobs/2.[taskID]

Job spool dir file contents

• Config
• Job configuration parameters
• environment */active_jobs_dir/*/environment
• All env variables to be set up for the job
• error
• Notices of severe errors during job startup
• exit_status
• Unix exit status for the job

Job spool dir retention

• Remember to turn this feature off!
• Puts extra load on SGE
• Takes up lots of space

Trigger SGE Debug Output

• Special ENV var(s) trigger debug output
• Some useful for admins, most useful only for Grid Engine Developers
• General process: 1. Source the “dl.sh” or “dl.csh” script found in $SGE_ROOT/util/ 2. Run command “dl <level>” 3. Levels 1,3,5 are most useful
• Try for yourself:
• $ source /$SGE_ROOT/util/dl.sh
• $ dl 1
• $ qrsh hostname

  The qmaster will produce (large amounts of) debugging output if SGE_DEBUG_LEVEL is set appropriately.  Source util/dl.sh and run sge_qmaster directly after then using `dl 4', for instance.  (I don't know what an appropriate debugging level is.)  Otherwise, strace, or a similar tool if it's not GNU/Linux, might show why the socket binding fails if this is on GNU/Linux.

   

It contains environment. 
*/active_jobs_dir/*/environment

It's hard to debug when something is running as deamon. So you need to prevent daemonizing with SGE_ND=1.
SGE_DEBUG_LEVEL defines the emount of information 

execd_param -- important KEEP_ACTIVE=ERROR