|
|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
| (slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix | |||||||
|
|
After submitting your job to Grid Engine you may track its status by the number or name. Both can be determined from the output of the qstat command. Sometimes job is stuck in queue, typically with State E or State AU
Reasons can be connected with the node (full filesystem, not running execd, not mounted NFS partition, etc) or with SGE.
You can get general classification of stuck jobs using command qstat -j
qstat -j
invalid error number
289, 290, 291, 292, 293, 294, 295, 296,
207
invalid error number
289, 290, 291, 292, 293, 294, 295, 296,
207, 305, 306, 307, 308, 309
Jobs can not run because queue instance is not contained in its hard queue list
305, 306, 307, 308, 309
invalid error number
305, 306, 307, 308, 309
qstat -pe ms -u "*"For example, Accelrys forms job name by prefixing MS_ to the name of the subdirectory in jobs directory
If job is no longer in the queue, the output of the command
qacct -j MS_YBJ2Iwere MS_YBJ2I is Accelrys Material Studio name for the job gives info why job failed.
|
|
Jobs for each server can be found at directory
$SGE_HOME/default/spool/ node_name/job_scriptsFor example:
[0]# ls
301
---sge server: /sge/default/spool/mysite52/job_scripts
[0]root@mysite17: # ll 301
-rwxr-xr-x 1 root root 1442 Jul 10 13:00 301*
---sge server: /sge/default/spool/mysite52/job_scripts
[0]root@mysite17: # cat 301
#!/bin/sh -f
# This file is submitted to the queuing system
# The command for the job submission:
# qsub -q mysite17 -S /bin/bash -V -N MS_8WX9Z -cwd -pe ms 10 ./sge.sh
# Appropriate environment should be set before the job submission
if [ -f /sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../share/license/data/lic_setup.sh ]; then
eval `/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../share/license/data/lic_setup.sh /sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../ -s sh`
fi
if [ -f /sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../share/bin/ms_setup.sh ]; then
eval `/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../share/bin/ms_setup.sh /sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../../../ -s sh`
fi
DSD_QUEUED_RUN=1
export DSD_QUEUED_RUN
cd "/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/jobs/8WX9Z"
awk '{print $1":"$2}' $PE_HOSTFILE > machines.LINUX
DSD_MachineList="/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/jobs/8WX9Z/machines.LINUX"
export DSD_MachineList
DSD_NumProc=10
export DSD_NumProc
/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/../../gwperl "/sge/Accelrys/MaterialsStudio6.0/etc/Gateway/root_default/dsd/servers/MatServer.pl" -standalone -fromdsd fromdsd.txt -todsd todsd.txt > stdout.txt
---sge server: /sge/default/spool/mysite52/job_scripts
qstat -u <name of the user with problem jobs>
or
qstat -u "*"
qstat -f
or, for even more detail:
qstat -F
If the "state" column in qstat -f has a big E, that host or queue is in an error state due to... well, something. Sometimes an error just occurs and marks the whole queue as "bad", which blocks all jobs from running in that queue, even though there is nothing otherwise wrong with it.
au - means that communication with node is broken. possible that execd daemon is not running. in this case you have -NA- in load average:
all.q@wx3481 BIP 0/0/8 -NA- lx24-amd64 auMaybe there is some network problem preventing the SGE master from communicating with the exec host, such as routing problems or a firewall misconfiguration. Once I have the case when there was a mismatch of hostnames
You can troubleshoot these things with qping, which will test whether the SGE processes on the master node and the exec nodes can communicate.
N.B.: remember, the execd process on the exec node is responsible for establishing a TCP/IP connection to the qmaster process on the master node, not the other way around. The execd processes basically "phones home". So you have to run qping from the exec nodes, not the master node!
Syntax example (I am running this on a exec node, and sheridan is the SGE master):
qping sheridan 6444 qmaster 1where 6444 is the port that qmaster is listening on, and 1 simply means that I am trying to reach a daemon. Can't reach it? Make sure your firewall has a hole on that port, that the routing is correct, that you can ping using the good old ping command, that the qmaster process is actually up, and so on.
Of course, you could ping the exec nodes from the master node, too, e.g. I can see if I can reach exec node kosh like this:
$ qping kosh 537 execd 1but why would you do such a crazy thing? execd is responsible for reaching qmaster, not the other way around.
If the above checks does not help, check the messages log in /var/log/sge_messages on the submit and/or master node:
$ tail /var/log/sge_messagesPersonally, I like running:
$ tail -f /var/log/sge_messagesbefore I submit the job, and then submit a job in a different window. The -f option will update the tail of the file as it grows, so you can see the message log change "live" as your job executes and see what's happening as things take place.
(Note that the above is actually a symbolic link I put in to the messages log in the qmaster spool directory, i.e. /opt/sge/default/spool/qmaster/messages .)
One thing that commonly goes wrong is permissions. Make sure that the user that submitted the job using qsub actually has the permissions to write error, output, and other files to the paths you specified.
See qsub -- Submitting Jobs To Queue Instances
maybe the problem is unique only to some nodes(s) or some queue(s)? To pin it down, try to run the job only on some specific node or queue:
$ qsub -i hostname=<node/host name> <command> -- <command params>$ qsub -l qname=<queue name> <other job params>Maybe you should also try to SSH into the problem nodes directly and run the job locally from there, as your own user, and see if you can get any more detail on why it fails.
(that is, qstat no longer shows it -- because it was sent to an exec host, ran, and exited), but something else isn't working right?
qacct -j <job number>(especially see the lines "failed" and "exit_status")
NOTE: You can also use:
$ qacct -jBut in this case you should pipe that to something like tail -n <number of lines> because you only want the end of the massive log that it will spit out.
Alternately, you can use:
$ qacct -j <job number>There are many other useful options, such as getting all jobs by a specific user, etc. See man qacct for them.
If the job is still running, use qstat instead of qacct above to get the same info.
NOTE: Use qmod -c <queue list> to clear the error state for a queue.
[0]root@mysite17: # qmod -c m12a.q m32a.q root@mysite17 changed state of "m12a.q@mysite52" (no error) Queue instance "m32a.q@mysite16" is already in the specified state: no error ---sge server: /sge/default/spool/qmaster
[1] root@mysite17: # qstat -f queuename qtype resv/used/tot. load_avg arch states --------------------------------------------------------------------------------- all.q@mysite16 BIP 0/0/32 0.00 lx24-amd64 --------------------------------------------------------------------------------- all.q@mysite17 BIP 0/0/12 0.06 lx24-amd64 --------------------------------------------------------------------------------- all.q@mysite52 BIP 0/0/12 12.02 lx24-amd64 --------------------------------------------------------------------------------- all.q@mysite53 BIP 0/0/80 39.72 lx24-amd64 --------------------------------------------------------------------------------- all.q@mysite54 BIP 0/0/80 0.02 lx24-amd64 --------------------------------------------------------------------------------- all.q@wx3481-ustc BIP 0/0/8 -NA- lx24-amd64 au --------------------------------------------------------------------------------- c12.q@mysite52 BIP 0/0/12 12.02 lx24-amd64 --------------------------------------------------------------------------------- c32.q@mysite16 BIP 0/0/32 0.00 lx24-amd64 --------------------------------------------------------------------------------- c32.q@mysite53 BIP 0/0/32 39.72 lx24-amd64 --------------------------------------------------------------------------------- c32.q@mysite54 BIP 0/0/32 0.02 lx24-amd64 E --------------------------------------------------------------------------------- c40.q@mysite53 BIP 0/0/40 39.72 lx24-amd64 --------------------------------------------------------------------------------- c40.q@mysite54 BIP 0/0/40 0.02 lx24-amd64 E --------------------------------------------------------------------------------- m12a.q@mysite52 BIP 0/12/12 12.02 lx24-amd64 --------------------------------------------------------------------------------- m32a.q@mysite16 BIP 0/0/32 0.00 lx24-amd64 --------------------------------------------------------------------------------- m40a.q@mysite54 BIP 0/0/40 0.02 lx24-amd64 E --------------------------------------------------------------------------------- m40b.q@mysite53 BIP 0/40/40 39.72 lx24-amd64qmod -c command in cost cases can clean the state (but not if the node is unreachable -- au):
qmod -c c40.q c32.q m40a.q root@mysite17 changed state of "c40.q@mysite54" (no error) Queue instance "c40.q@mysite53" is already in the specified state: no error Queue instance "c32.q@mysite16" is already in the specified state: no error root@mysite17 changed state of "c32.q@mysite54" (no error) Queue instance "c32.q@mysite53" is already in the specified state: no error root@mysite17 changed state of "m40a.q@mysite54" (no error)
Sometimes, the SGE master host will become so FUBARed that we have to resort to brute, traumatizing force to fix it. The following solution is equivalent to fixing a wristwatch with a bulldozer, but seems to cause more good than harm (although I can't guarantee that it doesn't cause long-term harm in favor of a short-term solution).
Basically, you wipe the database that keeps track of SGE jobs on the master host, taking any problem "stuck" jobs with it. (At least that's what I think this does...)
I've found this useful when:
qdelto wipe the jobs due to the same error.
The solution:
ssh sheridan su - service sgemaster stop cd /opt/sge/default/ mv spooldb spooldb.fubared mkdir spooldb cp spooldb.fubared/sge spooldb/ chown -R sgeadmin:sgeadmin spooldb service sgemaster start
Wipe spooldb.fubared when you are confident that you won't need its contents again.
Google matched content |
Oracle
Ubuntu Manpage qstat - show the status of Sun Grid Engine jobs and queues
sge-tutorial - The Center for Computational Biology and Bioinformatics - UT Austin Wikis
W. M. Keck Laboratory for Integrated Biology II
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|
|
You can use PayPal to to buy a cup of coffee for authors of this site |
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: March, 12, 2019
