Grid Scheduler

This is abandonware which corresponds to Sun SGE 6.2u5 with some bug fixes. Still used in major companies and universities. there are even new deployments as of 2014 despite the fact that development stopped in 2012.

Open Grid Scheduler/Grid Engine is released under the Sun Industry Standards Source License (SISSL). New code (new file) is licensed under the BSD license.

The fine print: Most of the code was taken from Sun Grid Engine (more specifically SGE 6.2 update 5 released in 2009), which was developed by Sun Microsystems. Using 6.2u5 as the starting point, we add new features and fixes to create Open Grid Scheduler/Grid Engine.

Looks like at one point Ubuntu included packages to install it. [ubuntu] Open Grid Scheduler qmaster host error

Open Grid Scheduler The official Open Source Grid Engine

Last Release: Grid Engine 2011.11p1

The latest release includes new features (e.g. Berkeley DB spooling improvement: removed dependency on NFSv4, hwloc support, GPU integration, ARM Linux port, gmake upgrade, Linux 3.x support, etc), together with other enhancements and bug fixes. See the release notes for more info!

Grid Engine 2011.11 is compatible with Sun Grid Engine 6.2u5. You can even run Grid Engine 2011.11 on a subset of nodes while the rest of the cluster runs the older 6.2u5 release - ie. as a drop-in replacement for 6.2u5.  Bugfix Release

The current bugfix & LTS (Long Term Support) release is version 6.2 update 5 patch 3 (SGE 6.2u5p3), which is based on Sun Grid Engine 6.2 update 5 (SGE 6.2u5). It was released on April 26, 2012 and only contains critical bug fixes.

Download Grid Engine/Grid Scheduler

See also: Quick Grid Engine Source Compile Guide.

Download: Grid Engine 2011.11 binary for x64 (now with the GUI installer) (available under the free SISSL license)

Build:

•  IzPack
• http://bioteam.net/2012/01/building-open-grid-scheduler-on-centos-rhel-6-2/

Compiling source code

There are several documents on the Web that can help

• Compiling Open Grid Scheduler - Grid Engine from Source
• Building Open Grid Scheduler on CentOS-RHEL 6.2 BioTeam

The primary difference is that our 6.2 OS includes an updated version of OpenSSL that breaks Grid Engine compilation. This document covers the patch necessary to complete the build on RHEL 6.2 and CentOS 6.2.

• Building A Grid Scheduler Cluster on Linux
• [gridengine users] Open grid compilation error

NEWS CONTENTS

• 200102 : Building Open Grid Scheduler on CentOS/RHEL 6.2 ( Building Open Grid Scheduler on CentOS/RHEL 6.2, )
• 200102 : Java, $JAVA_HOME and $PATH alterations ( )
• 200102 : BerkeleyDB version 4.4.20 ( )
• 200102 : Build binaries & manpages ( )
• 200102 : Edit Line #718 of scripts/distinst & set insthadoop=false to prevent failure due to missing "herd.jar": ( )
• 200102 : Edit near line #1551 to fix location of the libcrypto.so.1.0.0 file (it's looking in lib/ instead of lib64/ ) ( )

Old News ;-)

Building Open Grid Scheduler on CentOS/RHEL 6.2

chrisdag | Jan 12, 2012 | 27 comments

Contents

• Background & Disclaimer
• Skipped Build Items
• Differences between 6.2 and prior versions of CentOS and RHEL
• Build Environment
• Required Linux OS Packages
• Download Open Grid Scheduler 2011.11
• Download & Install External Dependencies
• IzPack 1.4.4
• Patch Grid Engine 2011.11 Source to support OpenSSL-1.0.0
• Customize the aimk.site file
• Customize the build.properties file
• Customize the ./clients/gui-installer/nbproject/project.properties file
• Hack up the aimk build file
• Build Grid Engine from Source
• Hack up the scripts/distinst binary staging script
• Edit the scripts/distinst.site configuration file
• Stage the distribution binaries by running the scripts/distinst script
• Build Grid Engine binary distribution archives
• Check /tmp/sge6_dist/ for the new archives

Background & Disclaimer

This blog page attempts to clean up and organize the scattered notes I maintain on how to build Grid Engine on various platforms. This post in particular is talking about Open Grid Scheduler version 2011.11 as built on CentOS 6.2 (a binary clone of RedHat Enterprise Linux RHEL 6.2).

First round of notes will cover building on x86-64 but I'll also try to document the same process for building 32bit executables as well.

Another reason for this document is that many of the online instructions for building Grid Engine take the "easy" way out by skipping the compilation of various features, services and subsystems. In particular I don't like refusing to compile with Java just because it's easier and I actually like having the java-based GUI installer present on my systems.

Skipped Build Items

The primary goal is to build as much of Grid Engine as possible. At this stage, the only things we are skipping are:
• ARCo
• sgeinspect binary
• Hadoop/Herd integration

Every other feature, service and 3rd party executable should be covered.

Differences between 6.2 and prior versions of CentOS and RHEL

The primary difference is that our 6.2 OS includes an updated version of OpenSSL that breaks Grid Engine compilation. This document covers the patch necessary to complete the build on RHEL 6.2 and CentOS 6.2.

Build Environment

Unless stated otherwise, assume that the build environment is a bare-bones CentOS 6.2 machine where I'll be building SGE as user "root" and installing many of the external dependencies into /opt/

Required Linux OS Packages

I started with a stripped down bare-bones CentOS 6.2 VM and had to install the following distro packages. I can't claim 100% that these are all the RPMs you will need but it's probably fairly comprehensive:

$ sudo yum -y install "openmotif openmotif-devel openmotif22 openssl-devel openssl-static make gcc \
autoconf pam-devel libXpm  libXpm-devel ncurses ncurses-devel unzip texinfo"

Download Open Grid Scheduler 2011.11

From the website: The 2011.11 release includes new features (e.g. Berkeley DB spooling improvement: removed dependency on NFSv4, hwloc support, GPU integration, ARM Linux port, gmake upgrade, Linux 3.0 support, etc), together with other enhancements and bug fixes.

See the release notes for more info!

# mkdir /opt/opengridscheduler-src
# cd /opt/opengridscheduler-src
# wget http://sourceforge.net/projects/gridscheduler/files/GE2011.11/GE2011.11.tar.gz/download?use_mirror=superb-sea2
# zcat GE2011.11.tar.gz | tar xvf -

Download & Install External Dependencies

There are a ton of them. Have patience.

Java, $JAVA_HOME and $PATH alterations

Visit http://www.oracle.com/technetwork/java/javase/downloads/index.html and grab a Java JDK. I used JDK 7u2 for x86-64 and selected the .tar.gz version for download.

# cd /opt/
# gunzip jdk-7u2-linux-x64.tar.gz
# tar xvf jdk-7u2-linux-x64.tar
# ln -s jdk1.7.0_02/ java

By making a symbolic link between /opt/jdk1.7.0_02 and /opt/java we can more easily set $JAVA_HOME to something convenient like "$JAVA_HOME=/opt/java".

We need to set some environment variables and PATH settings next. I chose to do this by making an /etc/profile.d/sge-build.sh entry with the following contents. How you set your path and environment variables depends on your personal preference and what shell you use. Note that the sge-build.sh file also includes info about Java ANT and JUNIT .jar files that will be installed later on.

## Java
export JAVA_HOME=/opt/java
export ANT_HOME=/opt/ant
export CLASSPATH=/opt/junit/junit3.8.1/junit.jar:/opt/ant/lib/ant.jar
export JUNIT_JAR=/opt/junit/junit3.8.1/junit.jar

 ## Path Tweaking
export PATH=$PATH:$JAVA_HOME/bin:$ANT_HOME/bin

ANT

Visit http://ant.apache.org/index.html and grab Apache Ant. The version used in this document is 1.8.2.

# cd /opt/
# gunzip apache-ant-1.8.2-bin.tar.gz
# tar xvf apache-ant-1.8.2-bin.tar
# ln -s apache-ant-1.8.2 ant

JUNIT

This is another dependency of the various Java bits within Grid Engine, in this case I'm using junit-3.8.1:

# cd /opt/
# mkdir junit
# cd /opt/junit/
# wget http://downloads.sourceforge.net/project/junit/junit/3.8.1/junit3.8.1.zip?use_mirror=superb-sea2
# unzip junit3.8.1.zip

We will need to reference the location of junit.jar in CLASSPATH and a few other locations, those are documented elsewhere on this page.

JavaCC

This is another dependency of the various Java bits within Grid Engine, this time from http://javacc.java.net/:

# cd /opt/
# mkdir other-sge-deps
# mkdir other-sge-deps/java
# cd /opt/other-sge-deps/java/
# wget http://java.net/projects/javacc/downloads/download/javacc-5.0.tar.gz
# gunzip javacc-5.0.tar.gz
# tar xvf javacc-5.0.tar

The configuration parameters that point to javacc are documented elsewhere on this page, the primary setting is within the build.properties file in the source directory.

IzPack 1.4.4

We need IZpack which is apparently a nice cross-platform Java tool for bundling application installers. I think, but am not 100% sure, that this dependency is required to build the graphical SGE GUI installer.

There is one small problem though. My working build environment seems to explicitly use an ancient version of IzPack that is not even available for download any more on the http://izpack.org/downloads/ website. Given that my build environment works with version 1.4.4 and I seem to also have much more modern versions laying around in "_disabled/" folders I'm guessing that SGE has a dependency on this specific (and old) version.

For convenience I've uploaded version 1.4.4 here for others to use if needed: http://bioteam.net/wp-content/uploads/misc/Izpack-1.4.4.tar.gz

# cd /opt/
# wget  http://bioteam.net/wp-content/uploads/misc/Izpack-1.4.4.tar.gz
# gunzip Izpack-1.4.4.tar.gz
# tar xvf Izpack-1.4.4.tar

The specific places where we mention the Izpack location in the SGE build & config files is highlighted elsewhere on this page.

BerkeleyDB version 4.4.20

It's actually important to use this (old) version of BerkeleyDB. It's the last version that supports RPC spooling to a remote BDB database and the SGE build process assumes that this functionality is present. In addition, there have been some indications on the mailing list that internal use of BDB for berkeley-db based spooling also assumes behavior and technical features associated with the older versions of the code.

Visit http://www.oracle.com/technetwork/database/berkeleydb/downloads/index.html and follow the link to "Previous Releases" where you can find the 4.4.20 version.

You will see below that we are building with "–enable-rpc" and installing into /opt/berkeley-db/ which is where the BDB bin/ and lib/ directories will end up.

# mkdir /opt/berkeley-db
# mv db-4.4.20.tar.gz berkeley-db/
# cd berkeley-db/
# gunzip -c db-4.4.20.tar.gz | tar xvf -
# cd db-4.4.20
# cd build_unix
# ../dist/configure --prefix=/opt/berkeley-db/ --enable-rpc
# make
# make install

Patch Grid Engine 2011.11 Source to support OpenSSL-1.0.0

If you see this error, you are hitting an issue with openssl-1.0 on your system:

You can see the bug report and patch at this URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=622849 and you can download the actual patch via this URL: http://bugs.debian.org/cgi-bin/bugreport.cgi?msg=10;filename=622849-openssl.patch;att=1;bug=622849

Place the patch file one level above the GE2011.11 source/ folder:

[root@grunt GE2011.11]# ls -l
total 1140
-rw-r--r--.  1 root root    6763 Jan 12 14:38 622849-openssl.patch
-rw-rw-r--.  1 dag  dag     5403 Nov 14 14:01 Changelog
-rw-rw-r--.  1 dag  dag  1121679 Nov 14 14:01 Changelog.SGE
drwxrwxr-x.  7 dag  dag     4096 Nov 14 14:02 doc
drwxrwxr-x.  2 dag  dag    20480 Nov 14 14:02 review
drwxrwxr-x. 12 dag  dag     4096 Jan 12 14:30 source

Now patch the actual ./source/libs/comm/cl_ssl_framework.c file …

[root@grunt GE2011.11]#  patch -p1 < 622849-openssl.patch
patching file source/libs/comm/cl_ssl_framework.c
[root@grunt GE2011.11]#

Customize the aimk.site file

The local file aimk.site file contains site-specific information about where your external dependencies are installed.

My current aimk.site has the following customizations (note that the info below is not the entire file, just the lines that I have added or changed..

Berkely-DB settings in aimk.site:

# BERKELEYDB_HOME the directory where the include and lib directory of
# Berkeley DB is installed
#
set BERKELEYDB_HOME = /opt/berkeley-db/
set BDB_INCLUDE_SUBDIR =
set BDB_LIB_SUBDIR =
set BDB_LIB_SUFFIX =

OpenSSL settings in aimk.site:

## dag (set values for openssl libs and include files)
set OPENSSL_SOVERSION = 1.0.0-10
set OPENSSL_HOME = /usr
set OPENSSL_LIB_DIR = /usr/lib64
set OPENSSL_EXTRA_LIBS = "-lkrb5 -lz "

JUNIT settings in aimk.site:

set JUNIT_JAR = '/opt/junit/junit3.8.1/junit.jar'

Customize the build.properties file

This file is where we set parameters related to the building of the java componants

JUNIT edits:

# junit jar file classpath
libs.junit.classpath=/opt/junit/junit3.8.1/junit.jar

JavaCC edits:

javacc.home=/opt/other-sge-deps/java/javacc-5.0

Customize the ./clients/gui-installer/nbproject/project.properties file

Yeah there is probably a better or cleaner way to do this but I found I had to edit the clients/gui-installer/nbproject/project.properties file in order to fix some issues with finding a certain iZpack/ant.jar file and the Java SWING layout stuff. Here are my changes

SWING edits in project.properties:

#Swing layout
#libs.swing-layout.classpath=${izpack.home}/../swing-layout-1.0.3.jar
libs.swing-layout.classpath=${izpack.home}/swing-layout-1.0.3.jar

ANT edits in project.properties:

#Ant
#libs.ant.classpath=${izpack.home}/../ant.jar
libs.ant.classpath=/opt/Izpack-1.4.4/ant.jar

Hack up the aimk build file

Note: We are really not supposed to be doing this but it's rare that I'm able to get through an SGE build without making minor alterations to the actual "aimk" compile script. This is where we can also get into personal preferences about how to build software so your own experience may be different,

Here are my edits…

I'm pretty sure this edit is meant to get around the fact that our libcrypto OpenSSL files are in "/usr/lib64″ instead of "/usr/lib" and the KFLAGS additions are meant to proactively avoid a new and different compile error that I forgot to document in my notes …

Near line #370 in ./aimk:

#set SECLIBS_STATIC    = "$OPENSSL_HOME/lib/libssl.a $OPENSSL_HOME/lib/libcrypto.a"                                                     
#set KLFLAGS           = "-L$OPENSSL_HOME/lib"                                                                                          
set SECLIBS_STATIC    = "$OPENSSL_HOME/lib64/libssl.a $OPENSSL_HOME/lib64/libcrypto.a"
set KLFLAGS           = "-L$OPENSSL_HOME/lib64 -lkrb5 -lz"

This next edit I believe is required to let the 'qtcsh' binary build properly, it complains unless we add "-ltermcap" …

Near line #3043 in ./aimk:

# Dag hack for qtcsh on CentOS 6.2 ...                                                                                                                      
# set SGE_LIBS = "-lsge -lpthread"                                                                                                                        
set SGE_LIBS = "-lsge -lpthread -ltermcap"

Build Grid Engine from Source

Whew! Now that all the pre-requisites, patching and hacking is done Grid Engine should ideally build smoothly from scratch at this point.

Refer to the README.build and README.aimk text files if needed. The commands that follow are taken straight from the build docs.

Prepwork

# ./aimk -only-depend
# scripts/zerodepend
# ./aimk depend

Build binaries & manpages

# ./aimk
# ./aimk -man

Hack up the scripts/distinst binary staging script

The "distinst" script is used to stage the newly built binaries into a location where the "mk_dist" script can finally start building .tar.gz archives for distribution. We need to make a few edits to it first …

Edit Line #718 of scripts/distinst & set insthadoop=false to prevent failure due to missing "herd.jar":

insthadoop=false

Edit near line #1551 to fix location of the libcrypto.so.1.0.0 file (it's looking in lib/ instead of lib64/ )

Note the two separate edits where we make changes from "lib" to "lib64/" below …

  elif [ -f $OPENSSLBASE/lib64/$libname ]; then
    #libname=$OPENSSLBASE/lib/$libname                                                                                                        
    libname=$OPENSSLBASE/lib64/$libname
  fi

Edit the scripts/distinst.site configuration file

We need to configure some values here to fix some OpenSSL and Berkeley-DB values.

# Base directory where the openssl binary and libraries reside
OPENSSLSOVERSION=1.0.0
OPENSSLBASE=/usr/

# Base directory where BDB resides
BERKELEYDBBASE=/opt/berkeley-db/

Stage the distribution binaries by running the scripts/distinst script

I decided my staging location was going to be "/opt/opengridscheduler-src/distinst-base/" so first we do:

# mkdir /opt/opengridscheduler-src/distinst-base/

And now we can actually run the distribution staging script!

# ./scripts/distinst -v -allall -bin -libs -basedir /opt/opengridscheduler-src/distinst-base/ -vdir GE2011.11

Build Grid Engine binary distribution archives

Now we can actually build distributable Grid Engine archives. The documentation recommends a symbolic link be placed inside your distinst "basedir" that points back to the "mk_dist" script:

# ln -s /opt/opengridscheduler-src/GE2011.11/source/scripts/mk_dist /opt/opengridscheduler-src/distinst-base/mk_dist

# cd /opt/opengridscheduler-src/distinst-base

And now we can actually run the command:

# ./mk_dist -vdir GE2011.11 -version GE2011.11 -common -doc -bin linux-x64

Check /tmp/sge6_dist/ for the new archives

Never had time to track down, change or care about this but by default it seems that "mk_dist" is placing our new binaries into /tmp/sge6_dist/ …

If things actually worked you should see something like this:

[root@grunt distinst-base]# ls -l /tmp/sge6_dist/
total 30384
-rw-r--r--. 1 root root 28162103 Jan 12 17:31 ge-GE2011.11-bin-linux-x64.tar.gz
-rw-r--r--. 1 root root  2946660 Jan 12 17:31 ge-GE2011.11-common.tar.gz
[root@grunt distinst-base]#
[root@grunt distinst-base]#

… and these are the standard "courtesy binary" archives that many of us are long familiar with.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019