Softpanorama May the source be with you, but remember the KISS principle ;-)	Home	Switchboard	Unix Administration	Red Hat	TCP/IP Networks	Neoliberalism	Toxic Managers
	(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix

Installation of SCE on small slusters

News	Grid Engine	Recommended Links	SGE implementations	Installation of SGE on CentOS 7	Installation of Engine Master	Installation of the Execution Hosts
SGE implementations	Sun SGE 6.2u5	SGE 6.2u7 (Oracle Grid engine)	Son of Grid Engine	Using the command line installer	Using GUI installer	Usage of NFS in Grid Engine
SGE Troubleshooting	Gridengine diag tool	Duke University Tools	Perl Admin Tools and Scripts	UNIVA Grid Engine	Humor	Etc

Introduction
Key Decisions
Gather the Necessary Information
Disk Space Requirements
$SGE_ROOT Directory
NFS export
File Access Permissions
Network Services
Choosing Between Classic Spooling and Database Spooling
Group IDs
Administration Hosts
Scheduler Profiles

Introduction

Whether you have installed previous versions of the Sun Grid Engine software or this is your first time, you must do some planning before you extract and install the software. This section describes the decisions that you must make, and, wherever possible, gives you criteria on which you can base your decisions.

We also assume that NFS is used for sharing files with master host.

Degree of sharing is not that important but generally $SGE_ROOT/$SGE_CELL should be shared. Efficiency consideration that are sited by many are overblown and without careful measurements and determining real bottleneck you might fall into classical trap called "premature optimization". And as Donald Knuth used to say "Premature optimization is the source of all evil". and long before him Talleyrand gave the following advice to young diplomats: "First and foremost, not too too much zeal". Just substitute "young diplomats" for novice SGE administrators.

The same issue applies to a choice between classic spooling vs. Berkeley DB. Without measurements the selection of Berkeley DB is fools gold.

Key Decisions

You must make several decisions before you can plan the installation:

First you need to chose the implementation. See SGE implementations. If you want to use an open source implementation you might consider Son of Grid Engine
Ensure that all users of the Grid Engine system have the same user names UID and GID on all submit and execution hosts. You need a consistent UID/GID mapping (bioteam.net):
- How you implement does not matter
  - What matters is that everything is globally consistent
  - Verify UID/GID cluster-wide for SGE admin account and others
- Make sure your chosen group ID range is really unique
  - SGE asks for a GID range to use internally
  - Used for resource utilization monitoring
  - Range is arbitrary but defines max # of jobs that can run on one host
- Verify shared file system mount options
  - Another thing that can get out of sync on a cluster and cause odd problems
  - Root squash OK
  - SETUID squash not OK
Decide the location of SGE installation directory and the degree of sharing you want. Location is defined by environment variable called SGE_ROOT. Typically this is /opt/sge. For small installations /opt/sge/default can be shared between master host and execution hosts via NSF. You can export less, you can export more. See Optimizing usage of NFS in Grid Engine
Decide on the queue structure. Queue structure is generally a method of classification of your hosts. For example you can have different queues for servers with different number of cores, for example: c12.q, c32.q, c40.q. You can have multiple classifications.
Decide whether NFS server will be the same as master host or a separate server or, possibly, NSF appliance.

Gather the Necessary Information

Write down and print key data for you installation so that you can refer to them easily and have lesser chances to mix things up when installing execution hosts. You can also try to use the following bullet point as as a simple checklist:

Select the location of $SGE_ROOT directory. With RPM-based installation you are usually better off taking default. For example /opt/sge. It would be nice to put this directory on shared by NFS SSD drive, but that his optional.
Cell name: Use default
Select the server that will serve as a master host. Pleas note that SGE does need a cluster for functioning. It is just a powerful batch scheduling system. For small installations it can also be NFS server.
Decide which servers will be execution hosts.
Decide under which account SGE will be installed root of sgeadmin. Generally you should install it as sgeadmin user as this is more flexible arrangement.
Select communication ports:
- sge_qmaster port number 6444 is recommended
- sge_execd port number 6445 is recommended
Select type of spooling. Spooling mechanism can be Berkeley DB or Classic spooling. Classic spooling is better for small clusters. Disadvantages of binary spooling may outweigh the benefits. On small cluster you should probably use classic spooling. Berkeley DB is nice if you need to perform in the range of 150 qsub submissions per second. Or tens of thousands of jobs per day … This is not a typical workload for small and medium clusters.
Select set of administration hosts and Submit hosts. Any execution host can also be a submit host and administrative host, but the question is whether you want this.
Decide upon SGE scheduler tuning profile (Normal, High, Max). Normal is OK for beginners, who do not have experience with SGe and small clusters. You can change it later so this not an irreversible decision, unlike choice of spooling.
Decide which installation method is best for you (interactive via an installation script or GUI installer, if it is available). For master host you should use command line method. It provides you with better documentation of your steps. For execution host your mileage may vary. But on many SGE distributions GUI installer is iether unavailable or broken.
If you have both you have a luxury to decide which installation method is best for you ( GUI installer, is pretty nice for execution hosts as it allows installing execution hosts on as many servers as you wish. Interactive command line installation of execution hosts can be easily automated using Expect or TeraTerm Macros

Disk Space Requirements

The Grid Engine has extremely minimal space requirements (for modern size of disk drives disk ). And it is even somewhat strange to discuss it, But many years ago 1GBdrive was huge and 200MB big. Generally it will be OK to use 1G partitions which is nothing in modern time. Here are minimal requirements as listed by Sun in 2010:

50 Mbytes for the installation files without any binaries
Between 60 and 100 Mbytes for each set of binaries The ideal disk space for Grid Engine system spool directories is as follows:
50-200 Mbytes for the master host spool directories
50-200 Mbytes for the Berkeley DB spool directories The spool directories of the master host and of the execution hosts are configurable and need not reside under the default location, sge-root.

$SGE_ROOT Directory

You must create a directory into which to load the contents of the distribution media or install RPMs. This directory is called the root directory, or $SGE_ROOT. When the Grid Engine system is running, this directory also will store the current SGE configuration and all other data that must be spooled to disk.

Note the $SGE_ROOT environment variable is used to refer to the directory into which the Sun Grid Engine software is installed. The $SGE_ROOT directory is the top level of the Grid Engine software directory tree. On startup, each Grid Engine execution host needs read access to the $SGE_ROOT/$SGE_CELL/common directory.

For ease of installation and administration, this directory should be readable on all hosts on which you intend to run the Grid Engine software installation procedure. For example, you can select a directory that is available across a network file system, such as NFS. If you choose to select file systems that are local to the hosts, you must copy the installation directory to each host before you start the installation procedure for the particular machine. See File Access Permissions for a description of required permissions.

NFS export

Generally the less you share the faster SGE operates and less overhead it provides for computational jobs But you should avoid too much zeal.

For small SGE instances and implementation that uses TAR files for distribution, the whole $SGE_ROOT directory should be NFS exported.
If you install using RPMs and they have dependencies, you better share just $SGE_ROOT/$SGE_CELL directory (/opt/sge/default).

See Optimizing usage of NFS in Grid Engine for details.

Note

Because changing the installation directory or the spool directories requires a new installation of the system, use extra care to select a suitable installation directory. Note that all important information from a previous installation can be preserved. By default, the installation procedure installs the Grid Engine software, man pages, spool areas, and the configuration files in a directory hierarchy under the installation directory.

File Access Permissions

If you install the software logged in as root, you might have a problem configuring root read/write access for all hosts on a shared file system. Therefore, you might have problems putting the $SGE_ROOT files onto a network-wide file system.

You can force Grid Engine software to run all Grid Engine system components through a non-root administrative user account, for example sgeadmin. With this setup, this particular user needs only read/write access to the shared $SGE_ROOT file system.

The installation procedure asks whether files should be created and owned by an administrative user account. If you answer "Yes" and provide a valid user name, files are created by this user. Otherwise, the user name under which you run the installation procedure is used. Create an administrative user, and answer "Yes" to this question.

Make sure in all cases that the account used for file handling on all hosts has read/write access to the $SGE_ROOT directory. Also, the installation procedure assumes that the host from which you access the Grid Engine software distribution media can write to the $SGE_ROOT directory.

Network Services

If you use tar file based installation you need to define SGE network services in /etc/services.

If those lines are missing (RPMs add those lines) then add lines manually

sge_qmaster     6444/tcp                # Grid Engine Qmaster Service
sge_qmaster     6444/udp                # Grid Engine Qmaster Service
sge_execd       6445/tcp                # Grid Engine Execution Service
sge_execd       6445/udp                # Grid Engine Execution Service

Master Host

The master host controls the Grid Engine system. This host runs the master daemon sge_qmaster.

The master host has pretty minimalistic requremnts for hardware and can run on modern desktops quite well:

The host must be a stable platform.
The host must not be excessively busy with other processing.
At least 60 to 120 Mbytes of unused main memory must be available to run the Grid Engine system daemons. For very large clusters that include many hundreds or thousands of hosts and tens of thousands of jobs in the system at any time, 1 GByte or more of unused main memory might be required and 2 CPUs might be beneficial.
The master host must be installed before execution, administration, or submit hosts.

Choosing Between Classic Spooling and Database Spooling

During the installation, you are given the option to choose between classic spooling and Berkeley DB spooling. Use classic spooling. Berkeley DB make sense only for more then 32 nodes and even this is depends on speed of your netwrok and how many small tasks SGE will be running simultaneously. At the beginning you will always be better off with classic spooling. When you learn the ropes you can always switch.

Group IDs

You need to provide a range of IDs that will be assigned dynamically for jobs. The range must be big enough to provide enough numbers for the maximum number of Grid Engine jobs running at a single moment on a single host.

A group ID is assigned to each Grid Engine job to monitor the resource utilization of the job. Each job will be assigned a unique ID while it is running. For example, a range of 20000-20100 allows 100 jobs to run concurrently on a single host. You can change the group ID range for your cluster configuration at any time, but the values in the UNIX group ID range must be unused on your system.

Administration Hosts

Operators and managers of the Grid Engine system use administration hosts to perform administrative tasks such as reconfiguring queues or adding Grid Engine users.

The master host installation script automatically makes the master host an administration host. During the master host installation process, you can add other administration hosts. You can also manually add administration hosts on the master host at any time after installation.

Submit Hosts Jobs can be submitted and controlled from submit hosts. The master host installation script automatically makes the master host a submit host.

Cluster Queues The installation procedure creates a default cluster queue structure, which is suitable for getting acquainted with the system. The default queue can be removed after installation.

Note No matter what directory is used for the installation of the software, the administrator can change most settings that were created by the installation procedure. This change can be made while the system is running. Consider the following when determining a queue structure:

Whether you need cluster queues for sequential, interactive, parallel, and other job types
Which queue instances to put on which execution hosts
How many job slots are needed in each queue For more detailed information on administering cluster queues, see Configuring Queues.

Scheduler Profiles

You can choose from three scheduler profiles during the installation process: normal, high, and max. You can use these predefined profiles as a starting point for Grid Engine tuning.

Using these profiles, you can optimize the scheduler for one or more of the following:

The amount of information that is tracked about a scheduling run
The load adjustment during a scheduling run
Interval scheduling (the default) or immediate scheduling You can choose from three scheduler profiles:
normal – This profile uses load adaptation and interval scheduling, and reports all the information that the scheduler gathers during the dispatch cycle. This profile is the starting point for most grids.
high – This profile is more appropriate for a large cluster, where throughput is more important than gathering and reporting all the information from the scheduler. This profile also uses interval scheduling. Use this profile if you want to get better performance at the cost of getting less information about your scheduling runs.
max – This profile disables all information gathering and reporting, enables immediate scheduling, and disables load adaptation. Immediate scheduling is very useful for sites with high throughput and very short running jobs. The advantage of immediate scheduling decreases as runtime of the jobs increases. This profile can be used in clusters of any size where only throughput is important and everything else is a lower priority.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019