|
|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
| (slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix | |||||||
The Environment Modules package provides the dynamic modification of a shell environment variables and first of all $PATH and $LD_LIBRARY_LOAD.
They are alsmost standard package on HPc clusters but can be used for other purposes too. In all cases when you deal with multiple version of sotare they are very helpful/
this package was written in the early 1990s and is used at universities and research centers to deploy multiple versions of application packages. The key functionality -- ability to use multiple versions of the program or package from the same account by just loading proper module is more useful and withstand the text of time.
The key idea is trivial: package manipulates shell environment giving the user the capability to modify environment to suit partioicylar package (or versin of the package) by using command load. And than when you no longer need to restore environment ot previous state by using command unload. In pronicple this functionality fdoes not reure anything fancy. loading can be done by sourcing bash scripts and unloading by restoring environment from the file saved when you loaded the module.Still the implementation was done in TCL and that makes in much less tranparent then if it would be woirttin in pure bash.
Typical installation consists of multiple module files stored in tree of directories, one for each version of software used. After the modules software package is initialized, the environment can be modified on a per-module basis using the module command, which interprets module file and modify environment variables such as $PATH, $LD_LIBRARY_LOAD, $MANPATH, and others. Unlike modified for specific application .bashrc all module files can be shared by all users on a system. You can think about module files as the more sophisticated method of sourcing bash scipts that set variables for application. All other functionality is peripheral ot his key functionality of the package.
As a user, you can add and remove module files from the current environment, but in reality this is really done (most users are really afraid of TCL ;-). In most installation module files are maintained centrally, which ahs its advantages and disadvantages. But that's how it is.
The environment changes contained in a module file can also be summarized through the module show command. yo can also diff environment befor and after loading of a particular module.
If you use a particular application all the time you can invoked the necessary modules modules from .bashrc file saving some typing. Othersize you can create source files for each application and store them in some directory in your home directory. for example /Rclib. In this case all modules that you need to load for say version 4.1 of CP2K application can be loaded with the command . ~/Rclib/cp2k
Environment Modules are "shell agnostic" and work with all major shells such as bash, ksh, zsh, sh, tcsh, and csh. But in reality bash dominates the field and trhat capability is not that useful.
They also implement the concept of "default" version, which is helpful.
All this explains why Environment Modules are now a part of most HPC cluster setup. It also inspired several alternative implementation such as lmod from University of Texas, which is written in LUA instead of TCL. LUA is more popular language that is known probably by more administrors then TCL but still it far from mainstream scripting language such as Perl or Python.
The modules system is based on modulefiles. Each modulfile specifies groups of environment settings that need to be set together. Modulefiles can be installed in a central location for general use, or in a user directory for personal use. Environment Modules modulefiles are written in the Tcl (Tool Command Language) and are interpreted by the modulecmd program.
modulefiles are created on per application per version basis. They can be dynamically loaded, unloaded, or switched. Along with the capability of using multiple
The default modules search path is in a hidden configuration file you can display with:
cat ${MODULESHOME}/init/.modulespath
The /etc/modulefiles directory used by some distributions (or any other directory) can be used after a build from source by modifying the ${MODULESHOME}/init/.modulespath file.
The ${MODULESHOME}/modulefiles/use.own module essentially performs these steps.
mkdir $HOME/privatemodules
cp ${MODULESHOME}/modulefiles/null $HOME/privatemodules
module use-append $HOME/privatemodules
The default strorage directory for modules is /etc/modulefiles directory. and it is populated with several modules during the installation of the package from RPM. The $HOME/privatemodules or another directory can be used instead along with "module use --append" or modification of the configuration file specifying the default modules search path.
The default modulefiles directory is empty initially. Copy the null module to the default modulefiles directory to have it shown by "module avail". The following uses the null and module-info modules to show use of a version file within a hierarchical organization and their affect on module avail and module show:
mkdir /etc/modulefiles/test
cp ${MODULESHOME}/modulefiles/null /etc/modulefiles/test/2.0
cp ${MODULESHOME}/modulefiles/module-info /etc/modulefiles/test/1.0
module avail
module show test
Set the first version as the default:
echo '#%Module' > /etc/modulefiles/test/.version echo 'set ModulesVersion "1.0"' >> /etc/modulefiles/test/.version module avail module show test module show test/2.0
Switch to the newer version as the default with:
rm /etc/modulefiles/test/.version echo '#%Module' > /etc/modulefiles/test/.version echo 'set ModulesVersion "2.0"' >> /etc/modulefiles/test/.version module avail module show test
After the above following a fresh install from source you would see:
$ module avail -------------------- /usr/local/Modules/3.2.10/modulefiles --------------------- dot module-git module-info modules null use.own ------------------------------- /etc/modulefiles ------------------------------- null test/1.0 test/2.0(default) --------------------------- /home/user/privatemodules -------------------------- null
$ module show test ------------------------------------------------------------------- /etc/modulefiles/test/2.0: module-whatis does absolutely nothing -------------------------------------------------------------------
$ module show test/1.0 ------------------------------------------------------------------- /etc/modulefiles/test/1.0: module-whatis returns all various module-info values +++ module-info +++++++++++++++++++++++++++++++ flags = 2 mode = display name = test/1.0 specified = test/1.0 shell = bash shelltype = sh version = test/1.0 user = advanced trace = 0 tracepat = -.* symbols = *undef* +++ info ++++++++++++++++++++++++++++++++++++++ hostname = localhost level = 1 loaded null = 0 library = /usr/local/lib/tcl8.6 nameofexecutable = sharedlibextension = .so tclversion = 8.6 patchlevel = 8.6.1 +++++++++++++++++++++++++++++++++++++++++++++++ -------------------------------------------------------------------
Environment Modules on Scientific Linux, CentOS, and RHEL distributions in the environment-modules package include modules.csh and modules.sh scripts for the /etc/profile.d directory that make modules initialization part of the default shell initialization.
This makes managing complex environments a bit less complicated.
For a source build the automation for all users can be manually configured.
/etc/profile.d/modules.sh from the etc/global/profile.modules file in the 3.2.10 modules build directory.
trap "" 1 2 3
case "$0" in
-bash|bash|*/bash) . /usr/local/Modules/default/init/bash ;;
-ksh|ksh|*/ksh) . /usr/local/Modules/default/init/ksh ;;
-zsh|zsh|*/zsh) . /usr/local/Modules/default/init/zsh ;;
*) . /usr/local/Modules/default/init/sh ;; # default
esac
trap 1 2 3
Copy the etc/global/profile.modules file from the 3.2.10 modules build directory to the system initialization directory:
sudo cp etc/global/profile.modules /etc/profile.d/modules.sh
Add a version 3.2.10 symbolic link for the above generic addresses:
cd /usr/local/Modules sudo ln -sT 3.2.10 default
A /etc/profile.d/modules.csh symbolic link to the etc/global/profile.modules file in the 3.2.10 modules build directory can enable automatic modules initialization for these users.
To get a usage list of module options type the following (the listing has been abbreviated to only those commands discussed on this page) :
% module help
Available Commands and Usage:
+ add|load modulefile [modulefile ...]
+ rm|unload modulefile [modulefile ...]
+ switch modulefile1 modulefile2
+ display modulefile [modulefile ...]
+ avail path [path]
+ list
+ help modulefile [modulefile ...]
To get help on a specific module use:
% module help [modulefile]
% module list
This lists all the modulefiles that are currently loaded into your environment.
% module avail
This option lists all the modulefiles that are available to be loaded. Notice that many of them have version numbers associated with them and that where there is more than one version, one is labeled as the default. You can also restrict this command to a single package; for example,
module avail visit
% module display [modulefile]
Use this command to see exactly what a given modulefile will do to your environment, such as what will be added to the PATH, MANPATH, etc., environment variables. This is synonymous with
% module show [modulefile]
% module load [modulefile1] [modulefile2] ...
This command adds one or more modulefiles to your current environment. It does so silently, unless there is a problem with a modulefile. If you load the generic name of a module, you will get the default version. To load a specific version, load the modulefile using its full specification. For example,
% module load visit
will do the same thing as
% module load visit//2.1.2
% module unload [modulefile]
This removes the listed module from the user's current environment. Modulefiles can be removed in any order. Note that this command will fail silently if the modulefile you specify is not already loaded.
% module switch [modulefile_old] [modulefile_new]
This command demonstrates the true advantage of using modules. Different versions of entire software packages can be replaced with a single module command.
This is synonymous with
% module swap [modulefile_old] [modulefile_new]
Note that the following command would be completely correct and appropriate if, before issuing it, you had issued a "module load" command for a non-default version of the VisIt software and you wish to switch to the default version:
module swap visit visit
Please refer to "module" man page for more details.
You can modify your environment so that certain modules are loaded whenever you log in. Put your changes in one of the following files, depending on your shell.
Here is an example of a .cshrc.ext file with commands to load modules. Notice that, modules specific to a platform are put inside of if-then blocks.
if ($NERSC_HOST == "hopper") then # Replace the following line with personal settings for Hopper module load fftw endifBack to Top
You can create and install your own modules for your convenience or for sharing software among collaborators. The module definition files can be in a project directory, your home directory, or any available file system. Make sure the UNIX file permissions grant access to all users who want to use the software. (Note: do not give write permissions in your home directory to anyone else!)
An example of such a modulefile "1.2.7" for the customized module "myzlib" is located at /global/project/projectdirs/mpccc/usg/modulefiles/hopper/myzlib. This modulefile includes a nice feature to detect the current compiler environment on Cray systems (such as pgi or intel, with the definifition of $LCcompiler), and adds the corresponding include and libaray paths automatically (with the definition of $XXX_POST_LINK_OPTS and $XXX_INCLUDE_OPTS) to the compiler wrappers. To use this module, just do the following:
% module use /global/project/projectdirs/modulefiles/hopper % module load myzlib/1.2.7
Note that the "module use" command adds this new directory before other module search paths (defined as $MODULEPATH), so modules defined in this custom directory will have precedence if there are other modules with the same name in the module search paths. If you prefer to have the new directory added at the end of $MODULEPATH, use "module use -a" instead of "module use" in the above command.
Google matched content |
...
|
|
|
|