VASP Performance optimization

Derived from

• Computational Platforms for VASP by Robert LORENZ Institut f ¨ur Materialphysik and Center for Computational Material Science Universit¨at Wien, Strudlhofgasse 4, A-1090 Wien, Austria
• VASP Performance Benchmark and Profiling - HPC Advisory Council

Vienna ab-initio simulation package version 5 (VASP 5) is an application package used for performing ab-initio quantum-mechanical molecular dynamics (MD).

Scalability of the package in cluster environment is limited due to limitation in parallization of algorithms used.   Both rackmount servers and blades are good candidates for High Performance Computing clusters running VASP, but certain questions remain unanswered: What hardware parameters are critical for VASP performance and which are not? What aspects of hardware architecture make it the ideal choice for running VASP?

First of all the ability to parallelize VASP is limitede. Unless the system and hardware are tuned to specific task, in cluster environment VASP does not scale well beyond 8 nodes or 128 cores and even that level of scalability is achievable only with Infiniband.  Without Infiniband 16 nodes are probably closer to a limit. Paradoxically if you use fewers cores of each server, for example 2 cores on 4 servers each, the speed is higher then on a single server with 8 cores. That suggests that for VASP memory bandwidth is a bottleneck. 

Among critical for VASP performance hardware parameters we can currently suggest the following (again this is a work in progress and our understanding of VASP so far is pretty superficial):

• CPU speed: the CPU throughput is somewhat important for the VASP performance, as many time consuming routines in vasp use BLAS/LAPACK; streaming methods(Intel’s SSE2) give an enormous performance boost. In any case speed should be higher then speed of memory which on modern servers is around 1.8GHz. But it is unclear how much higher is optimal. 

  • Number of cores per CPU is not important. moreover less might be better. Forget about 10 core per CPU and higher, they does not make much sense for VASP as bottleneck is  memory speed and bandwidth and more cores make this bottleneck more pronounced. So instead of traditions 8 core per CPU models 6 cores per CPU should be considered and 4 cores per CPU experimented with. That cut can cut costs without major degradation of performance.
     

  • Overall performance is limited by memory bandwidth. The second most important factor is the network latency and only then CPU speed. So more the double of the speed of memory (3.2 for 1.6GHz and 3.6 for 1.8GHz) will not produce much difference.  That exclude gains from over clocking. Increasing the CPU speed alone in not enough, you simultaneously need to increase network bandwidth and preserve low latency.
     

  • One need a fast data-path from network device to memory, which makes Infiniband a network architecture of choice.
     

• Memory size: VASP requires 512 - 1024 MByte  per CPU (core) minimum. 2GB is better. In this sense you might be OK with 64GB per 16 core node.
   

• Memory bandwidth: It is crucial factor; the computational speed of VASP depends on the sustained memory bandwidth. That means the Ivy Bridge architecture with its 1866 MHz memory is preferable.  And that memory should work at top speed (which limits the number of chips per socket to 4.
   

• HD size non critical, large enough to hold WAVECAR and CHG*. That allows to use SSD.
   

• The fast Fourier transformation  is an important VASP bottleneck. It can be ported to the GPU

How to improve performance?

• Use high performing settings (makefile, algorithm) included with the package.
• Use the best compiler available (Intel) & best possible hardware.
  • Use Intel BLAS/LAPACK
• Start from a makefile for a similar system
• Use appropriate compiler flags
• Try different CACHE SIZE settings. Per Peter Larsson (:

  The preprocessor option “-DCACHE_SIZE” influences the data structures used for computing fast Fourier transforms. My experience is that the best value is usually the one that is close to the L1 data cache of the CPU used. The chart below shows VASP run times using different cache size values, tested on an Intel Xeon E5345 processor with 32 KB of L1 data cache per core. Based on that value, we would use the initial guess of CACHE_SIZE=2000 (2000*16 bytes ≈ 32 KB). This choice give almost optimal performance, but 3000 and 4000 are marginally faster. AMD Opterons have 64KB of L1 cache per core, and there I, and others, have found CACHE_SIZE=5000 to be the optimal choice. The most important thing is to not settle with CACHE_SIZE=0 just because you don’t know what value to use (as is suggested in the comments of the source code). At least test a couple of values like 0, 2000 and 5000.

• Optimize MPI settings

Important parameters for cluster running vasp network

• Latency bound gives a lower bound for the MPI communication blocksize. It limits the maximum number of nodes. A typical Ethernet based network. (100MBit, 1000MBit) shows around 90μs latency. 
• Bandwidth bound the maximum transfer rate limits the communication speed. Vasp uses all–to–all communication
• Collision bound

Severe scalability limitation seen with Ethernet networks –1GbE, 10GbE and 40GbE performance starts to decline after 2 nodes.

QDR InfiniBand delivers the best performance for VASP among tested interfaces. Low latency is really important

• Up to 186% better performance than 40GbE on 8 nodes
• Over 5 times better performance than 10GbE on 8 nodes
• Over 8 times better performance than 1GbE on 8 nodes

QDR InfiniBand reduces the amount of time for MPI communications. With increase on nodes, MPI Communication time increases gradually, as the compute time reduces

• Communication pattern changes as more processes involved
  • 4 nodes: Majority of messages are concentrated at 64KB
  • 8 nodes: MPI_Alltoallv is the largest MPI time consumer, is largely concentrated at 64KB
  • 8 nodes: MPI_Bcast is the most frequently called MPI API, is largely concentrated at 4B
• Low latency in network communication is required to make VASP scalable
  • QDR InfiniBand delivers good scalability and provides lowest latency among the tested:
    • It is 186% faster versus 40GbE, over 5 times better than 10GbE and over 8 times than 1GbE on 8 nodes
  • Ethernet would not scale and become inefficient to run beyond 2 nodes
  • Mellanox messaging accelerations (SRQ and MXM) can provide benefit for VASP to run at scale
  • Heavy MPI collective communication occurred in VASP
• Enabling MXM in MPI enables better performance for VASP at high core counts
  • 47% higher job productivity than the untuned Open MPI run at 8 nodes
  • 19% higher job productivity than the SRQ-enabled Open MPI at 8 nodes
  • Special compilation flags for MPI are needed for VASP.
    • Compilation options for enabling MXM in Open MPI: -mca mtl mxm -mca btl_openib_free_list_num 8192 -mca btl_openib_free_list_inc 1024 -mca mpi_preconnect_mpi 1 -mca btl_openib_flags 9
    • Processor binding using dell_affinity.exe for all 3 cases 47% NPAR=8 19%

NEWS CONTENTS

• 20130726 : VASP Laboratory Computing Resource Center ( anl.gov )
• 20130726 : Benchmark Results for VASP ( Benchmark Results for VASP, )
• 20130726 : VASP Performance Benchmark and Profiling - HPC Advisory Council ( VASP Performance Benchmark and Profiling - HPC Advisory Council, )
• 20130726 : How to compile VASP - Micro and Nano Mechanics Group ( How to compile VASP - Micro and Nano Mechanics Group, )
• 200102 : How to compile VASP ( Micro and Nano Mechanics Group )
• 200102 : A study of improving the parallel performance of VASP by Baker, Matthew ( )
• 200102 : Abstract(summary) ( )
• 200102 : VASP on a GPU application to exact-exchange calculations of the stability of elemental boron hgpu.org ( 2011, Department of Physics, Carnegie Mellon University, Pittsburgh, PA 15213 )

Old News ;-)

[Jul 26, 2013] VASP Laboratory Computing Resource Center

anl.gov

VASP 5.2.11 and VASP 4.6.36 were compiled using the Intel-11.1 compilers and uses the Intel MKL libraries for a BLAS. As long as you run VASP with the "vasprun" that is included with the "+vasp-5.2.11" keyword, no further modification of your Softenv is necessary. VASP 5.2.11 can coexist with other MPICH compilers/libraries in your environment.

General notes:

... ... ...

• If you are running a calculation with only the gamma-point in your "integration" over reciprocal space, you can save a factor of two in elapsed time by using the gamma-point only version of VASP.

• The RMM-DIIS iterative matrix diagonalization algorithm (ALGO = Very_Fast) will give the best parallel performance.

• The parallel performance of VASP is very sensitive to the NPAR parameter on Fusion. You can improve performance by factor of two (or more) by using an optimum value of NPAR. Our tests with the medium Teragrid benchmark indicate that NPAR should be half the number of nodes for optimal performance.

• The LPLANE input variable should be set to ".FALSE." for best performance.

• VASP 4.6.36 is faster than VAPS.5.2.11. If you don't need the advanced features of VASP 5.2.11, you might consider using VASP.4.6.36.

Useful Tools:

• Vasp Data Viewer - A user recommended visualization tool for Windows.

• P4VASP - Another user recommend viz tool for VASP. This has been installed on Fusion but we are currently sorting out a few bugs in the code. Please email [email protected] if you are interested in using P4VASP.

• VESTA is a free crystal structure viewer and builder which can read and write POSCAR and CONTCAR files. VESTA can also visualize 3D data such are charge densities, potentials and orbitals from CHG, CHGCAR, PARCHG, LOCPOT and ELFCAR files from VASP.

Note that POSCAR and CONTCAR files for VASP.5.2.* can contain the element names on line 6, followed by the number of atoms for each element on line 7. This undocumented feature of VASP.5.2.* facilitates the input of coordinates and elements into VESTA.

• RINGS can extract pair distribution functions, mean square displacements and other properties from the trajectory files generated by VASP molecular dynamics runs.

• Graeme Henkelman's group, at UT Austin, maintain the TST tools for VASP. These tools provide methods for finding saddle points, evaluating transition state theory (TST) rate constants and Bader charge analysis with VASP.

• VMD can be used to visualize structures and trajectories from the VASP xml file. See http://www.uni-due.de/~hp0058/?file=vmdplugins.html&lang=en for more details.

Benchmark Results for VASP

Figure 1 - Performance and scalability of VASP-4.6.36 on Intel Westmere (X5650) comparing three binaries, Walltime (s) vs. Number of cores (NPAR=1). The speed up it is on the top of each bar.

Table 1 - Walltime of XRQTC Benchmarks on Intel Westmere (X5650) with Infiniband QDR (NPAR=1).

VASP Performance Benchmark and Profiling - HPC Advisory Council

How to compile VASP - Micro and Nano Mechanics Group

The binary (executable) file vasp can run in both serial mode (e.g. ./vasp) and parallel mode (e.g. mpiexec -np 4 vasp in a PBS script). The following table compares the time to run a simple benchmark case (one Au atom, LDA, ENCUT=400, ISMEAR=1, SIGMA=0.1, KPOINTS=21x21x21) using our executable here and the one available at /share/apps/vasp.4.6/bin/vasp. Our executable is about 70% faster.

How to compile VASP

Micro and Nano Mechanics Group

The following table compares the time to run the same benchmark case as above using both executables. Our executable shows speed up with multiple CPUs.

Number of CPUs	vasp compiled here	/share/apps/vasp.4.6/vasp	vasp another build (below)
1	76 (seconds)	75 (seconds)	48 (seconds)
2	59 (seconds)	72 (seconds)	34 (seconds)
4	35 (seconds)	64 (seconds)	29 (seconds)
8	37 (seconds)	65 (seconds)	31 (seconds)

... ... ...

This will create executable vasp in this directory. This time the executable vasp can not run interactively, but can only run in the queue through a PBS script (e.g. mpiexec --comm=ib -np 4 vasp).

Notice that here we need to specify the communication channel. (For mvapich1 we need to use --comm=ib and for mvapich2 use --comm=pmi.)

The following table provides the timing information for the same benchmark case studied above.

Number of CPUs	vasp compiled here
1	40 (seconds)
2	28 (seconds)
4	27 (seconds)
6	29 (seconds)
8	31 (seconds)
16	215 (seconds)

A Word of Caution: Make sure to run a few test cases to confirm your executable not only runs but produces the correct numerical results.

For example, we have found that on su-ahpcrc, function BRMIX (broyden.f) was giving serious errors. This was solved by changing the compilation options to "OFLAG=-O1 -mtune core2 -axW -unroll", and by changing "ICHARG = 0" to "ICHARG = 2" in INCAR. ("ICHARG=2" is the default when "ISTART=0" or if there are no CHG, CHGCAR, WAVECAR files in the folder.)

... ... ...

Intel MKL instructions

There are "official" instructions to compile VASP with the Intel Compiler family, named Using Intel MKL in VASP. Those instructions are unrelated with the present instructions but can be a good reference for future builds.

A study of improving the parallel performance of VASP by Baker, Matthew

Open Access Dissertation - ProQuest
2010, 46 pp. East Tennessee State University, M.S. degree.

Abstract(summary)

This thesis involves a case study in the use of parallelism to improve the performance of an application for computational research on molecules. The application, VASP, was migrated from a machine with 4 nodes and 16 single-threaded processors to a machine with 60 nodes and 120 dual-threaded processors. When initially migrated, VASP's performance deteriorated after about 17 processing elements (PEs), due to network contention. Subsequent modifications that restrict communication amongst VASP processes, together with additional support for threading, allowed VASP to scale up to 112 PEs, the maximum number that was tested. Other performance-enhancing optimizations that were attempted included replacing old libraries, which produced improvements of about 10%, and prefetching, which degraded, rather than enhanced, VASP performance.

VASP on a GPU application to exact-exchange calculations of the stability of elemental boron hgpu.org Maxwell Hutchinson, Michael Widom

2011, Department of Physics, Carnegie Mellon University, Pittsburgh, PA 15213

View Download (PDF) Source

General purpose graphical processing units (GPU's) offer high processing speeds for certain classes of highly parallelizable computations, such as matrix operations and Fourier transforms, that lie at the heart of first-principles electronic structure calculations. Inclusion of exact-exchange increases the cost of density functional theory by orders of magnitude, motivating the use of GPU's. Porting the widely used electronic density functional code VASP to run on a GPU results in a 5-20 fold performance boost of exact-exchange compared with a traditional CPU. We analyze performance bottlenecks and discuss classes of problems that will benefit from the GPU. As an illustration of the capabilities of this implementation, we calculate the lattice stability {alpha}- and {beta}-rhombohedral boron structures utilizing exact-exchange. Our results confirm the energetic preference for symmetry-breaking partial occupation of the {beta}-rhombohedral structure at low temperatures, but does not resolve the stability of {alpha} relative to {beta}.

Recommended Links

Softpanorama Recommended

Top articles

Sites

• VASP Performance Benchmark and Profiling - HPC Advisory Council
• Vasp performance - Google Search
• Computational Platforms for VASP
• Performance optimisation of VASP
• 3.7 Performance optimization of VASP
• Performance of Parallel VASP Jobs - Wolfgang
• Parallel performance of VASP
• HECToR " Optimising the performance of the VASP code on HECToR
• VASP - High Performance Computing Cluster at CWRU - Google Sites
• NERSC National Energy Research Scientific Computing Center
• NERSC National Energy Research Scientific Computing Center
• VASP at NERSC
• AAA Optimising the performance of the VASP code on HECToR
• VASP - High Performance Computing Cluster at CWRU

• Welcome to the HPC Knowledge Portal
• How to build VASP-4.6.36 and VASP-5.2.8 on Intel Westmere with Infiniband network
• VASP 5 Penn State
• Quantum Mechanical simulation package, VASP improved for multi ...
• hector.ac.uk

Compilation

• VASP 5.2 With Intel Compilers on 64-bit AMD Opteron Cluster CentOS Linux Lev Lafayette
• How to compile VASP on NSC's Triolith - Peter Larsson
• Compiling VASP.4.X, f90 compilers
• How to compile VASP - Peter Larsson
• UT theoretical chemistry code forum • View topic - Compiling VASP with bdrpro.F
• vasp - Computational Materials Physics - Universität Wien
• How to build VASP-4.6.36 and VASP-5.2.8 on Intel Westmere with Infiniband network

Forums

• UT theoretical chemistry code forum • View topic - Compiling VASP with bdrpro.F Texas university
• UT theoretical chemistry code forum • View topic - Compiling VASP with bdrpro.F

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019