down voteaccepted
As someone else said, TotalView is the standard for this. But it will cost you an arm and a leg.
The OpenMPI site has a great FAQ on MPI debugging. Item #6 in the FAQ describes how to attach GDB to MPI processes. Read the whole thing, there are some great tips.
|
|
||||
| Bulletin | Latest | Past week | Past month |
|
Up: Debugging Next: Starting the debugger when an error occurs Previous: Error handlersThe -dbg=<name of debugger> option to mpirun causes processes to be run under the control of the chosen debugger. For example, enter
mpirun -dbg=gdb or mpirun -dbg=gdb a.outinvokes the mpirun_dbg.gdb script located in the mpich/bin directory. This script captures the correct arguments, invokes the gdb debugger, and starts the first process under gdb where possible. There are five debugger scripts; ddd, gdb, xxgdb, ddd, and totalview. These may need to be edited depending on your system. There is another debugger script for dbx, but this one will always need to be edited as the debugger commands for dbx varies between versions. You can also use this option to call another debugger; for example, -dbg=mydebug. All you need to do is write a script file, mpirun_dbg.mydebug, which follows the format of the included debugger scripts, and place it in the mpich/bin directory. More information on using the Totalview debugger with mpich can be found in Section Debugging MPI programs with TotalView .
It is often convenient to have a debugger start when a program detects an error.
If mpich was configured with the option --enable-mpedbg, then
adding the command-line option -mpedbg to the program will cause mpich
to attempt to start a debugger (usually dbx or gdb) when an error
that generates a signal (such as SIGSEGV) occurs. For example,
mpirun -np 4 a.out -mpedbgIf you are not sure if your mpich provides this service, you can use -mpiversion to see if mpich was built with the --enable-mpedbg option. This feature may not be available with all devices.
Debugging SGI MPI Applications
TotalView can acquire processes started by SGI MPI, which is part of the Message Passing Toolkit (MPT) 1.3 and 1.4 packages.
Message queue display is supported by release 1.3 and 1.4 of the Message Passing Toolkit. See Displaying the Message Queue Graph for message queue display.
SGI MPI programs are normally started by using the mpirun command. You would use a similar command to start an MPI program under TotalView's control:
{ totalview | totalviewcli } mpirun -a mpirun-command-line
This invokes TotalView and tells it to show you the machine code for mpirun. Since you're not usually interested in debugging this code, use the Process > Go command to let the program run.
CLI Equivalent: dfocus p dgo
The SGI MPI mpirun command runs and starts all MPI processes. After TotalView acquires them, it asks if you want to stop them at startup. If you answer Yes, TotalView halts them before they enter the main program. You can then create breakpoints.
If you set a verbosity level that allows informational messages, TotalView also prints a message showing the name of the array and the value of the array services handle (ash) to which it is attaching.
Attaching to an SGI MPI Job
To attach to a running SGI MPI job, attach to the SGI MPI mpirun process that started the job. The procedure for attaching to an mpirun process is the same as the procedure for attaching to any other process. For details, see Attaching to Processes.
After you attach to the mpirun process, TotalView asks if you also wish to attach to slave MPICH processes. If you do, press Return or choose Yes. If you do not, select No.
If you choose Yes, TotalView starts the server processes and acquires all MPICH processes.
As an alternative, you can use the Group > Attach Subsets command to predefine what TotalView should do. For more information, see Attaching to Processes.
Debugging OpenMP code (Darryl Gove's blog)
Debugging OpenMP code
By Darryl Gove on Oct 10, 2007
The compiler flag -xopenmp enables the recognition of OpenMP directives. As a side -effect it also raises the optimisation level to -xO3. If you're trying to debug the code, then you'll not want the optimisation level raised then you can use the option -xopenmp=noopt which enables the recognition of OpenMP directives but does not increase the optimisation level.It's also worth compiling with the flags -xvpara and -xloopinfo which report parallelisation information.
Debugging OpenMP Applications TotalView User Guide (v6.3)
Debugging with DDT on the clusters - Research Computing
Chapter 13 Debugging OpenMP Programs (Sun Studio 12 Debugging a Program With dbx)
Debugging MPI By Dave Hadka | Water Programming: A ...
Debugging - High Performance Computing (HPC) - University Wiki
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|
|
You can use PayPal to to buy a cup of coffee for authors of this site |
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: March 12, 2019
