Cisco Logo


High Performance Computing Networking

In my last post, I talked about why MPI wrapper compilers are Good for you.  The short version is that it is faaar easier to use a wrapper compiler than to force users to figure out what compiler and linker flags the MPI implementation needs — because sometimes they need a lot of flags.

Hence, MPI wrappers are Good for you.  They can save you a lot of pain.

That being said, they can also hurt portability, as one user noted on the Open MPI user’s mailing list recently.

The user requested that Open MPI’s wrapper compiler support the same options as MPICH’s wrapper compiler.  As I said in my prior blog entry, on the surface, this is a very reasonable request.  After all, MPI is all about portability — so why not make the wrapper compilers be the same?

Unfortunately, there are several “like it or not, this is how the world works” kinds of reasons why wrapper compilers differ between MPI implementations.  Here’s a few of them:

>> 1. Wrapper compilers are not standardized.  Specifically: the MPI specification doesn’t mention wrapper compilers at all.  Hence, an implementation can do whatever it wants in terms of a wrapper compiler.

Indeed, some MPI implementations have wrapper compilers, some do not.  If memory serves, Cray MPI and IBM (AIX) MPI do not have wrapper compilers; you just add -lmpi (or something like it) when linking your MPI application, and all the Right magic occurs.

>> 2. If you have a wrapper compiler that understands its own command line options, you run the risk of accidentally intercepting command line options that were intended for the underlying compiler.

For example, in Open MPI, we chose a minimalist approach.  Open MPI’s “mpicc” only understands a single command line option: --showme.  All others are passed directly to the underlying compiler.  In this way, we minimize our chances of intercepting a real compiler option.

Other MPI implementations, such as MPICH2, went a different way.  They understand a handful of command line options (this is the exact portability point that the user was asking out).

>> 3. Code compiled by different C compilers can be compiled into a single executable and work just fine.  Except when it doesn’t.

Specifically, most C compiler vendors work quite hard to make their object code play nicely with others.  But compilers, just like any other piece of software, have bugs.  And these bugs can cause incompatibilities when compiling, linking, or running with code generated by other compilers (the run-time incompatibilities can be both incredibly subtle and exceedingly agonizing to debug).

Hence, the advice offered by the Open MPI team has always been that if you want to support different compilers, you should install Open MPI multiple times — once compiled by each different compiler that you want to install.

Many HPC installations use the Environment Modules package to help users manage switching between different MPI installations.

>> 4. Even though C code (usually) works between different compiler suites, C++ and Fortran have no such guarantees.  Their symbol-mangling schemes are usually quite different, making cross-linking impossible.  In practical terms, this means you usually need to install separate MPI installations for each C++ and Fortran compiler that you want to support.

Of course, many C++ and Fortran compilers emulate each others’ symbol-mangling schemes when given the appropriate command line flags.  But personally, I tend to distrust such things for the reasons noted in #3.

As noted above, these are annoying reasons.  The user advocate side of me screams “Just make it work!!”  But I’m not really sure how; there’s a lot of historical precedent and some technical issues outside the scope of MPI that would need to be solved first.

Suggestions welcome…

In an effort to keep conversations fresh, Cisco Blogs closes comments after 90 days. Please visit the Cisco Blogs hub page for the latest content.

6 Comments.


  1. On Cray XT5 and XE6, cc, CC and ftn are all effectively wrapper scripts that not only link in MPI but also BLAS, PAPI and other fun stuff for all possible compilers (Cray, GNU, PGI, Pathscale, Intel,…). All one has to do is load the proper modules and these scripts make the magic happen.

    For example, someone might do “module load PrgEnv-cray cce xt-libsci xt-papi perftools ugni pmi gni-headers dmapp” then “cc hello.c -o hello.x” and find that it works for a wide variety of “hello world” programs, including an MPI one, at least on a Cray XE6.

    Jeff

       0 likes

  2. The problem with wrappers is that they cause asymmetry in the configuration process because only one library can be The Most Important and thus allowed to use a wrapper. Unfortunately, “module” configurations are not portable and “unload” is consistently broken so you need to clear everything (e.g. log out) and reload only what you need. I think it is unfortunate that the standard did not specify a way for a script to discover how to compile and link.

       0 likes

    • I guess I don’t quite understand — wrappers can be layered. at least, Open MPI’s wrapper can be layered (I’m pretty sure MPICH2′s can be, too…?) by changing the underlying compiler command to be another wrapper compiler. Or, if that is not attractive, the wrapper can be used to extract the relevant flags. I don’t know if we made it clear in the docs, but Open MPI also supports the pkg-config system (e.g., “pkg-config ompi-c –cflags”).

      I’m also bummed to hear you say that “module unload” is usually broken. Is that a local configuration issue? I’ve seen complex modulefiles that do asymmetric things during load and unload, usually due to programmer error. FWIW: R. K. Owen told me recently that if a modulefile feels too complicated, it probably is.

      Admittedly, the whole system is not portable (e.g., the wrapper compiler flags, whether other MPI’s support pkg-config or not, etc.), which is a major bummer. :-\ But I do feel that it is correct for the MPI standard to stay away from specifying how to compile and link — every environment’s configure/build system is different. What we specify for POSIX might not play well with an IDE (e.g., Visual Studio, Eclipse, or Xcode).

         0 likes

  3. @Jed “mpicc -show” is pretty close to that but you should know how hard it is to do such a thing on every platform MPI aims to support. The MPI standard does not attempt to specify everything it could.

    A more important question is, “if someone cannot link an MPI program, should they be programming in parallel in the first place?” Everyone I know who can write correct parallel programs can figure out how to link against MPI. The converse is not true.

       0 likes

  4. @Jeff Squyres
    Both Open MPI and MPICH2 support pkg-config which is nice. Unfortunately, the pkg-config people made a mess of multiple configurations, so it works great as long as there is just one configuration of each package, but is a mess if you have many (built with different compilers, with and without debugging symbols, etc).

    I gave up on module unload after too many different systems on which it was buggy in the sense that unload did not completely clean up after itself. I never tracked down whether it was an upstream problem or just a “local configuration” problem that happened to affect every machine I used. In any case, I hate environment variables because they are fragile and you have to remember to reload them. I prefer to configure with the necessary absolute paths so that I can build and run multiple configurations without keeping long lists of modules around.

    I do not believe the MPI standard should specify how to compile and link, but I think it should specify a way for a script to determine how to compile and link. That could be specifying a machine-readable file format (even makefile variables) or an executable (e.g. mpi-config) that could be interrogated about flags for compiling and linking each language (mpicc etc might not exist, so “mpicc -show” isn’t the right name for this thing).

    Look at the autoconf, CMake, or PETSc BuildSystem scripts for figuring out how to use MPI. It’s a major hassle for no good reason.

    @Jeff Hammond
    I don’t want “mpicc -show” because I don’t want “mpicc” to be part of the standard. But I want a way for a script to learn how to link. This query is not complexity that is so fundamentally hard that it can’t be standardized and can only be taught by writing wiki pages for each computing center.

    “Everyone I know who can write correct parallel programs can figure out how to link against MPI.”

    This misses the point completely. Of course anyone competent can figure out how to link. This is not a problem that should require human interaction to solve, write down the answer, and then figure out how to inform every package they want to build about that answer. A person should be able to point to an “mpi-config” and the simple configure script from each project should be able to reliably compile and build with that MPI implementation.

       1 like

    • @Jed

      Both Open MPI and MPICH2 support pkg-config which is nice. Unfortunately, the pkg-config people made a mess of multiple configurations, so it works great as long as there is just one configuration of each package, but is a mess if you have many (built with different compilers, with and without debugging symbols, etc).

      Fair point. We initially didn’t do pkg-config support in OMPI because of that reason, and also because it didn’t offer (in our humble opinion) fine-enough granularity of the flags that you can query (e.g., you can’t separately query for the CPPFLAGS — which are not the same things as CFLAGS!).

      I gave up on module unload after too many different systems on which it was buggy…

      Bummer. But understandable.

      I do not believe the MPI standard should specify how to compile and link, but I think it should specify a way for a script to determine how to compile and link. That could be specifying a machine-readable file format (even makefile variables) or an executable (e.g. mpi-config) that could be interrogated about flags for compiling and linking each language (mpicc etc might not exist, so “mpicc -show” isn’t the right name for this thing).

      Hmm. How would such a file (for example) apply to an IDE? I can see how it might fit into make-based builds, and cmake, and scons, and … other script-based software-building environments that I have passing familiarity with.

      Is there a standard (even a de facto standard) that multiple IDEs use for “get a bunch of different compiler/linker flags from this file over here”?

         0 likes

  1. Return to Countries/Regions
  2. Return to Home
  1. All High Performance Computing Networking
  2. All Security
  3. Return to Home