Cisco Logo


High Performance Computing Networking

A while ago, Brock Palen tweeted me an MPI question: how does one send Standard Template Library (STL) C++ objects in MPI?

The problem that Brock is asking about is that STL objects tend to be variable size and type.  The whole point of the STL is to create flexible, easy-to-use “containers” of arbitrary types.  For example, STL lists allow you to create an arbitrary length list of a given type.

To cite a concrete example, let’s say that my application has an STL vector object named my_vector that contains a bunch of integers.  What parameters do I pass to MPI_SEND to send this beast?

The short answer is: use boost.mpi.  boost.mpi is a subsection of the Boost software package that supports several nice C++ language features with MPI, the easiest to cite of which is the ability to send and receive STL objects.

According to its documentation, boost.mpi “supports the majority of MPI 1.1.”  Ouch! What do you do if you need more than MPI-1.1 functionality?

Do not fear!  Since boost.mpi is just an abstraction layer over the native C MPI bindings, you can certainly mix the “raw” C MPI API in the same application with boost.mpi.

But I digress. Back to the original question…

The boost.mpi documentation has a nice tutorial explaining the basics.  For most STL types containing intrinsic C types (e.g., the vector of integers example from above), you can just send/receive them and boost.mpi does the right magic behind the scenes.  Something akin to:

boost::mpi::communicator world;
std::vector<int> my_vector;

if (world.rank() == 0) {
    my_vector.push_back(17);
    my_vector.push_back(38);
    world.send(1, 0, my_vector);
} else {
    world.recv(0, 0, my_vector);
}

For sending an receiving user-defined objects in an STL container, the Boost Serialization library can be used.  You essentially add a “serialize” method on your class, which then allows the boost.mpi innards to pack and unpack STL container contents properly.  

Refer to the boost.mpi and Boost Serialization documentation for more details.

In an effort to keep conversations fresh, Cisco Blogs closes comments after 90 days. Please visit the Cisco Blogs hub page for the latest content.

7 Comments.


  1. How does this work with one-sided, where remote unpacking must be done inside of MPI, rather than in the glue code between Boost::MPI and the MPI C bindings?

    It would be good to show potential users how to use MPI datatypes in the official API to communicate STL objects, rather than to punt this problem to “magical” Boost and its antiquated MPI support.

       0 likes

    • boost.mpi doesn’t work with one-sided. boost.mpi only “mostly supports MPI-1.1″.

      Since the MPI bindings are in C, they don’t have good native support for things like the STL. That’s why boost.mpi exists. I agree that its support is a bit antiquated at this point; it would be great if someone could update it to include a bit more than “most of MPI-1.1″.

      Additionallym such high-level concepts tend to not map well to hardware-offload scenarios. Meaning that one-sided operations of STL-like containers won’t map well to hardware-accelerated message passing solutions because the hardware won’t understand the (potentially) complex run-time created data layouts — such code paths will likely fall through to software code paths (i.e., potentially slow).

      Finally, don’t forget that STL containers are allowed to hide their exact internal memory structure, and are even allowed to change it. For this reason, constructing an MPI datatype may not be sufficient for sending / receiving an STL container as elegantly as the boost.mpi solution because a) of potential memory allocation issues at the receiver, and b) the memory may well be non-uniform at either the sender or receiver (i.e., the expanded typemap may not be N times a single datatype typemap).

         0 likes

  2. Regarding the documentation of Boost.MPI — here’s the version-invariant link: http://www.boost.org/doc/html/mpi.html

    Another trick you can use is to replace the version in URL with word “release” (this shall point to the docs for the newest version), i.e., instead of http://www.boost.org/doc/libs/1_48_0/doc/html/mpi.html use http://www.boost.org/doc/libs/release/doc/html/mpi.html (notice how “1_48_0″ was replaced with “release”).

    HTH!

       0 likes

  3. I am not sure how much I like or dislike this answer. Boost is cool, more people should use libraries, *cough* MPI *cough*, but C++ has portability issues and while some things like the STL are well supported across compilers I have had issues with Boost and the Intel C++ compiler which is our default. The get the entire thing to play nicely you really need to use g++ which is fine just limiting.

       0 likes

    • That is a fair point.

      Wearing my “MPI Forum” hat, I have to say that STL is a higher-level abstraction that lower-level communication libraries cannot possibly understand. MPI provides the building blocks for upper layer abstractions — like the STL — to take advantage of. boost.mpi is a good example of this, but as you mention, it has limitations.

      Wearing my “I’m an MPI apps programmer” hat, I have to say, that sucks. :-(

      FWIW, I think it would be great if someone developed a smaller, less all-encompassing MPI C++ class library (as compared to boost.mpi). Something that does a few things, and does them well — that would be nice. Case in point: it took me 4+ hours to write this blog entry because I have to download and install boost first (which took several attempts to get right). Don’t get me wrong — boost is a very, very nice tool. But it is ginormous and more than most MPI developers may need. A small, purpose-built MPI C++ library (that includes the ability to send/receive STL containers) could be quite awesome.

         0 likes

    • Brock, you should really report those issues as bugs to your compiler vendor (Intel in case of ICC) as they indicate non-compliance with the ISO C++ standard which Boost adheres to strictly.

      A vast majority (99%+) of the Boost libs relies on C++2003 or even C++1998. C++2011 has only begun to be considered recently and that usually in the context of providing a portable implementation of C++2011 features for C++2003 compilers such as Boost.Move (that being said, about 10 of Boost libraries have been incorporated into the C++2011 standard, so any quality issues with an implementation of Boost today indicate a potential issues in the future when C++2011 code and libs become more prevalent). It’s 2012 now, nearly a decade, so non-compliance isn’t really excused anymore.

      That being said, we live in an imperfect world, so the libs are regularly tested — see, for instance:
      http://www.boost.org/development/testing.html#Introduction
      http://www.boost.org/doc/libs/release/libs/math/doc/sf_and_dist/html/math_toolkit/main_overview/compilers_overview.html

         0 likes

  1. Return to Countries/Regions
  2. Return to Home
  1. All High Performance Computing Networking
  2. All Security
  3. Return to Home