Comments on: MCAPI and MPI

By: Jeff Squyres

Jeff Squyres — Mon, 12 Dec 2011 19:22:28 +0000

I just had a phone chat with one of the authors of the article, Sven Brehmer at Polycore Software. I’ll blog about the call in a little bit.

0 likes

By: Fab Tillier

Fab Tillier — Mon, 12 Dec 2011 18:28:17 +0000

I find it interesting that the whole premise of the article seems to be that MPI applications fundamentally use the dynamic process model. I’m not sure a lot of MPI applications could handle a “user’s inadvertent unplugging of a physical network cable”. In my experience, MPI apps expect the cluster they run on to be effectively static for the duration of the job, else they fail.

The article seems to boil down to “because MPI can run across the WAN, it is slow”, without acknowledging that the MPI libraries know and take advantage of the fastest interconnect available between any given processes.

0 likes

By: Fab Tillier

Fab Tillier — Mon, 12 Dec 2011 18:15:37 +0000

Microsoft Windows has had support for user-space to user-space copy for over a decade (since Windows 2000), allowing processes to move data in either direction via the ReadProcessMemory and WriteProcessMemory APIs.

Microsoft MPI takes advantage of this, though I don’t know if any other MPI libraries on Windows do (they really should).

0 likes

By: Jeff Squyres

Jeff Squyres — Mon, 12 Dec 2011 11:01:57 +0000

Yes, there were a few sweeping generalizations in that article that I found to be amusing.

0 likes

By: Jeff Squyres

Jeff Squyres — Mon, 12 Dec 2011 10:58:34 +0000

Correct. See also Jeff Hammond’s comments (and to confirm: XPMEM support is being developed in Open MPI as well).

0 likes

By: Jeff Hammond

Jeff Hammond — Sun, 11 Dec 2011 18:29:49 +0000

Ashley,

You might look at http://www.ipdps.org/ipdps2010/ipdps2010-slides/CAC/slides_cac_Mor10OptMPICom.pdf and related work on Nemesis in MPICH2 by INRIA and Argonne.

See also XPMEM (http://code.google.com/p/xpmem/), which is developed by some folks associated with OpenMPI.

On Blue Gene/P, MPI can exploit the static TLB map to directly access memory in other processes with no overhead, but this exists because of the unique properties of CNK, e.g. the bijective mapping of virtual and physical addresses.

0 likes

By: Jeff Hammond

Jeff Hammond — Sun, 11 Dec 2011 18:26:09 +0000

“Milliseconds matter. MCAPI can therefore be quick and responsive in a way that MPI can’t be.”

I find the insinuation that MPI has worse than millisecond latency to be rather hilarious.

0 likes

By: Ashley Pittman

Ashley Pittman — Sun, 11 Dec 2011 11:27:30 +0000

Actually, I take that back, it looks like both MPICH2 and OpenMPI now do this via knem, perhaps Jeff could confirm?

http://runtime.bordeaux.inria.fr/knem/

0 likes

By: Ashley Pittman

Ashley Pittman — Sat, 10 Dec 2011 13:49:26 +0000

Nobody has in-kernel userspace to userspace memory copy working again yet do they? Without this you have to use shared memory copy-in/copy-out buffers which halves the bandwidth.

At Quadrics we had two features here, firstly we could remap the whole of the BSS and heap allocators to shared memory so you could just memcpy() to and from remote address space and we had a modified kernel ptrace API that you could use to get the kernel to do direct userspace to userspace copy into a remote processes address space.

0 likes