How many ways to send?
Pop quiz, hotshot: how many types of sends are there in MPI?
Most people will immediately think of MPI_SEND. A few of you will remember the non-blocking variant, MPI_ISEND (where I = “immediate”).
But what about the rest — can you name them?
Here’s a hint: if I run “ls -1 *send*c | wc -l” in Open MPI’s MPI API source code directory, the result is 14. MPI_SEND and MPI_ISEND are two of those 14. Can you name the other 12?
Here’s a complete list:
- MPI_SEND: Standard send (a.k.a. “blocking”… but that’s not always true!)
- MPI_ISEND: Immediate (a.k.a. “non-blocking”) standard send
- MPI_SEND_INIT: Persistent standard send
- MPI_SSEND: Synchronous send
- MPI_ISSEND: Immediate synchronous send
- MPI_SSEND_INIT: Persistent synchronous send
- MPI_BSEND: Buffered send
- MPI_IBSEND: Immediate buffered send
- MPI_BSEND_INIT: Persistent buffered send
- MPI_RSEND: Ready send
- MPI_IRSEND: Immediate ready send
- MPI_RSEND_INIT: Persistent ready send
- MPI_SENDRECV: Standard send and receive
- MPI_SENDRECV_REPLACE: Standard send and receive into the same buffer
If you look closely, there’s really only 4 types of sends, but there’s 3 variants of each:
- Standard mode sends are what most people think of when they think of MPI sends. Standard mode sends may or may not block (depending on the underlying MPI implementation). When standard mode sends complete, it is guaranteed that it is safe for the application to use the buffer again. It is not guaranteed that the receiver has actually received the message yet (or even that it has been sent yet!).
- Synchronous mode sends are similar to standard mode sends, but provide one additional guarantee: a matching receive has been posted. This usually means that the receiver has at least started to receive the message.
- Buffered mode sends mean that the MPI will likely copy your message to internal storage before trying to send it. Buffered sends are evil. Don’t use these.
- Ready mode sends are an optimization. They are only correct when the application can guarantee that a matching receive has been posted. The thought is that the MPI implementation can take some shortcuts in sending if it knows that the receiver is already ready to receive this specific message.
For each of those 4 types, there’s 3 variants:
- The plain variant: exactly as described above.
- The immediate variant: also known as “non-blocking,” immediate calls return (more-or-less) immediately. The application will get an MPI_Request back that must be tested or waited on for completion. Progress on the send is occurring “in the background.”
- The persistent variant: a persistent send can be executed multiple times; they’re good for applications that need to send the same buffer repeatedly (e.g., sending boundary information in an iterative algorithm). The rationale is that the MPI implementation can incur the setup overhead for this particular send once, and then quickly initiate the actual sending mechanism each time after that.
The last 2 of the 14 are SENDRECV and SENDRECV_REPLACE. These are not really new modes / variants in themselves; they are actually both forms of the plain standard send. The interesting part about these two sends is that they are combined with a receive — but not necessary from the same recipient as the send!
For example, you can “send to the left” and “receive from the right” in a single call. Cool!
The REPLACE version allows you to use the same buffer to send and receive; MPI guarantees that the correct message will be sent and the receive buffer will contain the entire received message. It’s a good optimization for large messages; an application doesn’t need to use 2x the buffer space.