Which to use: tags or communicators?
A common question from new MPI developers is: which should I use to separate my messages — tags or communicators?
If you didn’t already know MPI offers two key abstractions for message passing:
- Message delineation. If you’re a TCP sockets programmer, you’re used to receiving streams of bytes. For example, if you try to receive 32 bytes, you might receive 17 bytes, meaning that you have to loop around and try again to receive the remaining 15 bytes. MPI doesn’t have streams; MPI only has atomic messages. For example, if you send 16 integers, the receiver will receive 16 integers (not 15 integers, not 17 bytes — they’ll receive the entire 16 integers all at once).
- Message separation. As mentioned in the first sentence, MPI offers two key mechanisms for separating messages: tags and communicators. We’ll dive into both in this blog post; I’ll explain the differences and when you might want to use one over the other.
The reason that people ask is because both tags and communicators are arguments to MPI’s point-to-point functions. For example:
- MPI_Send(buffer, count, datatype, dest_rank, tag, communicator)
- MPI_Recv(buffer, count, datatype, source_rank, tag, communicator, status)
Let’s start by defining both terms.
A communicator is an MPI term that means an ordered group of MPI processes with a unique communication context.
For example, when an MPI job is launched, all the processes in that job are put into a default communicator named MPI_COMM_WORLD (or “MCW”, for short). Each process gets a unique integer rank identifier in that communicator in the range [0, N) (where N is the number of processes in the job).
Communicators can be duplicated, and in the duplication process, can be subsetted and re-ordered. Hence, an individual MPI process can belong to any number of communicators, and may be a different rank in each of them.
A tag is simply an arbitrary integer that is used to delineate the matching of point-to-point messages.
Specifically, if a sender sends a message to MCW rank X on tag Y, then the receiver must match these specifications in order to receive the message. That is, receiver must specify MCW as its communicator, the same Y value for the tag (or the special wildcard MPI_ANY_TAG), and the sender’s rank in MCW (or the special wildcard MPI_ANY_SOURCE).
Hence, if the (rank, tag, communicator) tuple matches between the sender and receiver, the message will be delivered (…according to MPI’s ordering rules, but that’s a topic for a different blog entry).
So it looks like either (or both) of the tag and communicator arguments can be used to delineate the receipt of individual messages. When should you use each one? Here’s the rule of thumb I use:
- Communicators are heavy-weight objects. Creating a new communicator takes time and consumes internal MPI resources. I generally create new communicators for the following cases:
- When different abstraction layers within my application need “safe” communication scopes.
- When I need to subset the processes from a parent communicator (e.g., I have a significant operation that only needs to be performed by half the processes in MCW).
- When I need to re-order the processes from a parent communicator (e.g., my MPI processes have a logical ordering that is different than their “native” MCW rank).
- If I don’t need a communicator (by the above guidelines), I use tags.
For example, if I have an MPI application that uses 3 sub-libraries, then I ensure that each of my 3 sub-libraries creates and uses their own communicator. This will prevent the messages from one sub-library accidentally matching messages from a different sub-library that it knows nothing about.
Within each of those sub-libraries, for most messaging, I simply use tags. For example, I’ll use one tag for sending command messages, and a different tag for sending data messages. Or maybe I’ll use multiple different tags for commands and different types of data (depending on the application needs).
In short: use communicators when you need a whole new/safe communication scope, or you need to change your existing scope (e.g., subset and/or reorder the member processes). Use tags for everything else.