But just to clarify your point: hardware send queues are typically fairly deep, capable of holding thousands of pending sends. So the coalescing code in Open MPI probably won’t be triggered by just 4 sends. It’ll typically be triggered by ping-pong benchmarks (yet another reason benchmarks are not good indicators of real performance!) and other sending-many-thousands-of-sends-at-a-time types of codes.

   0 likes

For example let’s say you have this sequence of 4 messages:
MPI_Send(&i, 1, MPI_INTEGER, peer, tag, comm);
MPI_Send(&j, 1, MPI_INTEGER, peer, tag, comm);
MPI_Send(&i, 1, MPI_INTEGER, peer, tag, comm);
MPI_Send(&j, 1, MPI_INTEGER, peer, tag, comm);

The way you explained the implementation makes me thing that most likely the first message is dispatched immediately (and assuming that the send queue only accept 1 message) and the next 3 will be coalesced together and sent out.

Therefore this runtime solution is not solving *completely* the problem, if the developer would have coalesced the sends in the source code probably the performance will increase (also considering all the above considerations you discussed in the post).

thanks again.

   0 likes

For example, when the hardware send queue is full, but the application is still sending more new (short) messages. If successive new messages in this case are of the same MPI signature (e.g., to the same receiver on the same CID with the same tag), then Open MPI will coalesce the messages together.

Hence, when the send queue eventually opens up, there will be fewer entries placed on the queue than MPI messages that are actually sent.

Make sense?

This was a particularly ugly feature to implement and debug. But I’d be surprised if other OpenFabrics-based MPI implementations don’t do similar things (i.e., coalesce only when otherwise blocked from sending).

   0 likes

Since you are an expert, do you know whether MPI libraries are able to automatically coalesce messages? Some time ago I came across a parameter of the OpenMPI’s MCA framework called “btl_openib_use_message_coalescing” which should do that. However I have troubles figuring out how such thing is concretely implemented. To me there should be a kind of timer that waits “enough time” for several messages to be in the send buffer and then dispatches them all together to the receiver… but this would have a very bad impact on latency… and I don’t think this would even be allowed in MPI.

So the question remains!

   0 likes