You state that GPU buffers are similar to RDMA buffers. I wholehearted disagree. The former is a conscious choice by the user, while the latter is an internal implementation issue in the MPI library. A user of MPI should never know that they’re using RDMA under the covers, and memory registration is a burden that the MPI implementation takes on in exchange for better performance.

Said another way, a program using a GPU *knows* it’s using a GPU, and usually exactly which GPU vendor to boot. Such a program could just as easily call

MPI_Send(gpuPtr, NUM_ELEMENTS, OMPI_GPU_DOUBLE, dest, tag, comm);

You might argue that an implementation-specific datatype handle makes the program no longer portable, and I’ll counter that to allow using a GPU pointer in the first place requires knowledge that the MPI library supports GPU pointers, and loses portability just the same, but with potentially more difficult error manifestation (imagine treating a GPU pointer as a host pointer!)

Anyway, I think the end results will be better if the application is honest with the MPI library that it is using GPU buffers, rather than trying to deduce it from the pointer value. The portability issue could be resolved by the MPI Forum via standardizing the GPU datatypes, of defining a way of tagging a datatype’s buffer type.

Some other optimizations are GPUDirect RDMA (http://docs.nvidia.com/cuda/gpudirect-rdma/index.html) where the HCA could do peer-to-peer data transfers directly to/from GPU memory. This is independent of how the MPI library discovers that a buffer is a GPU buffer (whether by querying CUDA, or being told by the user).

   0 likes