0 likes

> currently one major API works across all O/Ses and that is Sockets

Just to be clear, this is not just Sockets, but synchronous Sockets (blocking or non-blocking, but not aio/overlapped). There is a lot to be gained by moving away from synchronous I/O, though the learning curve for async I/O can be steep. Throw in the concept of scatter/gather and it gets even steeper.

High performance asynchronous I/O applications in Windows can benefit from using I/O completion ports, rather than using per-I/O event objects and WaitForMultipleObjects (which has a limit to how many objects can be waited on concurrently). I/O completion ports allow aggregating completions from multiple files or sockets, and the application can poll the completion port, or block on it waiting for any I/O completion to be added. Multiple threads can poll events form an I/O completion port. MSMPI today uses I/O completion ports internally, so that we can block for completions if we exceed our polling limit. MSMPI supports blocking for completions for all of our communcation channels, SHM, NetworkDirect, and Sockets.

It would be great to allow CCI endpoints to be associated with a user’s I/O completion port, allowing users to get completions for CCI events side by side with their file completions, all through one function (GetQueuedCompletionStatus). There are design issues that you’ll need to work out, though (who provides the OVERLAPPED structure that identifies the I/O operation and is returned by GetQueueCompletionStatus, for example).

Cheers,
-Fab

   0 likes

Verbs works on most O/Ses and provides access to modern networking features, yet no one would argue that it is a simple API. We believe there is a middle ground.

Forgive me for not knowing the optimal I/O model for Windows, perhaps you could give me an example. CCI is inherently an asynchronous API (similar to MPI). You initiate a send (small message or RMA) and poll for completion. If the app prefers to block via a native O/S method (e.g. select(), poll(), epoll(), kqueue(), WSA*(), etc.), CCI can provide a native OS handle to the application. If CCI does not allow for a high performance implementation in Windows, we would be very interested in what changes CCI would need in order to provide one.

CCI gives the choice to the app. If the app does not want additional threads, it must poll for completions and to ensure progress if the underlying hardware does not provide progress. If the app does not want to burn the cycles to poll, it can block which requires someone to check for completions and signal the blocker. Whether that someone is a progress thread or the kernel is an implementation detail.

There are trade-offs for simplicity, portability, and performance and we hope that CCI provides the ability for the app to choose the best combination given its needs.

   0 likes

   0 likes

I also have doubts about whether an API that requires the underlying library to have threads (in order to signal/deliver events) is a good approach, as it requires surfacing a bunch of ‘knobs’ for the application to control the threading policy (number of threads, affinity, priority, etc).

Point being that simplicity, protability, and performance often conflict with one another.

-Fab

   0 likes

   0 likes