Cisco Logo

High Performance Computing Networking

Multiple readers have told me that it is difficult for them to understand and/or visualize the effects of latency on their HPC applications, particularly in modern NUMA (non-uniform memory access) and NUNA (non-uniform network access) environments.

Let’s breaks down the different levels of latency in a typical modern server and network computing environments.

Here’s a familiar analogy: your home.  You spend a lot of time at your home; it’s where you live and play.

You have friends and neighbors that live right near you in the same subdivision. You interact with these neighbors frequently, if for no other reason than they’re close by / easy and fast to get to.

But you also have friends in other subdivisions. They’re a several-minute drive across surface streets from your home.  You interact with these people, too, but you have to think and plan a little more vs. just walking next door.

And then you have friends who live far away — it takes a long time to get there, and involves travel over long-distance highways.

The distance-from-home analogy is pretty easy to understand:

So when you send a message to a peer (e.g,. MPI_SEND to another MPI process), consider with whom you’re communicating: are they next door, in the next subdivision, or in the next city?  That gives you an idea of the magnitude of the cost of communicating with them.

But let’s add another dimension here: caches and RAM.  Data locality is a major factor in performance, and is frequently under-appreciated.

Stay tuned; we’ll talk about data locality by extending this analogy in a future entry…


In an effort to keep conversations fresh, Cisco Blogs closes comments after 90 days. Please visit the Cisco Blogs hub page for the latest content.


  1. Here’s one more:

    And let’s not forget–Rear Admiral Grace Murray Hopper on Visualizing Nanoseconds:

  2. Exxxcellent.

  3. It’s worth pointing out that the same consideration of distance that data must move applies to cross-node interconnects as well, within a single HPC system. The effects this can have are pretty wide-ranging, and sometimes quite substantial. For instance, here’s a pile of recent papers address the issue:

  1. Return to Countries/Regions
  2. Return to Home
  1. All High Performance Computing Networking
  2. All Security
  3. Return to Home