Multiple readers have told me that it is difficult for them to understand and/or visualize the effects of latency on their HPC applications, particularly in modern NUMA (non-uniform memory access) and NUNA (non-uniform network access) environments.
Let’s breaks down the different levels of latency in a typical modern server and network computing environments.
Read More »
Tags: HPC, mpi, NUMA, NUNA
Process affinity is a hot topic. With commodity servers getting more and more complex internally (think: NUMA and NUNA), placing and binding individual MPI processes to specific processor, cache, and memory resources is becoming quite important in terms of delivered application performance.
MPI implementations have long offered options for laying out MPI processes across the resources allocated for the job. Such options typically included round-robin schemes by core or by server node. Additionally, MPI processes can be bound to individual processor cores (and even sockets).
Today caps a long-standing effort between Josh Hursey, Terry Dontje, Ralph Castain, and myself (all developers in the Open MPI community) to revamp the processor affinity system in Open MPI.
The first implementation of the Location Aware Mapping Algorithm (LAMA) for process mapping, binding, and ordering has been committed to the Open MPI SVN trunk. LAMA provides a whole new level of processor affinity control to the end user.
Read More »
Tags: HPC, hwloc, mpi, NUMA, NUNA, Open MPI, process affinity
At long last, we have released a stable, production-quality version of Hardware Locality (hwloc). Yay!
If you’ve missed all my prior discussions about hwloc, hwloc provides command line tools and a C API to obtain the hierarchical map of key computing elements, such as: NUMA memory nodes, shared caches, processor sockets, processor cores, and processing units (logical processors or “threads”). hwloc also gathers various attributes such as cache and memory information, and is portable across a variety of different operating systems and platforms.
In an increasing NUMA (and NUNA!) world, hwloc is a valuable tool for high performance.
Read More »
Tags: HPC, hwloc, NUMA, NUNA, process affinity
Traffic. It’s a funny thing. On my daily drive to work, I see (what appear to be) oddities and contradictions frequently. For example, although the lanes on my side of the highway are running fast and clear, the other side is all jammed up. But a half mile later, the other side is running fast and clear, and my lanes have been reduced to half-speed. A short distance further, I’m zipping along again at 55mph (ahem).
Sometimes the reasons behind traffic congestion are obvious. For example, when you drive through a busy interchange, it’s easy to understand how lots of vehicles entering and exiting the roadway can force you to slow down. But sometimes the traffic flow issues are quite subtle; congestion may be caused by a non-obvious confluence of second- and third-order effects.
The parallels from highway traffic to networking are quite obvious, but the analogy can go much deeper when you consider that modern computational clusters span multiple different networks — we’re entering an era of Non-Uniform Network Architectures (NUNAs).
Read More »
Tags: NUMA, NUNA, process affinity
I just ran across a great blog entry about SGE debuting topology-aware scheduling. Dan Templeton does a great job of describing the need for processor topology-aware job scheduling within a server. Many MPI jobs fit exactly within his description of applications that have “serious resource needs” — they typically require lots of CPU and/or network (or other I/O). Hence, scheduling an MPI job intelligently across not only the network, but also across the network and resources inside the server, is pretty darn important. It’s all about location, location, location!
Particularly as core counts in individual server are going up.
Particularly as networks get more complicated inside individual servers.
Particularly if heterogeneous computing inside a single server becomes popular.
Particularly as resources are now pretty much guaranteed to be non-uniform within an individual server.
These are exactly the reasons that, even though I’m a network middleware developer, I spend time with server-specific projects like hwloc — you really have to take a holistic approach in order to maximize performance.
Read More »
Tags: HPC, hwloc, mpi, NUMA, NUNA