On the other hand, I’m not sure I understand why you could not simply turn off swap (or not even mount any in the first place) for applications like this.

And it’s certainly possible to implement the UNIX API without demand paging (proof by construction, Thompson, Kernighan, Ritchie, 1970), so a HPC platform that isn’t based on Linux and still provides a completely compatible API shouldn’t be at all controversial.

   0 likes

1. The mbind(2) and set_mempolicy(2) pages don’t say anything about the policies being hints. I had therefore believed (apparently incorrectly) that the policies were binding (pardon the pun) — particularly when one of the policies has “STRICT” in its name and uses strong language about what it means.
2. Until you mentioned it, it would not have occurred to me that swapping in would be considered a new allocation. Sure, maybe that’s how it’s implemented on the back end, but I’m just a dumb user here; I didn’t think that a back-end implementation artifact would affect the policy that I previously set and had no upcall/notification when it had changed.

Perhaps I’m just a naive userspace guy, but I said that I wanted memory X to be bound to location Y; why should that change if the memory gets paged out?

FWIW, memory binding policies don’t change on Solaris or Windows if the memory gets paged out. When the memory is paged back in, the OS tries very hard to make it obey the original memory binding — only placing it elsewhere if it absolutely cannot place it where the original memory binding policy specified.

As for the “should we not use Linux?” comments; I believe that those comments are offered in the spirit of “Hmm.. that’s interesting to think about…” (including a few examples of how others have done it). I know just about everyone who has commented here; we all use and rely on Linux heavily every day. That doesn’t mean that Linux doesn’t have some warts that are worth talking about; potentially even resulting in the creation of a patch by a Linux kernel expert (which I clearly am not!).

   0 likes

regarding pagins, did you set vm.page-cluster to 0?

as for the linux bashing in comments: linux won, get over it. if you don’t like what linux currently does, where’s your patch?

   0 likes

@David Yes, “Linux must die” is rather over-the-top, but that is my general state-of-mind (Jeff S. can certainly vouch for this). I’m just trying to throw stones at the conventional wisdom that Linux is the only way to fly in HPC. The HPC community is pretty smart as far as computer users go, and we should not bind ourselves forever to an OS that was created by a grad student in 1983.

Kyle and I have both pointed out examples of attempts to use something other than Linux in HPC. BGL and BGP CNK are generally considered successful, BGP more so because it had more Linux-like features, including dlopen(). Most people I know think Catamount was a failure, but I don’t think the overarching design of Catamount was the reason. If Cray had done a better job of socializing the fact that Catamount was not Linux for a reason, it might have been easier to get people outside of Sandia to accept it. It seems there were implementation issues in many features of the Cray XT3, but I can’t speak with any authority on what they were.

What has not been said, but I think is important, is that any alternative to Linux in HPC should be generally POSIX-complaint. BGP CNK has this property, as it is derived from BSD. There are very few examples of properly designed HPC codes failing to run on BGP because of this. It is well-known that what BGP CNK prohibits are generally bad ideas in HPC anyways, e.g. oversubscription. I refuse to accept any argument that fork() and exec*() are good ideas in HPC codes.

   0 likes

For example, Open MPI really does need shared memory buffers to reside on specific NUMA nodes, or performance will plummet (relatively speaking). Paging out — such as suspending and resuming a job, as in your case — can be disastrous to performance. The problem only gets worse as core counts keep going up, potentially enabling sites to start allocating multiple jobs to individual compute nodes.

That being said, I’m planning a followup blog entry about this. I’ll include some stronger clarifications.

   0 likes

http://www.open-mpi.org/community/lists/hwloc-devel/2011/02/2012.php

On our HPC systems we require users to specify how much RAM per core they want (defaults to 1GB) and they have to request more if they need it. That limit is enforced by setting RLIMIT_AS for its child processes.

The scheduler won’t allocate jobs to a node for which memory is not available.

   0 likes

At best, users use process affinity and assume (the default) “preferred” memory placement will do the right thing. But in the face of physical memory (partially) full of page cache (often residual from the last job to finish), preferred page allocations will “go off-node” often enough to give annoyingly variable job performance. That’s what originally got us in to investigating adding memory binding to MPI. Even with swapin_readahead occasionally messing with binding placement, it’s still usually miles ahead of preferred placement. And, as you say, how many sites cause their jobs to page anyway? Yes, we do as a matter of scheduling policy but at a lot of sites paging would only be caused by the user and in that case, NUMA placement is probably a relatively minor concern.

So I guess I would be pointing out the shortcomings of the usual preferred NUMA node approach and promoting the large win that binding can provide (with the small caveat that it’s still not perfect).

   0 likes

In any event, as I understand it, the BlueGene CNK provides an offset-based virtual addressing system.

   0 likes

BTW, I think swap is one of the most useful OS features available to us for managing our HPC system in a smart way. We could replace it by reserving half our memory for suspended jobs but I doubt users would like that idea.

   0 likes

In general quantum chemistry does not require frequent malloc+free, and generally they are stack-like such that fragmentation shouldn’t be a huge issue.

I don’t see how fragmentation matters on BG/P anyways. There is no NUMA so fragmentation would only show up at the granularity of a cache line.

   0 likes