netlrts smp hangs at startup with some process/thread count configurations

[jobordner]

Compiling using ./build charm++ netlrts-linux-x86_64 smp and running, say, hello/3darray, hangs with some combinations of virtual nodes and threads-per-node both > 1, such as ./charmrun +p 16 ./hello 16 ++local ++ppn 8 (run on a single physical node). Tested on multiple platforms (Ryzen 9 and TACC Frontera).

It doesn't hang when using older versions of Charm++, so I tried git bisect and found the first bad commit to be b7b126dc6 (previous good is 76d1916de).

[ericjbohm]

Does the hang go away with +setcpuaffinity? I suspect the default binding in those configurations is piling the pthreads up on too few cores.

…

On Tue, Feb 4, 2025 at 3:33 PM James Bordner ***@***.***> wrote: Compiling using ./build charm++ netlrts-linux-x86_64 smp and running, say, hello/3darray, hangs with some combinations of virtual nodes and threads-per-node both > 1, such as ./charmrun +p 16 ./hello 16 ++local ++ppn 8 (run on a single physical node). Tested on multiple platforms (Ryzen 9 and TACC Frontera). It doesn't hang when using older versions of Charm++, so I tried git bisect and found the first bad commit to be b7b126d (previous good is 76d1916). — Reply to this email directly, view it on GitHub <#3868>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3HFH5RTA45KYRLAW6FB6L2OEW4HAVCNFSM6AAAAABWPSJ3YWVHI2DSMVQWIX3LMV43ASLTON2WKOZSHAZTCMZUGUYTONA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jobordner]

Does the hang go away with +setcpuaffinity?

I suspect the default binding in those configurations is piling the
pthreads up on too few cores.
…

It did help on the Ryzen9 (24 HT cores) but with the warning message "Multiple PEs assigned to same core, recommend adjusting processor affinity or passing +CmiSleepOnIdle to reduce interference".

It didn't help on TACC Frontera (56 HT cores per node). Output below.

./charmrun +p 16 ./hello 16 ++local ++ppn 8 +setcpuaffinity
Charmrun> scalable start enabled. 
Charmrun> started all node programs in 0.047 seconds.
Charm++> Running in SMP mode: 2 processes, 8 worker threads (PEs) + 1 comm threads per process, 16 PEs total
Charm++> The comm. thread both sends and receives messages
Converse/Charm++ Commit ID: v7.0.0-rc1-34-gb7b126dc6
Charm++ built without optimization.
Do not use for performance benchmarking (build with --with-production to do so).
Charm++ built with internal error checking enabled.
Do not use for performance benchmarking (build without --enable-error-checking to do so).
Isomalloc> Synchronized global address space.
Charm++> scheduler running in netpoll mode.
CharmLB> Load balancer assumes all CPUs are same.
Charm++> cpu affinity enabled. 

[jobordner]

To update, I've run 3darray hello multiple times (>30 up to 100) using all valid power-of-two configurations of processes (charm nodes) p and threads per process t. Tests were run on a single (56-core) node of TACC Frontera for changesets 76d1916de, b7b126dc6, and v8.0.1, each with and without +setcpuaffinity, and using a timeout of 5s (successful runs took at most about 0.05s)

The results:

• 76d1916de (last "good" changeset)
  • Virtually all runs were successful, either with or without +setcpuaffinity
• b7b126dc6 (first "bad" changeset)
  • without +setcpuaffinity: all runs with p>=2 and t>=8 hung
  • with +setcpuaffinity: all runs with p>=2 and t>=2 hung
• v8.0.1
  • without +setcpuaffinity, all runs with p>=2 and t>=8 hung
  • with +setcpuaffinity, runs with p>2 and t==4 hung > 90% of the time

So the p=2 t=4 configuration, say, (./charmrun +p 8 ./hello 8 ++mpiexec ++ppn 4 +setcpuaffinity) would seem to be a more reliable test case than the p=2 t=8 I used previously.

Note the diff between 76d1916de and b7b126dc6 is only about 500 lines and all in src/conv-core/cpuaffinity.C

[ericjbohm]

We almost always use explicit +pemap and +commap settings on supercomputers, so I think that is why we haven't really run into the issues with +setcpuaffinity at those scales. I could collect together and pass along the choices we use for frontera and frontier if that would be helpful.

…

On Wed, Feb 12, 2025 at 2:55 PM James Bordner ***@***.***> wrote: To update, I've run 3darray hello multiple times (>30 up to 100) using all valid power-of-two configurations of processes (charm nodes) p and threads per process t. Tests were run on a single (56-core) node of TACC Frontera for changesets 76d1916, b7b126d, and v8.0.1, each with and without +setcpuaffinity, and using a timeout of 5s (successful runs took at most about 0.05s) The results: - 76d1916 (last "good" changeset) - Virtually all runs were successful, either with or without +setcpuaffinity - b7b126d (first "bad" changeset) - without +setcpuaffinity: all runs with p>=2 and t>=8 hung - with +setcpuaffinity: all runs with p>=2 and t>=2 hung - v8.0.1 - without +setcpuaffinity, all runs with p>=2 and t>=8 hung - with +setcpuaffinity, runs with p>2 and t==4 hung > 90% of the time So the p=2 t=4 configuration, say, (./charmrun +p 8 ./hello 8 ++mpiexec ++ppn 4 +setcpuaffinity) would seem to be a more reliable test case than the p=2 t=8 I used previously. Note the diff between 76d1916 and b7b126d is only about 500 lines and all in src/conv-core/cpuaffinity.C — Reply to this email directly, view it on GitHub <#3868 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3HFHZZLNUR6QCQHEZ4Q5T2POYKHAVCNFSM6AAAAABWPSJ3YWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJUHAZDENBVGM> . You are receiving this because you commented.Message ID: ***@***.***>

[jobordner]

We almost always use explicit +pemap and +commap settings on
supercomputers, so I think that is why we haven't really run into the
issues with +setcpuaffinity at those scales.

I could collect together and pass along the choices we use for frontera and
frontier if that would be helpful.

Still hangs.

[ericjbohm]

We almost always use explicit +pemap and +commap settings on
supercomputers, so I think that is why we haven't really run into the
issues with +setcpuaffinity at those scales.
I could collect together and pass along the choices we use for frontera and
frontier if that would be helpful.

Still hangs.

On frontera, we would typically run namd like this:
+ppn 8 +pemap 6-53:2,7-53:2 +commap 0,2,4,1,3,5

or
+ppn 13 +pemap 4-55:2,5-55:2 +commap 0,2,1,3

[jobordner]

Thanks. After correcting "+ppn" to "++ppn" it still hangs.