Conversation
2c8a236 to
2fa5b0d
Compare
| end do | ||
|
|
||
| ! tridiagonal solve sweeping up and then down the column | ||
| #ifdef MPAS_OPENACC |
There was a problem hiding this comment.
It may be that we wouldn't lose much (any?) CPU performance if we were to move this directive below the k-loop (so, move it to line ~4066), and to simply use rho_pp(k, iCell) and rtheta_pp(k, iCell) on lines 4111 and 4113. This would allow us to eliminate the need for the RHO_PP and RTHETA_PP macros.
There was a problem hiding this comment.
Using a regional domain with 65094 cells on one CPU core, there does appear to be a slight CPU performance boost (~5%) with this proposed change; this was using the Intel 2025.3.2 compilers. However, the result could be different with other compilers or with more or fewer cells per MPI task.
There was a problem hiding this comment.
Thanks! Will also check this with the global runs.
There was a problem hiding this comment.
For the global real case, with 655362 cells, I'm noticing a 1.8% slowdown with intel/2025.2.1 and 4.4% slowdown with gnu/12.2.0
There was a problem hiding this comment.
For my regional test case using the GNU 14.3.0 compilers, I'm seeing a ~1% speedup with the proposed changes when running with 1 MPI task, and about a ~3% speedup when running with 128 MPI tasks (though based on just one run from each combination of code version and MPI task count).
Since the code region in question only accounts for a few percent of the total time integration time, I'd say any speedup or slowdown on the order of a few percent is probably negligible, and the benefit in the way of simpler code is probably good enough reason to go with the proposed code changes.
There was a problem hiding this comment.
Thanks for looking into it! I have modified commit 167ed to include these changes. And I think the PR is in otherwise a decent shape for review.
117568a to
f1f4c90
Compare
This commit optimizes the GPU performance of the relevant kernel in atm_advance_acoustic_step_work by splitting up a single parallel region into two separate regions. A single larger parallel region, especially with the presence of worker-level private variables ts and rs, results in higher register usage per thread, which in turn reduces the occupancy (number of threads executing at any given time on the GPU relative to max capacity) on the GPU. Splitting up into two parallel regionsm, and removing private variables ts and rs from the second region, reduces the register usage and improves the occupancy and performance in this instance. It is also required to save rs and ts to rho_pp and rtheta_pp respectively, in order for the second loop to retain the respective values. This optimization degrades performance on CPUs, and is addressed in the subsequent commit, Co-authored-by: Pranay Reddy Kommera <pranay.kommera@gmail.com>
…uilds The previous commit split up a parallel region into two as it is more optimal on GPUs, however this degrades performance on CPUs. This commit introduces a preprocessor directive that conditionally splits up the loops only in the case of an OPENACC GPU build, in order to preserve the current performance on CPUs. Co-authored-by: Michael Duda <duda@ucar.edu>
The second optimization involves swapping the sequential and vector loops in order to increase the DRAM bandwidth on GPUs. This optimization, however, results in performance degradation on CPUS. Hence, two preprocessor macros are introduced to try to preserve the same loop order on CPUs but interchange the loops on GPUs. Co-authored-by: Pranay Reddy Kommera <pranay.kommera@gmail.com>
This commit fuses two vertical level vector loops in order to optimize GPU performance. It does not noticeably degrade CPU performance. Co-authored-by: Pranay Reddy Kommera <pranay.kommera@gmail.com>
f1f4c90 to
a242478
Compare
2 participants
This PR introduces optimizations for the OpenACC port of atm_advance_acoustic_step_work: 3976. The table below lists the timings for a real, global 30km experiment on A100 GPU with the nvhpc.
For nvhpc gpu runs, -gpu=math_uniform is introduced as a build flag to ensure optimizations are bit-identical, and the we report the numbers using NV_ACC_TIME=1. The GPU runs are on 1 Derecho GPU node, using 1 A100 via 1 MPI task.
The numbers reported for the CPU runs with gnu and intel compilers use the newly-added timers local to this region, and are averaged across three runs. The CPU runs use a single derecho CPU node each, fully subscribed to 128 MPI tasks.
TODO: Add @Pranay-Reddy-Kommera as primary commit author
This work was completed in part at the NCAR/NLR/NOAA Open Hackathon, part of the Open Hackathons program. The authors would like to acknowledge OpenACC-Standard.org for their support.