Repository navigation
Add OpenMP pragmas to the stencil and DOF kernels - #103
Merged
Merged
Conversation
max-models
marked this pull request as ready for review
October 13, 2026 15:20
spossann
reviewed
Oct 13, 2026
Parallelizing only the first direction leaves threads idle when n1 is smaller than the number of threads (e.g. 256 threads on one rank). - dot and transpose (2D/3D): compute the indices of the first direction inside the i2 loop, so the loops over i1 and i2 can be collapsed. - axpy (2D/3D) and the 3D DOF kernels: collapse(2). - inner 3D: one partial sum per pair (i0, i1), still added in a fixed order, so the result does not depend on the number of threads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
spossann
added a commit
to struphy-hub/struphy
that referenced
this pull request
Oct 14, 2026
Corresponding feectools PR struphy-hub/feectools#103 Fix OpenMP directives so all 128 Struphy kernel modules compile with GNU Fortran using `struphy compile --openmp`, and compile the feectools kernels with OpenMP as well. - Repair parallel regions, variable scoping, and accumulation reductions. - Give each thread its own mutable spline-evaluation workspace to prevent data races. - Force rebuilding when OpenMP settings change, and record compiler settings only after successful compilation. - Add CI compilation and numerical regression checks, plus documentation for thread counts and reduction memory requirements. ## feectools - `struphy compile --openmp` now passes `--openmp` to `psydac-accelerate`, so the feectools kernels are threaded too (previously they were always compiled serially). - The `feectools` submodule points to the `add-openmp-pragmas` branch (struphy-hub/feectools#103), which adds OpenMP pragmas to the stencil `dot`, `transpose`, `axpy` and `inner` kernels (2D/3D) and the 3D DOF kernels. - The CI OpenMP step also runs `pytest --pyargs feectools.linalg.tests.test_openmp`. - `install.rst` mentions that the feectools kernels are compiled with OpenMP, and how to run their check. ## Validation - Fresh OpenMP compilation of all 128 kernel modules. - Compiled results match serial reference loops with 1, 2, and 4 threads. - 575 tests passed, including pusher, SPH, compilation, and numerical parity tests. - `console/tests/test_compile.py` (16 tests) passes, including the new check that `--openmp` is forwarded to `psydac-accelerate`. - Ruff and whitespace checks passed. Array reductions require temporary output arrays per thread; large problems may need additional memory and larger stack limits. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Stefan Possanner <stefan.possanner@ipp.mpg.de>
spossann
added a commit
to struphy-hub/struphy
that referenced
this pull request
Oct 21, 2026
Corresponding feectools PR struphy-hub/feectools#103 Fix OpenMP directives so all 128 Struphy kernel modules compile with GNU Fortran using `struphy compile --openmp`, and compile the feectools kernels with OpenMP as well. - Repair parallel regions, variable scoping, and accumulation reductions. - Give each thread its own mutable spline-evaluation workspace to prevent data races. - Force rebuilding when OpenMP settings change, and record compiler settings only after successful compilation. - Add CI compilation and numerical regression checks, plus documentation for thread counts and reduction memory requirements. ## feectools - `struphy compile --openmp` now passes `--openmp` to `psydac-accelerate`, so the feectools kernels are threaded too (previously they were always compiled serially). - The `feectools` submodule points to the `add-openmp-pragmas` branch (struphy-hub/feectools#103), which adds OpenMP pragmas to the stencil `dot`, `transpose`, `axpy` and `inner` kernels (2D/3D) and the 3D DOF kernels. - The CI OpenMP step also runs `pytest --pyargs feectools.linalg.tests.test_openmp`. - `install.rst` mentions that the feectools kernels are compiled with OpenMP, and how to run their check. ## Validation - Fresh OpenMP compilation of all 128 kernel modules. - Compiled results match serial reference loops with 1, 2, and 4 threads. - 575 tests passed, including pusher, SPH, compilation, and numerical parity tests. - `console/tests/test_compile.py` (16 tests) passes, including the new check that `--openmp` is forwarded to `psydac-accelerate`. - Ruff and whitespace checks passed. Array reductions require temporary output arrays per thread; large problems may need additional memory and larger stack limits. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Stefan Possanner <stefan.possanner@ipp.mpg.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corresponding struphy PR struphy-hub/struphy#757
Add OpenMP pragmas to the kernels struphy runs in its solver loops, and make
psydac-accelerate --openmpwork.Kernels
#$ omp parallel for collapse(2)over the two outer loops ofstencil_dot,stencil_transposeandstencil_axpy(2D and 3D), and of the 3D DOF kernels infeec/dof_kernels.py, so the threads sharen1 * n2rows instead ofn1(enough work for e.g. 256 threads on one rank). Indotandtransposethe indices of the first direction are computed inside thei2loop to make the loops perfectly nested.stencil_inner_3d: one partial sum per pair(i0, i1)withcollapse(2);stencil_inner_2d: one peri0(collapsing would need a work array the size of the vector). The partial sums are added in a fixed order afterwards, instead of an OpenMPreduction. A reduction combines the thread sums in arrival order, so repeated inner products differed in the last bits and CG results were not reproducible (this broketest_inverse_transpose_interaction, which compares two CG runs exactly). The result is now the same for every run and every number of threads; the kernel arguments (shared with the CUDA version) are unchanged.matvec/transposekernels and the field-evaluation kernels are unchanged.Compilation (
accelerate.py)--openmp: the makefile concatenates$(FLAGS)$(FLAGS_openmp)without a space, so pyccel received--compiler-family=GNU--openmp.make -B), since make only tracks source timestamps. The last setting is stored inaccelerate/.last_flags_openmp(git-ignored, removed by--cleanup).Tests
New
linalg/tests/test_openmp.py: runs the kernels with 1, 2 and 4 threads in fresh processes, on the CUDA parity cases plus larger grids and the 3D DOF kernels. Results must match the serial Python kernels (rtol=atol=1e-12) and be bitwise identical to the 1-thread compiled result. Skipped unless the kernels were compiled with--openmp.Validation (GNU Fortran,
--openmp)OMP_NUM_THREADS=4: all linalg and feec tests pass (8038 passed), on two consecutive runs, before and after thecollapse(2)change.test_openmp.pypasses.stencil_dot_3don a 48³ grid with degree 3: 3.7× faster with 4 threads (outer-loop version, idle machine).🤖 Generated with Claude Code