Skip to content

Add line numbering to gpu loops #1029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 35 commits into from
Nov 19, 2025
+11,181 −11,065
Merged

Add line numbering to gpu loops #1029

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
ed2bc5e
Correct line markers have been recovered
danieljvickers Nov 6, 2025
35c57eb
Made it through common
danieljvickers Nov 6, 2025
477a13b
Modified a few hundred macros
danieljvickers Nov 6, 2025
3c010f8
Finished with parallel loops
danieljvickers Nov 6, 2025
f8dd4cd
Removed duplicate parallel statements
danieljvickers Nov 6, 2025
fd482ed
This builds
danieljvickers Nov 6, 2025
ceec116
Removed all warnings
danieljvickers Nov 6, 2025
0ce8883
Spelling and Format
danieljvickers Nov 6, 2025
824585f
Updated macro API documentation
danieljvickers Nov 6, 2025
28a20bb
Spelling
danieljvickers Nov 6, 2025
5f768ec
Found an error in how FYPP processes that resulted in only openMP dir…
danieljvickers Nov 7, 2025
5ea3506
Resolved chemistry issues
danieljvickers Nov 7, 2025
1980851
Ben found my IO error
danieljvickers Nov 7, 2025
1d5e33c
Fixed issue with s_get_char_vol being elemental, which means that it …
danieljvickers Nov 7, 2025
723822d
All tests pass on wingtip
danieljvickers Nov 7, 2025
2eb6864
merge
danieljvickers Nov 16, 2025
9f946a0
Passes GNU tests
danieljvickers Nov 16, 2025
a098d5d
Upstream mnerge
danieljvickers Nov 16, 2025
07d558b
Formatting and passed all GNU tests
danieljvickers Nov 16, 2025
90639b2
refresh FFT code to master based on comment
danieljvickers Nov 16, 2025
2376abc
formatting:
danieljvickers Nov 16, 2025
0470192
Hard reset of file with controled search for macro changes
Nov 16, 2025
c451b20
Formatting
Nov 16, 2025
db852bf
Parenthesis change
Nov 16, 2025
8578364
Testing if reverting macros on the MPI file resolves seg fault issues
Nov 17, 2025
b521697
format
Nov 17, 2025
973ea73
Fixed old macros
Nov 17, 2025
2ede9a1
Undoing a deletion of a directive with and end loop
Nov 17, 2025
d5a3f74
Previous test passed except for gres resource errors, which are kind …
Nov 17, 2025
abf2f5a
Got errors with the first 9 loops, so now I removed 4 more loops. 5 t…
Nov 17, 2025
71f19b1
Last test pass, so I have narrowed the seg fault to 4 loops. Removed …
Nov 17, 2025
1875e3c
Previous test compiled, meaning the problem is one of the two remaini…
Nov 17, 2025
b5079f6
The previous test compiled. If this does not compile then I am certai…
Nov 17, 2025
31d913f
For some reason, the last test compiled. It changed with format. I wi…
Nov 17, 2025
30417cb
Readding a fix that I implemented earlier for bubbles that fails on N…
Nov 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/documentation/gpuParallelization.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,16 @@ Note: Ordering is not guaranteed or stable, so use key-value pairing when using

**Macro Invocation**

Uses FYPP eval directive using `#:call`
In order to parallelize a loop, simply place two macro calls on either end of the loop:

```C
#:call GPU_PARALLEL_LOOP(...)
$:$GPU_PARALLEL_LOOP(...)
{code}
#:endcall GPU_PARALLEL_LOOP
$:END_GPU_PARALLEL_LOOP()
```

This wraps the lines in `code` with parallelization calls to openACC or openMP, depending on environment and compiler settings.

Comment on lines +47 to +56
Copy link

@coderabbitai coderabbitai bot Nov 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Fix GPU_PARALLEL_LOOP docs to match new API and syntax

  • The invocation snippet should use $:GPU_PARALLEL_LOOP(...) (single $:), not $:$GPU_PARALLEL_LOOP(...).
  • The parameter table and example still describe the old #:call GPU_PARALLEL_LOOP(code=..., ...) style and a required code parameter, but the new API uses start/end wrappers without a code argument.

Consider updating this section so the parameters and examples reflect the new block-style $:GPU_PARALLEL_LOOP(...)\n{code}\n$:END_GPU_PARALLEL_LOOP() usage.

Also applies to: 99-108

🤖 Prompt for AI Agents 
In docs/documentation/gpuParallelization.md around lines 47 to 56 (and similarly
update lines 99 to 108), the example and parameter table document the old API
and use an incorrect invocation ($:$GPU_PARALLEL_LOOP); update the example to
use the correct single-prefix block style
$:GPU_PARALLEL_LOOP(...)\n{code}\n$:END_GPU_PARALLEL_LOOP(), remove any
references to a required code parameter or the #:call GPU_PARALLEL_LOOP(...)
form, and rewrite the parameter list and examples to describe the start/end
wrapper semantics (what parameters the start macro accepts and that code is
provided between the macro pairs). Ensure both the example snippet and the
parameter table reflect the new block-style API consistently.

**Parameters**

| name | data type | Default Value | description |
Expand Down
33 changes: 31 additions & 2 deletions src/common/include/acc_macros.fpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@
$:end_acc_directive
#:enddef

#:def ACC_PARALLEL_LOOP(code, collapse=None, private=None, parallelism='[gang, vector]', &
#:def ACC_PARALLEL_LOOP_OLD(code, collapse=None, private=None, parallelism='[gang, vector]', &
& default='present', firstprivate=None, reduction=None, reductionOp=None, &
& copy=None, copyin=None, copyinReadOnly=None, copyout=None, create=None, &
& no_create=None, present=None, deviceptr=None, attach=None, extraAccArgs=None)
Expand Down Expand Up @@ -161,6 +161,35 @@
$:acc_end_directive
#:enddef

#:def ACC_PARALLEL_LOOP(collapse=None, private=None, parallelism='[gang, vector]', &
& default='present', firstprivate=None, reduction=None, reductionOp=None, &
& copy=None, copyin=None, copyinReadOnly=None, copyout=None, create=None, &
& no_create=None, present=None, deviceptr=None, attach=None, extraAccArgs=None)
#:set collapse_val = GEN_COLLAPSE_STR(collapse)
#:set parallelism_val = GEN_PARALLELISM_STR(parallelism)
#:set default_val = GEN_DEFAULT_STR(default)
#:set private_val = GEN_PRIVATE_STR(private, False).strip('\n') + GEN_PRIVATE_STR(firstprivate, True).strip('\n')
#:set reduction_val = GEN_REDUCTION_STR(reduction, reductionOp)
#:set copy_val = GEN_COPY_STR(copy)
#:set copyin_val = GEN_COPYIN_STR(copyin, False).strip('\n') + GEN_COPYIN_STR(copyinReadOnly, True).strip('\n')
#:set copyout_val = GEN_COPYOUT_STR(copyout)
#:set create_val = GEN_CREATE_STR(create)
#:set no_create_val = GEN_NOCREATE_STR(no_create)
#:set present_val = GEN_PRESENT_STR(present)
#:set deviceptr_val = GEN_DEVICEPTR_STR(deviceptr)
#:set attach_val = GEN_ATTACH_STR(attach)
#:set extraAccArgs_val = GEN_EXTRA_ARGS_STR(extraAccArgs)
#:set clause_val = collapse_val.strip('\n') + parallelism_val.strip('\n') + &
& default_val.strip('\n') + private_val.strip('\n') + reduction_val.strip('\n') + &
& copy_val.strip('\n') + copyin_val.strip('\n') + &
& copyout_val.strip('\n') + create_val.strip('\n') + &
& no_create_val.strip('\n') + present_val.strip('\n') + &
& deviceptr_val.strip('\n') + attach_val.strip('\n')
#:set acc_directive = '!$acc parallel loop ' + &
& clause_val + extraAccArgs_val.strip('\n')
$:acc_directive
#:enddef

#:def ACC_ROUTINE(function_name=None, parallelism=None, nohost=False, extraAccArgs=None)
#:set parallelism_val = GEN_PARALLELISM_STR(parallelism)
#:assert isinstance(nohost, bool)
Expand Down Expand Up @@ -308,4 +337,4 @@
#:set acc_directive = '!$acc wait ' + clause_val + extraAccArgs_val.strip('\n')
$:acc_directive
#:enddef
! New line at end of file is required for FYPP
! New line at end of file is required for FYPP
58 changes: 57 additions & 1 deletion src/common/include/omp_macros.fpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@
$:omp_end_directive
#:enddef

#:def OMP_PARALLEL_LOOP(code, collapse=None, private=None, parallelism='[gang, vector]', &
#:def OMP_PARALLEL_LOOP_OLD(code, collapse=None, private=None, parallelism='[gang, vector]', &
& default='present', firstprivate=None, reduction=None, reductionOp=None, &
& copy=None, copyin=None, copyinReadOnly=None, copyout=None, create=None, &
& no_create=None, present=None, deviceptr=None, attach=None, extraOmpArgs=None)
Expand Down Expand Up @@ -196,6 +196,62 @@
$:omp_end_directive
#:enddef

#:def OMP_PARALLEL_LOOP(collapse=None, private=None, parallelism='[gang, vector]', &
& default='present', firstprivate=None, reduction=None, reductionOp=None, &
& copy=None, copyin=None, copyinReadOnly=None, copyout=None, create=None, &
& no_create=None, present=None, deviceptr=None, attach=None, extraOmpArgs=None)

#:set collapse_val = GEN_COLLAPSE_STR(collapse)
#:set parallelism_val = OMP_PARALLELISM_STR(parallelism)
#:set default_val = OMP_DEFAULT_STR(default)
#:set private_val = GEN_PRIVATE_STR(private, False).strip('\n') + GEN_PRIVATE_STR(firstprivate, True).strip('\n')
#:set reduction_val = GEN_REDUCTION_STR(reduction, reductionOp)
#:set copy_val = OMP_COPY_STR(copy)
#:set copyin_val = OMP_COPYIN_STR(copyin).strip('\n') + OMP_COPYIN_STR(copyinReadOnly).strip('\n')
#:set copyout_val = OMP_COPYOUT_STR(copyout)
#:set create_val = OMP_CREATE_STR(create)
#:set no_create_val = OMP_NOCREATE_STR(no_create)
#:set present_val = OMP_PRESENT_STR(present)
#:set deviceptr_val = OMP_DEVICEPTR_STR(deviceptr)
#:set attach_val = OMP_MAP_STR('always,tofrom', attach)
#:set extraOmpArgs_val = GEN_EXTRA_ARGS_STR(extraOmpArgs)
#:set clause_val = collapse_val.strip('\n') + parallelism_val.strip('\n') + &
& default_val.strip('\n') + private_val.strip('\n') + reduction_val.strip('\n') + &
& copy_val.strip('\n') + copyin_val.strip('\n') + &
& copyout_val.strip('\n') + create_val.strip('\n') + &
& no_create_val.strip('\n') + present_val.strip('\n') + &
& deviceptr_val.strip('\n') + attach_val.strip('\n')
#! Hardcoding the parallelism for now

#:if MFC_COMPILER == NVIDIA_COMPILER_ID or MFC_COMPILER == PGI_COMPILER_ID
#:set omp_start_directive = '!$omp target teams loop defaultmap(firstprivate:scalar) bind(teams,parallel) '
#:elif MFC_COMPILER == CCE_COMPILER_ID
#:set omp_start_directive = '!$omp target teams distribute parallel do simd defaultmap(firstprivate:scalar) '
#:elif MFC_COMPILER == AMD_COMPILER_ID
#:set omp_start_directive = '!$omp target teams distribute parallel do '
#:else
#:set omp_start_directive = '!$omp target teams loop defaultmap(firstprivate:scalar) bind(teams,parallel) '
#:endif

#:set omp_directive = omp_start_directive + clause_val + extraOmpArgs_val.strip('\n')
$:omp_directive
#:enddef

#:def END_OMP_PARALLEL_LOOP()

#:if MFC_COMPILER == NVIDIA_COMPILER_ID or MFC_COMPILER == PGI_COMPILER_ID
#:set omp_end_directive = '!$omp end target teams loop'
#:elif MFC_COMPILER == CCE_COMPILER_ID
#:set omp_end_directive = '!$omp end target teams distribute parallel do simd'
#:elif MFC_COMPILER == AMD_COMPILER_ID
#:set omp_end_directive = '!$omp end target teams distribute parallel do'
#:else
#:set omp_end_directive = '!$omp end target teams loop'
#:endif

$:omp_end_directive
#:enddef

#:def OMP_ROUTINE(function_name, nohost, extraOmpArgs)
#:assert isinstance(nohost, bool)
#:if nohost == True
Expand Down
35 changes: 32 additions & 3 deletions src/common/include/parallel_macros.fpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@

#:enddef

#:def GPU_PARALLEL_LOOP(code, collapse=None, private=None, parallelism='[gang, vector]', &
#:def GPU_PARALLEL_LOOP_OLD(code, collapse=None, private=None, parallelism='[gang, vector]', &
& default='present', firstprivate=None, reduction=None, reductionOp=None, &
& copy=None, copyin=None, copyinReadOnly=None, copyout=None, create=None, &
& no_create=None, present=None, deviceptr=None, attach=None, extraAccArgs=None, extraOmpArgs=None)

#:set acc_code = ACC_PARALLEL_LOOP(code, collapse, private, parallelism, default, firstprivate, reduction, reductionOp, copy, copyin, copyinReadOnly, copyout, create, no_create, present, deviceptr, attach, extraAccArgs)
#:set omp_code = OMP_PARALLEL_LOOP(code, collapse, private, parallelism, default, firstprivate, reduction, reductionOp, copy, copyin, copyinReadOnly, copyout, create, no_create, present, deviceptr, attach, extraOmpArgs)
#:set acc_code = ACC_PARALLEL_LOOP_OLD(code, collapse, private, parallelism, default, firstprivate, reduction, reductionOp, copy, copyin, copyinReadOnly, copyout, create, no_create, present, deviceptr, attach, extraAccArgs)
#:set omp_code = OMP_PARALLEL_LOOP_OLD(code, collapse, private, parallelism, default, firstprivate, reduction, reductionOp, copy, copyin, copyinReadOnly, copyout, create, no_create, present, deviceptr, attach, extraOmpArgs)

#if defined(MFC_OpenACC)
$:acc_code
Expand All @@ -36,6 +36,35 @@
#endif
#:enddef

#:def GPU_PARALLEL_LOOP(collapse=None, private=None, parallelism='[gang, vector]', &
& default='present', firstprivate=None, reduction=None, reductionOp=None, &
& copy=None, copyin=None, copyinReadOnly=None, copyout=None, create=None, &
& no_create=None, present=None, deviceptr=None, attach=None, extraAccArgs=None, extraOmpArgs=None)

#:set acc_directive = ACC_PARALLEL_LOOP(collapse, private, parallelism, default, firstprivate, reduction, reductionOp, copy, copyin, copyinReadOnly, copyout, create, no_create, present, deviceptr, attach, extraAccArgs)
#:set omp_directive = OMP_PARALLEL_LOOP(collapse, private, parallelism, default, firstprivate, reduction, reductionOp, copy, copyin, copyinReadOnly, copyout, create, no_create, present, deviceptr, attach, extraOmpArgs)

#if defined(MFC_OpenACC)
$:acc_directive
#elif defined(MFC_OpenMP)
$:omp_directive
#endif

#:enddef

#:def END_GPU_PARALLEL_LOOP()

#:set acc_end_directive = '!$acc end parallel loop'
#:set omp_end_directive = END_OMP_PARALLEL_LOOP()

#if defined(MFC_OpenACC)
$:acc_end_directive
#elif defined(MFC_OpenMP)
$:omp_end_directive
#endif

#:enddef

#:def GPU_ROUTINE(function_name=None, parallelism=None, nohost=False, cray_inline=False, extraAccArgs=None, extraOmpArgs=None)
#:assert isinstance(cray_inline, bool)
#:set acc_directive = ACC_ROUTINE(function_name=function_name, parallelism=parallelism, nohost=nohost, extraAccArgs=extraAccArgs)
Expand Down
2 changes: 1 addition & 1 deletion src/common/include/shared_parallel_macros.fpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,4 @@
#:endif
$:extraArgs_val
#:enddef
! New line at end of file is required for FYPP
! New line at end of file is required for FYPP
Loading
Loading