Skip to content

Environment file naming convention #67

Open
Open
Environment file naming convention#67
Labels
environmentIssues springing from the environment (in a general sense) where TSMP2 is executed.

Description

@kvrigor
kvrigor
opened on Apr 19, 2025

For context (all emphasis mine):

  1. Marco's original post

The ".2025" in default.2025.env kind of breaks machine independency, or at least—depending on the number's meaning—it'll often not be consistent with the "stage" of other machines.

Is it the STAGE, the thing that is specific for JSC? I am trying to bring this into Marvin (using the gompi version, which is at the moment at 2024a). Any objections against this?

I think it should not be part of the name. If no other ideas from anyone right now, I'll think of something (and propose as a commit).

  1. My reply

2025 is originally associated with the JSC software stage, and here I thought to expand its meaning to the default stable software stack for some machine X this year. This relates to an important issue: how do we decide on the update schedule of the TSMP2 software stack? This is another big discussion but I would postpone it for now.

  1. Marco's response

So 2025 means the same as stage? Even as we now have STAGE==2024a for Marvin and 24.04 for Ubuntu?

Adding it to the env filename is no good, because, given the dot separator, awk'ing the Ubuntu case breaks positions (vielleicht important for the WFE).

I'll regard it as a sort of release then… but we have actual releases for that. I'd rather like to see the default sourcing of default.env (no number) and update the number in jsc.2025.gnu.openmpi, uni-bonn.2024a.gnu.openmpi (potential commit upcoming) and ubuntu.24.04.gnu.openmpi… oh no! the Stage is only known after sourcing the script (and the awk issue)! This is one of those problems I alluded to! (But to be fair, I did not solve that for the Marvin case either.)

Activity

  1. kvrigor commented on Apr 19, 2025

    @kvrigor
    kvrigor
    on Apr 19, 2025
    MemberAuthor

    To reiterate: 2025 is not necessarily the software stage. Rather, it influences which software stage to use on a given machine. It has to be interepreted in the context of the environment file name, e.g.

    • default.2025.env: Load the appropriate software stack for the TSMP2 release in 2025.
      • jsc.2025.gnu.openmpi: Stages/2025 is the stable software stack on JSC in 2025.
      • uni-bonn.2024a.gnu.openmpi: 2024a is the stable software stack on Marvin in 2025.
      • ubuntu.24LTS.gnu.openmpi: 24.04 LTS is the most recent stable Ubuntu release in 2025.

    How about this folder and naming structure? What's nice about this is it becomes crystal clear which machines does TSMP2 support on a given release year. The structure naturally filters out stale machines/environments.

    env
├── 2025
│   ├── default.2025.env
│   ├── jsc.2025.gnu.openmpi
│   ├── ubuntu.24LTS.gnu.openmpi
│   └── uni-bonn.2024a.gnu.openmpi
├── 2026
│   ├── default.2026.env
│   ├── jsc.2026.gnu.openmpi
│   ├── ubuntu.26LTS.gnu.openmpi
│   └── uni-bonn.2026a.gnu.openmpi
.
.
.
│
└── YYYY
    ├── default.YYYY.env
    ├── machine.stage.compiler.mpilib
    └── ...
  2. kvrigor
    added
    build_tsmp2
    on Apr 19, 2025

  3. mvhulten commented on Apr 28, 2025

    @mvhulten
    mvhulten
    on Apr 28, 2025
    Contributor

    Yes, that is more clear. I like it.

    A caveat is that (when you expand its meaning to the default stable software stack for some machine X this year), there is an artificial transition between those software stacks (on every 1 January 00:00). I suppose this is a general problem of year(-only) versioning and that if generally there is at most one release (supposing I can see the env/???? directories as releases) per year. I suppose that it is okay if, e.g., release 2026 is only in February 2026 for JSC and in April 2026 for Uni Bonn.

  4. mvhulten commented on Apr 28, 2025

    @mvhulten
    mvhulten
    on Apr 28, 2025
    Contributor

    Earlier I suggested this renaming:

    Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        renamed:    jsc.2024.intel.psmpi -> jsc.2024.iimpi
        renamed:    jsc.2025.gnu.openmpi -> jsc.2025.gompi
        renamed:    jsc.2025.intel.psmpi -> jsc.2025.iimpi
        renamed:    uni-bonn.gnu.openmpi -> uni-bonn.2023a.gompi

    I did not push it fwd, because—even though it looks cleaner—it is not per se an improvement.

  5. AGonzalezNicolas commented on Apr 29, 2025

    @AGonzalezNicolas
    AGonzalezNicolas
    on Apr 29, 2025
    Contributor

    @mvhulten what does the "a" on uni-bonn.2023a.gompi stand for?

  6. mvhulten commented on Apr 29, 2025

    @mvhulten
    mvhulten
    on Apr 29, 2025
    Contributor

    The "a" is part of the version number of the iimpi and gompi compiler toolchain Lmod packages (that include the compiler and accompanying MPI implementation). On Marvin current versions are iimpi/2023a and gompi/2024a.

  7. kvrigor commented on Apr 29, 2025

    @kvrigor
    kvrigor
    on Apr 29, 2025
    MemberAuthor

    A good naming would be unambiguously differentiate among these combinations:

    And there are a lot more combinations out there. While I also like the concise syntax, I'm afraid it's not expressive enough to easily accommodate new combinations.

  8. mvhulten commented on Apr 30, 2025

    @mvhulten
    mvhulten
    on Apr 30, 2025
    Contributor

    In the current scheme, there is no perfect disambiguation either (if you stick with e.g. .intel). This is fine, because the disambiguation is already in the files. We just have to make it unlikely to have collisions in filenames. If this would occur, names can be adjusted then (without breaking the general scheme ORG.RELEASE.COMPILER.MPI). If you have a collision, maybe we made the choice in env files to large. Maybe you should replace the classic Intel compiler with ifx instead of renaming jsc.2025.intel.psmpi. The commit and the fact that it is saved into the install and will be saved into the build directory is enough for reproducibility.

    Or you introduce jsc.2025.intelllvm.psmpi and optionally rename the original. You can see on the way.

    In this Intel compiler example I'd not change the filename or introduce a new file because Intel/2024.2.0 contains both ifort and ifx and we don't have time to validate if all components including libraries and utilities stick with one or the other.* Who cares anyway? If you do care, you can find out by deep inspection in the build directory.

    I think the current scheme is a fine compromise: simple looking filenames, but not too likely to get to a naming conflict. It think it's expressive enough.

    Forget the toolchain Lmod meta package names; they could be loaded on some machines, but are just confusing for the user.

    I think also this generalisation of stage is fine.

    Sorry, too many ramblings. Bottom line is that in my opinion it's okay to close this issue.

    * edit: Maybe I'm wrong about not being able to follow which compiler is used in all the details. I suppose CMake tells the rest what to use. Then they should use it.

  9. kvrigor commented on Apr 30, 2025

    @kvrigor
    kvrigor
    on Apr 30, 2025
    MemberAuthor

    Maybe you should replace the classic Intel compiler with ifx instead of renaming jsc.2025.intel.psmpi. The commit and the fact that it is saved into the install and will be saved into the build directory is enough for reproducibility.

    Or you introduce jsc.2025.intelllvm.psmpi and optionally rename the original.

    jsc.2025.intelllvm.psmpi is good. We can't use ifx since we're using a mix of C/C++/Fortran compilers.

    In this Intel compiler example I'd not change the filename or introduce a new file because Intel/2024.2.0 contains both ifort and ifx and we don't have time to validate if all components including libraries and utilities stick with one or the other.* Who cares anyway? If you do care, you can find out by deep inspection in the build directory.

    IMO it's safe to be explicit by default. Opaqueness can bite back really hard (e.g. see this related eCLM issue)

    I think the current scheme is a fine compromise: simple looking filenames, but not too likely to get to a naming conflict. It think it's expressive enough.

    Yes, though I'd prefer to have a shorter naming scheme if possible. We can think about it later when we have actually more combinations to support.

    Bottom line is that in my opinion it's okay to close this issue.

    Not yet; I plan to do the env folder restructuring on another PR. Also I think the filenaming issue is not that urgent; let's wait for inspiration to strike and come back to this later 😜

  10. AGonzalezNicolas
    added
    environment file name
    model ID
    on May 5, 2025
  11. kvrigor
    added a parent issue on May 16, 2025
  12. github-project-automation
    added this to TSMP2: Towards Stages/2026 supporton Jan 13, 2026
  13. github-project-automation
    moved this to Todo in TSMP2: Towards Stages/2026 supporton Jan 13, 2026
  14. kvrigor
    moved this from New features to Housekeeping in TSMP2: Towards Stages/2026 supporton Jan 13, 2026
  15. kvrigor
    added
    environmentIssues springing from the environment (in a general sense) where TSMP2 is executed.
    and removed
    environment file name
    model ID
    build_tsmp2
    on May 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    environmentIssues springing from the environment (in a general sense) where TSMP2 is executed.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions