How to act on the $MODULEPATH when the checkloader runs

[MarcoMagl]

Hello,

In our center we are using ReFrame for multiple purposes. One of them is to ensure that the different modules we have from different software stacks we deployed year after year deliver consistent performance.

In order to make our ReFrame setup more consistent, we want to use a specific version of ReFrame (this is important for what I am trying to achieve) to test the different software stacks we have. Put differently, whatever the stack I want to test is, I want to use the same ReFrame executable. On top of this, I want to be able to test the different versions of the modules that we have deployed.

With a conceptual example:

Suppose I am launching ReFrame in an environement where $MODULEPATH is release/releaseA in which no PyTorch is available.
Imagine I have another software stack that I can access by setting $MODULEPATH tp release/releaseB where we have deployed three different versions of PyTorch: PyTorchB1, PyTorchB2, PyTorchB3

To test these 3 versions, we launch a test (located with $RFM_CHECK_SEARCHPATH) like this:

reframe --module-path=release/releaseB --run 

to test the different version of PyTorch, we use something like this in the test definition:

@rfm.simple_test
class Subscheduling(rfm.RunOnlyRegressionTest):
    module_info = parameter( lxp_find_modules("PyTorch/"))

what lxp_find_modules does is something like this:

def lxp_find_modules(sustr_in_module_name):
    import reframe.core.runtime as rt
    ms = rt.runtime().modules_system
    return ms.available_modules(substr_in_module_name)

it will perform (in our case) some module avail commands and we expect it to return ["PyTorchB1", "PyTorchB2", "PyTorchB3"].

In the present case however, lxp_find_modules("PyTorch/") will return nothing. This is, I think, because the list of test is constructed in cli.py in line 1213 for ReFrame 4.8.2:

    loader = RegressionCheckLoader(check_search_path,
                                   check_search_recursive,
                                   external_vars,
                                   options.skip_system_check,
                                   options.skip_prgenv_check)

which occurs before the different module options are taken into account which start in line 1511 still in cli.py. So the function ms.available_modules is not aware of the --module-path argument that we provided.

Is there a way to access the options.module_paths (cli.py, line 1546) from rt.runtime() or anything else?

[vkarak]

I think you have a point. We should be setting the MODULEPATH before loading the tests. Unfortunately you can't access the --module-path option from the runtime as it's not associated with a configuration option (maybe that's also bug; I don't see the reason why). So the only way at the moment is to rather set/unset the MODULEPATH explicitly without relying on --module-path.