Optionally Re-run the tests but not the workflow to help develop tests

[yfarjoun]

Slightly different from #174 (not a duplicate)

Once I have a workflow and code set up, I iterate on the tests that I want to add, adding md5sum contains elements to the yml file, but not actually changing the workflow or the code behind it.

It would be helpful if there was a way to just run the "lower part" of each workflow test.

In the envisioned implementation the user would

• provide an "old" working directory (that was preserved using --kwd)
• state that they do not want the workflows to be rerun ( --use_old_workflow_outputs)

This would cause pytest workflow to use the old working directory, not delete file in it, skip the first step of each test (running the workflow) and evaluate the rest of the tests as if that step succeeded.

[rhpvorderman]

I think this can be collapsed in a single command line flag that provides the old TMP folder where all the workflows are run. This then also disables launching any workflows and simply the workflow directory provided is used to run all the tests again. I think this is doable to implement without too much code.

So now we just need a nice flag. I am thinking of --reuse-workflow-dir <WORKFLOW_DIR>. @yfarjoun, @DavyCats , @Redmar-van-den-Berg any suggestions or feedback?

[Redmar-van-den-Berg]

I think this would be a nice addition, especially for slow tests where you run your entire pipeline with some fake data, which can take a few minutes each time.

[wholtz]

This is functionality that I would also find useful.

As for the name of the flag, --reuse-workflow-dir <WORKFLOW_DIR> could be interpreted as simply reusing the the directory as the place to execute the workflow. I will suggest --reevaluate-workflow-dir <WORKFLOW_DIR> as it is more descriptive of the action being performed.

[rhpvorderman]

@wholtz Excellent suggestion. I agree. That is a nice descriptive flag.

[stianlagstad]

I would very much like a flag like --reevaluate-workflow-dir <WORKFLOW_DIR>, to allow quicker iterations of the tests (not the workflow). Any initial ideas on how to implement this, to get me started?