Skip to content

Regression testing of Modelica and ModelicaTest

Jump to bottom
Thomas Beutlich edited this page Feb 29, 2020 · 27 revisions

Documentation

Steps

Regression testing consists of various steps to be performed as automated as possible.

Run the pedantic check

  1. Load libraries Modelica and ModelicaTest in Dymola and verify that you also loaded the Dymola-specific ModelicaServices

  2. Clone the Release checks repository

git clone https://github.com/modelica-tools/msl-release

  1. Open ReleaseChecks.mo in Dymola

  2. Run the pedantic check on Modelica and ModelicaTest

ReleaseChecks.printExecutables(libraries={"Modelica", "ModelicaTest"})
  1. Check the created file "log_failed.txt" for all models that fail the pedantic translation and create a new gitHub issue.

Simulate all example and test models

  1. Open a git bash and run
git config --get remote.origin.url
git rev-parse --short HEAD

to determine the Git revision and Git URL of the repository under test.

  1. Simulate all example and test models with a tolerance of 1e-6
ReleaseChecks.simulateExecutables(libraries={"Modelica", "ModelicaTest"}, incrementalRun=false, tolerance=1e-6, gitURL="https://github.com/modelica/ModelicaStandardLibrary.git", gitRevision="add from above", description="Reg test MSL v4.0.0-rc.1")

Check the options of ReleaseChecks.simulateExecutables. For example, if you want to keep the simulation MAT files containing the trajectories of all variables, you can set the option keepResultFile. Or, you probably want to set the compiler to differ from the default value.

The simulation results will be written to the output directories "modelica://Modelica/Resources/Reference/Modelica" and "modelica://ModelicaTest/Resources/Reference/ModelicaTest", respectively. In case these directories already contain simulation log files or simulation result files from previous runs, they will be deleted first.

  1. Repeat the simulation for all failed models with an increased tolerance
ReleaseChecks.simulateExecutables(libraries={"Modelica", "ModelicaTest"}, incrementalRun=true, tolerance=1e-5, gitURL="https://github.com/modelica/ModelicaStandardLibrary.git", gitRevision="add from above", description="Reg test MSL v4.0.0-rc.1")
ReleaseChecks.simulateExecutables(libraries={"Modelica", "ModelicaTest"}, incrementalRun=true, tolerance=1e-4, gitURL="https://github.com/modelica/ModelicaStandardLibrary.git", gitRevision="add from above", description="Reg test MSL v4.0.0-rc.1")
  1. Search the two output directories for files "check_failed.log", "translate_failed.log" and "simulate_failed.log" and create new GitHub issues.

Run the test

  1. Download the reference result files to be used as the base-line. It is assumed that this directory is in parallel to the MSL root directory.

See the Reference results page how to obtain the reference results.

  1. Get the CSV Compare tool

Download or build Compare.exe from the csv-compare repository. It is recommended to use/build the latest version, supporting the "--failedonly" option.

  1. Copy Compare.exe to the MSL root directory

  2. In the MSL root directory, delete the file "log.txt" in case it already exists from a previous run

  3. Run the comparison with the "bitmap" option

Compare.exe --mode csvTreeCompare --override --bitmap --tolerance 2e-3 --delimiter "," --verbosity 2 .\Modelica\Resources\Reference\ ..\MAP-LIB_ReferenceResults\ --logfile logall.txt --comparisonflag --reportdir C:\temp\MSL400\
  1. Identify the failed models

The workflow is to repeat the comparison run without the "bitmap" option only for the failed models. We need to copy all directories containing the file "compare_failed.log" to a separate location, while maintaining the directory structure. If you prefer, you can create a script for this job, or use a file manager as Total Commander with particular plugins.

For example, in Total Commander:

  • Delete C:\compare\Failed
  • Find all directories (ALT+F7) containing a file "compare_failed.log" using the content plugin FileInDir

grafik

  • Feed the search result to listbox (ALT+L)
  • Select the found directories (CTRL+A) and copy the directory structure to C:\compare\Failed\ using the packer plugin TreeCopyPlus (ALT+F5)

grafik

  1. Repository creation

Create a new git repository (e.g., "MSL400RC1FAILED") at GitHub clone this empty repository to C:\compare\MSL400RC1FAILED, for example.

  1. Repeat the comparison without "bitmap" option, but with the "failedonly" option
Compare.exe --mode csvTreeCompare --override --tolerance 2e-3 --delimiter "," --verbosity 2 C:\compare\Failed ..\MAP-LIB_ReferenceResults\ --logfile log.txt --failedonly --comparisonflag --reportdir c:\compare\MSL400RC1FAILED\
  1. Upload
  • Add all CSS, HTML and JS files to the repository, commit and push.
  • In the seetings page of the GitHub repository, setup GitHub Pages to create a static website from the master branch

grafik

  1. Repeat for ModelicaTest

Repeat steps 4 to 9 for ModelicaTest.

Clone this wiki locally