Skip to content

Regression testing of Modelica and ModelicaTest

Jump to bottom
Thomas Beutlich edited this page Feb 29, 2020 · 27 revisions

Documentation

Steps

Regression testing consists of various steps to be performed as automated as possible.

Run the pedantic check

  1. Load libraries Modelica and ModelicaTest in Dymola and verify that you also loaded the Dymola-specific ModelicaServices

  2. Clone the Release checks repository

git clone https://github.com/modelica-tools/msl-release

  1. Open ReleaseChecks.mo in Dymola

  2. Run the pedantic check on Modelica and ModelicaTest

ReleaseChecks.printExecutables(libraries={"Modelica", "ModelicaTest"})
  1. Check the created file "log_failed.txt" for all models that fail the pedantic translation and create a new gitHub issue.

Simulate all example and test models

  1. Open a git bash and run
git config --get remote.origin.url
git rev-parse --short HEAD

to determine the Git revision and Git URL of the repository under test.

  1. Simulate all example and test models with a tolerance of 1e-6
ReleaseChecks.simulateExecutables(libraries={"Modelica", "ModelicaTest"}, incrementalRun=false, tolerance=1e-6, gitURL="https://github.com/modelica/ModelicaStandardLibrary.git", gitRevision="add from above", description="Reg test MSL v4.0.0-rc.1")

The simulation results will be written to the output directories "modelica://Modelica/Resources/Reference/Modelica" and "modelica://ModelicaTest/Resources/Reference/ModelicaTest", respectively. In case these directories already contain simulation log files or simulation result files from previous runs, they will be deleted first.

  1. Repeat the simulation for all failed models with an increased tolerance
ReleaseChecks.simulateExecutables(libraries={"Modelica", "ModelicaTest"}, incrementalRun=true, tolerance=1e-5, gitURL="https://github.com/modelica/ModelicaStandardLibrary.git", gitRevision="add from above", description="Reg test MSL v4.0.0-rc.1")
ReleaseChecks.simulateExecutables(libraries={"Modelica", "ModelicaTest"}, incrementalRun=true, tolerance=1e-4, gitURL="https://github.com/modelica/ModelicaStandardLibrary.git", gitRevision="add from above", description="Reg test MSL v4.0.0-rc.1")
  1. Search the two output directories for files "check_failed.log", "translate_failed.log" and "simulate_failed.log" and create new GitHub issues.

Run the test

Download CSV files

wget -r --level=0 -np -nH --cut-dirs=3 -A csv http://www.ltx.de/download/MA/Compare_MSL_v3.2.3-rc.2/2018-12-15_230315_Modelica_v3.2.3-rc.2_2019

Run the comparison with bitmaps

d:\ITI-Projekte-NoBak\csv-compare\Modelica_ResultCompare\bin\x64\Release\Compare.exe --mode csvTreeCompare --override --bitmap --tolerance 2e-3 --delimiter "," --verbosity 2 e:\compare\2018-12-15_230315_Modelica_v3.2.3-rc.2_2019\ t:\Referencemodels\Libraries\ModelicaStandardLibrary\ReferenceResults\v3.2.2+build.0-beta.3\ --logfile log.txt --comparisonflag --reportdir e:\compare\MSL323ALL\

Check the failed ones

In TC:

  • find all directories containing a file compare_failed.log using content plugin FileInDir (Load saved search settings "Failed CSV comparisons")
  • put search result to listbox
  • select directories and copy directory structure to e:\compare\Failed\ using packer plugin TreeCopyPlus

Run the comparison without bitmaps

d:\ITI-Projekte-NoBak\csv-compare\Modelica_ResultCompare\bin\x64\Release\Compare.exe --mode csvTreeCompare --override --tolerance 2e-3 --delimiter "," --verbosity 2 e:\compare\Failed\ t:\Referencemodels\Libraries\ModelicaStandardLibrary\ReferenceResults\v3.2.2+build.0-beta.3\ --logfile log.txt --comparisonflag --reportdir e:\compare\MSL323FAILED\

Upload

  • Create git repo and push e:\compare\MSL323FAILED\

Same for ModelicaTest

Clone this wiki locally