Release Testing System

The GeoDMS release testing system is a Python-based framework for running automated regression tests against different versions of the GeoDMS software. The testing framework is implemented in the GeoDMS-Test repository and allows developers to verify that new versions don't break existing functionality.

overview

The release testing system consists of:

batch/full.py - the main test runner script
batch/generic/regression.py - experiment management and the HTML report
batch/generic/profiler.py - profiling and the profile page of each test
batch/run_detached.ps1 - starts a run as a detached Windows process, so it survives the session that started it
batch/publish_results.py - publishes the report of a version on geodms.nl (see publishing test results)

The system runs a comprehensive suite of tests including operator tests, polygon tests, storage tests, network model tests, land use model tests, and GUI tests.

adding a new version to the release testing system

To test a new version of the GeoDMS, you need to:

1. install the GeoDMS version

Install the GeoDMS version you want to test. The Windows installers place the executables in:

%ProgramFiles%/ObjectVision/GeoDms{VERSION}

For example, for the MSBuild flavour of 20.26.0:

C:\Program Files\ObjectVision\GeoDms20.26.0.m

The Linux flavour (.l) is the .deb package, installed at /opt/ObjectVision/GeoDms{VERSION}.l in the WSL distro of the Windows account that runs the tests.

2. configure your machine (one-time)

Copy the gitignored settings template and adjust paths for your drives / user:

cd C:\path\to\GeoDMS-Test\batch
copy local_settings.json.template local_settings.json
notepad local_settings.json

See configuring local machine parameters below for the schema.

3. run the test script

Navigate to the batch folder in the GeoDMS-Test repository and run the test script with the version parameter. The path of the checkout must not contain a space (C:\dev\..., not C:\Users\Jip Claassens\...): GeoDmsRun gets the configuration paths unquoted, so a space makes every test fail within a second.

cd C:\path\to\GeoDMS-Test\batch
python full.py -version 20.26.0.m      # an installed release: .m MSBuild, .c CMake, .l Linux
python full.py -version 17.4.6         # versions before 20 have no flavour suffix
python full.py -version local          # the local CMake Release build

A full round takes hours. Start it detached, from the root of the checkout, so that it is not killed together with the terminal or session that started it:

powershell -ExecutionPolicy Bypass -File batch\run_detached.ps1 -Version 20.26.0.m

The output goes to {ResultsBaseDir}\_Temp\run_{version}_{time}.out, with the process id beside it.

re-running tests for an existing version

To re-run some tests of a version that has already been tested (e.g. after fixing a test or updating test data), name them with -tests. Their stored results (.bin) are replaced, and the report of that version is regenerated at the end. From the root of the checkout:

powershell -ExecutionPolicy Bypass -File batch\run_detached.ps1 -Version 20.26.0.c -Tests t2000_hestia

-tests matches substrings of the test names, so t200 also matches t2000; use a precise stem. A round starts by wiping the intermediates of that version, so a test that needs the output of another one (t405_2 and t405_3 need t405_1, t641_2 needs t641_1) is re-run together with it, e.g. -tests t405. To keep the previous measurement, copy the .bin and the log of the test first.

To continue an interrupted round, add -resume. To rebuild only the report from the stored results, use -report-only.

optional parameters

Parameter	Description
`-version`	the GeoDMS version to test, e.g. `20.26.0.m`, `20.26.0.c`, `20.26.0.l`, `17.4.6`, or a local build: `local`, `local-msbuild-release`, `local-linux-release`
`-tests`	comma-separated substrings of the names of the tests to run; omit it to run the full suite
`-report-only`	run nothing; regenerate the HTML report, and the profile pages of the version, from the stored results
`-resume`	continue an interrupted round of the same version, keeping its intermediates
`-all-versions`	show every result folder as a report column, not only the full GitHub releases plus the version under test
`-no-gui`	skip the GUI tests (t1630, t1640, t1642) on any flavour
`-linux-gui`	also run the GUI tests on `.l`
`-sp`	run GeoDmsRun with `/SP` (performance logging, GeoDMS 20.16.0 and later)
`-MT1`, `-MT2`, `-MT3`	multithreading settings `S1` or `C1`, `S2` or `C2`, `S3` or `C3`; default `S1 S2 S3`

python full.py -h gives the full and current list.

how the version lookup works

The get_geodms_paths function in full.py maps the -version string to the GeoDMS executables:

20.26.0.m and 20.26.0.c, and versions before 20 without a suffix such as 17.4.6, resolve to the installed %ProgramFiles%/ObjectVision/GeoDms{version} folder.
20.26.0.l resolves to /opt/ObjectVision/GeoDms20.26.0.l in WSL; every command runs through wsl --.
local and local-{flavor} (cmake-release, cmake-debug, msbuild-release, msbuild-debug, linux-release, linux-debug) resolve to a build in the GeoDMS source tree above ProfilerDir, or to the folders given under LocalBuilds in local_settings.json — handy for testing the working tree without going through the installer.

The report scripts (regression.py and profiler.py) always come from batch/generic/ of the checkout.

prerequisites

Before running the tests, ensure:

Python 3.x is installed and available in your PATH
GeoDMS version is either installed via the official installer (for numeric versions) or built locally (for -version local)
Source data directories are configured in batch/local_settings.json:
- RegressionTestsSourceDataDir — location of regression test data
- LocalDataDir — local data directory for test outputs
- ResultsBaseDir — where the result folders and the HTML reports are stored
Project snapshots are available in the prj_snapshots subdirectory of RegressionTestsSourceDataDir

configuring local machine parameters

Machine-specific paths live in batch/local_settings.json, which is gitignored so each working copy has its own without conflicting with others. A committed local_settings.json.template shows the expected schema and sane defaults.

one-time setup per working copy

cd C:\path\to\GeoDMS-Test\batch
copy local_settings.json.template local_settings.json
notepad local_settings.json   # adjust drive letters / user names

Example local_settings.json:

{
    "_comment": "OVSRV10 — Maarten's dev box",
    "RegressionTestsSourceDataDir":    "F:/SourceData/RegressionTests",
    "RegressionTestsAltSourceDataDir": "F:/SourceData",
    "LocalDataDir":                    "C:/LocalData",
    "ProfilerDir":                     "C:/dev/GeoDMS_2026/profiler",
    "LocalBuildDir":                   "C:/dev/GeoDMS_2026/build/windows-x64-release/bin"
}

local_settings.json.template lists the other keys, among them ResultsBaseDir and PublishSite (see publishing test results).

resolution order

Each setting is resolved in this priority order (highest wins):

Environment variable with the same name as the JSON key (e.g. LocalDataDir)
local_settings.json in batch/
Built-in default in _BUILT_IN_DEFAULTS (top of full.py)

So for one-off overrides without editing the file:

$env:LocalDataDir = "D:\TestRunData"
python full.py -version local

test results

Test results are stored under ResultsBaseDir (on OVSRV05 C:/LocalData/GeoDMS_Test_Results), in a folder per version and flavour named after the version with underscores: 20_26_0_m, 20_26_0_c, 20_26_0_l, 17_4_6.

Each folder holds:

bin/ - the stored result of each test; the report is built from these
log/ - the GeoDMS log of each test
html/ - the profile page of each test: memory, CPU and disk use over the run, compared with earlier versions
result/ - indicator results and file comparisons

The HTML report of a run is reports/{version}___{oldest version}.html, e.g. reports/20_26_0_m___16_0_5.html, with a column for each full release plus the version under test.

available tests

The full regression test suite includes:

Test ID	Description
t010	Operator test - tests all GeoDMS operators
t020	Polygon families compare (geos, bg, bp, cgal, dms)
t050	Storage write (Shapefile polygon)
t060	BAG Snapshot (GeoPackage)
t100	Network connect
t101	Network OD PC4 dense
t102	Network OD PC6 sparse
t151	BL-RD coordinate conversion
t200	Grid Poly2Grid
t300	XML read/parse
t301	BAG residential type
t405	NetworkModel PBL (prepare, run, indicator)
t410	NetworkModel EU
t611	LUS demo 2023
t641	RSopen regression test
t710	2UP model
t720	2BURP model
t810	ValLuisa Czech
t910	CUSA2 Africa
t1630	GUI expand test
t1640	Value info test
t1642	Value info group_by
t1742	Command statistics

adding new tests

To add a new test to the regression suite, add an entry in the get_experiments() function:

regression.add_exp(
    exps, 
    name=f"{result_folder_name}__t{NUMBER}_{TEST_NAME}",
    cmd=f"{geodms_paths['GeoDmsRunPath']} /L{result_paths['results_log_folder']}/{TEST_NAME}.txt /{MT1} /{MT2} /{MT3} {path_to_dms} {item_to_calculate}",
    exp_fldr=f"{result_paths['results_folder']}",
    env=env_vars,
    log_fn=f"{result_paths['results_log_folder']}/{TEST_NAME}.txt",
    indicator_results_file=f"{result_paths['results_folder']}/{TEST_NAME}.txt"
)

troubleshooting

version not found

If you get an error that the version is not found, verify:

The GeoDMS is installed in the expected location
The version string matches the folder name exactly (e.g., "18.0.3" not "v18.0.3")

missing source data

Ensure the regression test source data is available at the configured paths. Contact the GeoDMS team for access to the test data repository.

permission errors

Run the test script with administrator privileges if you encounter permission issues writing to the output directories.

every test fails within a second

The path of the checkout contains a space; GeoDmsRun gets the configuration paths unquoted. Use a checkout in a path without spaces.

a run stops without an error and its log ends mid-line

The run was killed together with the terminal or session that started it. Start long runs with run_detached.ps1.

t405_2 and t405_3 time out on `.l` after about six minutes

WSL has too little memory. WSL settings are per Windows account: without a .wslconfig, WSL gets half of the RAM and a small swap, while these tests need about 50 GB of RAM plus swap on Linux. On OVSRV05, give the account that runs the tests a C:\Users\{account}\.wslconfig with memory=48GB, swap=128GB and a swap file of its own on D: (swapFile=D:\\WSL\\swap_{account}.vhdx), then run wsl --shutdown.

publishing test results

Before a version is set to Latest on GitHub, its report is published on https://geodms.nl/tests/reports/ and linked from the release notes. The report links, per result column, to ../{folder}/log/{test}.txt and ../{folder}/html/{test}.html, and to nothing else on the site. So publish the report and exactly those files; bin/, result/ and *.sample.csv stay on the test machine.

batch/publish_results.py does that through WinSCP:

cd C:\path\to\GeoDMS-Test\batch
python publish_results.py 20.26.0 --own-only            # dry run: lists what would go up, no network
python publish_results.py 20.26.0 --own-only --preview  # WinSCP logs in and lists what it would upload
python publish_results.py 20.26.0 --own-only --upload   # uploads the files, then the report

It takes the newest report of the version, the one written after the last flavour finished. If results changed after that, e.g. by a re-run, first regenerate the reports with python full.py -version {version}.{flavour} -report-only, .m last.
It stages the linked files as hard links in {ResultsBaseDir}/_publish/stage and lets WinSCP synchronise them with /public_html/tests on ftp.geodms.nl: a file that the site lacks or holds in another size goes up, nothing is deleted, and the report goes last.
--own-only keeps it to the folders of that version. Without it, the other columns of the report are brought up to date as well, which can be several GB.
WinSCP logs in with a site saved in WinSCP; set its name as PublishSite in local_settings.json or as an environment variable. The password stays in WinSCP. Saved sites belong to a Windows account.

The firewall of the hosting provider (Mangelot Hosting) blocks an address that sends many requests in a short time or fails to log in, for the website and the FTP server alike, and such a block can last for hours. So do not check the site with a request per file; use --preview. When blocked, unblock the address in the Mangelot Hosting customer account (log in from another network, e.g. a phone) or open a support ticket.

releasing a version

Run the round of each flavour (.m, .c, .l), detached.
Check the report: the Windows flavours all green, and duration and memory in line with the current Latest release.
Publish the report: python publish_results.py {version} --own-only --upload.
Add [Test report](https://geodms.nl/tests/reports/{report}.html) at the end of the release notes, after a --- line.
Set the release to Latest: gh release edit v{version} --repo ObjectVision/GeoDMS --prerelease=false --latest, and make sure the notes no longer call it a pre-release.

Release Testing System

overview

adding a new version to the release testing system

1. install the GeoDMS version

2. configure your machine (one-time)

3. run the test script

re-running tests for an existing version

optional parameters

how the version lookup works

prerequisites

configuring local machine parameters

one-time setup per working copy

resolution order

test results

available tests

adding new tests

troubleshooting

version not found

missing source data

permission errors

every test fails within a second

a run stops without an error and its log ends mid-line

t405_2 and t405_3 time out on .l after about six minutes

publishing test results

releasing a version

see also

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

t405_2 and t405_3 time out on `.l` after about six minutes