diff --git a/man/manual-premium.md b/man/manual-premium.md index 07eeab86094..3553acc1b91 100644 --- a/man/manual-premium.md +++ b/man/manual-premium.md @@ -776,6 +776,120 @@ You can write comments about a suppression as follows: // cppcheck-suppress warningid ; some comment // cppcheck-suppress warningid // some comment +# Generating and using a baseline + +When you first run Cppcheck on an existing codebase it's common to get a +large number of warnings. A "baseline" lets you suppress all of today's +warnings and see only *new* warnings introduced from now on. + +This works by asking Cppcheck to include a content-based `hash` for every warning +(computed from the surrounding code, not the line number), then converting today's +warnings into an XML suppressions file keyed on `id` + `fileName` + `hash`. Because +the hash is based on content rather than line number, warnings stay suppressed even +after unrelated lines above them are added or removed. A warning only reappears if +the code it actually points at changes, or a new warning shows up elsewhere. + +## 1. Generate the baseline + +Run Cppcheck with `--xml` and capture stderr (where Cppcheck writes its XML) to a +file: + +```sh +cppcheck --enable=style --xml src 2> baseline-results.xml +``` + +Use whatever combination of `--enable`/defines/include paths you normally +analyze the project with — the suppressions you get out only cover the +checks you ran. + +## 2. Convert the results into a suppressions file + +A [script](https://github.com/cppcheck-opensource/cppcheck/blob/main/tools/generate-baseline-suppressions.py) can be used to generate the baseline: +```sh +python3 generate-baseline-suppressions.py baseline-results.xml suppressions.xml +``` + +This produces a `suppressions.xml` like: + +```xml + + + + uninitvar + src/file1.c + 12345678 + + +``` + +`suppressions.xml` needs to be shared by everyone who runs Cppcheck on this +codebase, including CI. + +Having a script instead of using some special cppcheck flags has the advantages: + * it's flexible. You can tweak it if needed. + * there is fewer flags for us to maintain and document, and for you to learn. + +## 3. Use the baseline on future runs + +```sh +cppcheck --enable=style --xml --suppress-xml=suppressions.xml src +``` + +Only warnings that aren't in the baseline are reported: new warnings in changed +code, and warnings for checks/files that weren't covered when the baseline was +generated. + +## 4. Refreshing the baseline + +Re-run steps 1-2 whenever you want to accept the current state as the new +baseline (e.g. after cleaning up a batch of warnings, or deliberately accepting a +new one). Regenerating overwrites `suppressions.xml` with an entry for every +warning present at that time. + +## The baseline doesn't need to be kept in sync + +Once code that a baselined warning pointed at is fixed, refactored away, or +deleted, its `suppress` entry in `suppressions.xml` becomes dead: nothing will +ever match it again. You don't need to go find and remove it. + +Cppcheck's `unmatchedSuppression` check (part of `--enable=all`) normally warns +about suppressions that never matched anything, on the theory that a +suppression nobody needs is probably a mistake. But it does *not* fire for +`suppress` entries that carry a `` — which is every entry the baseline +script generates. So a baseline file with plenty of dead entries produces no +noise, and developers never need to prune it by hand — it only needs to be +regenerated (step 4) when you deliberately want to reset what's accepted. + +## Caveats + +- A warning can only be suppressed this way if it carries a `hash` attribute in + the XML output. Almost all checks compute one. Critical errors, i.e. syntax + errors do not get hash and must be fixed. Certain information messages do not + get hash neither. +- Changed Cppcheck options might produce new warnings that are not suppressed + by the baseline. +- Cppcheck upgrades don't affect the hash directly (it isn't version-tagged), + but if a new release for instance changes a check's message wording, the hash + changes with it — so upgrading Cppcheck can resurrect baselined warnings for + checks whose messages were reworded, even though nothing in the analyzed code + changed. + +## Strategies for gradually shrinking the baseline + +A baseline makes it possible to adopt Cppcheck in CI immediately without +fixing everything first, but nothing about it enforces that the accepted set +of warnings actually shrinks over time. + +Generic advice: + +- **Prioritize by severity** Checks like `uninitvar` or + `nullPointer` are more valuable to clear than `style` warnings; a baseline + makes it possible to drive the highest-severity checks to zero first while + deliberately leaving lower-risk ones suppressed longer. +- **Be careful** Every fix is a code change, and every code change carries some + risk of introducing a new bug. It can make sense to leave some things suppressed + to minimize the risk that bugs are introduced in working code. + # XML output Cppcheck can generate output in XML format. Use `--xml` to enable this format. diff --git a/man/manual.md b/man/manual.md index 6227ce0abb0..e1ccd7ad59c 100644 --- a/man/manual.md +++ b/man/manual.md @@ -777,6 +777,120 @@ You can write comments about a suppression as follows: // cppcheck-suppress warningid ; some comment // cppcheck-suppress warningid // some comment +# Generating and using a baseline + +When you first run Cppcheck on an existing codebase it's common to get a +large number of warnings. A "baseline" lets you suppress all of today's +warnings and see only *new* warnings introduced from now on. + +This works by asking Cppcheck to include a content-based `hash` for every warning +(computed from the surrounding code, not the line number), then converting today's +warnings into an XML suppressions file keyed on `id` + `fileName` + `hash`. Because +the hash is based on content rather than line number, warnings stay suppressed even +after unrelated lines above them are added or removed. A warning only reappears if +the code it actually points at changes, or a new warning shows up elsewhere. + +## 1. Generate the baseline + +Run Cppcheck with `--xml` and capture stderr (where Cppcheck writes its XML) to a +file: + +```sh +cppcheck --enable=style --xml src 2> baseline-results.xml +``` + +Use whatever combination of `--enable`/defines/include paths you normally +analyze the project with — the suppressions you get out only cover the +checks you ran. + +## 2. Convert the results into a suppressions file + +A [script](https://github.com/cppcheck-opensource/cppcheck/blob/main/tools/generate-baseline-suppressions.py) can be used to generate the baseline: +```sh +python3 generate-baseline-suppressions.py baseline-results.xml suppressions.xml +``` + +This produces a `suppressions.xml` like: + +```xml + + + + uninitvar + src/file1.c + 12345678 + + +``` + +`suppressions.xml` needs to be shared by everyone who runs Cppcheck on this +codebase, including CI. + +Having a script instead of using some special cppcheck flags has the advantages: + * it's flexible. You can tweak it if needed. + * there is fewer flags for us to maintain and document, and for you to learn. + +## 3. Use the baseline on future runs + +```sh +cppcheck --enable=style --xml --suppress-xml=suppressions.xml src +``` + +Only warnings that aren't in the baseline are reported: new warnings in changed +code, and warnings for checks/files that weren't covered when the baseline was +generated. + +## 4. Refreshing the baseline + +Re-run steps 1-2 whenever you want to accept the current state as the new +baseline (e.g. after cleaning up a batch of warnings, or deliberately accepting a +new one). Regenerating overwrites `suppressions.xml` with an entry for every +warning present at that time. + +## The baseline doesn't need to be kept in sync + +Once code that a baselined warning pointed at is fixed, refactored away, or +deleted, its `suppress` entry in `suppressions.xml` becomes dead: nothing will +ever match it again. You don't need to go find and remove it. + +Cppcheck's `unmatchedSuppression` check (part of `--enable=all`) normally warns +about suppressions that never matched anything, on the theory that a +suppression nobody needs is probably a mistake. But it does *not* fire for +`suppress` entries that carry a `` — which is every entry the baseline +script generates. So a baseline file with plenty of dead entries produces no +noise, and developers never need to prune it by hand — it only needs to be +regenerated (step 4) when you deliberately want to reset what's accepted. + +## Caveats + +- A warning can only be suppressed this way if it carries a `hash` attribute in + the XML output. Almost all checks compute one. Critical errors, i.e. syntax + errors do not get hash and must be fixed. Certain information messages do not + get hash neither. +- Changed Cppcheck options might produce new warnings that are not suppressed + by the baseline. +- Cppcheck upgrades don't affect the hash directly (it isn't version-tagged), + but if a new release for instance changes a check's message wording, the hash + changes with it — so upgrading Cppcheck can resurrect baselined warnings for + checks whose messages were reworded, even though nothing in the analyzed code + changed. + +## Strategies for gradually shrinking the baseline + +A baseline makes it possible to adopt Cppcheck in CI immediately without +fixing everything first, but nothing about it enforces that the accepted set +of warnings actually shrinks over time. + +Generic advice: + +- **Prioritize by severity** Checks like `uninitvar` or + `nullPointer` are more valuable to clear than `style` warnings; a baseline + makes it possible to drive the highest-severity checks to zero first while + deliberately leaving lower-risk ones suppressed longer. +- **Be careful** Every fix is a code change, and every code change carries some + risk of introducing a new bug. It can make sense to leave some things suppressed + to minimize the risk that bugs are introduced in working code. + # XML output Cppcheck can generate output in XML format. Use `--xml` to enable this format. diff --git a/releasenotes.txt b/releasenotes.txt index 1652fe61215..b84ac0bfe01 100644 --- a/releasenotes.txt +++ b/releasenotes.txt @@ -20,7 +20,8 @@ Changed interface: - Infrastructure & dependencies: -- compile_commands.json - flexible handling of -isystem, --sysroot and -I flags through the script tweak-compile_commands.py. +- flexible handling of -isystem, --sysroot and -I flags in compile_commands.json through the script tweak-compile_commands.py. +- baseline functionality. Other: - Added configuration file for Microsoft.GSL (Guideline Support Library). diff --git a/tools/generate-baseline-suppressions.py b/tools/generate-baseline-suppressions.py new file mode 100755 index 00000000000..762a2ef3642 --- /dev/null +++ b/tools/generate-baseline-suppressions.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Convert a cppcheck XML results file into a cppcheck XML suppressions file. + +Usage: generate-baseline-suppressions.py + +Each in the results file that has a non-zero "hash" attribute is +converted into a entry using the error's id, the file of its +first (the primary location used by cppcheck's suppression +matching) and the hash. +""" +import sys +import xml.etree.ElementTree as ET + + +def convert(results_path, suppressions_path): + tree = ET.parse(results_path) + root = tree.getroot() + errors_el = root.find('errors') + + seen = set() + suppressions = [] + for error in errors_el.findall('error'): + error_id = error.get('id') + error_hash = error.get('hash') + location = error.find('location') + if error_id is None or location is None: + continue + if not error_hash or error_hash == '0': + continue + file_name = location.get('file') + key = (error_id, file_name, error_hash) + if key in seen: + continue + seen.add(key) + suppressions.append(key) + + out_root = ET.Element('suppressions') + for error_id, file_name, error_hash in suppressions: + suppress = ET.SubElement(out_root, 'suppress') + ET.SubElement(suppress, 'id').text = error_id + ET.SubElement(suppress, 'fileName').text = file_name + ET.SubElement(suppress, 'hash').text = error_hash + + ET.indent(out_root, space=' ') + out_tree = ET.ElementTree(out_root) + out_tree.write(suppressions_path, encoding='UTF-8', xml_declaration=True) + print(f'Wrote {len(suppressions)} suppression(s) to {suppressions_path}') + + +if __name__ == '__main__': + if len(sys.argv) != 3: + print(f'Usage: {sys.argv[0]} ', file=sys.stderr) + sys.exit(1) + convert(sys.argv[1], sys.argv[2]) diff --git a/tools/readme.md b/tools/readme.md index 3a2de757199..249a0c0bfc6 100644 --- a/tools/readme.md +++ b/tools/readme.md @@ -84,6 +84,18 @@ message. Script to compare the error IDs in the expected `testrunner` output (without executing it) with the `--errorlist` output. It will report missing test coverage for an ID and missing IDs in the `--errorlist` output. +### * tools/generate-baseline-suppressions.py + +Script that converts a Cppcheck XML results file (`--xml`) into a Cppcheck XML suppressions file, so that +existing warnings can be suppressed as a baseline and only new warnings are reported afterwards. For each +`` in the results file that has a non-zero `hash` attribute, it writes a `` entry with the +error's id, the file of its first `` and the hash. Usage: +```shell +$ cppcheck --xml --xml-version=2 ./src 2> results.xml +$ python tools/generate-baseline-suppressions.py results.xml suppressions.xml +``` +The generated `suppressions.xml` can then be passed to Cppcheck with `--suppress-xml=suppressions.xml`. + ### * tools/tweak-compile-commands.py Script to tweak `-isystem`/`--sysroot`/`-I` options in a `compile_commands.json` file, for example to make