[snehabandla]

Summary

Adds tests/explanation_corpus, a data-driven set of Rego policies and inputs
for explanation (causality) and partial-evaluation tests to share. Later PRs
can add expected explanations or partial-evaluation results to existing cases
instead of writing new ones. The value each case expects is fixed now, checked
on every regorus evaluator and against OPA.

The change is test-only: no production code, no new dependencies. It is built
with the rvm feature, which is on by default.

Details

• Cases. 52 cases with 150 examples in three YAML files, documented in
  tests/explanation_corpus/README.md. A case is one policy and query; its
  examples are inputs with an expected value or error. Each case needs at least
  two distinct expected outcomes, so every matching input has a non-matching
  twin.
• Coverage. Complete, partial-set, partial-object and function rules;
  else chains; defaults; multiple definitions; not; some ... in; every;
  membership; comprehensions; helper rules and cross-package calls; with;
  builtins; Undefined vs false vs empty vs null; an absent input document;
  conflicts; runtime errors; and Rego v0 syntax. corpus_is_well_formed fails if any construct has no case
  and prints the coverage counts.
• Evaluators. corpus_values runs every example on the interpreter and on
  the RVM in run-to-completion and suspendable modes.
• OPA check. When REGORUS_OPA points to an OPA binary,
  corpus_matches_opa runs opa eval on every example and compares the result
  with the expected value. An expected error must be an OPA evaluation error;
  a parse, compile or usage error from OPA fails the test. Without the
  variable the test does nothing.
• Errors. want_error is a substring of the regorus error message. It
  must not occur in the policy source, because error messages quote the
  failing line.
• Strict builtin errors. A case can set strict_builtin_errors. It is
  applied to both regorus evaluators and passed to OPA as
  --strict-builtin-errors.
• Hooks for later features.
  • unknowns lists the input paths partial evaluation should treat as
    unknown.
  • secrets lists values that a report must not show unredacted. Each one
    must appear in the case's input or data.
  • explanation (per example) and partial_eval (per case) are reserved and
    rejected for now. The PR that adds a feature replaces the field with its
    typed expectation and a check.

Known divergences

Where regorus differs from OPA today, the expected value is OPA's, and a
divergence entry pins what regorus currently returns. The test checks the
pinned value. Once the behaviour is fixed, the test fails, and the fix PR must
remove the entry. Pinned today:

• Membership (x in coll) on an Undefined collection returns false instead
  of Undefined, so not x in coll succeeds. This affects both evaluators.
• The RVM compiler doesn't support with yet.
• The RVM currently returns Undefined, instead of a conflict error, when
  complete-rule or function definitions produce different values.
• The RVM keeps the last value for a conflicting partial-object key instead of
  raising an error (RVM: partial object codegen gaps (constant-key, multi-level, duplicate-key) #719).
• The RVM currently accepts every k, v in input.limits { v <= data.max[k] }
  when the check fails for the current key, or when the key is missing from
  data.max.
• In strict mode, the RVM currently returns Undefined for a failing builtin
  call, such as count(5) or division by zero, instead of an error.

Compatibility

None. This change adds test code only.

Testing

• cargo test --test mod explanation_corpus passes with default features and
  with --features explanations, with REGORUS_OPA set to OPA 1.2.0. OPA
  agrees with all 150 expected values.
• cargo test --test mod (all 182 tests) passes with default features and with
  --features rego-extensions. With --no-default-features the corpus compiles
  out and the remaining tests pass.
• Pre-commit hook (cargo xtask pre-commit: build, fmt and clippy, including
  bindings).
• The checks catch errors in the corpus:
  • Changing an expected value fails corpus_values and corpus_matches_opa.
  • Removing or altering a divergence entry fails corpus_values.
  • A case without a non-matching twin, or an expected error text that also
    occurs in the policy, fails corpus_is_well_formed.
  • A policy that OPA can't parse fails corpus_matches_opa.
  • Treating an absent input as null on the RVM fails corpus_values.
• cargo fmt --check; cargo clippy --all-features --all-targets -D warnings;
  cargo clippy --no-default-features --lib --tests -D warnings.

[Anand Krishnamoorthi (anakrish)]

LGTM!

Thanks for focusing on tests for such a complex feature!