Open-source tools for reproducing, diagnosing, and testing order, fulfillment, and inventory integration failures using explicit evidence and deterministic scenarios.
An order is marked complete after one shipment, but units remain outstanding. Reproduce the failure, inspect the supporting records, test corrected behavior, and share a report that explains the result.
SafetyMP reliability thesis — automation depends on dependable operations. AI-enabled decisions inherit risk from the records and integrations beneath them. This kit explores how reproducible scenarios, explicit evidence, and inconclusive verdicts can surface order, fulfillment, and inventory discrepancies before teams rely on faulty signals. It tests integrations, not AI models or live merchant systems. See the portfolio thesis.
Try the live demo · Source repository · Apache-2.0 license
Scope: Early v0.1 toolkit and frontend demo. The Node CLI and library include 24 synthetic scenarios. The browser demo uses a separate six-fixture evaluator. No merchant accounts are connected, and no production integration has been validated by the demo. No versioned package release is published.
Jump to: Who it helps · Demo · Integration testing · Quick start · Verification · Documentation · Contributing
| Team | Use the kit to |
|---|---|
| Integration agencies | Turn recurring support cases into synthetic reproductions and share evidence with clients and connector vendors. |
| Connector engineers | Test completion, synchronization, and inventory publication behavior against explicit expectations in your own test workflow. |
| Warehouse and fulfillment software teams | Examine partial shipments, duplicate events, missing orders, and stale inventory observations across system boundaries. |
| Merchant technical teams | Understand what the available records support and prepare a reproducible issue for an integration partner. |
Start with a synthetic case before adapting a test to your own normalized evidence. The toolkit accepts files and supplied outcomes; connecting systems and collecting their records remain your responsibility.
The browser demo needs no installation, account, API key, or backend. Its records are synthetic, and it has no telemetry or analytics.
Follow the partial-shipment example:
- Run the faulty example: 5 ordered, 3 fulfilled, 2 outstanding, but the example reports the order complete.
- Inspect the assertion and its exact supporting records.
- Run the corrected example against the same evidence and policy.
- Compare FAIL → PASS, then download JSON or a standalone HTML report.
Also explore normal synchronization delay, persistent missing orders, an inventory publication buffer, stale warehouse evidence, and duplicate shipment events. Search and category filters help you find a case; policy controls show how assumptions affect a result. See the demo walkthrough.
The demo holds up to 12 reports in memory. Refresh clears those reports; downloads preserve the selected run. Browser examples and CLI scenarios are separate fixtures, so their quantities and coverage can differ.
The CLI and library support three workflows:
| Workflow | What you supply | What you get |
|---|---|---|
| Reproduce a scenario | One of the 24 versioned synthetic scenarios | Deterministic findings and assertion verdicts for the declared case |
| Test integration behavior | A scenario plus outcomes or normalized captures produced by your own integration test | Assertions against the scenario's expectations, with evidence and uncertainty retained |
| Diagnose captured records | Normalized JSON, JSONL, or CSV evidence and an explicit evaluation time | Supported findings, missing prerequisites, and inspectable reports |
The implemented checks cover order presence, fulfillment quantities and deadlines, inventory comparison, and freshness. The toolkit includes schema validation, a virtual clock, JSON/HTML/JUnit reports, and qualified report comparison. Use the CLI and library guide for faulty/corrected integration examples and CI exit codes, and the import guide for capture formats and coverage declarations.
A scenario can pass because it correctly detects an expected operational problem. Findings describe supported operational facts. Assertion verdicts judge whether an expectation was met. A missing-order scenario can therefore have a supported missing-order finding and passing assertions.
Missing, stale, ambiguous, or incomplete evidence can make a result INCONCLUSIVE. Coverage and mappings are caller declarations. Agreement between records does not prove physical stock accuracy, and passing assertions do not certify a connector.
Use Node 22.23.2 and npm 10. Run these commands from a new checkout:
git clone https://github.com/SafetyMP/commerce-reliability-kit.git
cd commerce-reliability-kit
npm ci --ignore-scripts
npm run build
node dist/packages/cli/index.js demo partial-shipment --output ./my-first-reportOpen my-first-report/report.html. The same directory contains JSON and JUnit reports. Existing report files are never overwritten; choose a new output directory when repeating the command.
To run the browser demo locally from the same checkout:
npm --prefix apps/demo ci --ignore-scripts
npm --prefix apps/demo run devOpen the local URL printed by Vite. For a production build and local preview:
npm --prefix apps/demo run build
npm --prefix apps/demo run previewThe root toolkit and frontend have separate committed lockfiles. Installation needs registry access or a populated npm cache; evaluation runs locally. The npm package remains private and is not published to the registry.
After installing both sets of dependencies, run from the repository root:
npm run check
node acceptance/run-tests.mjs
npm run build
node acceptance/frontend-check.mjs
node --test scripts/publication-check.test.mjs
node scripts/publication-check.mjs --checkThese commands cover toolkit and contract tests, frontend tests and typechecking, builds, and publication checks. There is no lint command configured. The GitHub workflow validates changes and deploys the checked static artifact only from main. A green workflow does not establish production connector compatibility.
See development for individual commands and deployment for repository-subpath builds, browser acceptance, and rollback.
- Synthetic demo: no merchant accounts, customer credentials, production API calls, or external commerce updates.
- Your own captures: toolkit reports include accepted input evidence. Review identifiers and free text before sharing; strict schemas and import filtering do not guarantee anonymity. Never attach raw customer exports or secrets to public issues.
- Current scope: live collectors, provider adapters, hosted monitoring, automatic corrections, general bundles, fractional conversions, returns, B2B/EDI, and arbitrary transformations are outside the current implementation.
Report vulnerabilities through the private route in SECURITY.md. Read limitations, compatibility, and the dated source register before applying a result to a real integration.
Useful contributions include synthetic failure scenarios, reproducible correctness bugs, clearer provider-mapping assumptions, and improvements to the demo or documentation. Include the expected behavior, supporting evidence, and relevant scenario/policy versions. See scenario authoring and CONTRIBUTING.md.
Support · Code of conduct · Governance · Roadmap · Changelog
The open toolkit and scenarios are independently useful. A future commercial service may offer managed collection, hosting, collaboration, notifications, and support; those services are not implemented or required to use this repository. See the commercial boundary.
Part of the SafetyMP portfolio.
Original code, documentation, and synthetic fixtures are licensed under Apache-2.0. Preserve NOTICE and third-party attribution. No vendor endorsement or certification is claimed.
| Topic | Start here |
|---|---|
| Browser scenarios and report comparison | Demo walkthrough |
| CLI commands, library API, integration tests, and exit codes | CLI and library guide |
| Normalized captures and evidence declarations | Import guide |
| Adding a reproducible synthetic case | Scenario authoring |
| Component boundaries and supported behavior | Architecture · Compatibility |
| Setup, tests, and build commands | Development · CI notes |
| Pages publication, browser checks, and rollback | Deployment |
| Limitations and source uncertainty | Limitations · Sources |
