tsdive is a Python library and command-line tool that checks process
time series (historian and sensor archives) for gaps, bad quality codes,
clipped ranges, timestamp faults and unit mismatches, then runs baseline,
control-chart and PCA monitoring on the same window. It reads single-tag
parquet archives, built from CSV or parquet exports with ingest or
through a source adapter (docs/SOURCES.md), and writes
text, JSON, Python objects or a static HTML page. A check with no answer
raises a typed error naming the check. It also runs randomized two-setting
trials on one unit (switchback).
tsdive needs Python 3.12 or newer. Install it, then write the demo
archives the examples read into data/:
pip install "git+https://github.com/NorthernLightx/tsdive.git"
tsdive demo dataGuides and reference: https://northernlightx.github.io/tsdive/.
Wheels are on the
releases page. Add
[ml] to the install for the IsolationForest detector (scikit-learn) or
[mcp] for the MCP server. The package is not on PyPI.
The library and CLI make no network calls. The dataset fetch scripts
under scripts/ download on request and print their size first.
Commands in the order you run them on an archive. A window is START/END,
START/PT5H, PT5H/END, or a date for one UTC day. Every analysis
command and ingest take --json and print one JSON object instead of
text, opening with result_kind and tsdive_version. --no-color and
the NO_COLOR environment variable turn colour off. Transcripts are
trimmed to the lines the text reads.
Data checks and statistics for one window. The headline carries the numbers you read first.
$ tsdive profile data/demo/fic101_demo.parquet \
--window "2024-03-30T20:00:00Z/2024-03-31T06:00:00Z" \
--tz Europe/London --flatline
demo:FIC101.PV FIC-101 flow
coverage 0.933 GOOD 561/562 censored yes gaps 1 flatline NOT ASSESSED
window 2024-03-30 20:00:00Z -> 2024-03-31 06:00:00Z (10 h)
contract TIME_WEIGHTED RECORDED NONE stepped no digest d59d433d9c62
units m3/h -> cubic meters per hour
Coverage
coverage 0.933 valid 0.998 gaps 1 data-loss gaps 1 longest 40 min
2024-03-30 23:00:00Z -> 23:40:00Z 40 min unknown (no rule matched)
Quality
GOOD 561 UNCERTAIN 1 BAD 0
unmapped codes, treated UNCERTAIN: SENSOR DRIFT
Range
clipped 0.0516 censored yes
Values GOOD n=561
min 60.86 p05 61.20 median 62.32 p95 100.0 max 100.0
constant run 2024-03-31 02:01:00Z -> 02:29:00Z 28 min n=29
Flatline
NOT ASSESSED (window censored; saturated != frozen)Omit --window to profile the whole archive. The constant run is the
half hour at full scale.
Finds where the tag's level changed, so you can pick a baseline window.
$ tsdive segment data/demo/fic101_demo.parquet
demo:FIC101.PV 6 segments 5 breakpoints censored yes
window 2024-03-30 20:00:00Z -> 2024-03-31 06:00:00Z usable 561
method PELT L2 on MAD-scaled values penalty 18.99 (3*log n) min-size 10
# start end n median mad
3 2024-03-31 00:10:00Z 2024-03-31 02:00:00Z 111 62.56 0.1826
4 2024-03-31 02:01:00Z 2024-03-31 02:29:00Z 29 100.0 0
5 2024-03-31 02:30:00Z 2024-03-31 03:56:00Z 86 61.37 0.1696Segment 4 is the half hour at full scale. A larger --penalty gives
fewer segments.
Flags samples outside a baseline built from another window.
$ tsdive screen data/demo/fic101_demo.parquet \
--baseline "2024-03-30T20:00:00Z/2024-03-31T01:00:00Z" \
--window "2024-03-31T01:00:00Z/2024-03-31T06:00:00Z"
demo:FIC101.PV flagged 29 of 300 (9.7%)
baseline 2024-03-30 20:00:00Z -> 2024-03-31 01:00:00Z GOOD 261 censored no
window 2024-03-31 01:00:00Z -> 06:00:00Z
method MAD center 62.32 scale 0.5488 k 3.0 limits [60.67, 63.97]
Flagged
2024-03-31 02:01:00Z -> 02:29:00Z 29 samples--mode <parquet> computes one baseline per regime of a MODE tag, such as
segment --mode-out writes. --method moving-range replaces the MAD scale.
A censored baseline, or one that overlaps the window, raises an error in
screen, spc and mspc.
Individuals control chart with three Nelson rules on the same baseline
as screen. A rule with no hits prints zero.
$ tsdive spc data/demo/fic101_demo.parquet \
--baseline "2024-03-30T20:00:00Z/2024-03-31T01:00:00Z" \
--window "2024-03-31T01:00:00Z/2024-03-31T06:00:00Z"
demo:FIC101.PV 37 rule hits in 300 samples
baseline 2024-03-30 20:00:00Z -> 2024-03-31 01:00:00Z GOOD 261 censored no
window 2024-03-31 01:00:00Z -> 06:00:00Z
limits center 62.32 sigma 0.5488 lcl 60.67 ucl 63.97
basis individuals 3-sigma
BEYOND_3SIGMA 29
2024-03-31 02:01:00Z -> 02:29:00Z 29 samples
RUN_9_SAMESIDE 6
2024-03-31 01:08:00Z 9 consecutive points on one side of center 62.32
TREND_6 2
2024-03-31 01:25:00Z 6 consecutively increasing/decreasing pointsPCA T2 and SPE over two or more archives on one grid.
$ tsdive mspc \
data/demo/fic101_demo.parquet data/demo/tic101_demo.parquet \
--baseline "2024-03-30T20:00:00Z/2024-03-30T23:00:00Z" \
--window "2024-03-31T04:00:00Z/2024-03-31T06:00:00Z"
demo:FIC101.PV, demo:TIC101.PV T2 breaches 70 SPE breaches 108 of 121 rows
baseline 2024-03-30 20:00:00Z -> 23:00:00Z rows 181 coverage 1.000
window 2024-03-31 04:00:00Z -> 06:00:00Z rows 121 coverage 1.000
model rate 60 s (declared) components 1 of 2 explained 0.9767
limits T2 3.759 SPE 0.2859 (empirical q0.99)
contributors not ranked (2 tags; top-3 would list every one)
T2 breaches 70
2024-03-31 04:00:00Z
SPE breaches 108
2024-03-31 04:00:00ZThe demo temperature stops tracking the flow at 04:00Z. SPE carries the residual.
Reports what changed between two periods, over every tag of a unit. The tables do not say why.
$ tsdive compare \
data/demo/fic101_demo.parquet data/demo/tic101_demo.parquet \
--before "2024-03-30T20:00:00Z/2024-03-30T23:00:00Z" \
--after "2024-03-31T04:00:00Z/2024-03-31T06:00:00Z"
demo 2 tags refused 0 pairs 1 of 1 clearing
before 2024-03-30 20:00:00Z -> 23:00:00Z (3 h)
after 2024-03-31 04:00:00Z -> 06:00:00Z (2 h)
grid rate 60 s (declared) coverage 1.000 -> 1.000
Tags that changed
tag quality sigma spread flagged changed at
TIC101.PV ok +0.8 x6.0 62% 2024-03-31 04:18:00Z
FIC101.PV ok +0.4 x0.5 0% 2024-03-31 04:00:00Z
Pairs that decoupled (Pearson on first differences, 95% block bootstrap)
pair before after delta interval
FIC101.PV ~ TIC101.PV 0.69 0.17 -0.53 [-0.76, -0.26]
Joint structure (PCA fitted on before, 2 of 2 tags aligned)
components 1 of 2 explained 0.9767 -> 0.5129 on after
rows 121 T2 breaches 70 SPE breaches 108
SPE contributors TIC101.PV 50% FIC101.PV 50%The temperature's spread is 6 times the before period's, 62% of its samples
fall outside its before-period baseline, and its coupling to the flow drops
from 0.69 to 0.17 with an interval excluding 0. The PCA model fitted on the
before period keeps 51% of the after period's variance. With two tags the
residual splits evenly between them. --top N sets how many rows each table
prints.
Runs one TOML plan over many archives and writes ledger.json,
ledger.txt and report.html (tables and one plot per tag) to
-o DIR (default tsdive-run/ beside the plan). Options are the
step's own flags. Globs resolve against the plan file.
examples/plans/demo.toml, in a clone:
archives = ["../../data/demo/*.parquet"]
window = "2024-03-31T01:00:00Z/2024-03-31T06:00:00Z"
baseline = "2024-03-30T20:00:00Z/2024-03-31T01:00:00Z"
before = "2024-03-30T20:00:00Z/2024-03-30T23:00:00Z"
after = "2024-03-31T04:00:00Z/2024-03-31T06:00:00Z"
steps = ["profile", "segment", "screen", "spc", "mspc", "compare"]
[options.profile]
flatline = true
tz = ["Europe/London"]$ tsdive run examples/plans/demo.toml
run demo.toml 2 archives 6 steps profiles 2 findings 7 refusals 1
REFUSAL mspc demo:FIC101.PV, demo:TIC101.PV
[MspcAlignmentError] aligned coverage 0.867 below required 0.95; refusing to
interpolate across tags
wrote examples/plans/tsdive-run/ledger.json
examples/plans/tsdive-run/ledger.txt
examples/plans/tsdive-run/report.htmlThe baseline holds the 40-minute outage, so the aligned grid covers 0.867
and mspc raises MspcAlignmentError. ledger.txt opens with this text,
then one row per tag, every profile and every finding. --strict exits 2
on an error row and 3 on a refusal row. tsdive report-html <parquet...> --window START/END -o report.html writes the same page without a plan.
A wide export, one column per tag, takes two commands:
tsdive ingest export.csv --wide --timestamp-col ts --quality-suffix _q \
--init-meta meta/ --source-id plant1
tsdive ingest export.csv --wide --timestamp-col ts --quality-suffix _q \
--out archive/plant1/ --meta-dir meta/ --tz Europe/LondonThe first writes one template per tag with the optional keys of
docs/SCHEMA.md left null. The second writes one
archive per tag, reading quality from <tag>_q. A filled template:
{
"identity": {"source_id": "plant1", "point_id": "FIC101.PV"},
"name": "FIC-101 flow",
"unit_raw": "m3/h",
"eng_range_zero": 0.0,
"eng_range_span": 100.0,
"sample_rate_s": 3.0,
"role": "PV"
}A single-tag export uses --out FILE --meta FILE (see tsdive ingest --help).
import tsdive
p = tsdive.profile("data/demo/fic101_demo.parquet",
"2024-03-30T20:00:00Z/2024-03-31T06:00:00Z")
p.physics.coverage.coverage # 0.933
p.render() # the plain text the CLI prints, no colour
s = tsdive.screen("data/demo/fic101_demo.parquet",
"2024-03-30T20:00:00Z/2024-03-31T01:00:00Z", # baseline
"2024-03-31T01:00:00Z/2024-03-31T06:00:00Z") # window
s.to_dict()["n_flagged"] # 29; s.render() is the text tsdive screen printstsdive-mcp serves profile, segment, screen, spc, compare and
switchback_analyze to an MCP client over stdio, with the same arguments
and fields as --json. Tools and result shapes are in docs/MCP.md.
Every read runs these checks.
| Check | Reported as | Error |
|---|---|---|
tag identity is (source_id, point_id), not the display name |
the headline: identity, then name | SchemaError: no tsdive.meta or a missing key |
| timestamps are time-zone aware, stored in UTC | Timestamps: audited, duplicates, non-monotonic |
SchemaError: naive timestamps at ingest; NonMonotonicIndex: index runs backwards |
| DST transition inside the window | DST (<zone>), with --tz |
none |
| vendor quality codes kept verbatim, severity derived | Quality: the severity counts and unmapped codes |
SchemaError: no quality column and no --assume-quality |
| digital state codes are states, not values | value nulled, code kept in quality |
SchemaError: string value on a tag that is not role: MODE |
| data loss separated from compression silence | Coverage: coverage, gaps, longest, per-gap class and rule |
none; an unmatched gap is unknown |
| values pinned at the engineering range | Range: clipped, censored |
InsufficientQuality: censored window used as a baseline |
| unit spelling resolves through one alias table | units <raw> -> <canonical> or unresolved (null) |
UnresolvedUnitError; IncomparableUnitsError: reference-condition unit with no declared state |
| statistics use GOOD rows only | Values GOOD n=... |
InsufficientQuality: no GOOD sample |
| baseline values spread | scale, sigma |
ZeroSpreadBaseline: scale 0 |
General anomaly-detection libraries score the values they are given. tsdive checks what the archive did to those values first.
| Tool | What it does | What tsdive does differently |
|---|---|---|
| PyOD | outlier detectors over numeric feature arrays | reads timestamps and quality codes before any detector runs, and raises SchemaError on an archive with no metadata, naive timestamps or no quality column |
| ADTK | rule-based and unsupervised anomaly detection on pandas series | reports coverage, gap classes and clipping for the window, and raises InsufficientQuality instead of scoring a censored one |
| Darts | forecasting models with anomaly scores from forecast residuals | takes baselines from validated reference windows and raises ValueError when a baseline overlaps the window it screens |
| aeon | machine-learning toolkit for time-series tasks | states the sampling contract of every window and raises IncomparableSamplingError across contracts |
| Seeq, TrendMiner | commercial analytics servers connected to a live historian | runs offline on exported archives and replays identically under a recorded provenance fingerprint |
The checks ran over the 3W dataset (Petrobras, 1,119 offshore well instances, 14,347 tags) with zero refusals. Every detector number in the studies sits beside a position-in-record control and a false-alarm floor, and under the own-history design the control ranks above every detector: profile, detectors, Chronos-Bolt zero-shot, conformal alarm. Protocol in docs/EVAL.md, reproduction in docs/DATA.md.
No fault diagnosis. No causal inference outside a
switchback trial. No writes to any process or
historian. No alarm limits, notifications or real-time path. Deferred work
is in docs/SCOPE.md and docs/ROADMAP.md.
uv sync # editable install plus the dev group
make test # uv run pytest, offline
make lint # uv run ruff check .
make reference # refusal cases, byte for byte
make bench # BENCHMARKS.md, byte for byte
make check # all of the above
make api-diff # API breaks since the last release tagApache-2.0. The runtime dependencies (numpy, pandas, pyarrow, pint) carry
BSD or Apache licences; the tep extra's pyreadr is AGPL-3.0.