feat: box plot chart, a measure's distribution per group

CHANGELOG.md

@@ -12,11 +12,17 @@ while the major version is 0, minor bumps may include breaking changes and say s
   - `increase_color`, `decrease_color` and `total_color` take green, amber, red or any `#RRGGBB`, and replace Superset's stock green, red and grey on every release. `show_value`, `show_legend`, `number_format`, axis titles, `x_label_rotation` (0, 45 or 90) and `x_label_format` set the rest.
   - On Superset 6.1.0 or later, `steps` lists a bridge's values in its own order and `closing` names the row that closes it, drawn last as the running total in the total colour: `"steps": ["FY2025", "Price", "Volume"], "closing": "FY2026"`. The first step rises from zero; `"opening": "FY2025"` draws the opening row as a total too, on a dashboard whose theme keeps zero on the value axis (`{"echartsOptionsOverridesByChartType": {"waterfall": {"yAxis": {"scale": false}}}}` in its JSON), which `check` and `apply` verify (`waterfall_opening_axis`). Older releases draw a running total after every step, so `check`, `apply` and `plan` refuse `steps` there. `total_label`, `increase_label` and `decrease_label` name the bars on 6.1.0 and warn before it.
   - `apply`'s data check says when a bridge's opening, step or closing has no row, a value isn't in `steps`, or the closing row doesn't equal the steps added up (Superset draws the sum either way).
-- The design brain is version 9. A waterfall gets `chart.waterfall-additive` (warn: `AVG`, `MIN`, `MAX` or `COUNT_DISTINCT` steps don't add up), `chart.waterfall-steps` (warn: past ~12 bars), `chart.waterfall-order` (info: a categorical waterfall without `steps` reads A to Z), `chart.waterfall-colors` (info for Superset's stock colours, whose green is 2.2:1 against the panel; warn for a colour under 3:1) and `chart.waterfall-axis-titles` (info: Superset puts the titles over wide or rotated tick labels). `advise --fix` fills a waterfall's `show_value` when it has at most 12 bars, and `number_format` `,.0f` on a count. The brief's chart-choice guide says when a waterfall is the right form.
-- `tools/extract_panel_contract.py` extracts the waterfall's options for the params contract from its control panel at each release; `tools/ci_live_waterfall.py` builds a bridge on a virtual dataset in CI.
+- A box plot (`"type": "box_plot"`, Superset's Box Plot): a measure's distribution in each group, its median, quartiles, whiskers and outliers, such as the daily sales of each month. Compiled, decompiled, compared by `plan` and checked by `apply` on 4.1.4, 5.0.0 and 6.1.0.
+  - `distribute_across` holds the columns whose rows are the observations (a time column at `time_grain`, e.g. a day each), `groupby` the columns that get a box each, `metrics` the measure.
+  - `whiskers` is `"tukey"` (Superset's default), `"min_max"` (no outliers) or two percentiles such as `[5, 95]`, which every release takes.
+  - `number_format`, `x_label_format`, axis titles, `x_label_rotation` and `color_scheme` set the rest; `row_limit` is a Superset 6.0.0 control, and warns before it, where a dashboard's query still stops at it but Explore leaves it out.
+  - `apply`'s data check says when the observations reach the row limit (10,000 unset on Superset 6.x), so the boxes miss some.
+- The design brain is version 9. A waterfall gets `chart.waterfall-additive` (warn: `AVG`, `MIN`, `MAX` or `COUNT_DISTINCT` steps don't add up), `chart.waterfall-steps` (warn: past ~12 bars), `chart.waterfall-order` (info: a categorical waterfall without `steps` reads A to Z), `chart.waterfall-colors` (info for Superset's stock colours, whose green is 2.2:1 against the panel; warn for a colour under 3:1) and `chart.waterfall-axis-titles` (info: Superset puts the titles over wide or rotated tick labels). `advise --fix` fills a waterfall's `show_value` when it has at most 12 bars, and `number_format` `,.0f` on a count. A box plot gets `chart.box-plot-observations` (warn: grouped by a period at that period's grain, each box holds a few observations) and `chart.box-plot-groups` (warn, with `--profile`: past ~20 boxes). `chart.ordinal-order` reads a box plot's groups, and with `--profile` stands down for a column the dataset reports numeric or temporal. The brief's chart-choice guide says when a waterfall is the right form, and when a box plot beats a line of P50 and P90.
+- `tools/extract_panel_contract.py` extracts the waterfall's and the box plot's options for the params contract from their control panels at each release; `tools/ci_live_waterfall.py` builds a bridge on a virtual dataset in CI.
 
 ### Fixed
 
+- `decompile` reads Superset's stored defaults for data zoom and the series limit (`zoomable` false on time-axis and mixed charts, no `series_limit` on a pivot) as unset; before, `params not preserved` named them on charts saved in Explore. A zoomable chart or a series limit is still named.
 - `plan` reads a `trend_color` written as the hex of a named shade (`#1B7F3B` for green) as equal to the dashboard, which decompile reads back as the name; before, the chart showed as changed after every apply. The waterfall's colours compare the same way.
 
 - `decompile` (and so `adopt`) names more chart settings it can't carry when they differ from Superset's defaults: big-number font sizes and a hidden trendline, a smooth or step line, horizontal bars on a time axis, a pie's radius and hidden labels, a funnel's labels, tooltip and percentage calculation, treemap labels, a heatmap's axis sort, legend, margins, label intervals and value bounds, a histogram's normalize setting, and any chart's currency format. Before, they were dropped without a note.

README.md

@@ -119,7 +119,7 @@ as-is.
   automatically. Dashboards Chartwright did not create are never overwritten;
   to bring a hand-built one under a spec, decompile it and build it at a new
   slug.
-- **What a spec can express**: 16 chart types, metrics as you write them,
+- **What a spec can express**: 17 chart types, metrics as you write them,
   per-chart filters, a native filter bar, tabs, markdown notes, a header and
   footer shown on every tab, and layouts you can draw as ASCII sketches.
 - **Environment promotion**: specs name their data (connection, schema,

chartwright/compiler.py

@@ -70,6 +70,7 @@
     "treemap": "treemap_v2",
     "mixed": "mixed_timeseries",
     "waterfall": "waterfall",
+    "box_plot": "box_plot",
 }
 # 'bar' and 'timeseries_bar' share a viz_type; the compiler marks categorical
 # bars in params so the decompiler can tell them apart (x column not temporal
@@ -498,6 +499,8 @@ def metric(m: str):
             p["number_format"] = chart.number_format
     elif t == "waterfall":
         _waterfall_params(chart, p, metric)
+    elif t == "box_plot":
+        _box_plot_params(chart, p, ds, metric)
 
     if isinstance(chart, _SeriesDisplay):
         _series_display_params(chart, p, metric)
@@ -845,6 +848,53 @@ def _waterfall_params(chart, p: dict, metric) -> None:
         p["x_axis_time_format"] = chart.x_label_format
 
 
+# whiskers -> whiskerOptions, the Box Plot panel's choices (BoxPlot/controlPanel.ts, 4.1.4
+# and 5.0.0 :83-99, 6.1.0 :84-102). The select is free-form, and the boxplot operator
+# reads any 'lo/hi percentiles' (operators/boxplotOperator.ts:28, all three releases), so
+# [5, 95] works on 4.1.4 too, whose choices don't list it (seen rendering).
+WHISKER_OPTIONS = {"tukey": "Tukey", "min_max": "Min/max (no outliers)"}
+
+
+def whisker_options(whiskers) -> str:
+    if isinstance(whiskers, list):
+        return f"{whiskers[0]}/{whiskers[1]} percentiles"
+    return WHISKER_OPTIONS[whiskers]
+
+
+def _box_plot_params(chart, p: dict, ds, metric) -> None:
+    """BoxPlot/controlPanel.ts at 4.1.4, 5.0.0 and 6.1.0: the observations are the rows
+    of `columns` (Distribute across), a box per `groupby` value, and the boxplot
+    post-processing turns them into quartiles (BoxPlot/buildQuery.ts:29-57). row_limit
+    is a 6.0.0 control."""
+    p["columns"] = list(chart.distribute_across)
+    p["groupby"] = list(chart.groupby)
+    p["metrics"] = [metric(m) for m in chart.metrics]
+    # Written even as Tukey: without it buildQuery adds no boxplot post-processing.
+    p["whiskerOptions"] = whisker_options(chart.whiskers)
+    if chart.time_grain:
+        # buildQuery buckets a column by the grain only if temporal_columns_lookup marks
+        # it, a control Explore fills from the dataset (BoxPlot/buildQuery.ts:38-50).
+        p["time_grain_sqla"] = chart.time_grain
+        lookup = {c: True for c in chart.distribute_across if ds.is_temporal(c) is not False}
+        if lookup:
+            p["temporal_columns_lookup"] = lookup
+    if chart.row_limit:
+        p["row_limit"] = chart.row_limit
+    if chart.number_format:
+        p["number_format"] = chart.number_format
+    if chart.x_label_format:
+        p["date_format"] = chart.x_label_format
+    if chart.x_label_rotation is not None:
+        p["x_ticks_layout"] = TICK_LAYOUTS[chart.x_label_rotation]
+    # sections.titleControls, as the axis charts write them (_y_axis_params).
+    if chart.x_axis_title:
+        p["x_axis_title"] = chart.x_axis_title
+        p["x_axis_title_margin"] = 50 if chart.x_label_rotation else 30
+    if chart.y_axis_title:
+        p["y_axis_title"] = chart.y_axis_title
+        _y_title_layout(p)
+
+
 def _x_label_params(chart: _AxisChart, p: dict) -> None:
     # Emitted only when set, so pre-feature bundles stay byte-identical.
     # x_axis_time_format and xAxisLabelRotation: every chart panel at 4.1.4, 5.0.0 and 6.1.0.

chartwright/decompile.py

@@ -20,7 +20,7 @@
 from .compiler import (
     BACKGROUND, COLUMN_CONFIG_KEYS, CONTRIBUTION_VALUES, FOOTER_PREFIX, HEADER_PREFIX, HEADER_SIZE,
     LEGEND_TYPES, ROW_UNITS_PER_SPEC_UNIT, SDC_BAR_MARKER, STACK_VALUES, VIZ_TYPE,
-    bridge_totals_sql, parse_steps_order_sql,
+    WHISKER_OPTIONS, bridge_totals_sql, parse_steps_order_sql,
 )
 from .spec import (
     ADHOC_AGGREGATES, BRIDGE_TOTAL, DEPENDENCY_PARENT_TYPES, FORMAT_COLOR_HEX, FORMAT_TEXT_HEX,
@@ -60,7 +60,8 @@
     "left_margin", "show_percentage", "show_values", "value_bounds",
     "xscale_interval", "yscale_interval", "column", "bins", "normalize",
     "show_value", "slice_id", "url_params", "percent_calculation_type",
-    "show_tooltip_labels", "tooltip_label_type", "show_total", "x_ticks_layout", SDC_BAR_MARKER,
+    "show_tooltip_labels", "tooltip_label_type", "show_total", "x_ticks_layout", "zoomable",
+    "series_limit", "series_limit_metric", SDC_BAR_MARKER,
 }
 
 # Of the keys above, those a user changes from the value an untouched chart stores:
@@ -105,6 +106,9 @@
     # Waterfall/controlPanel.tsx: a closing total (show_total, 6.1.0 :134) and Superset's
     # own tick layout (x_ticks_layout, 4.1.4 :130, 6.1.0 :210).
     "show_total": (True,), "x_ticks_layout": ("auto",),
+    # sharedControls.tsx: data zoom off (zoomable, 6.1.0 :422) and no series limit
+    # (series_limit and series_limit_metric, the box plot's at all three releases).
+    "zoomable": (False,), "series_limit": (None, "", 0), "series_limit_metric": (None, "", [], {}),
 }
 # A currency format has no default: unset reads as {} or every part empty.
 _UNSET_WHEN_EMPTY = {"currency_format"}
@@ -120,7 +124,7 @@
 
 # Spec chart types with a color_scheme field (spec._ColorSchemeMixin).
 _COLOR_SCHEME_TYPES = ("timeseries_line", "timeseries_bar", "timeseries_area", "timeseries_scatter",
-                       "bar", "pie", "histogram", "funnel", "treemap", "mixed")
+                       "bar", "pie", "histogram", "funnel", "treemap", "mixed", "box_plot")
 _REVERSE_HEADER_SIZE = {v: k for k, v in HEADER_SIZE.items()}
 _REVERSE_OPACITY = {"opacityLow": "low", "opacityMedium": "medium", "opacityHigh": "high"}
 
@@ -378,6 +382,49 @@ def _waterfall_to_spec(p: dict, out: dict, losses: list, name: str, metric_one)
     return read
 
 
+_WHISKER_SPEC = {v: k for k, v in WHISKER_OPTIONS.items()}
+_PERCENTILES = re.compile(r"^(\d+)/(\d+) percentiles$")  # the boxplot operator's own reading
+
+
+def _box_plot_to_spec(p: dict, out: dict, losses: list, name: str, metric_one) -> set[str] | None:
+    """A box plot's params into out; the params it read, or None when the chart can't be
+    a spec chart (an observation or group the spec can't name: a SQL column)."""
+    ms = [m for m in (metric_one(m) for m in (p.get("metrics") or [])) if m]
+    across, groupby = p.get("columns") or [], p.get("groupby") or []
+    if not ms or not across:
+        losses.append(Loss(name, "box plot needs metrics + distribute across; chart skipped"))
+        return None
+    if not all(isinstance(c, str) for c in [*across, *groupby]):
+        losses.append(Loss(name, "box plot over SQL columns the spec can't express; chart skipped"))
+        return None
+    out.update({"metrics": ms, "distribute_across": list(across)})
+    if groupby:
+        out["groupby"] = list(groupby)
+    whiskers = p.get("whiskerOptions") or "Tukey"
+    percentiles = _PERCENTILES.match(whiskers) if isinstance(whiskers, str) else None
+    if whiskers in _WHISKER_SPEC:
+        if whiskers != "Tukey":
+            out["whiskers"] = _WHISKER_SPEC[whiskers]
+    elif percentiles and int(percentiles.group(1)) < int(percentiles.group(2)) <= 100:
+        out["whiskers"] = [int(percentiles.group(1)), int(percentiles.group(2))]
+    else:
+        losses.append(Loss(name, f"whisker option {whiskers!r} not preserved (Tukey on re-apply)"))
+    if p.get("time_grain_sqla"):
+        out["time_grain"] = p["time_grain_sqla"]
+    if p.get("number_format") not in (None, "", "SMART_NUMBER"):
+        out["number_format"] = p["number_format"]
+    if p.get("date_format") not in (None, "", "smart_date"):
+        out["x_label_format"] = p["date_format"]
+    for key in ("x_axis_title", "y_axis_title"):
+        if isinstance(p.get(key), str) and p[key].strip():
+            out[key] = p[key]
+    _tick_layout_to_spec(p, out, losses, name)
+    if p.get("row_limit"):
+        out["row_limit"] = p["row_limit"]
+    return {"columns", "whiskerOptions", "date_format", "x_axis_title", "y_axis_title",
+            "x_ticks_layout"}
+
+
 def _annotations_to_spec(p: dict, out: dict, losses: list, name: str) -> None:
     """FORMULA annotation layers -> spec annotations; any other layer (events,
     intervals, other charts' series) is named as a loss."""
@@ -1015,6 +1062,10 @@ def keep_row_limit() -> None:
         waterfall_read = _waterfall_to_spec(p, out, losses, name, metric_one)
         if waterfall_read is None:
             return None
+    elif spec_type == "box_plot":
+        box_plot_read = _box_plot_to_spec(p, out, losses, name, metric_one)
+        if box_plot_read is None:
+            return None
 
     mapped_here = {"combineMetric", "conditional_formatting", "rowTotals", "colTotals", "colSubTotals"} if spec_type == "pivot_table" else (
         {"order_by_cols", "timeseries_limit_metric", "series_limit_metric",
@@ -1049,6 +1100,8 @@ def keep_row_limit() -> None:
         mapped_here = mapped_here | {"show_total"}
     if spec_type == "waterfall":
         mapped_here = mapped_here | waterfall_read
+    if spec_type == "box_plot":
+        mapped_here = mapped_here | box_plot_read
     if spec_type in _AXIS_TYPES:
         _x_labels_to_spec(p, out, losses, name)
         mapped_here = mapped_here | _axis_to_spec(p, out, losses, name, spec_type)

chartwright/design/brief.py

@@ -38,7 +38,7 @@ def render_brief(audience: str, overlay: Overlay | None = None) -> str:
         "",
         f"- Height budget per tab: {p.fold_units} units (~{p.fold_units * 40}px). Over budget -> tabs or prune.",
         f"- KPI band: {p.kpi_row_min}-{p.kpi_row_max} big numbers, {p.kpi_height} units tall, first band on the page.",
-        f"- Axis charts (timeseries/bar/heatmap/histogram/waterfall): >= {p.min_axis_height} units tall, 8 is the comfortable default.",
+        f"- Axis charts (timeseries/bar/heatmap/histogram/waterfall/box_plot): >= {p.min_axis_height} units tall, 8 is the comfortable default.",
         f"- At most {p.max_row_charts} axis charts per row; below 3/12 width a chart is unreadable.",
         "- Pie/donut: >= 5/12 wide, >= 8 tall. Heatmap: >= 5/12 wide (7/12 when many columns), >= 6 tall.",
         f"- Vertical bars: <= {p.vbar_max_categories} categories, then flip horizontal; an ordered axis "

chartwright/design/content.py

@@ -70,7 +70,7 @@
 NUMBER_FORMAT_TYPES = frozenset({
     "big_number_total", "big_number_trend", "timeseries_line", "timeseries_bar",
     "timeseries_area", "timeseries_scatter", "bar", "pie", "pivot_table", "heatmap", "funnel",
-    "treemap", "waterfall"})
+    "treemap", "waterfall", "box_plot"})
 
 _MARK = re.compile(r"/\* cw:(std|end) ([A-Za-z0-9][A-Za-z0-9_-]*)(?: ([0-9a-f]{12}))? \*/")
 

chartwright/design/guidelines/chart-choice.md

@@ -9,6 +9,7 @@
 | Several measures per item | `table` or `pivot_table`, not grouped bars. Pivots: <= 3 total dimensions; column-dim values x metrics <= ~15 rendered columns |
 | Two dimensions, one measure | `heatmap` (keep the grid under ~400 cells) |
 | Distribution of one column | `histogram`, 20-30 bins; trim long tails with a WHERE filter and say so in the title |
+| A distribution per group | `box_plot`: median, quartiles, whiskers and outliers of many observations each (daily sales by month) |
 | Staged conversion | `funnel`, 3-8 ordered stages |
 | Hierarchical share | `treemap`, at most 2 levels |
 | A bridge between two totals | `waterfall`: the opening, 3-12 steps that add up, the closing; `steps` and `closing` keep its order |
@@ -39,6 +40,13 @@
   in `steps` (6.1.0+). An opening total needs a dashboard theme that keeps zero
   on the axis (see `opening` in the schema); without `opening`, the first step
   rises from zero. Set the three colours and `show_value`.
+- Show the distribution, not the average, when the spread is the story
+  (delivery times, daily sales, latency): an average hides skew and outliers.
+  A `box_plot` beats a line of P50 and P90 across categories or a few periods,
+  and whenever the outliers matter; keep the line for a trend over many
+  periods. Give each box many observations: `distribute_across` a time column
+  at P1D, `groupby` the month. Superset colours each box by its group: pin one
+  colour to the group labels in `dashboard.label_colors`.
 - One dominant category flattening its siblings is the data talking: note it
   or filter it, don't hide it.
 - Every chart with a time axis should tolerate the dashboard time filter;

chartwright/design/model.py

@@ -29,12 +29,14 @@
 # "9" = the waterfall: chart.waterfall-additive and chart.waterfall-steps (warn) can
 # newly block a strict gate, chart.waterfall-order, chart.waterfall-colors and
 # chart.waterfall-axis-titles are new findings, and default.value-labels and
-# default.count-format now fill a waterfall's show_value and number_format.
+# default.count-format now fill a waterfall's show_value and number_format. The box
+# plot: chart.box-plot-observations and chart.box-plot-groups (warn), and
+# chart.ordinal-order now reads a box plot's groups.
 DESIGN_BRAIN_VERSION = "9"
 
 KPI_TYPES = {"big_number_total", "big_number_trend"}
 TIMESERIES_TYPES = {"timeseries_line", "timeseries_bar", "timeseries_area", "timeseries_scatter"}
-AXIS_TYPES = TIMESERIES_TYPES | {"bar", "heatmap", "histogram", "mixed", "waterfall"}
+AXIS_TYPES = TIMESERIES_TYPES | {"bar", "heatmap", "histogram", "mixed", "waterfall", "box_plot"}
 # Charts that are neither KPI nor axis-bearing; a contract test asserts the
 # three sets exactly cover CHART_TYPES, so a new chart type fails CI until
 # someone consciously classifies it (and reviews which rules apply).