Skip to content

fix(export): preserve legacy non-object event data in CSV - #208

Merged
ErikBjare merged 2 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/csv-legacy-nonobject-data
Nov 17, 2026
Merged

ErikBjare merged 2 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/csv-legacy-nonobject-data

Conversation

@TimeToBuildBob

Copy link
Copy Markdown
Contributor

Problem

Previously accepted list/scalar event payloads break the entire CSV stream. A list payload produces HTTP 200 headers and then AttributeError: list has no get while consuming the body; numeric/boolean payloads fail during key discovery.

Change

  • Select the existing JSON data column before key discovery whenever any exported event has non-object data.
  • Preserve every row and payload without rewriting stored events or changing ingestion policy. Object-only exports retain their existing columns.
  • Add 12 regressions, including full Flask streamed-body consumption, legacy falsey payloads, explicit aw-core null normalization, object-only header collisions/formula protection/Unicode/precision and wide-object fallback. Include them in make test.
  • Separately retain Python 3.9 on Ubuntu 22.04, matching the bounded runner repair in fix(api): reject nonfinite heartbeat pulse windows #207; ubuntu-latest now selects unsupported Ubuntu 26.04.

Verification

  • RED on master 994ea98: list-data export fails during response.get_data with AttributeError.
  • 42 focused tests passed; package-native make test: 156 passed.
  • mypy: all 32 source files passed; changed-path Black/Ruff and git diff checks passed. csv_export.py covered 100%.
  • No live server, personal data, migrations or ingestion-policy changes.

Coordination

#201 changes exact JSON archives and #173 changes performance; neither edits csv_export.py. Their Makefile test-list edits may need reconciliation. Maintainer merge and regression replay on master remain outstanding.

@greptile-apps

greptile-apps Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The behavior change is limited to bucket CSV downloads.

Fix the large-integer export failure before merging.

Findings

  1. P1 Large integers stop CSV downloads ▶

Summary

The PR preserves legacy non-object event data by choosing a JSON data column before reading object keys.

  • Bucket CSV exports keep every event when payloads are not objects.
  • Finite JSON numbers stay unchanged in CSV cells.

Reviews (3) · Last reviewed commit: "fix(export): keep numeric JSON cells un-..." · Reviewed by Greptile

@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

CI-green and mergeable — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.

@TimeToBuildBob

TimeToBuildBob commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 AI code review

The PR changes events_to_csv() to select the JSON data column before discovering keys whenever an event has non-dictionary data. It adds CSV export tests for legacy payloads and object-only behavior, and includes the new test file in make test.

Safe to merge — no P0/P1 findings

Confidence 5/5

✅ No findings. The diff looks correct to me on this pass.

Files changed (3) — the diff as I read it
  • Makefile — Adds tests/test_csv_export.py to the pytest command in the test target.
  • aw_server/csv_export.py — Detects non-dictionary event data before key discovery and selects the JSON fallback for the entire export.
  • tests/test_csv_export.py — Adds streamed export and exporter tests for legacy scalar/list payloads, falsey values, null normalization, object columns, and wide-object fallback.
Previous review passes
commit score findings engine when
d7eba98a3e2d 5/5 0 subscription 2026-10-28 17:03 UTC

Reviewed 5faf152d821f · panel: codex:gpt-6.1-sol + claude-code:sonnet · subscription engine · 21s · about this reviewer

Maintainer commands

@TimeToBuildBob review (own line) — fresh review · @TimeToBuildBob fix — a worker acts on the findings. Once per comment; 👀 = received.

Use the existing JSON-column fallback before key discovery whenever a stored payload is not an object. Preserve every row without changing ingestion or historical data; cover streamed Flask responses and object-only compatibility.

Git-Session-Id: 01c28659-e76d-44c6-aa3a-ef006c0b6f44
@TimeToBuildBob
TimeToBuildBob force-pushed the fix/csv-legacy-nonobject-data branch from d7eba98 to 5faf152 Compare November 6, 2026 15:33
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

Rebased onto master 37f501d and pushed 5faf152 with an explicit force-with-lease. The conflict was only in the old runner-pin commit: I dropped that now-superseded commit and retained master's Ubuntu 24.04 / Python 3.9 settings and actions/checkout@v7. The export fix and all 12 CSV regressions are unchanged (git range-diff confirms the implementation commit is equivalent).

Verification after rebase: native make test 156 passed, including all 12 CSV cases; csv_export.py coverage 100%; mypy passed all 32 source files; changed-path Black and git diff --check passed. GitHub now reports the PR mergeable; fresh CI is running, not yet all green. The earlier Ubuntu 22.04 bullet in the description is superseded by this rebase.

@TimeToBuildBob

This comment has been minimized.

Comment thread aw_server/csv_export.py Resolved
Formula neutralization prefixed ' to negative numbers such as -5, so a scalar data cell no longer round-tripped through json.loads. Finite JSON numbers are now exported unchanged; strings that look like formulas are still neutralized.

Git-Session-Id: 7044b809-3df1-480f-88a9-e5e27e03138f
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread aw_server/csv_export.py
text = json.dumps(value, separators=(",", ":"), ensure_ascii=False)
# Keep -5 / -1.5 as valid JSON numbers; a leading "'" would break json.loads.
is_number = isinstance(value, (int, float)) and not isinstance(value, bool)
if is_number and math.isfinite(value):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Large integers stop CSV downloads

_json now calls math.isfinite on integers, which raises OverflowError for valid JSON integers such as 10**400. An event with data={"n": 10**400} therefore stops a CSV export after the header, even though json.dumps can preserve the value.

Treat integers as finite directly and call math.isfinite only for floats. Add a large-integer regression.

Suggested change
if is_number and math.isfinite(value):
if is_number and (isinstance(value, int) or math.isfinite(value)):

@ErikBjare
ErikBjare merged commit beefc8d into ActivityWatch:master Nov 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants