Repository navigation
feat(operator): cut a numeric column into bins - #8481
Conversation
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8481 +/- ##
=========================================
Coverage 91.55% 91.56%
- Complexity 5112 5132 +20
=========================================
Files 1291 1293 +2
Lines 57359 57414 +55
Branches 7502 7506 +4
=========================================
+ Hits 52516 52570 +54
- Misses 2933 2934 +1
Partials 1910 1910
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 490 | 0.299 | 20,044/27,087/27,087 us | 🔴 +7.8% / 🔴 +102.5% |
| 🟢 | bs=100 sw=10 sl=64 | 1,050 | 0.641 | 94,406/106,226/106,226 us | 🟢 -18.6% / 🔴 -25.5% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,183 | 0.722 | 846,810/869,732/869,732 us | ⚪ within ±5% / 🔴 -18.8% |
Baseline details
Latest main 2e98e44 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 490 tuples/sec | 517 tuples/sec | 1,046 tuples/sec | -5.2% | -53.2% |
| bs=10 sw=10 sl=64 | MB/s | 0.299 MB/s | 0.316 MB/s | 0.638 MB/s | -5.4% | -53.2% |
| bs=10 sw=10 sl=64 | p50 | 20,044 us | 18,587 us | 10,039 us | +7.8% | +99.7% |
| bs=10 sw=10 sl=64 | p95 | 27,087 us | 28,360 us | 13,374 us | -4.5% | +102.5% |
| bs=10 sw=10 sl=64 | p99 | 27,087 us | 28,360 us | 15,946 us | -4.5% | +69.9% |
| bs=100 sw=10 sl=64 | throughput | 1,050 tuples/sec | 1,044 tuples/sec | 1,408 tuples/sec | +0.6% | -25.5% |
| bs=100 sw=10 sl=64 | MB/s | 0.641 MB/s | 0.637 MB/s | 0.86 MB/s | +0.6% | -25.4% |
| bs=100 sw=10 sl=64 | p50 | 94,406 us | 90,129 us | 78,599 us | +4.7% | +20.1% |
| bs=100 sw=10 sl=64 | p95 | 106,226 us | 130,559 us | 86,338 us | -18.6% | +23.0% |
| bs=100 sw=10 sl=64 | p99 | 106,226 us | 130,559 us | 108,934 us | -18.6% | -2.5% |
| bs=1000 sw=10 sl=64 | throughput | 1,183 tuples/sec | 1,215 tuples/sec | 1,457 tuples/sec | -2.6% | -18.8% |
| bs=1000 sw=10 sl=64 | MB/s | 0.722 MB/s | 0.742 MB/s | 0.889 MB/s | -2.7% | -18.8% |
| bs=1000 sw=10 sl=64 | p50 | 846,810 us | 821,794 us | 766,492 us | +3.0% | +10.5% |
| bs=1000 sw=10 sl=64 | p95 | 869,732 us | 851,035 us | 804,229 us | +2.2% | +8.1% |
| bs=1000 sw=10 sl=64 | p99 | 869,732 us | 851,035 us | 850,445 us | +2.2% | +2.3% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,408.46,200,128000,490,0.299,20043.86,27086.57,27086.57
1,100,10,64,20,1904.97,2000,1280000,1050,0.641,94406.12,106226.09,106226.09
2,1000,10,64,20,16913.18,20000,12800000,1183,0.722,846810.44,869731.88,869731.88
carloea2
left a comment
There was a problem hiding this comment.
Additional boundary cases reproduced with compiled local tests.
The same gap as the timestamp parts, on the other kind of column. Grouping by a continuous number puts almost every row in a group of its own, so counts per age bracket or per price band could not be asked for; a user wanting them had to write a Python UDF. Two cuts, because they answer different questions. Equal width divides the span into bins of one size; equal frequency cuts at the quantiles, so the bins hold about as many rows as each other. The second is why the operator reads the whole table before it emits: a quantile is not known until the last row has arrived. The bin reads as its own range, "(2.5, 5.0]", which is a label to group by rather than a number to do arithmetic on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An equal-width cut has no edges to find when every cell of the column is empty, and none when no row arrived at all, so pandas raises before the suffix that keeps a hole a hole can run. Both renderings now skip the cut in that case and let the suffix turn the holes into the empty bin labels they already are. The equal-frequency cut does not raise on the same table, but it goes through the same guard: one shape of answer for a table with nothing in it, whichever way it was asked to cut. Two tests run the exported script over an all-empty column, an empty table and a column with one hole, since the defect is what pandas does at runtime rather than what the operator writes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@carloea2 the finding is fixed: a column with nothing in it skips the cut, so the rows keep null bin labels. Would you take another look when you have a moment? |
carloea2
left a comment
There was a problem hiding this comment.
The all missing column case is handled now. Looks good.
…-integration Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
carloea2
left a comment
There was a problem hiding this comment.
All 13 existing tests and formatting checks pass. A new parity test using the real table-operator lifecycle fails on stored NaNs in nullable Arrow input; the mixed-value control passes.
A nullable Float64 column read from Arrow can hold NaN as a value, which notna() reports present, so an all-NaN column reached pd.cut and raised on duplicate edges while the engine path left it uncut. The guard now tests a float64 view. Adds a stored-NaN case to the runtime test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
left a comment
There was a problem hiding this comment.
The stored NaN fix passes. All 14 focused tests pass, including my original reproduction run through the real UDF lifecycle and generated export for both equal-width and equal-frequency binning. Mixed finite values agree; all stored NaNs produce empty bins. The UDF probe uses the #8766 SDK. Formatting passes. No new finding in this update.
sbt -batch 'WorkflowOperator/testOnly *BinningOpDescSpec -- -oDF' 'scalafmtCheckAll'
Python's open() defaults to the locale's encoding, which is cp1252 on Windows, while the spec writes these files as UTF-8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
left a comment
There was a problem hiding this comment.
All 13 focused tests and formatting checks pass on a fresh detached checkout. The binning implementation is unchanged from the stored-NaN fix verified earlier; this update fixes the test driver's encoding. The remaining CI failure is LoopIntegrationSpec, covered by #8938. Looks good. Leaving the draft status unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
left a comment
There was a problem hiding this comment.
Binning fails on valid LONG values.
pandas builds edges in float64, which cannot count by one past 2**53, so an equal-width cut raised on duplicate edges and a quantile cut merged the rows into one bin. An integer column is now cut as its distance from the value nearest zero, and that value is added back to the labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cutting every integer column as a distance changed the labels pandas rounds, so 723.062 could read 723.0625. The shifted cut is now used only where the plain one raises or loses a bin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@carloea2 Large LONG values now keep separate bins in both paths, and ordinary columns keep pandas' labels. Could you take another look? |
carloea2
left a comment
There was a problem hiding this comment.
Equal-width binning still loses precision at the Int64 limits. The branch also conflicts with current main.
…e same bin Comparing category counts missed Int64 values near the limit, where the plain cut keeps all bins but puts every row in one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala
|
@carloea2 may you take another look? |
carloea2
left a comment
There was a problem hiding this comment.
Retested both generated paths at the positive and negative Int64 limits. The 52 Python cases and 13 Scala tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
carloea2
left a comment
There was a problem hiding this comment.
All 13 BinningOpDescSpec tests pass locally. The macOS frontend job failed on the registration modal test timing out after 20 seconds; 7,235 tests passed. @mengw15, could you rerun that job? https://github.com/apache/texera/actions/runs/42942174175/job/130266447647
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The new run after ccf3205 is all green. |
# Conflicts: # common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala
|
@aglinxinyuan The macOS frontend job failed on |
What changes were proposed in this PR?
An operator that cuts a numeric column into bins and adds the bin a row fell in as a column of its own.
Two cuts, because they answer different questions. Equal width divides the span into bins of one size; equal frequency cuts at the quantiles, so the bins hold about as many rows as each other. The second is why the operator reads the whole table before it emits: a quantile is not known until the last row has arrived.
The bin reads as its own range,
(2.5, 5.0], which is a label to group by rather than a number to do arithmetic on.Any related issues, documentation, discussions?
Not part of #8325: that set makes the operators Texera already had exportable. This adds a new one, which implements the trait from the start rather than gaining it. The trait is in #8327, which has merged.
Closes #8479, the task this change is the whole of.
The same gap as #8478, on the other kind of column. Grouping by a continuous number puts almost every row in a group of its own, so counts per age bracket or per price band could not be asked for; a user wanting them had to write a Python UDF.
How was this PR tested?
Ten unit tests in the operator's own spec, over the schema it declares and the Python it emits on both paths. The parity harness runs it on four configurations, one per cut plus a hostile column name and an empty cell, comparing the engine's answer against the exported script's.
The operator in the editor, cutting
scoreinto four equal-width bins:Each row's bin beside the value it came from, named by the range it fell in rather than by a number:
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)