Repository navigation
Conversation
|
This replaces stale-closed #8186. The problem still reproduces on current main. The patch passes its regression, 349 broader tests and Ruff checks. /request-review @eugenegujing |
|
/request-review @eugenegujing |
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8955 +/- ##
=========================================
Coverage 91.56% 91.57%
Complexity 5132 5132
=========================================
Files 1293 1293
Lines 57423 57426 +3
Branches 7506 7506
=========================================
+ Hits 52582 52585 +3
Misses 2933 2933
Partials 1908 1908
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 531 | 0.324 | 18,921/24,003/24,003 us | 🟢 -7.7% / 🔴 +98.2% |
| 🟢 | bs=100 sw=10 sl=64 | 1,274 | 0.778 | 75,370/100,262/100,262 us | 🟢 -15.1% / 🔴 +22.6% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,467 | 0.896 | 684,977/746,286/746,286 us | 🔴 +6.3% / 🟢 -7.7% |
Baseline details
Latest main af58667 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 531 tuples/sec | 547 tuples/sec | 1,084 tuples/sec | -2.9% | -51.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.324 MB/s | 0.334 MB/s | 0.662 MB/s | -3.0% | -51.0% |
| bs=10 sw=10 sl=64 | p50 | 18,921 us | 17,923 us | 9,548 us | +5.6% | +98.2% |
| bs=10 sw=10 sl=64 | p95 | 24,003 us | 26,011 us | 12,811 us | -7.7% | +87.4% |
| bs=10 sw=10 sl=64 | p99 | 24,003 us | 26,011 us | 15,432 us | -7.7% | +55.5% |
| bs=100 sw=10 sl=64 | throughput | 1,274 tuples/sec | 1,202 tuples/sec | 1,467 tuples/sec | +6.0% | -13.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.778 MB/s | 0.733 MB/s | 0.895 MB/s | +6.1% | -13.1% |
| bs=100 sw=10 sl=64 | p50 | 75,370 us | 77,894 us | 74,441 us | -3.2% | +1.2% |
| bs=100 sw=10 sl=64 | p95 | 100,262 us | 118,115 us | 81,800 us | -15.1% | +22.6% |
| bs=100 sw=10 sl=64 | p99 | 100,262 us | 118,115 us | 103,712 us | -15.1% | -3.3% |
| bs=1000 sw=10 sl=64 | throughput | 1,467 tuples/sec | 1,474 tuples/sec | 1,517 tuples/sec | -0.5% | -3.3% |
| bs=1000 sw=10 sl=64 | MB/s | 0.896 MB/s | 0.9 MB/s | 0.926 MB/s | -0.4% | -3.2% |
| bs=1000 sw=10 sl=64 | p50 | 684,977 us | 677,196 us | 725,911 us | +1.1% | -5.6% |
| bs=1000 sw=10 sl=64 | p95 | 746,286 us | 702,159 us | 764,152 us | +6.3% | -2.3% |
| bs=1000 sw=10 sl=64 | p99 | 746,286 us | 702,159 us | 808,901 us | +6.3% | -7.7% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,376.35,200,128000,531,0.324,18921.44,24002.85,24002.85
1,100,10,64,20,1569.74,2000,1280000,1274,0.778,75370.01,100262.30,100262.30
2,1000,10,64,20,13630.68,20000,12800000,1467,0.896,684976.95,746285.94,746285.9495fdbdc to
2743266
Compare
|
/request-review @mengw15 Could you review the large binary hash routing? The regression and broader tests pass locally, and CI is green. |
|
/request-review @sshiv012 @parshimers |
eugenegujing
left a comment
There was a problem hiding this comment.
LGTM. One small thing: the test only checks Tuple.hash, but the KeyError actually came out of HashBasedShufflePartitioner._route. There's already test_same_key_routes_to_same_receiver_deterministically in test_partitioners.py using an INTEGER key, might be worth copying it with a LARGE_BINARY key so the partitioner path is covered too.
|
@eugenegujing Added the partitioner-level LARGE_BINARY test you suggested, including a non-BMP URI. It reproduces the original KeyError. The focused suites give 144 passes; the existing Windows epoch-hashing test remains separate in #8957. Ruff passes, and CI has restarted. |
0f4e58d to
0285b24
Compare
|
@mengw15, could you add the frontend label so those CI checks run too? The full frontend suite passes locally, but I don't have permission to add the label. |
|
/request-review @mengw15 Refreshed onto main with partitioner coverage. All required checks are green. Could you review this head? |
|
@carloea2 Can you resolve the merge conflict? Once it's resolved, I'll have a pass on this PR. |
0285b24 to
90173c3
Compare
|
@kunwp1 The conflict with main is resolved. The large binary routing tests pass locally. Could you take a look? |
What changes were proposed in this PR?
Hash LARGE_BINARY URI keys using Java-compatible UTF-16 code units. This supports tuple hashing and shuffle routing for ordinary and supplementary Unicode URIs, including null fields.
Rebased on upstream main at af58667, preserving the signed byte hashing from #8250.
Any related issues, documentation, discussions?
Closes #8185
Closes #8243
Replaces #8186 and consolidates #8244.
How was this PR tested?
The five LARGE_BINARY regressions fail on main with KeyError and pass with this change. From
amberafter runningbash bin/python-proto-gen.shfrom the repo root:386 passed, 9 skipped, and 1 expected failure on Windows. The excluded timestamp test also fails on main and is tracked in #8957.
Previous frontend verification on the earlier commit:
7,020 tests passed across 248 files, with one skip. Formatting and lint passed. A maintainer can add the frontend label to run that coverage in CI.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Codex