Skip to content

feat(pyamber): hash large binary tuples - #8955

Open
carloea2 wants to merge 1 commit into
apache:mainfrom
carloea2:codex/feature-large-binary-hash
Open

carloea2 wants to merge 1 commit into
apache:mainfrom
carloea2:codex/feature-large-binary-hash

Conversation

@carloea2

@carloea2 carloea2 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Hash LARGE_BINARY URI keys using Java-compatible UTF-16 code units. This supports tuple hashing and shuffle routing for ordinary and supplementary Unicode URIs, including null fields.

Rebased on upstream main at af58667, preserving the signed byte hashing from #8250.

Any related issues, documentation, discussions?

Closes #8185
Closes #8243

Replaces #8186 and consolidates #8244.

How was this PR tested?

The five LARGE_BINARY regressions fail on main with KeyError and pass with this change. From amber after running bash bin/python-proto-gen.sh from the repo root:

python -m pytest -q -m 'not integration' src/test/python/core/models src/test/python/core/architecture/sendsemantics -k 'not (TestTuple and test_hash) or large_binary or test_hash_binary_with_high_byte_matches_java or test_hash_treats_none_fields_as_zero'
python -m ruff check src/main/python src/test/python
python -m ruff format --check src/main/python src/test/python

386 passed, 9 skipped, and 1 expected failure on Windows. The excluded timestamp test also fails on main and is tracked in #8957.

Previous frontend verification on the earlier commit:

corepack yarn install --immutable
corepack yarn test:ci
corepack yarn format:ci

7,020 tests passed across 248 files, with one skip. Formatting and lint passed. A maintainer can add the frontend label to run that coverage in CI.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Codex

@carloea2

Copy link
Copy Markdown
Contributor Author

This replaces stale-closed #8186. The problem still reproduces on current main. The patch passes its regression, 349 broader tests and Ruff checks.

/request-review @eugenegujing

@carloea2

Copy link
Copy Markdown
Contributor Author

/request-review @eugenegujing

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @Ma77Ball
    You can notify them by mentioning @Ma77Ball in a comment.

@codecov-commenter

codecov-commenter commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.57%. Comparing base (af58667) to head (90173c3).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #8955   +/-   ##
=========================================
  Coverage     91.56%   91.57%           
  Complexity     5132     5132           
=========================================
  Files          1293     1293           
  Lines         57423    57426    +3     
  Branches       7506     7506           
=========================================
+ Hits          52582    52585    +3     
  Misses         2933     2933           
  Partials       1908     1908           
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø) Carriedforward from af58667
agent-service 99.56% <ø> (ø) Carriedforward from af58667
amber 85.31% <ø> (ø) Carriedforward from af58667
computing-unit-managing-service 64.02% <ø> (ø) Carriedforward from af58667
config-service 87.50% <ø> (ø) Carriedforward from af58667
file-service 82.16% <ø> (ø) Carriedforward from af58667
frontend 96.43% <ø> (ø) Carriedforward from af58667
notebook-migration-service 83.77% <ø> (ø) Carriedforward from af58667
pyamber 98.50% <100.00%> (+<0.01%) ⬆️
workflow-compiling-service 78.30% <ø> (ø) Carriedforward from af58667

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 6 better · 🔴 3 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main af58667 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 531 0.324 18,921/24,003/24,003 us 🟢 -7.7% / 🔴 +98.2%
🟢 bs=100 sw=10 sl=64 1,274 0.778 75,370/100,262/100,262 us 🟢 -15.1% / 🔴 +22.6%
🔴 bs=1000 sw=10 sl=64 1,467 0.896 684,977/746,286/746,286 us 🔴 +6.3% / 🟢 -7.7%
Baseline details

Latest main af58667 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 531 tuples/sec 547 tuples/sec 1,084 tuples/sec -2.9% -51.0%
bs=10 sw=10 sl=64 MB/s 0.324 MB/s 0.334 MB/s 0.662 MB/s -3.0% -51.0%
bs=10 sw=10 sl=64 p50 18,921 us 17,923 us 9,548 us +5.6% +98.2%
bs=10 sw=10 sl=64 p95 24,003 us 26,011 us 12,811 us -7.7% +87.4%
bs=10 sw=10 sl=64 p99 24,003 us 26,011 us 15,432 us -7.7% +55.5%
bs=100 sw=10 sl=64 throughput 1,274 tuples/sec 1,202 tuples/sec 1,467 tuples/sec +6.0% -13.2%
bs=100 sw=10 sl=64 MB/s 0.778 MB/s 0.733 MB/s 0.895 MB/s +6.1% -13.1%
bs=100 sw=10 sl=64 p50 75,370 us 77,894 us 74,441 us -3.2% +1.2%
bs=100 sw=10 sl=64 p95 100,262 us 118,115 us 81,800 us -15.1% +22.6%
bs=100 sw=10 sl=64 p99 100,262 us 118,115 us 103,712 us -15.1% -3.3%
bs=1000 sw=10 sl=64 throughput 1,467 tuples/sec 1,474 tuples/sec 1,517 tuples/sec -0.5% -3.3%
bs=1000 sw=10 sl=64 MB/s 0.896 MB/s 0.9 MB/s 0.926 MB/s -0.4% -3.2%
bs=1000 sw=10 sl=64 p50 684,977 us 677,196 us 725,911 us +1.1% -5.6%
bs=1000 sw=10 sl=64 p95 746,286 us 702,159 us 764,152 us +6.3% -2.3%
bs=1000 sw=10 sl=64 p99 746,286 us 702,159 us 808,901 us +6.3% -7.7%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,376.35,200,128000,531,0.324,18921.44,24002.85,24002.85
1,100,10,64,20,1569.74,2000,1280000,1274,0.778,75370.01,100262.30,100262.30
2,1000,10,64,20,13630.68,20000,12800000,1467,0.896,684976.95,746285.94,746285.94

@carloea2
carloea2 force-pushed the codex/feature-large-binary-hash branch from 95fdbdc to 2743266 Compare October 15, 2026 07:08
@carloea2

Copy link
Copy Markdown
Contributor Author

/request-review @mengw15

Could you review the large binary hash routing? The regression and broader tests pass locally, and CI is green.

@github-actions
github-actions Bot requested a review from mengw15 October 15, 2026 22:42
@carloea2

Copy link
Copy Markdown
Contributor Author

/request-review @sshiv012 @parshimers

@eugenegujing eugenegujing left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One small thing: the test only checks Tuple.hash, but the KeyError actually came out of HashBasedShufflePartitioner._route. There's already test_same_key_routes_to_same_receiver_deterministically in test_partitioners.py using an INTEGER key, might be worth copying it with a LARGE_BINARY key so the partitioner path is covered too.

@carloea2

Copy link
Copy Markdown
Contributor Author

@eugenegujing Added the partitioner-level LARGE_BINARY test you suggested, including a non-BMP URI. It reproduces the original KeyError. The focused suites give 144 passes; the existing Windows epoch-hashing test remains separate in #8957. Ruff passes, and CI has restarted.

@github-actions github-actions Bot added engine dependencies Pull requests that update a dependency file ddl-change Changes to the TexeraDB DDL frontend Changes related to the frontend GUI ci changes related to CI docs Changes related to documentations infra common platform Non-amber Scala service paths agent-service pyright-language-service Changes under pyright-language-service/ (Pyright LSP bridge) labels Oct 30, 2026
@carloea2
carloea2 force-pushed the codex/feature-large-binary-hash branch from 0f4e58d to 0285b24 Compare November 4, 2026 02:52
@github-actions github-actions Bot removed engine dependencies Pull requests that update a dependency file ddl-change Changes to the TexeraDB DDL labels Nov 4, 2026
@github-actions github-actions Bot removed frontend Changes related to the frontend GUI ci changes related to CI docs Changes related to documentations infra common platform Non-amber Scala service paths agent-service pyright-language-service Changes under pyright-language-service/ (Pyright LSP bridge) labels Nov 4, 2026
@carloea2

carloea2 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

@mengw15, could you add the frontend label so those CI checks run too? The full frontend suite passes locally, but I don't have permission to add the label.

@carloea2

carloea2 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

/request-review @mengw15

Refreshed onto main with partitioner coverage. All required checks are green. Could you review this head?

@kunwp1
kunwp1 self-requested a review November 16, 2026 21:55
@kunwp1

kunwp1 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

@carloea2 Can you resolve the merge conflict? Once it's resolved, I'll have a pass on this PR.

@carloea2
carloea2 force-pushed the codex/feature-large-binary-hash branch from 0285b24 to 90173c3 Compare November 18, 2026 23:23
@carloea2

Copy link
Copy Markdown
Contributor Author

@kunwp1 The conflict with main is resolved. The large binary routing tests pass locally. Could you take a look?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hash partitioning crashes on LARGE_BINARY keys Hash shuffle crashes on large binary fields

4 participants