Repository navigation
Acknowledge spooled segments once decoded - #645
Open
aminghadersohi wants to merge 1 commit into
Open
aminghadersohi wants to merge 1 commit into
aminghadersohi wants to merge 1 commit into
Conversation
SegmentIterator acknowledged a spooled segment only when the caller asked for the row after its last row. A read that stops at the last row, such as fetchone() or SQLAlchemy's scalar()/first()/one() on a single-row result, never acknowledged the final segment, so the server kept it in the spooling storage until its expiry pruning. Acknowledge each spooled segment as soon as its rows are decoded. A segment whose download or decode fails is still not acknowledged and is retried.
|
Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #644.
SegmentIteratoracknowledged a spooled segment only when the caller asked for the row after its last row. Reads that stop at the last row never acknowledged the final segment, so the server kept it in spooling storage until expiry pruning. That coversfetchone()and SQLAlchemy'sscalar()/first()/one()on a single-row result.This change acknowledges each spooled segment as soon as its rows are decoded. The rows are already in memory at that point. A segment whose download or decode fails is still not acknowledged and is retried, so
test_segment_iterator_retries_failed_segment_without_skipping_itis unchanged and still passes.Tests:
test_segment_iterator_acknowledges_last_segment_without_reading_past_it: fails on master, passes with this change.tests/unit/test_client_spooling.py: 24 passed..scalar()reads.Non-technical explanation
With the spooling protocol, reading a result without asking for a row past its end (for example
fetchone()or SQLAlchemyscalar()on a single-row result) left the last result segment in the server's spooling storage until it expired. The client now releases each segment as soon as it has read it.Release notes
( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text: