Skip to content

Acknowledge spooled segments once decoded - #645

Open
aminghadersohi wants to merge 1 commit into
trinodb:masterfrom
aminghadersohi:ack-decoded-spooled-segments
Open

aminghadersohi wants to merge 1 commit into
trinodb:masterfrom
aminghadersohi:ack-decoded-spooled-segments

Conversation

@aminghadersohi

@aminghadersohi aminghadersohi commented Sep 26, 2026 •

Copy link
Copy Markdown

Description

Fixes #644.

SegmentIterator acknowledged a spooled segment only when the caller asked for the row after its last row. Reads that stop at the last row never acknowledged the final segment, so the server kept it in spooling storage until expiry pruning. That covers fetchone() and SQLAlchemy's scalar()/first()/one() on a single-row result.

This change acknowledges each spooled segment as soon as its rows are decoded. The rows are already in memory at that point. A segment whose download or decode fails is still not acknowledged and is retried, so test_segment_iterator_retries_failed_segment_without_skipping_it is unchanged and still passes.

Tests:

  • New test_segment_iterator_acknowledges_last_segment_without_reading_past_it: fails on master, passes with this change.
  • tests/unit/test_client_spooling.py: 24 passed.
  • Verified live against Trino 483 with spooling on AWS S3: no segment objects remain after .scalar() reads.

Non-technical explanation

With the spooling protocol, reading a result without asking for a row past its end (for example fetchone() or SQLAlchemy scalar() on a single-row result) left the last result segment in the server's spooling storage until it expired. The client now releases each segment as soon as it has read it.

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

* Fix the last spooled result segment not being acknowledged when reading
  stops at the final row, for example with `fetchone()` or SQLAlchemy
  `scalar()` on a single-row result, leaving it in spooling storage until
  it expires.
  ({issue}`644`)

SegmentIterator acknowledged a spooled segment only when the caller asked
for the row after its last row. A read that stops at the last row, such as
fetchone() or SQLAlchemy's scalar()/first()/one() on a single-row result,
never acknowledged the final segment, so the server kept it in the spooling
storage until its expiry pruning.

Acknowledge each spooled segment as soon as its rows are decoded. A segment
whose download or decode fails is still not acknowledged and is retried.
@cla-bot

cla-bot Bot commented Sep 26, 2026

Copy link
Copy Markdown

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to cla@trino.io. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Final spooled segment is not acknowledged when reading stops at the last row

1 participant