Repository navigation
Conversation
minio-go v7.2.1 copies the object in FGetObject and never closes the body. Go 1.27 does not finalize response bodies, so every successful Bucket download keeps that connection until exit. Download with GetObject and close the reader. Keep the etag match. Write to a hashed part file, the same scheme minio-go uses, and rename it into place only after the full copy and a successful file close. A failed copy removes the part file. Assisted-by: Grok/grok-4.7 Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
Member
|
Thanks for tracking this down. Since #2203 bumped minio-go to v7.4.1, Client.FGetObject upstream already closes the reader (and adds checksum verification), this fix is no longer needed. |
Contributor
Author
Yes. Closing this pull. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
S3-compatible Bucket downloads use minio-go v7.2.1
Client.FGetObject. That method copies the GET body and never closes it.getObjectsays the caller must close. Go 1.27 does not finalize response bodies, so each successful download keeps the connection until the process exits.MinioClient.FGetObjectstill stats the object and setsIf-Matchfrom that etag. It then downloads withGetObjectand closes the reader. Bytes are written to a part file whose name is the sha256 of the base name and etag, the same scheme minio-go uses, so a parallel fetch of another key ending in.part.miniodoes not share that path. The part file is renamed into place only after the copy matches the stat size and the file close succeeds. A failed copy deletes the part file and leaves the destination untouched. A reader close error is logged and does not replace the download error. A writable-file close error is returned on the success path and logged when the copy already failed.minio-go stays at v7.2.1. #2183 moved it from v7.2.0 to v7.2.1 for region
ap-southeast-6. The upstream close arrived in minio-go #2272 together with checksum verification, released in v7.3.0. This patch does not take that release.GCP (#2053) and Azure (#2200) already close their download bodies the same way.
The call has been in the Bucket provider since March 2022 (
ed6c6ebc).Tests
TestMinioClient_FGetObject_ClosesResponseBodyfetches 120000 bytes, checks etagabc123andIf-Match: "abc123", writes a nested path, and requires the GET body to close. On the previous code the body was still open after 2 seconds.TestMinioClient_FGetObject_CopyErrorserves 5 bytes withContent-Length: 64. The call fails, the destination is absent, no*.part.miniofile remains, and the GET body closes. On the previous code the body stayed open.go test -race ./internal/bucket/minio/passed.make tidy fmt vet && make testpassed.