Skip to content

Terminate GET forwarding after SplitInfo - #4256

Closed
roman-khimov with Copilot wants to merge 3 commits into
masterfrom
copilot/test-raw-object-get-head-replacement-complex-objec
Closed

roman-khimov with Copilot wants to merge 3 commits into
masterfrom
copilot/test-raw-object-get-head-replacement-complex-objec

Conversation

Copilot AI commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Raw range GETs for complex objects could fail with SDK rc24 because forwarding continued after a SplitInfo response. SplitInfo must be the final message in the response stream.

  • Stream handling
    • Treat SplitInfo as terminal in both GET forwarding paths.
    • Forward SplitInfo, then stop reading downstream messages.
  • Regression coverage
    • Verify a raw 0:10 range forwards SplitInfo but not a trailing chunk.
SplitInfo → forward → end stream

Co-authored-by: roman-khimov <22092804+roman-khimov@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix test_raw_object_get_head failure for complex object Terminate GET forwarding after SplitInfo Oct 28, 2026
Copilot AI requested a review from roman-khimov October 28, 2026 12:21

@roman-khimov roman-khimov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This touches forwarding code, but it means another SN is sending split info and some data after it. It shouldn't be doing that, please fix it instead. @copilot.

Copilot AI commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

This touches forwarding code, but it means another SN is sending split info and some data after it. It shouldn't be doing that, please fix it instead. @copilot.

I traced the response paths. A locally generated SplitInfo is sent by sendStatusGetResponse and immediately returned; that path cannot append payload. The only paths that can emit SplitInfo followed by data are the two proxy paths, when the downstream stream contains both. The committed change makes our node's outgoing raw GET stream terminal at SplitInfo. If you mean a particular non-proxying producer path, could you share which node/log trace shows it? I don't see another local producer that can send that sequence.

@roman-khimov roman-khimov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's true that this happens only when forwarding. Nodes serve local requests like this:

2026-10-28T11:36:55.965Z	debug	get/get.go:206	serving request...	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": true, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.965Z	debug	get/get.go:249	requested object is virtual	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": true, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.965Z	debug	get/assemble.go:12	can not assemble the object	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": true, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.965Z	debug	get/get.go:210	return result directly	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": true, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}

While forwarding node does this:

2026-10-28T11:36:55.961Z	debug	get/get.go:206	serving request...	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.961Z	debug	get/container.go:12	trying to execute in container...	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.961Z	debug	get/remote.go:16	processing node...	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20063"]}
2026-10-28T11:36:55.964Z	debug	object/server.go:1172	failed to get object from remote node	{"error": "unexpected EOF"}
2026-10-28T11:36:55.964Z	debug	get/remote.go:35	remote call failed	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20063"], "error": "localhost:20063: unexpected EOF"}
2026-10-28T11:36:55.964Z	debug	get/remote.go:16	processing node...	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20076"]}
2026-10-28T11:36:55.965Z	debug	object/server.go:1172	failed to get object from remote node	{"error": "unexpected EOF"}
2026-10-28T11:36:55.965Z	debug	get/remote.go:35	remote call failed	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20076"], "error": "localhost:20076: unexpected EOF"}
2026-10-28T11:36:55.965Z	debug	get/remote.go:16	processing node...	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20055"]}
2026-10-28T11:36:55.965Z	debug	object/server.go:1172	failed to get object from remote node	{"error": "reading the response failed: rpc error: code = Canceled desc = context canceled"}
2026-10-28T11:36:55.965Z	debug	get/remote.go:35	remote call failed	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20055"], "error": "localhost:20055: reading the response failed: rpc error: code = Canceled desc = context canceled"}
2026-10-28T11:36:55.965Z	debug	get/container.go:66	interrupt placement iteration by context	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "error": "context canceled"}
2026-10-28T11:36:55.965Z	debug	get/get.go:265	operation finished with error	{"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "error": "status: code = 2049 message = object not found"}

So it effectively forwards (at least tries to, client closes the stream soon) the same split responses from all other container nodes. The fix presented here will help, but we already do have code parsing split info and handling various cases around handleSplitInfoAndRespond(). Please adjust it to stop iterating over nodes once we have a split info result for raw request.

@copilot

Co-authored-by: roman-khimov <22092804+roman-khimov@users.noreply.github.com>
@roman-khimov

Copy link
Copy Markdown
Member

Related to #4232 somewhat.

Copilot AI commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

It's true that this happens only when forwarding. Nodes serve local requests like this:

2026-10-28T11:36:55.965Z	debug	get/get.go:206	serving request...	{"component": "Object.Get service", "addres...

Updated in 305c079: raw SplitInfo forwarded through the container GET transport now returns the terminal-response signal, which stops placement iteration instead of probing more nodes. Added a regression test confirming the next node is not contacted.

@roman-khimov

Copy link
Copy Markdown
Member

Closing in favor of #4266.

@roman-khimov
roman-khimov deleted the copilot/test-raw-object-get-head-replacement-complex-objec branch October 30, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_raw_object_get_head[REP PLACEMENT-complex object] failure

2 participants