Repository navigation
Terminate GET forwarding after SplitInfo - #4256
roman-khimov with Copilot wants to merge 3 commits into
Conversation
Co-authored-by: roman-khimov <22092804+roman-khimov@users.noreply.github.com>
roman-khimov
left a comment
There was a problem hiding this comment.
This touches forwarding code, but it means another SN is sending split info and some data after it. It shouldn't be doing that, please fix it instead. @copilot.
I traced the response paths. A locally generated SplitInfo is sent by |
roman-khimov
left a comment
There was a problem hiding this comment.
It's true that this happens only when forwarding. Nodes serve local requests like this:
2026-10-28T11:36:55.965Z debug get/get.go:206 serving request... {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": true, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.965Z debug get/get.go:249 requested object is virtual {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": true, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.965Z debug get/assemble.go:12 can not assemble the object {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": true, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.965Z debug get/get.go:210 return result directly {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": true, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
While forwarding node does this:
2026-10-28T11:36:55.961Z debug get/get.go:206 serving request... {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.961Z debug get/container.go:12 trying to execute in container... {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]"}
2026-10-28T11:36:55.961Z debug get/remote.go:16 processing node... {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20063"]}
2026-10-28T11:36:55.964Z debug object/server.go:1172 failed to get object from remote node {"error": "unexpected EOF"}
2026-10-28T11:36:55.964Z debug get/remote.go:35 remote call failed {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20063"], "error": "localhost:20063: unexpected EOF"}
2026-10-28T11:36:55.964Z debug get/remote.go:16 processing node... {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20076"]}
2026-10-28T11:36:55.965Z debug object/server.go:1172 failed to get object from remote node {"error": "unexpected EOF"}
2026-10-28T11:36:55.965Z debug get/remote.go:35 remote call failed {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20076"], "error": "localhost:20076: unexpected EOF"}
2026-10-28T11:36:55.965Z debug get/remote.go:16 processing node... {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20055"]}
2026-10-28T11:36:55.965Z debug object/server.go:1172 failed to get object from remote node {"error": "reading the response failed: rpc error: code = Canceled desc = context canceled"}
2026-10-28T11:36:55.965Z debug get/remote.go:35 remote call failed {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "address group": ["localhost:20055"], "error": "localhost:20055: reading the response failed: rpc error: code = Canceled desc = context canceled"}
2026-10-28T11:36:55.965Z debug get/container.go:66 interrupt placement iteration by context {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "error": "context canceled"}
2026-10-28T11:36:55.965Z debug get/get.go:265 operation finished with error {"component": "Object.Get service", "address": "AT9bBfsUoPwrjANfx77MTDegBDqSMDNHBmTrqYp7XHdG/ED3NtqZCNvPf59rpRCx9GAVjn1XvVbdu7Xe5oFBDG5z5", "raw": true, "local": false, "with session": false, "with bearer": false, "request": "GET_RANGE", "original range": "[0:10]", "error": "status: code = 2049 message = object not found"}
So it effectively forwards (at least tries to, client closes the stream soon) the same split responses from all other container nodes. The fix presented here will help, but we already do have code parsing split info and handling various cases around handleSplitInfoAndRespond(). Please adjust it to stop iterating over nodes once we have a split info result for raw request.
Co-authored-by: roman-khimov <22092804+roman-khimov@users.noreply.github.com>
|
Related to #4232 somewhat. |
Updated in |
|
Closing in favor of #4266. |
Raw range GETs for complex objects could fail with SDK rc24 because forwarding continued after a SplitInfo response. SplitInfo must be the final message in the response stream.
0:10range forwards SplitInfo but not a trailing chunk.