Overview
When VDiff writes a table's progress to _vt.vdiff_table, the report JSON embeds sampled mismatched rows. For a table with many mismatches and/or large column values (e.g. a large JSON/BLOB), that UPDATE ... report=... statement can exceed max_allowed_packet and fail with errno 1153.
The real problem is the recovery path. A SQL error's text includes the full failing statement (mysql/sqlerror.SQLError.Error appends during query: <query>), so the error string is itself larger than max_allowed_packet. VDiff then records that error via:
update _vt.vdiff set state='error', last_error=left(%s, 1024) (sqlUpdateVDiffState), andinsert into _vt.vdiff_log(...) values (..., %s).
left() bounds only the stored value, not the statement: the %s literal still carries the full oversized error (including the echoed report payload), so these recording statements also exceed max_allowed_packet and cannot be sent (errno 2006 broken pipe / errno 1153). saveErrorState then retries indefinitely.
Result
The VDiff is left in state='started' with an empty last_error, looping roughly every 60s. VDiff show reports it as still running, and any orchestration polling for completion waits indefinitely — with no indication of the cause. The tablet log repeats, at WARNING:
Failed to persist vdiff error state: ... broken pipe (errno 2006) ... during query:
update _vt.vdiff set state='error', last_error=left('... (errno 1153) ... during query: update _vt.vdiff_table set ... report=<very large> ...')
Root cause
The only unbounded part of a SQL error is the echoed query. Embedding the full error string as a literal into the error-recording statements is what overflows them, so the VDiff can never persist its own terminal state.
Proposed fix
At the report-write sites, rebuild the error from the structured *sqlerror.SQLError fields (message, errno, sqlstate), dropping the during query echo, and name the failing table. The recorded error becomes small and meaningful, so the VDiff transitions to error with an informative last_error instead of silently looping.
Affected versions
Observed on v22; present on all currently supported releases. Backport candidates: release-23.0, release-24.0.
Overview
When VDiff writes a table's progress to
_vt.vdiff_table, thereportJSON embeds sampled mismatched rows. For a table with many mismatches and/or large column values (e.g. a large JSON/BLOB), thatUPDATE ... report=...statement can exceedmax_allowed_packetand fail with errno 1153.The real problem is the recovery path. A SQL error's text includes the full failing statement (
mysql/sqlerror.SQLError.Errorappendsduring query: <query>), so the error string is itself larger thanmax_allowed_packet. VDiff then records that error via:update _vt.vdiff set state='error', last_error=left(%s, 1024)(sqlUpdateVDiffState), andinsert into _vt.vdiff_log(...) values (..., %s).left()bounds only the stored value, not the statement: the%sliteral still carries the full oversized error (including the echoed report payload), so these recording statements also exceedmax_allowed_packetand cannot be sent (errno 2006 broken pipe / errno 1153).saveErrorStatethen retries indefinitely.Result
The VDiff is left in
state='started'with an emptylast_error, looping roughly every 60s.VDiff showreports it as still running, and any orchestration polling for completion waits indefinitely — with no indication of the cause. The tablet log repeats, at WARNING:Root cause
The only unbounded part of a SQL error is the echoed query. Embedding the full error string as a literal into the error-recording statements is what overflows them, so the VDiff can never persist its own terminal state.
Proposed fix
At the report-write sites, rebuild the error from the structured
*sqlerror.SQLErrorfields (message, errno, sqlstate), dropping theduring queryecho, and name the failing table. The recorded error becomes small and meaningful, so the VDiff transitions toerrorwith an informativelast_errorinstead of silently looping.Affected versions
Observed on v22; present on all currently supported releases. Backport candidates:
release-23.0,release-24.0.