Online DDL: classify vreplication errors as resumable vs. unrecoverable before failing a migration

[mattlord]

Overview

reviewRunningMigrations (go/vt/vttablet/onlineddl/executor.go) treats vreplication state Error on a running vrepl Online DDL migration as terminal: it cancels the migration and marks it failed. But "state Error" conflates two very different situations:

• Genuinely unrecoverable errors — isUnrecoverableError (FAILED_PRECONDITION vterrors, the known-fatal MySQL errnos: schema mismatch, bad data, etc.). Failing the migration is correct.
• Errors made terminal only by the --vreplication-max-time-to-retry-on-error same-error window — a long network partition to the source, a source tablet unavailable for an extended stretch, a slow-but-alive applier stalling. The underlying stream is fully resumable: the _vt.vreplication row and _vt.copy_state checkpoints are intact, and a plain Workflow start resumes the copy from the stored lastpk.

The asymmetry is what makes the current behavior expensive: cancel is destructive (a later RETRY starts a fresh shadow table and re-copies everything, discarding potentially days of multi-shard row-copy work) while resume is nearly free. #20922 removed the throttling-induced instance of this (the vplayer stall detector firing during sustained throttling), but any other resumable-class error that repeats past the window still converts a healthy, resumable migration into failed.

Suggested approach

When the executor sees vreplication state Error on a running migration, classify before acting:

1. Unrecoverable class → cancel/fail as today.
2. Resumable class → keep the migration alive with an operator-visible "stuck: vreplication stream errored but is resumable" condition (probably remaining in running with a liveness message, since there is no "stuck" migration status and inventing one has its own semantics — interaction with the stale-migration reaper needs care), and either leave resumption to the operator or auto-issue the equivalent of Workflow start on a slow, bounded backoff.

Design questions to settle:

• Auto-resume vs. operator bound. The stream reached terminal state precisely because the operator bounded retries with --vreplication-max-time-to-retry-on-error; the executor auto-resuming overrides that judgment. It likely wants an opt-in (ddl_strategy flag) and/or a bounded resume budget, rather than unconditional auto-resume.
• Structured error classification. The executor currently learns the error from the message string. The robust fix is a structured error class carried with the vreplication row: today errVPlayerStalled carries no vterrors code, and vterrors.Wrap does not support errors.Is chain traversal, so nothing can classify these errors except by string matching. A small cross-component contract (e.g. an error-class column or a coded message convention) would benefit non-DDL consumers of stream state too.
• This is a behavior change in failure handling and needs a release-note callout when implemented.

References

• Bug Report: VReplication vplayer stall detector fires while the workflow is throttled, terminally failing resumable Online DDL migrations #20922 — the incident analysis (a routine node replacement → sustained throttling → stall error → terminal window → migration failed while Workflow start would have resumed it).
• VReplication: don't count throttled time in the vplayer stall deadline #20925 — the vplayer stall fix for Bug Report: VReplication vplayer stall detector fires while the workflow is throttled, terminally failing resumable Online DDL migrations #20922, which removes the throttling-specific false positive; this issue covers the remaining class.