SwitchTraffic allows switching writes before reads, leaving a Reshard uncompletable

[pedroalb]

Overview

SwitchTraffic (WorkflowSwitchTraffic) switches exactly the tablet types passed in the request. When a caller switches writes (PRIMARY) before reads (REPLICA/RDONLY) have been switched, writes move to the target while the source still serves reads. This is both a correctness footgun and, for a Reshard, leaves the workflow impossible to complete.

What happens

For a Reshard, if PRIMARY is switched while a read type is still on the source (e.g. an RDONLY served-type partition that still routes to the source - even in keyspaces that have no RDONLY tablets at all), the workflow is frozen and writes are on the target, but MoveTablesComplete/Complete refuses:

cannot complete workflow because you have not yet switched all read and write traffic

because getWorkflowState → GetCellsWithShardReadsSwitched reports RdonlyCellsNotSwitched > 0. The reshard is stuck: writes are already switched (so you can't cleanly reverse), yet Complete won't proceed until the remaining read type is also switched.

Separately, even outside Reshard, switching writes ahead of reads points those reads at a source that no longer receives writes - reads go stale.

Why writes can be switched first today

WorkflowSwitchTraffic switches whatever TabletTypes are requested (reads first, then writes within a call), with no precondition that reads are switched before writes. So a --tablet-types=primary call (or tooling that switches REPLICA+PRIMARY but never RDONLY) reaches this state. The default in_order:RDONLY,REPLICA,PRIMARY avoids it only when all types are switched in one call.

Proposal

Refuse a forward PRIMARY switch when any read type (REPLICA or RDONLY) would remain on the source after the request, i.e., it is neither already switched nor being switched in the same call - unless --force is given. Error message points the user to switch reads first or include them in the same request (--tablet-types=rdonly,replica,primary).

This makes the safe rdonly → replica → primary order the default and prevents leaving a cluster in an uncompletable/stale-read state, while --force preserves an escape hatch for advanced/recovery flows.

Environment

• Vitess v22 (observed), 4 cells.
• Keyspaces with a vestigial RDONLY served-type partition and zero RDONLY tablets are especially affected: the read switch for rdonly is "invisible" (no tablets/traffic) yet still gates Complete.

PR

A proposed implementation (with a unit test) is in the linked PR. Filing for discussion. Happy to adjust the scope (e.g. also gate reverse-direction write switches, or make the block hard rather than --force-overridable).