fix(state_sync): skip interval removal when there is nothing to remove - #6027
Open
avakili-voleon wants to merge 1 commit into
Open
fix(state_sync): skip interval removal when there is nothing to remove#6027avakili-voleon wants to merge 1 commit into
avakili-voleon wants to merge 1 commit into
Conversation
`remove_intervals` unconditionally called `insert_append`, which raises "Cannot construct source query from an empty DataFrame" when there are no intervals to remove. With `remove_shared_versions=True` the rows to write are rebuilt by querying the `_intervals` table for rows sharing the target name/versions. When none of the affected snapshots have any rows there, for example an un-backfilled snapshot version held by another environment, the resulting set is empty and the insert fails. This surfaced as a non-zero exit code at the end of an otherwise successful prod restatement, after the model batches had already been backfilled and committed. Return early instead, matching the guards already present on the neighbouring insert paths `add_snapshots_intervals` and `_push_snapshot_intervals`. Fixes SQLMesh#5909 Signed-off-by: Amir Vakili <AVakili@Voleon.com>
avakili-voleon
marked this pull request as ready for review
September 4, 2026 19:12
Collaborator
|
Hey @avakili-voleon -- this looks good to me. If you're up for it, I think another test in sqlmesh plan --auto-apply
sqlmesh plan dev --skip-backfill --auto-apply
sqlmesh plan --restate-model test.incremental_model --auto-applyand assert it doesn't crash Something like this (AI Output): def test_prod_restatement_plan_with_unbackfilled_dev_version(tmp_path: Path):
"""
Scenario:
Prod is built. A breaking change is planned to devenv with --skip-backfill,
so devenv holds a different snapshot version with no interval rows. The
local files are restored to match prod, then prod is restated.
Outcome:
RestatementStage tries to clear devenv intervals for that other version,
finds none in `_intervals`, and no-ops instead of crashing on an empty
insert. Prod restatement still applies; devenv still has no intervals.
"""
original_model = """
MODEL (
name test.incremental_model,
kind INCREMENTAL_BY_TIME_RANGE (
time_column "date"
),
start '2024-01-01',
cron '@daily'
);
select account_id, date from test.external_table;
"""
models_dir = tmp_path / "models"
models_dir.mkdir()
(models_dir / "model.sql").write_text(original_model)
config = Config(model_defaults=ModelDefaultsConfig(dialect="duckdb"))
ctx = Context(paths=[tmp_path], config=config)
engine_adapter = ctx.engine_adapter
engine_adapter.create_schema("test")
df = pd.DataFrame(
{
"account_id": [1001, 1002, 1003, 1004, 1005],
"date": ["2024-01-01", "2024-01-02", "2024-01-03", "2024-01-04", "2024-01-05"],
}
)
columns_to_types = {
"account_id": exp.DataType.build("int"),
"date": exp.DataType.build("date"),
}
external_table = exp.table_(table="external_table", db="test", quoted=True)
engine_adapter.create_table(table_name=external_table, target_columns_to_types=columns_to_types)
engine_adapter.insert_append(
table_name=external_table, query_or_df=df, target_columns_to_types=columns_to_types
)
ctx.plan(auto_apply=True, no_prompts=True)
assert engine_adapter.fetchone("select count(*) from test.incremental_model") == (5,)
(models_dir / "model.sql").write_text("""
MODEL (
name test.incremental_model,
kind INCREMENTAL_BY_TIME_RANGE (
time_column "date"
),
start '2024-01-01',
cron '@daily'
);
select 1 as account_id, date from test.external_table;
""")
ctx.load()
ctx.plan(environment="dev", skip_backfill=True, auto_apply=True, no_prompts=True)
dev_snapshot_id = ctx.get_snapshot("test.incremental_model").snapshot_id
assert not ctx.state_sync.get_snapshots([dev_snapshot_id])[dev_snapshot_id].intervals
(models_dir / "model.sql").write_text(original_model)
ctx.load()
prod_snapshot_id = ctx.get_snapshot("test.incremental_model").snapshot_id
assert prod_snapshot_id != dev_snapshot_id
engine_adapter.execute("delete from test.external_table where date = '2024-01-01'")
ctx.plan(
start="2024-01-01",
end="2024-01-02",
restate_models=["test.incremental_model"],
auto_apply=True,
no_prompts=True,
)
assert engine_adapter.fetchone("select count(*) from test.incremental_model") == (4,)
assert engine_adapter.fetchone(
"select count(*) from test.incremental_model where date = '2024-01-01'"
) == (0,)
assert not ctx.state_sync.get_snapshots([dev_snapshot_id])[dev_snapshot_id].intervals
plan = ctx.plan_builder("prod").build()
assert not plan.has_changes
assert not plan.missing_intervals
``` |
cmgoffena13
self-requested a review
September 9, 2026 01:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #5909.
remove_intervalsunconditionally calledinsert_append, which raises "Cannot construct source query from an empty DataFrame" when there is nothing to remove.With
remove_shared_versions=True(used when restating prod) the rows to write are rebuilt by querying_intervalsfor rows sharing the target name/versions. If none of the affected snapshots have rows there — for example an un-backfilled snapshot version held by another environment — the set is empty and the insert fails. SinceRestatementStageruns after the backfill, this surfaced as a non-zero exit code at the end of an otherwise successful prod restatement, aborting any script wrapping it.Return early instead, matching the guards already present on the neighbouring insert paths
add_snapshots_intervalsand_push_snapshot_intervals.Test Plan
test_remove_interval_no_matching_intervalsandtest_remove_interval_empty_input; both fail without this changeplanreports no changes, and the restated models hold the expected rowsmake stylepassesmake fast-testpassesBoth
make styleandmake fast-testalso report one failure unrelated to this change:tests/web/test_main.pyimportshttpx2, which is not declared insetup.py. It reproduces on a cleanmain.Checklist
make styleand fixed any issuesmake fast-test)