Repository navigation
fix(transaction): keep only the first requirement on each value - #3701
xanderbailey wants to merge 2 commits into
Conversation
Each action derives its requirements from the transaction-local table, so a second action asserts state that an earlier action in the same transaction wrote. Catalogs check requirements against the transaction base, so two fast appends in one transaction failed with "Requirement failed: Branch or tag `main` not found". The first requirement on each value is the one derived from the base; later ones on the same value are now dropped. Refs are matched by name. Prior art: apache#3075, which took the same approach alongside other changes.
|
@blackmwk or @laskoviymishka would you be able to take a look here, this I think is the last PR before we can have RTAS |
mbutrovich
left a comment
There was a problem hiding this comment.
Thanks @xanderbailey for picking this up. The fix works for two appends in one transaction. My question is whether the merge rule holds up for the RTAS replace action this is meant to unblock.
| // Catalogs check against the base, so keep only the first requirement per value. | ||
| fn merge_requirements( | ||
| existing_requirements: &mut Vec<TableRequirement>, | ||
| requirements: Vec<TableRequirement>, | ||
| ) { | ||
| for requirement in requirements { | ||
| if !existing_requirements | ||
| .iter() | ||
| .any(|existing| Self::asserts_same_value(existing, &requirement)) | ||
| { | ||
| existing_requirements.push(requirement); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| fn asserts_same_value(left: &TableRequirement, right: &TableRequirement) -> bool { | ||
| match (left, right) { | ||
| ( | ||
| TableRequirement::RefSnapshotIdMatch { r#ref: left, .. }, | ||
| TableRequirement::RefSnapshotIdMatch { r#ref: right, .. }, | ||
| ) => left == right, | ||
| _ => std::mem::discriminant(left) == std::mem::discriminant(right), | ||
| } | ||
| } |
There was a problem hiding this comment.
Keeping the first requirement on each value gives the base value only when every action asserts each value it changes. The RTAS replace may not do that. build_replacement removes main. Java's forReplaceTable asserts only the table UUID, and it skips the ref assertion for a replace while still suppressing later ones on that ref.
If the replace action follows Java, the fast append after it derives RefSnapshotIdMatch { ref: "main", snapshot_id: None } from the staged table (the table after the earlier actions in the transaction). That becomes the first requirement on main, and the commit fails. I reproduced this against MemoryCatalog with an action that emits RemoveSnapshotRef { main } and only UuidMatch, followed by a fast append on a table that already has a snapshot:
CatalogCommitConflicts => Requirement failed: Branch or tag `main` already exists
How do you plan to have the replace action emit its requirements? Could we filter against the base in do_commit instead, where self.table is in scope? apply already checks each requirement against the staged table it came from (L121-L123). So a requirement that fails on the base guards a value that an earlier action in the transaction changed, and that action decides how to guard it. For today's actions this keeps the same requirements as the PR. It also handles an action that doesn't assert what it changes, and it removes merge_requirements and asserts_same_value. Line 128 goes back to existing_requirements.extend(requirements), and the commit is built like this:
// Each action derives its requirements from the staged table, but the catalog checks
// them against the base. A requirement that fails on the base guards a value an earlier
// action in this transaction changed, and that action decides how to guard it.
let mut requirements = Vec::with_capacity(existing_requirements.len());
for requirement in existing_requirements {
if requirement.check(Some(self.table.metadata())).is_ok()
&& !requirements.contains(&requirement)
{
requirements.push(requirement);
}
}
let table_commit = TableCommit::builder()
.ident(self.table.identifier().to_owned())
.updates(existing_updates)
.requirements(requirements)
.build();Whichever rule we keep, could we add the replace-then-append case as a MemoryCatalog test? Here is the action I used:
#[derive(Clone)]
struct RemoveMainAssertingUuidOnly;
#[async_trait]
impl TransactionAction for RemoveMainAssertingUuidOnly {
type State = ();
fn new_state(&self) -> Self::State {}
async fn commit(&self, _state: &mut (), table: &Table) -> Result<ActionCommit> {
Ok(ActionCommit::new(
vec![TableUpdate::RemoveSnapshotRef {
ref_name: MAIN_BRANCH.to_string(),
}],
vec![TableRequirement::UuidMatch {
uuid: table.metadata().uuid(),
}],
))
}
async fn cleanup(self, _state: (), _table: &Table, _status: CommitStatus) {}
}There was a problem hiding this comment.
The replace action will assert main at the base snapshot, so a concurrent commit to main conflicts and the transaction retries against the refreshed table. With that, keeping the first requirement holds for replace-then-append, I'll happily add that test in the next PR if that works? Is it better for my to put that PR up now to see where we're going?
There was a problem hiding this comment.
I'll happily add that test in the next PR if that works? Is it better for my to put that PR up now to see where we're going?
No worries. Not a blocker for me.
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn test_commit_two_fast_appends_in_one_transaction() { |
There was a problem hiding this comment.
The description says the current schema and sort order ids hit the same failure, but only the fast-append case goes through a catalog. Could we add a MemoryCatalog test for update_schema().add_column(...) followed by replace_sort_order() on the new column? Without the fix it fails with Requirement failed: Current schema id does not match, because replace_sort_order asserts the staged schema id.
Which issue does this PR close?
What changes are included in this PR?
Two fast appends in one transaction currently fail with
Requirement failed: Branch or tag main not found. Each action builds its requirements from the transaction's staged table, but the catalog checks them against the base, so the second append asserts amainthe catalog hasn't seen yet. Same thing happens with the current schema and sort order ids.This keeps only the first requirement on each value. Needed for RTAS so the replace and the append can go in one commit.
This matches what PyIceberg does in
Transaction._stage(apache/iceberg-python#1631, apache/iceberg-python#4315).Similar to #3075 which went stale.
Are these changes tested?
Yes, a unit test and a
MemoryCatalogtest with two appends in one transaction.AI Disclosure
Used to investigate and write tests