feat(format): reserve a writer flag for versioned inverted indices by xgbtxy · Pull Request #10459 · lance-format/lance

xgbtxy · 2026-10-29T14:42:19Z

Summary

Reserve writer-only bit 20, FLAG_VERSIONED_INVERTED_INDEX, for snapshots containing an inverted index segment with IndexMetadata.index_version >= 3.

Refs #9670. A Pylance 8.0.1 writer can load a newer dataset, filter out an unsupported FTS index, and commit an append without that index's metadata. This proposal defines the compatibility boundary that makes such a writer refuse the dataset before committing.

Format contract

Set the bit atomically with the affected index metadata, using the complete index section for ordinary commits, maintenance, clone, and restore. Preserve unsupported segments or refuse the operation. Clear the bit only when no affected segment remains.
Match the inverted index type and its declared IndexMetadata version, independently of posting_format_version. Other index types at version 3 are unaffected.
Leave reader flags unchanged. A client can still read supported data and indexes, but must recognize this capability before writing. This also excludes older writers that can read v3 FTS but do not know the new flag.
Existing unflagged snapshots and stale handles are not retroactively protected; deployments must upgrade or stop incompatible writers before activation.

Bit 20 avoids the allocations currently proposed in #10358/#10164 (18) and #10431 (19). This PR leaves those positions unassigned in its base contract and does not approve either proposal.

Validation

Reproduced the unflagged index loss using official Pylance 8.0.1, 13.0.0, and 14.0.2 wheels.
On isolated copies, selected the latest manifest by its decoded version and changed only its writer feature word to 1 << 20. Both Dataset.insert and write_dataset(mode="append") refused before commit on 8.0.1 and 14.0.2. Version and rows stayed unchanged; 14.0.2 could still query the FTS index after every attempt. This is a synthetic compatibility probe, not an implemented writer change.
A standalone probe, commands, and results are available separately from this format-only diff.
Compiled all 15 protobuf schemas before and after the proposal. Descriptors excluding source comments are byte-identical; no protobuf field or wire layout changes.
Proto comment-style check and the repository's pinned pre-commit hooks passed on the exact proposal commit 98244d2b86c6ebc67d7ef15ae8d399187dff5981.

Scope

This is the format proposal for PMC review under the format voting process. It changes only the specification and matching protobuf comments. Runtime flag support, regression tests, and the migration-guide warning will follow after the contract is accepted; this PR does not close the runtime bug.

AI assistance: OpenAI Codex assisted with investigation, drafting, and validation. The commit includes an Assisted-by trailer.

Specify writer-only bit 20 for inverted index segments whose declared IndexMetadata version is at least 3. Require complete index metadata to be preserved across manifest publications or the write to be refused. Keep reader capability checks separate and document the upgrade and historical-snapshot limitations. Runtime support follows the format vote. Refs: lance-format#9670 Assisted-by: OpenAI Codex (GPT-6)

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

github-actions · 2026-10-29T14:42:43Z

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — 0 of 3 required approvals


Approvals	none (0/3)
Latest commit approved by	none — one PMC member must approve the latest commit
Vetoes	none
Voting period	elapsed — ended Tue 2026-11-03 14:42 UTC (06:42 PST)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

lance-gatekeeper

⚠️ Gate recommendation: approve with a non-blocking risk.

The writer-only capability remains appropriate for the released-client index-loss path in #9670. Its activation and preservation rules cover unflagged snapshots and stale writers. Runtime enforcement and regression tests remain follow-up work before activation.

Keep capability allocations distinct when integrating the pending format proposals.

lance-gatekeeper · 2026-11-11T07:28:06Z

 | `2^17`   | `FLAG_OBJECT_METADATA`         | No              | Yes             | The snapshot contains [clustering definitions, references, or a nonzero allocation high-water mark](clustering.md), nonempty [fragment metadata](index.md#fragment-metadata), or nonempty [index segment metadata](../index/index.md#index-segment-metadata). Reserved, unsupported until writers implement the maintenance rules. Readers may ignore this metadata. |
 | `2^18`   | `FLAG_ROW_ADDR_INDICES`         | No              | Yes             | A stable-row-ID table contains an index that stores physical row addresses. Older writers could treat its entries as stable row IDs and silently extend coverage without indexing rewritten fragments. |
 | `2^19`   | `FLAG_INLINE_DELETIONS`         | Yes             | Yes             | Some fragment stores its deletion vector in the manifest (`DeletionFile.inline`) rather than in a file under `_deletions/`. A reader without this flag would look for a file that does not exist. |
+| `2^20`   | `FLAG_VERSIONED_INVERTED_INDEX` | No              | Yes             | The index section contains an inverted index with declared index version 3 or later. Writers must preserve unsupported index segments or refuse the operation. See [Versioned inverted indices](#versioned-inverted-indices). Reserved and unsupported until implemented. |


Bit 20 is also reserved in the current specification diffs of #10358 (FLAG_BTREE_KEY_LAYOUT), #10428 (FLAG_RETENTION_LEASES), and #10780 (FLAG_LOCAL_FRAGMENT_ALIASES). Reusing the bit would let a client recognize one capability while accepting manifests that require another capability's preservation or reader rules.

These proposals are still pending, and this base rejects bit 20, so this is a non-blocking integration risk. Keep separate bit allocations as the proposals land, before enabling their implementations.

lance-gatekeeper

⚠️ Gate recommendation: approve with a non-blocking risk.

The writer-only capability remains appropriate for the released-client index-loss path in #9670. Its activation and preservation rules cover unflagged snapshots and stale writers. The generated binding comments now match the proposal; runtime enforcement and regression tests remain follow-up work before activation.

The bit 20 allocation risk remains. Keep distinct allocations across the pending format proposals before enabling their implementations.

Copilot AI balanced review requested due to automatic review settings October 29, 2026 14:42

claude Bot reviewed Oct 29, 2026

View reviewed changes

github-actions Bot added enhancement New feature or request A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). labels Oct 29, 2026

Copilot AI reviewed Oct 29, 2026

This comment was marked as outdated.

Sign in to view

lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. and removed K-approved Latest Gatekeeper recommendation permits acceptance. labels Oct 29, 2026

Merge branch 'main' into spec/fts-writer-feature-9670

c4362a2

lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Nov 11, 2026

lance-gatekeeper Bot reviewed Nov 11, 2026

View reviewed changes

lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 11, 2026

chore(proto): sync generated lance.table.rs comments

d4b882f

lance-gatekeeper Bot removed K-risk Latest Gatekeeper recommendation includes a non-blocking risk. K-approved Latest Gatekeeper recommendation permits acceptance. labels Nov 14, 2026

lance-gatekeeper Bot reviewed Nov 14, 2026

View reviewed changes

lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Nov 14, 2026

Navigation Menu

Repository navigation

feat(format): reserve a writer flag for versioned inverted indices - #10459

feat(format): reserve a writer flag for versioned inverted indices#10459
xgbtxy wants to merge 3 commits into
lance-format:mainfrom
xgbtxy:spec/fts-writer-feature-9670

xgbtxy commented Oct 29, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Oct 29, 2026 •

edited

Loading

Format specification vote

Uh oh!

This comment was marked as outdated.

Uh oh!

lance-gatekeeper Bot left a comment

Uh oh!

lance-gatekeeper Bot Nov 11, 2026

Uh oh!

lance-gatekeeper Bot left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

xgbtxy commented Oct 29, 2026

Summary

Format contract

Validation

Scope

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Format specification vote

Uh oh!

This comment was marked as outdated.

Uh oh!

lance-gatekeeper Bot left a comment

Choose a reason for hiding this comment

Uh oh!

lance-gatekeeper Bot Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

lance-gatekeeper Bot left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Oct 29, 2026 •

edited

Loading