Make IterTokens able to represent all locations within a node, not just value/link

[imlvts]

to_next_get_val resumes from focus_iter_token whenever it is not NODE_ITER_INVALID, rather than starting from the focus, and so does to_next_k_path. So the token is not scratch space: it is a promise that an iteration is in progress and positioned where the reader expects. Several movement operations broke that promise, and after each a following to_next_val carried on from the wrong place and reported that nothing was left:

• descend_first_byte (and so to_next_step) stored the token next_items had already advanced past the item it descended into, and NODE_ITER_FINISHED on the branch that did not move.
• descend_first_k_path leaves a token that belongs to the k-path walk, which to_next_k_path legitimately resumes from.
• reset cleared the token only on the branch that popped an ancestor, so a zipper whose whole subtrie lives in one node kept a spent one.
• ascend, ascend_byte, ascend_until and ascend_until_branch restored or left a token that no longer described the focus.

Clearing the token everywhere is the obvious fix and costs ~10% on a to_next_step walk, because to_next_sibling_byte resumes from the same field and to_next_step alternates the two. Instead IterOwner records which walk the token belongs to: the k-path pair claims it, every ascent and descend_first_byte disown it, reset clears it, and to_next_get_val restarts unless the token is its own. The sibling path keeps its fast resume. descend_first_byte stores the token before its child-descent block, which may replace it with the child's own; storing after clobbers it. No API change.

Squashed from five commits on the bugfixes branch, all one defect. Regression test: read_zipper_to_next_val_after_every_movement, nine routes onto the same location followed by a full value walk, plus the reset and four-ascent sequences and a k-path walk. Fails before this change.

lean/FINDINGS.md finding 2 on the lean-fuzzer-restage branch.

[imlvts]

Alternative: store iteration owner as a part of the iteration token, if there's a bit to spare.

[luketpeterson]

I don't like the concept of a "token owner". The idea behind the iter token is that it's a compact but redundant representation of the node_path. If that correspondence is broken the token should be cleared. Conceptually there should never be a situation where the node_path disagrees with the token, regardless of how the token got to the value it has.

[luketpeterson]

This is ready to merge into master. It'll take the agent a bit to back-port it to the old API.

[adamv-symbolica]

Fable 5.1

Regressions

Common root cause: descend_first_byte, to_next_sibling_byte (single after_focus step) and
the ascend* family (via reascend_iter_token) now trust focus_iter_token, where master
recomputed it, while some paths still leave a token that does not describe the focus.

R1. A failed or exhausted k-path walk leaves NODE_ITER_FINISHED at the base focus.
k_path_internal returns false with focus_iter_token = NODE_ITER_FINISHED. The next
descend_first_byte returns None, to_next_step returns false with paths remaining, and
LineListNode::ascend_iter_token reads FINISHED as "end of the final item", so an ascent yields a
token for the wrong position.

R2. reset() does not clear the token when there are no ancestors. Only the
ancestors.pop() Some arm sets NODE_ITER_INVALID; in a single-node trie the stale token
survives. The original PR body listed this bug; the rewrite dropped the fix and commented out the
read_zipper_to_next_val_after_reset_matches_fresh_zipper test in bad0396.

R3. k_path_internal consumes NODE_ITER_INVALID tokens popped from ancestors.
descend_to_internal, descend_to_byte, descend_indexed_byte, to_sibling and regularize
push INVALID; the PR moved the re-derivation out of the loop, so after a pop it calls
next_items(NODE_ITER_INVALID, ..). Debug: asserts in LineListNode::next_items. Release: silent
duplicate or wrong path. Miri (release profile) reports no UB.

R4. A nonexistent focus plus descend_first_byte / to_next_sibling_byte stores a lower-bound
token; later ascents misuse it. iter_token_for_path returns 0, FINISHED or
ITER_TOKEN_SLOT_1_BIT for missing paths. The PR stores that as the focus token and
reascend_iter_token passes it to ascend_iter_token, whose non-debug assert!s fire in
LineListNode (release-mode panic). In DenseByteNode the debug_assert_eq!(byte_count, 1)
fires, and in release the root token makes to_next_sibling_byte miss real siblings.

Pre-existing bugs (fail on both branches)

• P1. to_next_k_path yields a duplicate when the slot-0 key is a prefix of the slot-1 key
  ({a, abcd}). The new after_focus contract says "not below the focus", but
  LineListNode::next_items emits key1 here.
• P2. to_next_sibling_byte / to_prev_sibling_byte escape a zipper rooted at a one-byte path
  inside a multi-key node (prefix_buf.len() == 0 is checked instead of path().len() == 0).
• P3. to_next_val after an exhausted k-path walk: at the root it returns false; inside a
  subtree it skips the rest of that subtree. Same FINISHED-at-base token as R1, consumed by
  to_next_get_val, which already existed on master.
• P4. DenseByteNode::get_sibling_of_child(.., next=false) overflows mask_i -= 1
  (debug-only panic).

Tests

use pathmap::PathMap;
use pathmap::zipper::*;

// ---------------------------------------------------------------- R1

#[test]
fn r1a_failed_descend_first_k_path_then_descend_first_byte() {
    let m: PathMap<()> = [&b"ab"[..]].into_iter().collect();
    let mut z = m.read_zipper();
    assert!(!z.descend_first_k_path(3));
    assert_eq!(z.path(), b"");
    assert_eq!(z.descend_first_byte(), Some(b'a')); // PR: None
}

#[test]
fn r1b_exhausted_to_next_k_path_then_to_next_step() {
    let m: PathMap<()> = [&[0u8, 0, 0, 1, 0, 2, 2, 2, 1, 2, 1, 0, 3][..]].into_iter().collect();
    let mut z = m.read_zipper();
    assert!(z.to_next_val());
    assert!(!z.to_next_k_path(5));
    assert_eq!(z.path(), &[0, 0, 0, 1, 0, 2, 2, 2]);
    assert!(z.to_next_step());                         // PR: false
    assert_eq!(z.path(), &[0, 0, 0, 1, 0, 2, 2, 2, 1]);
}

#[test]
fn r1c_exhausted_k_path_then_ascend_then_descend_first_byte() {
    let m: PathMap<()> = [&b"ab"[..], b"wxyz"].into_iter().collect();
    let mut z = m.read_zipper();
    z.descend_to(b"w");
    assert!(z.descend_first_k_path(1));
    assert_eq!(z.path(), b"wx");
    assert!(!z.to_next_k_path(1));
    assert_eq!(z.path(), b"w");
    assert!(z.ascend_byte());
    assert_eq!(z.path(), b"");
    assert_eq!(z.descend_first_byte(), Some(b'a'));    // PR: Some(b'w')
}

// ---------------------------------------------------------------- R2

#[test]
fn r2_reset_without_ancestors_keeps_stale_token() {
    let m: PathMap<()> = [&[24u8][..]].into_iter().collect();
    let mut z = m.read_zipper();
    assert!(z.to_next_step());
    z.reset();
    assert_eq!(z.descend_first_byte(), Some(24));      // PR: None
    z.reset();
    assert!(z.to_next_step());                         // PR: false
}

// ---------------------------------------------------------------- R3

#[test]
fn r3a_descend_to_then_to_next_k_path_across_node_boundary() {
    let m: PathMap<()> = [&[0u8, 0, 1, 1][..], &[0, 1, 0, 1, 0, 0], &[0, 1, 1, 0, 0, 0, 1, 0]].into_iter().collect();
    let mut z = m.read_zipper();
    z.descend_to(&[0, 1, 1]);
    assert!(!z.to_next_k_path(2));                     // PR debug: assert; release: true
    assert_eq!(z.path(), &[0]);
}

#[test]
fn r3b_move_to_path_then_to_next_k_path() {
    let m: PathMap<()> = [&[3u8, 2, 2][..], &[3, 2, 3, 1, 2, 3, 0], &[3, 2, 3, 3]].into_iter().collect();
    let mut z = m.read_zipper();
    z.move_to_path(&[3, 2, 3, 3]);
    assert!(!z.to_next_k_path(4));                     // PR debug: assert; release: true
    assert_eq!(z.path(), b"");
}

#[test]
fn r3c_descend_to_descend_until_then_to_next_k_path() {
    let m: PathMap<()> = [&[0u8, 0, 0, 0, 1, 0, 1][..], &[0, 1, 1, 1]].into_iter().collect();
    let mut z = m.read_zipper();
    z.descend_to(&[0, 1]);
    assert!(z.descend_until());
    assert_eq!(z.path(), &[0, 1, 1, 1]);
    assert!(!z.to_next_k_path(4));                     // PR debug: assert `is_used::<1>()`
    assert_eq!(z.path(), b"");
}

// ---------------------------------------------------------------- R4

#[test]
fn r4a_nonexistent_before_key0_descend_first_byte_then_ascend_panics() {
    let m: PathMap<()> = [&b"b"[..], b"c"].into_iter().collect();
    let mut z = m.read_zipper();
    z.descend_to(b"a");
    assert!(!z.path_exists());
    assert_eq!(z.descend_first_byte(), None);
    assert!(z.ascend_byte());                          // PR (release too): panic
    assert_eq!(z.descend_first_byte(), Some(b'b'));
}

#[test]
fn r4b_nonexistent_after_key1_descend_first_byte_then_ascend() {
    let m: PathMap<()> = [&b"ab"[..], b"wxyz"].into_iter().collect();
    let mut z = m.read_zipper();
    z.descend_to(b"z");
    assert_eq!(z.descend_first_byte(), None);
    assert!(z.ascend_byte());
    assert_eq!(z.descend_first_byte(), Some(b'a'));    // PR: Some(b'w')
}

#[test]
fn r4c_nonexistent_below_zipper_root_then_ascend_until_branch_panics() {
    let m: PathMap<()> = [&[1u8, 12, 4, 5, 4][..], &[13, 9, 5, 15, 5, 0, 11, 14, 13, 3, 12, 0, 4], &[15, 3, 14, 15, 0, 8, 7]].into_iter().collect();
    let mut z = m.read_zipper_at_path(&[1, 12, 4, 5, 4]);
    z.descend_to(&[12, 12]);
    z.descend_to(&[173, 37, 23]);
    assert_eq!(z.descend_first_byte(), None);
    assert_eq!(z.ascend_until_branch(), 5);            // PR (release too): panic
    assert!(z.at_root());
}

#[test]
fn r4d_nonexistent_then_to_next_step_panics() {
    let m: PathMap<()> = [&[0u8][..], &[7], &[14, 5]].into_iter().collect();
    let mut z = m.read_zipper_at_path(&[14, 5]);
    z.descend_to_byte(11);
    z.descend_to_byte(13);
    assert_eq!(z.descend_first_byte(), None);
    assert!(z.ascend_byte());
    assert!(!z.to_next_step());                        // PR (release too): panic
}

#[test]
fn r4e_nonexistent_then_to_next_val_escapes_zipper_root() {
    let m: PathMap<()> = [&[7u8, 167, 36, 166, 110][..]].into_iter().collect();
    let mut z = m.read_zipper_at_path(&[7, 167, 36, 166, 110]);
    z.descend_to(&[45, 47, 220]);
    assert_eq!(z.descend_first_byte(), None);
    assert_eq!(z.ascend(1), 1);
    assert!(!z.to_next_val());                         // PR: true
}

#[test]
fn r4f_dense_nonexistent_below_leaf_then_sibling_missed() {
    let m: PathMap<()> = [&[10u8][..], &[20], &[30]].into_iter().collect();
    let mut z = m.read_zipper();
    z.descend_to(&[10, 99]);
    assert_eq!(z.descend_first_byte(), None);
    assert!(z.ascend_byte());
    assert_eq!(z.path(), &[10]);
    assert_eq!(z.to_next_sibling_byte(), Some(20));   // PR: None
}

#[test]
fn r4g_dense_multibyte_nonexistent_then_ascend_debug_assert() {
    let m: PathMap<()> = [&[24u8][..], &[49, 69], &[54], &[73, 209, 145, 207], &[124]].into_iter().collect();
    let mut z = m.read_zipper();
    z.move_to_path(&[133, 52, 64, 90]);
    assert_eq!(z.to_next_sibling_byte(), None);
    assert_eq!(z.ascend(2), 2);                        // PR debug: assert byte_count == 1
    assert_eq!(z.path(), &[133, 52]);
}

// ---------------------------------------------------------------- pre-existing

#[test]
fn pre_p1_k_path_duplicate_when_key0_is_prefix_of_key1() {
    let m: PathMap<()> = [&b"a"[..], b"abcd"].into_iter().collect();
    let mut z = m.read_zipper();
    assert!(z.descend_first_k_path(1));
    assert_eq!(z.path(), b"a");
    assert!(!z.to_next_k_path(1));                     // both: true, path "a" again
}

#[test]
fn pre_p2_sibling_moves_escape_zipper_root() {
    let m: PathMap<()> = [&[4u8][..], &[5]].into_iter().collect();
    let mut z = m.read_zipper_at_path(&[4]);
    assert_eq!(z.to_next_sibling_byte(), None);        // both: Some(5)
    let m: PathMap<()> = [&[14u8][..], &[15]].into_iter().collect();
    let mut z = m.read_zipper_at_path(&[15]);
    assert_eq!(z.to_prev_sibling_byte(), None);        // both: Some(14)
}

#[test]
fn pre_p3a_to_next_val_after_exhausted_k_path_at_root() {
    let m: PathMap<()> = [&b"ab"[..], b"wxyz"].into_iter().collect();
    let mut z = m.read_zipper();
    assert!(z.descend_first_k_path(1));
    assert!(z.to_next_k_path(1));
    assert!(!z.to_next_k_path(1));
    assert!(z.at_root());
    assert!(z.to_next_val());                          // both: false
    assert_eq!(z.path(), b"ab");
}

#[test]
fn pre_p3b_to_next_val_after_exhausted_k_path_in_subtree() {
    let m: PathMap<()> = [&[1u8, 2, 1, 0, 0, 0, 2, 1, 1, 1, 2, 3, 1, 2][..], &[2, 2, 1, 2, 3, 2, 3], &[3, 3, 3, 2, 3, 0, 0, 3]].into_iter().collect();
    let mut z = m.read_zipper();
    assert_eq!(z.descend_indexed_byte(1), Some(2));
    assert!(z.descend_to_existing_byte(2));
    assert!(!z.to_next_k_path(1));
    assert_eq!(z.path(), &[2]);
    assert!(z.to_next_val());
    assert_eq!(z.path(), &[2, 2, 1, 2, 3, 2, 3]);       // both: [3, 3, 3, 2, 3, 0, 0, 3]
}

#[test]
fn pre_p4_to_prev_sibling_debug_overflow_in_dense_node() {
    let m: PathMap<()> = [&[0u8, 0, 2][..], &[0, 1], &[1], &[1, 2], &[2], &[2, 0, 1], &[2, 2, 1], &[2, 2, 2]].into_iter().collect();
    let mut z = m.read_zipper();
    assert!(z.descend_first_k_path(1));
    assert_eq!(z.to_prev_sibling_byte(), None);        // both (debug): subtract overflow
}