Object id PR 1a: no home PE in the id, 49-bit packed index, hashed ids with per-process tranches, delivery by id

[lvkale]

PR 1a of the object id redesign (#3994), stacked on objid-redesign; design in doc/objid64-design.md sections 1-6 on that branch. Merges into the integration branch, not the reviewed line.

What changes

• Layout: tag 3 | collection 12 (build-time CMK_OBJID_COLLECTION_BITS) | payload 49. CMK_OBJID_HOME_BITS is gone; no PE number lives in the id. Bounded arrays pack the index into the whole payload (16 bits before). Unbounded arrays get a hash key of the index plus a number unique within the array; the key width is set per run from the process count and +objid_expand (default 8) and travels with every checkpoint.
• One home per element: homePe(idx) and homePe(id) agree by construction (map for packed arrays; index hash to a process for hashed arrays). The cache gets its manager back-pointer to compute it.
• Minting: hashed ids come synchronously from a per-process tranche with one atomic cursor; the per-PE counter and its silent overflow are gone. Refills (the allocator) are PR 1b; until then exhausting the initial tranche aborts with a message.
• Delivery by id: receive-side misses go to homePe(id) via handleUnknownByID (Aditya's e417584 folded in); lookupIdx loses its linear scan; requestLocationOnce stops repeated detours; the "two hops then home" rule is enabled.
• LB global update now carries the element's true epoch from the emigrating PE.
• Restart: ids never change; directories are rebuilt (restore() informs the home; the __FAULT__ cache pup is dropped). Restarting with a different number of processes aborts for hashed arrays until PR 1b; packed arrays restart on any count.
• New test tests/charm++/unbounded_restart.

Tested on reconverse-darwin-arm8 (production), 1-3 processes: pingpong, megatest, hello/4darray, startupTest, anytime_bcastred, unbounded_migration, the three load_balancing tests with RotateLB, chkpt (same count; 4->2 and 2->4 PEs), unbounded_restart, and a 200,000-element unbounded insert from one PE. Found on the way, not caused here (reproduced on the unmodified build): a 2-process x 2-PE checkpoint restart hangs after "CkRestartMain done"; TreeLB asserts when a program calls migrateMe under an active balancer (program error).

Confidence tests and fixes added 2026-09-30

Five test programs, each run single-process and under lcrun with 2 and 4 processes on macOS (in a single process every hashed-kind home is PE 0, so the multi-process runs are the ones that exercise hashed-kind homes):

• tests/charm++/objid_home: from every PE, homePe(idx) == homePe(id), packed ids computable on PEs that never saw the index, hashed ids obey the key/unique split and land in the creating process's tranche, ids and homes unchanged after two migrations.
• tests/charm++/objid_insert (both kinds): insertion from every PE, synchronous [inline] local insertion, cold sends by index, sends over one- and two-hop-stale caches, [createhome] demand creation, ckDestroy.
• tests/charm++/anytime_bcastred -u: the existing migration stress on an unbounded array; the reconverse stress job runs it too.
• tests/charm++/objid_sections: ckmulticast sections on a hashed-kind array, migrating every round.
• tests/charm++/objid_bound: bound arrays over a hashed-kind array under migrateMe and under +balancer RotateLB/GreedyLB, including a never-inserted bound array reached only through [createhome] (deliverInline's bound-sibling demand creation, the last user of the reverse lookup).

Fixes they found, each its own commit:

• 6d0029d: the "after two hops go home" rule in recvMsg redirected to the home on the home PE itself, so the home re-sent the message to itself forever (this PR's regression; failed anytime_bcastred in CI).
• d7a76ab (pre-existing): a [createhome] request from a PE with no id never got a reply, so the requester's buffered messages were never delivered. bufferForCreation now also requests the location by index.
• c2ce757 (pre-existing, upstream main too): deliverInline delivered to the null pointer looked up before demand-creating a bound sibling.

Not fixed, to file separately: a direct ckDestroy() inside a broadcast entry method segfaults in CkArrayBroadcaster::attemptDelivery (base runtime too); the tests use thisProxy[thisIndex].ckDestroy().

🤖 Generated with Claude Code

[adityapb]

Review of this PR at dca0730 against doc/objid64-design.md. These come from reading the code; I haven't reproduced them yet. Most of the change matches the design and looks right. Two demand-creation regressions and the CMK_GLOBAL_LOCATION_UPDATE change should be fixed before merge.

Must fix

1. Demand creation is lost when the message reaches the home by id (regression for bounded arrays).
recvMsg now always calls handleUnknownByID (ckarray.C:1874). e417584 tried tryLookupIdx first and only routed by id when the index was unknown. handleUnknownByID has no demand-creation branch: at the home it parks the message in bufferedIDMsgs and calls requestLocation(id), which does nothing on the home.

• Example: bounded 1D array with a [createhome] entry; element 5 never inserted; its home is PE 1. PE 0 sends arr[5].poke(). PE 0 has the id (packed index), so it forwards the message to PE 1 (every message counts as small at the default +messageBufferingThreshold). PE 1 parks it, and element 5 is never created. On objid-redesign, PE 1 recovers the index and creates the element (handleUnknown → bufferForCreation).
• [createhere]: a forwarded createhere message that lands where the element is unknown is now parked. Before, the element was created on the sending PE.
• objid_insert phase 4 misses this because every PE sends revive, including the home. The home's own send creates the element, which releases the parked messages. A variant where only non-home PEs send would catch it.
• Suggested fix: in handleUnknownByID, when the message asks for creation and the index can be recovered (compressor, or a local record), go through handleUnknown(idx). Re-creating a deleted element of an unbounded array still needs an id → index lookup at the home, since the reverse scan is gone.

2. requestLocationAtHome (d37b78c) makes the home reply with an empty entry.
CkLocMgr::requestLocation(idx, peToTell) (cklocation.C:2842) replies whenever it finds an id for the index, even if the location is unknown. For bounded arrays the id is always found, so the reply is CkLocEntry::nullEntry (id 0, pe -1). Before this PR, bounded arrays never sent the index-keyed request, so this couldn't happen.

• Example: bounded array; a [createhere] send from a non-home PE R to a missing element. The home tells R to create it, then answers R's location request with the null entry, because R hasn't informed the home yet.
• Debug builds: CkAssert(e.pe != -1) in CkLocMgr::updateLocation aborts on R every time.
• Production builds: the index listeners call sendBufferedMsgs(idx, 0, -1), so any messages R holds for element id 0 are sent to PE -1. Id 0 is a real element: index 0 of a bounded array, or the first hashed id on process 0.
• Hashed kind: once an element has been deleted (reclaimRemote keeps the idx → id binding), the same reply overwrites R's idx2id[idx] with 0.
• Not caught because testing used production builds and the new test only uses [createhome].
• Suggested fix: reply only when cache->getPe(id) != -1. Otherwise push onto bufferedLocationRequests, which updateLocation already answers once the location is known.

3. CMK_GLOBAL_LOCATION_UPDATE: the update no longer happens before the program resumes.
UpdateLocation is now empty (cklocation.C:93) and emigrate broadcasts the new location instead (cklocation.C:3247). The comment in CentralLB (CentralLB.C:1049-1065, still present) explains that every PE, the destination included, must learn the new location before the move is acted on. Now the resume broadcast from PE 0 and each migration's broadcast from its source travel independently, so a PE can resume and send with a stale location.

• GPU zerocopy: the receiver aborts with "Current PE does not match the destination PE..." (ckrdmadevice.C:259, also line 1230) on builds without CMK_GPU_COMM. That's the configuration the GPU line enables this flag for.
• Debug builds: the extra hop hits CkAssert(false) in multiHop (cklocation.C:2964).
• Cost: one group broadcast per migrated element, so migrations × PEs messages per LB step, where the old path sent none.
• Design section 4.4 keeps UpdateLocation and makes it id-keyed, so this departs from the doc. The commit message lists no build with this flag.
• Suggested fix: keep UpdateLocation synchronous and id-keyed. Either carry the element's true epoch in MigrateInfo from the source's LB stats, or keep the fabricated epoch for ordering and the broadcast to correct it.

Smaller

• decompress uses (1 << bits) - 1 (ckarrayindex.h:426). With the 49-bit budget one dimension can now need 31 bits, which is undefined for a signed shift. ((CmiUInt8)1 << bits) - 1 matches what compress does with 1UL.
• pupLayout is added to CkPupROData (ckcheckpoint.C:432) with no version marker, so restarting from a checkpoint written before this PR misreads _numReadonlyMsgs silently instead of failing clearly.
• The restart abort at cklocation.C:2623 also fires when the process count is unchanged but packedByNode != CkMyNode() (e.g. 2×2 → 2×1 PEs, where PE 0's copy is restored everywhere). The message says "different process count", which is misleading there.
• The uniqueBits < 16 check in initLayout (cklocation.C:2376) runs at every startup, whether or not a hashed array exists. With --ampi-only's CMK_OBJID_COLLECTION_BITS=29 (payload 32), every run above 8,192 processes would abort. Moot until AMPI is supported, but checking at the first hashed CkLocMgr would be safer.
• pendingLocReqs entries stay stuck when the home drops the request (CkLocCache::requestLocation(id, pe) drops unknown ids). It mostly clears through multiHop, and section 4.3 says PR 1b removes requestLocationOnce anyway.

Looks right

• Layout, masks and static_asserts.
• homePe(idx) == homePe(id) by construction for both kinds; every homePe(id) caller passes the payload (array_element_id()), not the full ObjID.
• restore() now informs the home, replacing the __FAULT__ cache pup.
• Tranche cursor: lock-free fetch_add, and a restore only raises it.
• cf394e6 (home no longer redirects to itself) and ba609ba (re-lookup after demand creation; a real null dereference on main too).

[lvkale]

This merged on the 2026-10-01 approval, ahead of Aditya's 2026-10-04 review above. The merged head (dca0730) is exactly the commit that review examined, so its three must-fix items are now in objid-redesign unaddressed:

1. demand creation lost when a message reaches the home by id (handleUnknownByID has no creation branch);
2. requestLocation(idx, peToTell) replying with a null entry for bounded arrays;
3. CMK_GLOBAL_LOCATION_UPDATE no longer synchronous before the program resumes.

A fix PR against objid-redesign follows, with a test for each item and the smaller points from the same review; Aditya's approval is required on it. Until it lands, please do not merge #4015 or #4021. objid-redesign is an integration branch with no users, so nothing on the reviewed line is affected.