Repository navigation
Decode Omnicorp shared counts as ASCII decimal, not little-endian ints - #193
Merged
Merged
Conversation
The deployed shared_counts.lmdb stores each count as decimal text carried over from the upstream Redis strings (e.g. b"10584"), but the worker read values with int.from_bytes(..., "little"). Every emitted num_publications was inflated: b"1" became 49 and b"10584" became 224281309233, which also saturated the literature weight in scoring. Parse values with int() and raise on anything that isn't a plain run of ASCII digits so a store in an unexpected format fails loudly. The test fixture now writes the real format, and a new test uses raw key/value bytes read from the production artifact. Fixes #192 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S2yuRDWyMqCZeQLyysBbxA
Codecov Report✅ All modified and coverable lines are covered by tests.
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
cbizon
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #192.
Problem
shared_count_query()decodedshared_counts.lmdbvalues withint.from_bytes(raw, "little"), but the deployed artifact stores each count as ASCII decimal text carried over from the upstream Redis strings. Every emittednum_publicationswas therefore inflated, e.g.:b"1"b"80"b"10584"(Metformin × Diabetes Mellitus)This also saturated the literature weight in scoring, so it affects ranking.
Confirming the stored format
Read directly from the deployed LMDB files:
The per-curie counts match the values reported from CI/TEST/PROD in #192, and every sampled shared-count value is decimal text. The stored data is correct; the reader was wrong.
Changes
workers/aragorn_omnicorp/worker.py: parse values withint(raw); anything that isn't a plain run of ASCII digits (binary bytes, sign, whitespace, underscores, empty) raisesValueErrorrather than being guessed at. Docstrings updated to describe the real format.tests/unit/aragorn/test_aragorn_omnicorp.py: the fixture now writes ASCII decimal (the old one wrote little-endian bytes, so it only tested the reader against its own assumption). Adds a test using raw key/value bytes from the production artifact, including267_25657 → b"10584", and a parametrized test that malformed values are rejected.A
ValueErroris caught by the overlay's existing outertry/except, which logs it and returns the message without Omnicorp co-occurrence edges rather than emitting wrong counts.Testing
tests/unit/aragorn/test_aragorn_omnicorp.py: 20 passed. Against the old decoder, 9 of these fail.black --checkpasses.tests/unit/arax/test_query_parity.py, identical with and without this change, so they're unrelated (likely the local Python 3.13 vs the targeted 3.12).After deploy, the #192 reproduction request should return
num_publications = 10584. Expect rankings to shift, since the inflated counts were feeding the scorer.Generated by Claude Code