Skip to content

Decode Omnicorp shared counts as ASCII decimal, not little-endian ints - #193

Merged
maximusunc merged 1 commit into
develfrom
claude/omnicorp-decimal-shared-counts
Oct 29, 2026
Merged

maximusunc merged 1 commit into
develfrom
claude/omnicorp-decimal-shared-counts

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

Fixes #192.

Problem

shared_count_query() decoded shared_counts.lmdb values with int.from_bytes(raw, "little"), but the deployed artifact stores each count as ASCII decimal text carried over from the upstream Redis strings. Every emitted num_publications was therefore inflated, e.g.:

Stored Decoded before this PR
b"1" 49
b"80" 12,344
b"10584" (Metformin × Diabetes Mellitus) 224,281,309,233

This also saturated the literature weight in scoring, so it affects ranking.

Confirming the stored format

Read directly from the deployed LMDB files:

metformin: {'index': 25657, 'pmc': 27317}  diabetes: {'index': 267, 'pmc': 614535}
pair raw: b'10584'
first 10 entries:
   b'0_1' b'80'
   b'0_10' b'2293'
   b'0_100' b'27082'
   ...

The per-curie counts match the values reported from CI/TEST/PROD in #192, and every sampled shared-count value is decimal text. The stored data is correct; the reader was wrong.

Changes

  • workers/aragorn_omnicorp/worker.py: parse values with int(raw); anything that isn't a plain run of ASCII digits (binary bytes, sign, whitespace, underscores, empty) raises ValueError rather than being guessed at. Docstrings updated to describe the real format.
  • tests/unit/aragorn/test_aragorn_omnicorp.py: the fixture now writes ASCII decimal (the old one wrote little-endian bytes, so it only tested the reader against its own assumption). Adds a test using raw key/value bytes from the production artifact, including 267_25657 → b"10584", and a parametrized test that malformed values are rejected.

A ValueError is caught by the overlay's existing outer try/except, which logs it and returns the message without Omnicorp co-occurrence edges rather than emitting wrong counts.

Testing

  • tests/unit/aragorn/test_aragorn_omnicorp.py: 20 passed. Against the old decoder, 9 of these fail.
  • black --check passes.
  • Full unit suite: 10 failures in tests/unit/arax/test_query_parity.py, identical with and without this change, so they're unrelated (likely the local Python 3.13 vs the targeted 3.12).

After deploy, the #192 reproduction request should return num_publications = 10584. Expect rankings to shift, since the inflated counts were feeding the scorer.


Generated by Claude Code

The deployed shared_counts.lmdb stores each count as decimal text carried
over from the upstream Redis strings (e.g. b"10584"), but the worker read
values with int.from_bytes(..., "little"). Every emitted num_publications
was inflated: b"1" became 49 and b"10584" became 224281309233, which also
saturated the literature weight in scoring.

Parse values with int() and raise on anything that isn't a plain run of
ASCII digits so a store in an unexpected format fails loudly. The test
fixture now writes the real format, and a new test uses raw key/value
bytes read from the production artifact.

Fixes #192

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuRDWyMqCZeQLyysBbxA
@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 53.94%. Comparing base (1d5ed6c) to head (d3fe9a3).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines Coverage Δ
workers/aragorn_omnicorp/worker.py 76.10% <100.00%> (+0.17%) ⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 010f7ef...d3fe9a3. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc requested a review from cbizon October 29, 2026 00:19
@maximusunc
maximusunc merged commit 7ae7772 into devel Oct 29, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/omnicorp-decimal-shared-counts branch October 29, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Omnicorp emits impossible shared-publication counts; likely LMDB encoding/decoding mismatch

3 participants