Repository navigation
Read only the data a model needs - #7
Merged
Merged
Conversation
Every model, fit and validation that needed a few datasets built a dict of the whole catalogue (some 8,300 files) to look them up. They now read only the records they need through optmat._catalogue, and the catalogue, when loaded, reuses the records already read. molecular_rate_model reads 395 data files instead of 8,671. Contract tests serve synthetic datasets through a curated fixture that overlays the catalogue lookups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gas registries built every gas to return one: default_model built all fifteen fits (each with its Kramers-Kronig absorption term), and the ionisation parameters, the atomic and molecular rates and the Raman parameters built every species and every model. Each is now built on demand and cached; fitted_model and the gas published_model read one record, as the solid published_model does. The functions that list everything keep their results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ionisation_rate_coverage and molecular_rate_coverage read only the rates files, so a caller can tell an atom from a molecule, and list a family's model names, without building any model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
data/catalogue_index.json lists each source record's size, the materials and quantities of its datasets and the materials of its formulas, derived from the records alone by tools/catalogue_index.py. datasets(material=, quantity=), quantity_datasets and formula_sources read only the records it lists, instead of every record (air_model: 799 files read, now 19). The index is checked against the records on first use (one stat each): a record that is new or changed size is read again, and a CatalogueIndexWarning names any whose entries the index gets wrong, so an out-of-date index in a source install is slower, never wrong. A test keeps the committed index equal to the records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source's datasets are built from its record's entries when first asked for: Source.dataset(id) builds one, Source.datasets (now a cached property) builds all, and the catalogue loads its records with load_source(path, defer=True). load_source still builds and checks every dataset by default. A source's citation, errata and formulas now cost no data file: the gas models of He read 179 files rather than 1,731, and every record 762 rather than 8,333. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/test_data_reads.py runs the entry points a propagation code uses in a fresh process and fails if one reads more than about twice what it reads now, so a full-catalogue load cannot return unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Documentation build overview
106 files changed ·
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This was referenced Oct 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Models read only the data they need. No value, model or stored fit changes.
A model that needed a few datasets used to read the whole catalogue, about 8,300 files. On a cold disk, or under an on-access scanner such as Microsoft Defender, that took tens of seconds. solprop's start-up spent most of its time here.
Data files opened, fresh process
molecular_rate_model("N2")ionisation_rate_model("He")default_model("He")raman_response("N2")air_model()models("He")sources()(every record)Cold start: solprop's optmat start-up calls, timed on fresh installs, on a macOS laptop with Defender:
Most of the remaining cold time is first-time module loading, which is the same on both.
Commits
_catalogue.by_key/datasetinstead of{d.key: d for d in datasets()}. The catalogue reuses records already read, so none is read twice. Contract tests now inject synthetic datasets through a sharedcuratedfixture intests/conftest.py, which overlays the catalogue lookups.fitted_model(name), plus a gaspublished_model(name)like the solid one. The functions that list everything are unchanged.ionisation_rate_coverage/molecular_rate_coverage. The gases each rate model covers, read from the rates file alone (2 files). They let a caller tell an atom from a molecule without building models; solprop will use them.data/catalogue_index.jsonis written bytools/catalogue_index.py. For each record it holds the record's size, the materials and quantities of its datasets, and the materials of its formulas. Searches by material, quantity or formula read only the records it lists.tests/test_catalogue_index.pykeeps it equal to the records.statper record. A new record, or one whose size changed, is re-read from its TOML.CatalogueIndexWarningnames records the index gets wrong, so an out-of-date index in a source install is slower, never wrong.Source.datasetsbecomes a cached property built from the newSource.entries, andSource.dataset(id)builds one dataset. The catalogue loads records withload_source(path, defer=True).load_sourcestill builds and checks every dataset by default.tests/test_data_reads.pyruns the entry points a propagation code uses in a fresh process. It fails if one opens more than about twice what it opens now.Public API (snapshot re-recorded, CHANGELOG updated)
Source.entries,Source.dataset,load_source(..., defer=),fitted_model, gaspublished_model,ionisation_rate_coverage,molecular_rate_coverage,CatalogueIndexWarning.Source.datasetsis a property, not a constructor field. Nothing outsideload_sourceconstructsSource.No OEP: the maintainer approved an exception for this optimisation.
Checks
The full
poe checkpassed locally: 21,507 tests with 1 skipped, 100% statement and branch coverage, and a clean docs build.🤖 Generated with Claude Code