Repository navigation
Build pydantic validators on first use to reduce memory - #159
Conversation
Importing s2python built the validators and serializers of all 126 models, which costs about 10 MB of memory, while applications typically use only a few of them. - Generate the models with an S2BaseModel base class that sets defer_build=True. - The message classes copied config and fields from the generated models by reference, so pydantic modified the generated models when the subclasses were created. That went unnoticed as long as the generated models were built first, but broke with deferred building. Copy them with the new copy_config() and copy_field() helpers instead. The JSON schemas of all public models are unchanged. On an ARM device with pydantic 2.7.4 memory use drops from 10.3 MB to 6.7 MB.
|
The rationale totally makes sense to me and I like the defer_build approach to achieve it. I would like to have @sebastiaan-la-fleur have a look at it because he set up the whole pydantic based class generations. What do you think of decreasing memory consumption this way and do you see any unwanted side effects? |
|
Hea guys! @jorritn happy to have a look at this and I think it is a good idea for me to review it. Unfortunately I am currently with vacation. Will have a look when I get back on the 20th of October! |
|
@dirkjanfaber Thanks for the PR! Could you add: Everything else looks great! Thanks for a great addition to improve useability! |
|
Thanks @sebastiaan-la-fleur! Added |
|
@dirkjanfaber Fix released as part of https://github.com/flexiblepower/s2-python/releases/tag/v0.10.2 |
3 participants
Reduce memory use by building pydantic validators on first use
Importing s2python currently builds the pydantic validators and serializers of all 126 models (65 generated + 61 message classes). That costs about 10 MB of memory per process, while applications typically use only a handful of messages.
On embedded devices (like our GX devices) running several services that use s2python, this adds up.
Changes
S2BaseModelbase class withdefer_build=True(via--base-classinci/generate_s2.sh), so a model is built the first time it is used.model_configandmodel_fields[...]from the generated models by reference.model_config["validate_assignment"] = Truechanged the config of the generatedmodel, and pydantic rewrote the reused
FieldInfoobjects (e.g. the generatedDDBCOperationMode.Idended up typeduuid.UUIDinstead ofID). This went unnoticed while the generated models were built before being modified, but with deferred building their validation changed. The newcopy_config()andcopy_field()helpers copy them instead.Verification
Notes
gen_s2.pycontains a manual change that is not in the specification:populate_by_name=TrueonTransition(Fix for transition argument #113). Regenerating would drop it; this PR keeps it. It should probably move into the specification.gen_s2.pyonly changes the base class. A full regeneration with datamodel-code-generator 0.32.0 produces the same code apart from formatting.ddbc/andombc/use CRLF line endings, while the rest of the repo uses LF, and there's no.gitattributes.)