Repository navigation
Add a shared mixin for state that cannot be pickled - #4357
Conversation
Locks, thread-locals and caches have to be dropped when pickling and rebuilt afterwards. Each class has handled this with its own __getstate__ and __setstate__, and new locks keep landing without it. Add _TransientStateMixin and port FsspecFileIO and PyArrowFileIO to it.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @rambleraptor, nice cleanup! Centralizing the transient-state handling removes a whole class of 'added a lock, forgot pickling' bugs, and I verified the motivation on main: a FileIO with an OAuth2AuthManager in its properties fails to pickle today with TypeError: cannot pickle '_thread.lock' object, and pickles fine with this PR. Ran the pickle round-trips locally (all protocols, deepcopy, fresh and used instances) and everything holds.
| def _init_transient(self) -> None: | ||
| """Create the attributes listed in `_transient_attrs`.""" | ||
|
|
||
| def __getstate__(self) -> dict[str, Any]: |
There was a problem hiding this comment.
Nit: __getstate__/__setstate__ here only round-trip self.__dict__, so a subclass that uses __slots__ (and doesn't mirror its slots into __dict__) would silently lose its slotted state on pickle. The hand-written versions on main had the same behavior, so this is not a regression, but since this is now the central mechanism future adopters might hit it. Maybe worth a line in the docstring that slotted subclasses need their own handling.
kevinjqliu
left a comment
There was a problem hiding this comment.
LGTM
BTW I dug into the root cause:
issue: RestCatalog stores the live AuthManager object in the FileIO properties (rest/__init__.py#L673-L674), only so the S3 signer can get an auth header (fsspec.py#L135-L136). that makes FileIO pickling depend on every auth manager being picklable. this PR fixes OAuth2, but EntraAuthManager and custom auth managers still fail.
follow up: don't put the AuthManager in FileIO properties. the auth config is already there, so the signer can create its own AuthManager from it with AuthManagerFactory.create and rebuild it after unpickling. the legacy oauth2 path might needs separate handling.
Rationale for this change
I've commented on a number of PRs about how some of their changes will cause an object to no longer be pickle-able (https://docs.python.org/3/library/pickle.html) This can potentially be a breaking change for users, so we should avoid it when possible!
I don't love that we have to keep creating
__getstate__and__setstate__methods throughout the codebase. I've created a mixin that allows developers to define which variables cannot be pickled and then the mixin centralized the__getstate__and__setstate__changes for us.Are these changes tested?
I've added some tests and we're expecting the rest of the suite to work properly.
Are there any user-facing changes?