Repository navigation
fix(io): break PyArrowFileIO reference cycle so filesystems are freed by refcounting - #4070
richacode007-byte wants to merge 2 commits into
Conversation
|
We hit #4069 in a long-running service: on Python 3.14 a web app whose request handlers call We tested this PR against it, and it fixes half. With the cycle collector off, 20 × (
The other half is a second cycle with the same shape, in the scan planner. The planner also holds Standalone reproduction (local warehouse, so it counts surviving import gc, tempfile, weakref
from functools import lru_cache
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.io.pyarrow import PyArrowFileIO
from pyiceberg.table import DataScan, ManifestGroupPlanner
planners = []
def run(label):
wh = tempfile.mkdtemp()
cat = load_catalog("c", type="sql", uri=f"sqlite:///{wh}/c.db",
warehouse=f"file://{wh}", init_catalog_tables="true")
cat.create_namespace("ns")
cat.create_table("ns.t", pa.schema([("a", pa.int64())])).append(pa.table({"a": [1]}))
cat.load_table("ns.t").scan().to_arrow()
gc.collect()
gc.disable()
live = lambda: sum(type(o) is PyArrowFileIO for o in gc.get_objects())
before = live()
for _ in range(20):
cat.load_table("ns.t").scan().to_arrow()
print(f"{label:15s} surviving FileIOs after 20 loads: {live() - before}")
gc.enable()
gc.collect()
run("released")
# This PR, in effect: the per-instance cache keeps only a weakref to the FileIO.
init = PyArrowFileIO.__init__
def weak_init(self, properties=None):
init(self, properties or {})
ref = weakref.ref(self)
self.fs_by_scheme = lru_cache(lambda s, n=None: ref()._initialize_fs(s, n))
PyArrowFileIO.__init__ = weak_init
run("this PR")
# The second cycle: drop the planner's cached partition_filters after planning.
planner_init, plan_files = ManifestGroupPlanner.__init__, DataScan.plan_files
def tracked_init(self, *a, **k):
planner_init(self, *a, **k)
planners.append(weakref.ref(self))
def plan_and_unpin(self):
files = plan_files(self)
for ref in planners:
if (p := ref()) is not None:
p.__dict__.pop("partition_filters", None)
return files
ManifestGroupPlanner.__init__, DataScan.plan_files = tracked_init, plan_and_unpin
run("this PR + planner")So a full fix for #4069 probably needs this PR plus breaking the planner's cycle, e.g. building |
Closes #4069
Rationale for this change
PyArrowFileIO.__init__caches filesystems withlru_cache(self._initialize_fs). The cache wraps a bound method, which holds a strong reference back toself. Sinceselfalso holds the cache, everyPyArrowFileIOis part of a reference cycle.So when the last user reference to a FileIO goes away, refcounting can't free it. The FileIO and every filesystem it cached stay alive until Python's cycle collector happens to run, along with their open connections and connection pools (S3, GCS, ADLS, ...). In long-running processes that create many catalogs or tables, this shows up as connections and memory that pile up for no visible reason.
This PR replaces the bound-method cache with a small module-level factory,
_fs_by_scheme_cache(file_io). It returns anlru_cache-wrapped closure that holds only aweakrefto the FileIO. That removes the cycle, so the FileIO and its cached filesystems are freed by plain refcounting as soon as they're unused.fs_by_schemestill behaves exactly as before: same signature, same caching, andcache_info()still works.__getstate__is unchanged.__setstate__rebuilds the cache through the same factory, so unpickled instances don't have the cycle either.PyArrowFileIO._initialize_fson the class keep working, because the closure calls it through the instance.Out of scope, possible follow-ups:
FsspecFileIOmay have the same pattern.SqlCatalogcreates a second FileIO, which is worth checking separately.Are these changes tested?
Yes. Two new tests in
tests/io/test_pyarrow.py:test_pyarrow_file_io_freed_by_refcounting: turns off the cycle collector (gc.disable(), restored afterwards), creates a FileIO, fills the cache withfs_by_scheme("file", None), drops the FileIO, and checks that aweakrefto it is now dead.test_pyarrow_file_io_pickle_round_trip_keeps_cache_and_refcounting: pickles and unpickles a FileIO, checks the restored copy still resolves aLocalFileSystemand caches it, and checks it's also freed by refcounting alone.Both tests fail on
mainand pass with this change. The existingtest_pyarrow_file_io_fs_by_scheme_cachetest, which usescache_info(), still passes.make testpasses (4255 passed, 5 skipped), andmake lintpasses.Are there any user-facing changes?
No API changes. The only difference users will see is that a
PyArrowFileIOand its filesystems (with their connections) are released as soon as nothing references them, instead of waiting for the cycle collector.