Skip to content

fix(io): break PyArrowFileIO reference cycle so filesystems are freed by refcounting - #4070

Open
richacode007-byte wants to merge 2 commits into
apache:mainfrom
richacode007-byte:fix/4069-pyarrow-fileio-ref-cycle
Open

richacode007-byte wants to merge 2 commits into
apache:mainfrom
richacode007-byte:fix/4069-pyarrow-fileio-ref-cycle

Conversation

@richacode007-byte

Copy link
Copy Markdown

Closes #4069

Rationale for this change

PyArrowFileIO.__init__ caches filesystems with lru_cache(self._initialize_fs). The cache wraps a bound method, which holds a strong reference back to self. Since self also holds the cache, every PyArrowFileIO is part of a reference cycle.

So when the last user reference to a FileIO goes away, refcounting can't free it. The FileIO and every filesystem it cached stay alive until Python's cycle collector happens to run, along with their open connections and connection pools (S3, GCS, ADLS, ...). In long-running processes that create many catalogs or tables, this shows up as connections and memory that pile up for no visible reason.

This PR replaces the bound-method cache with a small module-level factory, _fs_by_scheme_cache(file_io). It returns an lru_cache-wrapped closure that holds only a weakref to the FileIO. That removes the cycle, so the FileIO and its cached filesystems are freed by plain refcounting as soon as they're unused.

  • fs_by_scheme still behaves exactly as before: same signature, same caching, and cache_info() still works.
  • __getstate__ is unchanged. __setstate__ rebuilds the cache through the same factory, so unpickled instances don't have the cycle either.
  • Tests that patch PyArrowFileIO._initialize_fs on the class keep working, because the closure calls it through the instance.

Out of scope, possible follow-ups:

  • FsspecFileIO may have the same pattern.
  • SqlCatalog creates a second FileIO, which is worth checking separately.

Are these changes tested?

Yes. Two new tests in tests/io/test_pyarrow.py:

  • test_pyarrow_file_io_freed_by_refcounting: turns off the cycle collector (gc.disable(), restored afterwards), creates a FileIO, fills the cache with fs_by_scheme("file", None), drops the FileIO, and checks that a weakref to it is now dead.
  • test_pyarrow_file_io_pickle_round_trip_keeps_cache_and_refcounting: pickles and unpickles a FileIO, checks the restored copy still resolves a LocalFileSystem and caches it, and checks it's also freed by refcounting alone.

Both tests fail on main and pass with this change. The existing test_pyarrow_file_io_fs_by_scheme_cache test, which uses cache_info(), still passes. make test passes (4255 passed, 5 skipped), and make lint passes.

Are there any user-facing changes?

No API changes. The only difference users will see is that a PyArrowFileIO and its filesystems (with their connections) are released as soon as nothing references them, instead of waiting for the cycle collector.

@richacode007-byte

Copy link
Copy Markdown
Author

@ragnard @yiftizur @gardenia Pls review the code

@richacode007-byte

Copy link
Copy Markdown
Author

@ragnard @yiftizur @gardenia Pls review the code

@awkyu

awkyu commented Nov 17, 2026

Copy link
Copy Markdown

We hit #4069 in a long-running service: on Python 3.14 a web app whose request handlers call load_table(...) + scan against an S3-compatible store ran its workers out of file descriptors (RLIMIT_NOFILE 1024). Each surviving FileIO pins its S3FileSystem, and each of those holds an eventfd. Once a worker was full, even the S3 client's hostname lookup failed (curlCode: 6, Could not resolve hostname).

We tested this PR against it, and it fixes half. With the cycle collector off, 20 × (load_table + scan().to_arrow()) on a SqlCatalog over a local S3 endpoint (pyiceberg 0.12.0, pyarrow 24.0.0, Python 3.14.7):

eventfds left over filesystems built
0.12.0 as released +40 40
with this PR's change +20 40

The other half is a second cycle with the same shape, in the scan planner. ManifestGroupPlanner.partition_filters (pyiceberg/table/__init__.py, the @cached_property returning KeyDefaultDict(self._build_partition_projection)) stores a bound method of the planner in the planner's own __dict__:

planner -> __dict__["partition_filters"] -> KeyDefaultDict -> bound method -> planner

The planner also holds io (the table's FileIO), so every scan leaves the table's FileIO, and its filesystem, alive until the collector runs. With this PR applied, all 20 surviving FileIOs were pinned this way. Dropping only that partition_filters entry after planning took the leftover eventfds from +20 to +1.

Standalone reproduction (local warehouse, so it counts surviving PyArrowFileIO objects rather than eventfds; needs only pyiceberg, pyarrow, sqlalchemy):

import gc, tempfile, weakref
from functools import lru_cache

import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.io.pyarrow import PyArrowFileIO
from pyiceberg.table import DataScan, ManifestGroupPlanner

planners = []


def run(label):
    wh = tempfile.mkdtemp()
    cat = load_catalog("c", type="sql", uri=f"sqlite:///{wh}/c.db",
                       warehouse=f"file://{wh}", init_catalog_tables="true")
    cat.create_namespace("ns")
    cat.create_table("ns.t", pa.schema([("a", pa.int64())])).append(pa.table({"a": [1]}))
    cat.load_table("ns.t").scan().to_arrow()
    gc.collect()
    gc.disable()
    live = lambda: sum(type(o) is PyArrowFileIO for o in gc.get_objects())
    before = live()
    for _ in range(20):
        cat.load_table("ns.t").scan().to_arrow()
    print(f"{label:15s} surviving FileIOs after 20 loads: {live() - before}")
    gc.enable()
    gc.collect()


run("released")

# This PR, in effect: the per-instance cache keeps only a weakref to the FileIO.
init = PyArrowFileIO.__init__


def weak_init(self, properties=None):
    init(self, properties or {})
    ref = weakref.ref(self)
    self.fs_by_scheme = lru_cache(lambda s, n=None: ref()._initialize_fs(s, n))


PyArrowFileIO.__init__ = weak_init
run("this PR")

# The second cycle: drop the planner's cached partition_filters after planning.
planner_init, plan_files = ManifestGroupPlanner.__init__, DataScan.plan_files


def tracked_init(self, *a, **k):
    planner_init(self, *a, **k)
    planners.append(weakref.ref(self))


def plan_and_unpin(self):
    files = plan_files(self)
    for ref in planners:
        if (p := ref()) is not None:
            p.__dict__.pop("partition_filters", None)
    return files


ManifestGroupPlanner.__init__, DataScan.plan_files = tracked_init, plan_and_unpin
run("this PR + planner")
released        surviving FileIOs after 20 loads: 40
this PR         surviving FileIOs after 20 loads: 20
this PR + planner surviving FileIOs after 20 loads: 0

So a full fix for #4069 probably needs this PR plus breaking the planner's cycle, e.g. building partition_filters from a function that doesn't capture the planner, or not caching it on the instance. The PR also conflicts with main at the moment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PyArrowFileIO is only freed by the cycle collector, so S3 connections pile up on Python 3.14

2 participants