Repository navigation
[SPARK-62017][CONNECT][PYTHON] Report the types, flags, metadata and fields that the array, map, field and struct types hold to Python's garbage collector, as PySpark's __dict__ holds them - #866
Closed
HyukjinKwon wants to merge 1 commit into
Conversation
…fields that the array, map, field and struct types hold to Python's garbage collector, as PySpark's __dict__ holds them PySpark's ArrayType, MapType, StructField and StructType are plain Python classes whose __dict__ holds the type objects that they are made of, their flags, the metadata dict of a field and the fields and names lists of a struct. So a reference cycle through them is collected: - T.ArrayType(t) of a DecimalType t with t.job = job and job.held = the array; - a map of such a key or value; - a field of such a type, or of a metadata dict that holds job; - a struct of such a field, also of add. The drop-in held them in Rust (KeptType, Py<PyAny>, Py<PyDict>, Py<PyList>), and the four classes had no __traverse__, so the collector did not see them and the cycle was never collected, as SPARK-61971 found of the other holders of Python objects. Now the four classes report them in __traverse__ and drop the kept type objects, and the name of a field that is not a str, in __clear__. Their lists, dicts and flags are cleared by their own types. The classes have a __dict__, so the collector can traverse an object of them while it is created, before its value is written: as the other such classes, each has a GcReady that __traverse__ checks first. A KeptType reports its object only if its lock is free, and drops it after the lock is released, as the drop may run Python code. Co-authored-by: Isaac <no-reply@databricks.com>
dongjoon-hyun
approved these changes
Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
PySpark's
ArrayType,MapType,StructFieldandStructTypeare plain Python classes. Their__dict__holds the type objects that they are made of, their flags, the metadata dict of a field, and thefieldsandnameslists of a struct, so the garbage collector collects a reference cycle through them. The drop-in holds them in Rust (KeptType,Py<PyAny>,Py<PyDict>,Py<PyList>), and the four classes had no__traverse__. The collector did not see them, so such a cycle was never collected, as SPARK-61971 found of the other holders.Now:
__traverse__.__clear__drops the kept type objects, and a field's name that is not a str. The lists, dicts and flags are cleared by their own types.__dict__, so the collector can traverse one of their objects while it is created, before its value is written. As the other such classes, each has aGcReadythat__traverse__checks first.KeptTypereports its object only if its lock is free, which keeps the object alive otherwise. It drops the object after the lock is released, as the drop may run Python code.Why are the changes needed?
So that a reference cycle through a type that an array, a map, a field or a struct holds is collected, as in PySpark: before, it leaked.
Does this PR introduce any user-facing change?
Yes. A cycle such as
job.held = ArrayType(t)of aDecimalTypetwitht.job = jobis collected, as pyspark-client 4.3.0 collects it. Before, it was never freed. Map keys and values, fields, metadata dicts, structs andStructType.addbehave the same way. A child that is a singleton, such asLongType(), keeps the cycle in both clients.How was this patch tested?
test_cycles_through_the_types_that_a_type_holds_are_collected_as_pyspark(test_dropin_offline.py) covers the eight cycles of reviewer C's probep_gc_more2.py. It fails on master, and its body passes on pyspark-client 4.3.0.test_objects_traversed_while_they_are_created_do_not_crashmakes the four types, their deep copies,toNullableanddf.schema, at a collection on every second allocation. Without theGcReadychecks of__traverse__, it crashes with a segmentation fault (a mutation check).p_gc_more2.pyandp_gc_types_ctl.py, run on both clients against a Spark 4.3.0 server: the data-type cases are collected on both. The instance-dict cycles of the four classes (t.ref = t) are still collected.cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;cargo test -p pyspark-rs --libasrust.ymlruns it: 15 passed;cargo test -p apache-spark-connect -p apache-spark-connect-core --release: 1706 passed, 18 ignored.checkandformat --check,test_ruff_excludes.pyandscripts/audit_no_stubs.sh;gen_type_stubs.py --pyspark-src <v4.3.0>/python --checkandgen_fn_dispatch.py --check;test_type_stubs_offline.pywith mypy 2.4.1: 12 passed;coverage.yml: 6446 passed and 77 skipped. The skips are those of master: 68 for "not a public constructor", 8 that needSPARK_REMOTE, andtest_type_stubs_offline.py, which needs mypy and ran above.test_dropin_e2e.pyagainst a Spark 4.3.0 server: 328 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.