Skip to content

[SPARK-62017][CONNECT][PYTHON] Report the types, flags, metadata and fields that the array, map, field and struct types hold to Python's garbage collector, as PySpark's __dict__ holds them - #866

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62017-gc-data-types
Closed

HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62017-gc-data-types

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

PySpark's ArrayType, MapType, StructField and StructType are plain Python classes. Their __dict__ holds the type objects that they are made of, their flags, the metadata dict of a field, and the fields and names lists of a struct, so the garbage collector collects a reference cycle through them. The drop-in holds them in Rust (KeptType, Py<PyAny>, Py<PyDict>, Py<PyList>), and the four classes had no __traverse__. The collector did not see them, so such a cycle was never collected, as SPARK-61971 found of the other holders.

Now:

  • The four classes report them in __traverse__.
  • __clear__ drops the kept type objects, and a field's name that is not a str. The lists, dicts and flags are cleared by their own types.
  • The classes have a __dict__, so the collector can traverse one of their objects while it is created, before its value is written. As the other such classes, each has a GcReady that __traverse__ checks first.
  • A KeptType reports its object only if its lock is free, which keeps the object alive otherwise. It drops the object after the lock is released, as the drop may run Python code.

Why are the changes needed?

So that a reference cycle through a type that an array, a map, a field or a struct holds is collected, as in PySpark: before, it leaked.

Does this PR introduce any user-facing change?

Yes. A cycle such as job.held = ArrayType(t) of a DecimalType t with t.job = job is collected, as pyspark-client 4.3.0 collects it. Before, it was never freed. Map keys and values, fields, metadata dicts, structs and StructType.add behave the same way. A child that is a singleton, such as LongType(), keeps the cycle in both clients.

How was this patch tested?

  • test_cycles_through_the_types_that_a_type_holds_are_collected_as_pyspark (test_dropin_offline.py) covers the eight cycles of reviewer C's probe p_gc_more2.py. It fails on master, and its body passes on pyspark-client 4.3.0.
  • test_objects_traversed_while_they_are_created_do_not_crash makes the four types, their deep copies, toNullable and df.schema, at a collection on every second allocation. Without the GcReady checks of __traverse__, it crashes with a segmentation fault (a mutation check).
  • The probes p_gc_more2.py and p_gc_types_ctl.py, run on both clients against a Spark 4.3.0 server: the data-type cases are collected on both. The instance-dict cycles of the four classes (t.ref = t) are still collected.
  • The Rust gates:
    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;
    • cargo test -p pyspark-rs --lib as rust.yml runs it: 15 passed;
    • cargo test -p apache-spark-connect -p apache-spark-connect-core --release: 1706 passed, 18 ignored.
  • The Python gates:
    • ruff 0.14.0 check and format --check, test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • gen_type_stubs.py --pyspark-src <v4.3.0>/python --check and gen_fn_dispatch.py --check;
    • test_type_stubs_offline.py with mypy 2.4.1: 12 passed;
    • the guards of SPARK-61851.
  • The offline suite of coverage.yml: 6446 passed and 77 skipped. The skips are those of master: 68 for "not a public constructor", 8 that need SPARK_REMOTE, and test_type_stubs_offline.py, which needs mypy and ran above.
  • test_dropin_e2e.py against a Spark 4.3.0 server: 328 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…fields that the array, map, field and struct types hold to Python's garbage collector, as PySpark's __dict__ holds them

PySpark's ArrayType, MapType, StructField and StructType are plain Python classes whose __dict__
holds the type objects that they are made of, their flags, the metadata dict of a field and the
fields and names lists of a struct. So a reference cycle through them is collected:
- T.ArrayType(t) of a DecimalType t with t.job = job and job.held = the array;
- a map of such a key or value;
- a field of such a type, or of a metadata dict that holds job;
- a struct of such a field, also of add.

The drop-in held them in Rust (KeptType, Py<PyAny>, Py<PyDict>, Py<PyList>), and the four classes
had no __traverse__, so the collector did not see them and the cycle was never collected, as
SPARK-61971 found of the other holders of Python objects.

Now the four classes report them in __traverse__ and drop the kept type objects, and the name of a
field that is not a str, in __clear__. Their lists, dicts and flags are cleared by their own
types. The classes have a __dict__, so the collector can traverse an object of them while it is
created, before its value is written: as the other such classes, each has a GcReady that
__traverse__ checks first. A KeptType reports its object only if its lock is free, and drops it
after the lock is released, as the drop may run Python code.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants