Skip to content

[SPARK-62016][CONNECT][PYTHON] Treat user-defined type objects as PySpark does: keep them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString() - #865

Closed
HyukjinKwon wants to merge 1 commit into
apache:masterfrom
HyukjinKwon:SPARK-62016-udt-objects-as-pyspark

Conversation

@HyukjinKwon

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Four changes, each in its own commit, that make the drop-in treat user-defined type (UDT) objects as pyspark-client 4.3.0 does. A last commit renames a helper that scripts/audit_no_stubs.sh refused. With SPARK-61999 (#848), the toNullable of an array, a map or a struct is as_nullable of the objects, and a type of metadata or user attributes pickles of them, which keeps the UDT objects of part 2 too. With SPARK-62004 (#849), the schema that a copy or an unpickle of a DataFrame requested is converted as a schema of the server (part 3).

1. The simple strings of the mllib vector and matrix types (core DataType::simple_string). The VectorUDT and MatrixUDT of pyspark.mllib.linalg are named vector and matrix, as those of pyspark.ml.linalg already were (SPARK-61690). A cast to them shows VECTOR and MATRIX, as PySpark's CastExpression.__repr__ does. Other UDTs keep udt, which is PySpark's UserDefinedType.simpleString().

2. A UDT object is kept in the types that hold it until a use needs its JSON (types.rs). PySpark's ArrayType, MapType, StructField and StructType keep the type objects that they are given as they are, and read one only for a use that needs its JSON or its conversion. The drop-in lowered the child into a core type when it made the type, which calls jsonValue(). So the type of a UDT whose module() or sqlType() is not implemented, such as UserDefinedType() itself, raised NOT_IMPLEMENTED when it was made, by its constructor, add or a setter.

  • Now the child is lowered leniently when it is made (lenient_type). A type that does not lower is kept as its object, with the null type standing for it in the snapshot.
  • A use that needs the JSON reads the object again through KeptType::live, so it raises then, as PySpark's does: json(), the plans of a cast, DataFrame.to, from_json, a table, a Java UDF, a UDTF, a reader's schema, and data of no rows.
  • The uses that PySpark answers from the objects fall back to them when the strict lowering fails:
    • simpleString and needConversion;
    • ==, by the attributes, as PySpark's __dict__ comparison;
    • the hash;
    • pickles, copies and deep copies;
    • toNullable;
    • the tree string, which names a stand-in by its object's typeName();
    • names and add.
  • DataFrame.to of such a schema raises the type's error at each request, where it raised UNSUPPORTED_OPERATION of the schema.

3. The schemas of the server make new UDT objects (types::schema_to_py). PySpark converts the types of the server with UserDefinedType.fromJson, which makes a new object of the class, UDT() (python/pyspark/sql/types.py:1986-2005). The drop-in made them from a process-wide registry of the objects that the client lowered, by their JSON. So:

  • df.schema["p"].dataType was the user's own object, with its attributes;
  • lowering an unrelated UDT of the same JSON later changed the schema of an untouched DataFrame;
  • a class whose __init__ takes arguments did not raise, where PySpark's df.schema, df.dtypes and df.collect() raise TypeError.

Now the schema of a DataFrame and its dtypes, the schemas of the rows that collect, toLocalIterator, head and take convert, those of toPandas and _to_table, and _data_type_of_proto make each UDT a new object of its class. The array, map, field or struct that holds it keeps that object. The registry stays for the types that the client makes of core types (SPARK-61917). DataFrame.to keeps the struct that it is given as its cached schema, as SPARK-61962 made it, so its schema holds copies of the user's UDT objects, as PySpark's.

4. A cast renders the simpleString() of its type object (core CastTarget::Shown, Column::cast_shown; pyspark-rs cast_column). PySpark's CastExpression.__repr__ shows simpleString().upper() of the type object. The core rendered the simple string of its core type, so:

  • a UDT of its own simpleString() showed UDT, where PySpark shows CAST(x AS SCALE2) and STRUCT<A:SCALE2,B:SCALE10>;
  • a type of a Python subclass showed its class's (CAST(id AS BOOLEAN) where PySpark shows CAST(id AS SUB)), a known difference that this removes.

The core has a hidden cast target that sends the type and renders a given string. The drop-in uses it with the object's simpleString() when the type holds a UDT or a type of a Python subclass. A cast to a type that does not lower now is made, renders its simpleString(), and raises at each request, as PySpark's.

Why are the changes needed?

So that UDT objects behave as in pyspark-client 4.3.0:

  • the types of a UDT of no JSON are made;
  • the schemas of DataFrames do not share the user's UDT objects, nor change with other types;
  • a UDT of arguments raises where PySpark's does;
  • casts show PySpark's strings.

Does this PR introduce any user-facing change?

Yes. With u = PtUDT(); u.marker = 1, a df of createDataFrame of a schema of u, a ScaleUDT(2) whose simpleString() is scale2, and an ArgUDT whose __init__ takes a scale:

PySpark 4.3.0, and now Before
ArrayType(UserDefinedType()) ArrayType(UserDefinedType(), True); its json() raises [NOT_IMPLEMENTED] module() is not implemented. raises that error
df.schema["p"].dataType is u False, a new PtUDT() without the attribute True
df.schema of a schema of ArgUDT(2) TypeError: ... missing 1 required positional argument: 'scale' the user's ArgUDT(2)
F.col("x").cast(ScaleUDT(2)) Column<'CAST(x AS SCALE2)'> Column<'CAST(x AS UDT)'>
F.col("x").cast(pyspark.mllib.linalg.VectorUDT()) Column<'CAST(x AS VECTOR)'> Column<'CAST(x AS UDT)'>

How was this patch tested?

  • Offline tests, of official PySpark 4.3.0's strings and errors. Their bodies pass on pyspark-client 4.3.0, but for the requests to the mock server, and they fail on master:

    • test_casts_to_the_linalg_types_show_their_simple_strings_as_pyspark (test_dropin_offline.py);
    • test_types_of_a_user_defined_type_of_no_json_keep_it_as_pysparks (test_dropin_types_columns_offline.py), of three such UDTs and seven containers: reprs, simpleString, needConversion, ==, hash, copies, pickles, json(), toInternal, the tree string, add, toNullable and the setters;
    • test_casts_show_the_simple_strings_of_their_type_objects_as_pysparks (test_dropin_types_columns_offline.py).
  • E2E tests against a Spark 4.3.0 server. They pass on pyspark-client 4.3.0 and fail on master:

    • test_uses_of_the_json_of_a_type_of_a_user_defined_type_of_no_json_raise_its_error;
    • test_schemas_of_the_server_make_new_user_defined_type_objects_as_pyspark.

    test_maps_with_user_defined_keys_collect_their_deserialized_keys_as_pyspark defines its UDT at module level, which fromJson imports, so that it passes on pyspark-client 4.3.0 too.

  • Rust tests:

    • simple_string_of_the_ml_udts_matches_pyspark (types_behavior.rs);
    • cast_shown_renders_its_shown_type_and_sends_its_type (core column.rs).
  • Probes against the same server, of both clients, compared line by line. All match, but for createDataFrame of local rows or a pandas DataFrame of a UDT of no JSON, which is left for later:

    • 417 container operations of three UDTs of no JSON;
    • the uses of their JSON;
    • the schemas of the server;
    • the cast reprs of 10 types.
  • The gates below ran on the branch merged with master at 7ddb02d, with the commit of the stacked GC PR on top of this one's.

  • The Rust gates:

    • cargo fmt --all --check;
    • cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;
    • RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;
    • cargo test -p pyspark-rs --lib as rust.yml runs it: 15 passed;
    • cargo test -p apache-spark-connect -p apache-spark-connect-core --release as rust.yml runs it: 1706 passed, 18 ignored.
  • The Python gates:

    • ruff 0.14.0 check and format --check, test_ruff_excludes.py and scripts/audit_no_stubs.sh;
    • gen_type_stubs.py --pyspark-src <v4.3.0>/python --check and gen_fn_dispatch.py --check;
    • test_type_stubs_offline.py with mypy 2.4.1: 12 passed;
    • the guards of SPARK-61851.
  • The offline suite of coverage.yml: 6446 passed and 77 skipped. The skips are those of master: 68 for "not a public constructor", 8 that need SPARK_REMOTE, and test_type_stubs_offline.py, which needs mypy and ran above.

  • test_dropin_e2e.py against a Spark 4.3.0 server: 328 passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 5.5

This pull request and its description were written by Isaac.

…park does: keep them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString()

PySpark's CastExpression.__repr__ shows the simpleString() of the type in upper case
(python/pyspark/sql/connect/expressions.py:1004-1016). The VectorUDT and MatrixUDT of
pyspark.mllib.linalg have the simple strings "vector" and "matrix", as those of pyspark.ml.linalg.
The core's DataType::simple_string named only the pyspark.ml.linalg classes (SPARK-61690), so the
drop-in's F.col("x").cast(pyspark.mllib.linalg.VectorUDT()) showed Column<'CAST(x AS UDT)'>,
where PySpark shows Column<'CAST(x AS VECTOR)'>. The same held in an array, a map and a struct,
and for try_cast.

The core now names the two mllib classes too. Another user-defined type keeps "udt", which is
PySpark's UserDefinedType.simpleString() unless the class has its own.

PySpark's ArrayType, MapType, StructField and StructType keep the type objects that they are given
as they are, and read one only for a use that needs its JSON (json(), a plan) or its conversion
(toInternal). So ArrayType(UserDefinedType()) is made, with the repr
ArrayType(UserDefinedType(), True), as are the types of a UserDefinedType whose module() or
sqlType() is not implemented. Their json() raises [NOT_IMPLEMENTED] module() is not implemented.
The drop-in lowered the child into a core type when it made the type, which calls jsonValue(), so
the constructors, add and the setters raised that error.

Now the child is lowered leniently when it is made (types::lenient_type). A type that does not
lower is kept as its object, with the null type standing for it in the snapshot, and an array, a
map or a struct of one stands as the types that it holds. A use that needs the JSON reads the
object again through KeptType::live, so it raises the type's error then, as PySpark's does. The
uses that PySpark answers from the objects fall back to them when the strict lowering fails:
- simpleString, needConversion and the hash;
- == and !=, by the attributes, as PySpark's __dict__ comparison;
- pickles, copies and deep copies, of the objects;
- toNullable;
- the tree string, which names a stand-in by its object's typeName();
- StructType's names and add.

DataFrame.to of such a schema raises the type's error at each request, where it raised
UNSUPPORTED_OPERATION of the schema (dataframe.rs).

PySpark converts the types of the server with UserDefinedType.fromJson, which makes a new object
of the class of each user-defined type, UDT() (python/pyspark/sql/types.py:1986-2005). So
df.schema["p"].dataType is a new PtUDT(), not the object that the schema of createDataFrame held,
nor of its attributes, and a class whose __init__ takes arguments raises
TypeError: __init__() missing 1 required positional argument for df.schema, df.dtypes and
df.collect(). The drop-in made the types of the server of a process-wide registry of the objects
that the client lowered, by their JSON (data_type_to_py). So:
- df.schema["p"].dataType was the user's object, with its attributes;
- lowering an unrelated ArrayType(PtUDT()) later changed the df.schema of an untouched DataFrame;
- a class of arguments did not raise, and its rows were deserialized by the user's object.

Now types::schema_to_py converts the types of the server. Each user-defined type is a new object
of its class, of the parser's fromJson, which the array, the map, the field and the struct that
hold it keep, without lowering them again. It is used for:
- the schema of a DataFrame and its dtypes;
- the schemas of the rows of collect, toLocalIterator, head and take;
- the schemas of toPandas and _to_table;
- the types of _data_type_of_proto.

The registry stays for the types that the client makes of core types (SPARK-61917).

DataFrame.to now keeps the struct that it is given as its cached schema, as PySpark's
res._cached_schema = schema, where it kept only a schema that is not a struct. So schema, dtypes,
columns and printSchema read the object as it is then, of the attributes of the user-defined types
that it holds, and a later edit of the struct shows in them, as in PySpark.
test_schema_cache_to_uses_supplied_struct_schema now expects PySpark's behavior.
test_maps_with_user_defined_keys_collect_their_deserialized_keys_as_pyspark defines its UDT at
module level, which fromJson imports, so that it passes on pyspark-client 4.3.0 too.

docs/compatibility.md says so, where it said that a struct of fromJson holds the UDT last made,
which neither client does.

PySpark's CastExpression.__repr__ shows self._data_type.simpleString().upper() of the type object
(python/pyspark/sql/connect/expressions.py:1004-1016). The core renders a cast by the simple string
of its core type, which has no Python class. So the drop-in showed:
- UDT for a user-defined type of its own simpleString(): CAST(x AS UDT) where PySpark shows
  CAST(x AS SCALE2), and STRUCT<A:UDT,B:UDT> for STRUCT<A:SCALE2,B:SCALE10>;
- the simple string of the class's core type for a type of a Python subclass: CAST(id AS BOOLEAN)
  where PySpark shows CAST(id AS SUB).

The core's CastTarget has a hidden Shown(DataType, String) target, which the plan sends as the
type and the render shows as the string, and Column::cast_shown makes it. cast_column of
pyspark-rs uses it with the simpleString() of the type object when the type holds a user-defined
type or a type of a Python subclass (types::shown_simple_string); other casts are unchanged.

A cast to a type that does not lower now, such as ArrayType(UserDefinedType()), is made, renders
its simpleString(), and raises the error of its JSON at each request, as PySpark's does; it raised
that error at the call.

The known difference of the cast repr of a type of a subclass is removed, and docs/compatibility.md
says so.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant