Repository navigation
[SPARK-62016][CONNECT][PYTHON] Treat user-defined type objects as PySpark does: keep them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString() - #865
Closed
HyukjinKwon wants to merge 1 commit into
Closed
HyukjinKwon wants to merge 1 commit into
HyukjinKwon wants to merge 1 commit into
Conversation
…park does: keep them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString()
PySpark's CastExpression.__repr__ shows the simpleString() of the type in upper case
(python/pyspark/sql/connect/expressions.py:1004-1016). The VectorUDT and MatrixUDT of
pyspark.mllib.linalg have the simple strings "vector" and "matrix", as those of pyspark.ml.linalg.
The core's DataType::simple_string named only the pyspark.ml.linalg classes (SPARK-61690), so the
drop-in's F.col("x").cast(pyspark.mllib.linalg.VectorUDT()) showed Column<'CAST(x AS UDT)'>,
where PySpark shows Column<'CAST(x AS VECTOR)'>. The same held in an array, a map and a struct,
and for try_cast.
The core now names the two mllib classes too. Another user-defined type keeps "udt", which is
PySpark's UserDefinedType.simpleString() unless the class has its own.
PySpark's ArrayType, MapType, StructField and StructType keep the type objects that they are given
as they are, and read one only for a use that needs its JSON (json(), a plan) or its conversion
(toInternal). So ArrayType(UserDefinedType()) is made, with the repr
ArrayType(UserDefinedType(), True), as are the types of a UserDefinedType whose module() or
sqlType() is not implemented. Their json() raises [NOT_IMPLEMENTED] module() is not implemented.
The drop-in lowered the child into a core type when it made the type, which calls jsonValue(), so
the constructors, add and the setters raised that error.
Now the child is lowered leniently when it is made (types::lenient_type). A type that does not
lower is kept as its object, with the null type standing for it in the snapshot, and an array, a
map or a struct of one stands as the types that it holds. A use that needs the JSON reads the
object again through KeptType::live, so it raises the type's error then, as PySpark's does. The
uses that PySpark answers from the objects fall back to them when the strict lowering fails:
- simpleString, needConversion and the hash;
- == and !=, by the attributes, as PySpark's __dict__ comparison;
- pickles, copies and deep copies, of the objects;
- toNullable;
- the tree string, which names a stand-in by its object's typeName();
- StructType's names and add.
DataFrame.to of such a schema raises the type's error at each request, where it raised
UNSUPPORTED_OPERATION of the schema (dataframe.rs).
PySpark converts the types of the server with UserDefinedType.fromJson, which makes a new object
of the class of each user-defined type, UDT() (python/pyspark/sql/types.py:1986-2005). So
df.schema["p"].dataType is a new PtUDT(), not the object that the schema of createDataFrame held,
nor of its attributes, and a class whose __init__ takes arguments raises
TypeError: __init__() missing 1 required positional argument for df.schema, df.dtypes and
df.collect(). The drop-in made the types of the server of a process-wide registry of the objects
that the client lowered, by their JSON (data_type_to_py). So:
- df.schema["p"].dataType was the user's object, with its attributes;
- lowering an unrelated ArrayType(PtUDT()) later changed the df.schema of an untouched DataFrame;
- a class of arguments did not raise, and its rows were deserialized by the user's object.
Now types::schema_to_py converts the types of the server. Each user-defined type is a new object
of its class, of the parser's fromJson, which the array, the map, the field and the struct that
hold it keep, without lowering them again. It is used for:
- the schema of a DataFrame and its dtypes;
- the schemas of the rows of collect, toLocalIterator, head and take;
- the schemas of toPandas and _to_table;
- the types of _data_type_of_proto.
The registry stays for the types that the client makes of core types (SPARK-61917).
DataFrame.to now keeps the struct that it is given as its cached schema, as PySpark's
res._cached_schema = schema, where it kept only a schema that is not a struct. So schema, dtypes,
columns and printSchema read the object as it is then, of the attributes of the user-defined types
that it holds, and a later edit of the struct shows in them, as in PySpark.
test_schema_cache_to_uses_supplied_struct_schema now expects PySpark's behavior.
test_maps_with_user_defined_keys_collect_their_deserialized_keys_as_pyspark defines its UDT at
module level, which fromJson imports, so that it passes on pyspark-client 4.3.0 too.
docs/compatibility.md says so, where it said that a struct of fromJson holds the UDT last made,
which neither client does.
PySpark's CastExpression.__repr__ shows self._data_type.simpleString().upper() of the type object
(python/pyspark/sql/connect/expressions.py:1004-1016). The core renders a cast by the simple string
of its core type, which has no Python class. So the drop-in showed:
- UDT for a user-defined type of its own simpleString(): CAST(x AS UDT) where PySpark shows
CAST(x AS SCALE2), and STRUCT<A:UDT,B:UDT> for STRUCT<A:SCALE2,B:SCALE10>;
- the simple string of the class's core type for a type of a Python subclass: CAST(id AS BOOLEAN)
where PySpark shows CAST(id AS SUB).
The core's CastTarget has a hidden Shown(DataType, String) target, which the plan sends as the
type and the render shows as the string, and Column::cast_shown makes it. cast_column of
pyspark-rs uses it with the simpleString() of the type object when the type holds a user-defined
type or a type of a Python subclass (types::shown_simple_string); other casts are unchanged.
A cast to a type that does not lower now, such as ArrayType(UserDefinedType()), is made, renders
its simpleString(), and raises the error of its JSON at each request, as PySpark's does; it raised
that error at the call.
The known difference of the cast repr of a type of a subclass is removed, and docs/compatibility.md
says so.
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Four changes, each in its own commit, that make the drop-in treat user-defined type (UDT) objects as pyspark-client 4.3.0 does. A last commit renames a helper that
scripts/audit_no_stubs.shrefused. With SPARK-61999 (#848), the toNullable of an array, a map or a struct isas_nullableof the objects, and a type of metadata or user attributes pickles of them, which keeps the UDT objects of part 2 too. With SPARK-62004 (#849), the schema that a copy or an unpickle of a DataFrame requested is converted as a schema of the server (part 3).1. The simple strings of the mllib vector and matrix types (core
DataType::simple_string). TheVectorUDTandMatrixUDTofpyspark.mllib.linalgare namedvectorandmatrix, as those ofpyspark.ml.linalgalready were (SPARK-61690). A cast to them showsVECTORandMATRIX, as PySpark'sCastExpression.__repr__does. Other UDTs keepudt, which is PySpark'sUserDefinedType.simpleString().2. A UDT object is kept in the types that hold it until a use needs its JSON (
types.rs). PySpark'sArrayType,MapType,StructFieldandStructTypekeep the type objects that they are given as they are, and read one only for a use that needs its JSON or its conversion. The drop-in lowered the child into a core type when it made the type, which callsjsonValue(). So the type of a UDT whosemodule()orsqlType()is not implemented, such asUserDefinedType()itself, raisedNOT_IMPLEMENTEDwhen it was made, by its constructor,addor a setter.lenient_type). A type that does not lower is kept as its object, with the null type standing for it in the snapshot.KeptType::live, so it raises then, as PySpark's does:json(), the plans of a cast,DataFrame.to,from_json, a table, a Java UDF, a UDTF, a reader's schema, and data of no rows.simpleStringandneedConversion;==, by the attributes, as PySpark's__dict__comparison;toNullable;typeName();namesandadd.DataFrame.toof such a schema raises the type's error at each request, where it raisedUNSUPPORTED_OPERATIONof the schema.3. The schemas of the server make new UDT objects (
types::schema_to_py). PySpark converts the types of the server withUserDefinedType.fromJson, which makes a new object of the class,UDT()(python/pyspark/sql/types.py:1986-2005). The drop-in made them from a process-wide registry of the objects that the client lowered, by their JSON. So:df.schema["p"].dataTypewas the user's own object, with its attributes;__init__takes arguments did not raise, where PySpark'sdf.schema,df.dtypesanddf.collect()raiseTypeError.Now the schema of a DataFrame and its
dtypes, the schemas of the rows thatcollect,toLocalIterator,headandtakeconvert, those oftoPandasand_to_table, and_data_type_of_protomake each UDT a new object of its class. The array, map, field or struct that holds it keeps that object. The registry stays for the types that the client makes of core types (SPARK-61917).DataFrame.tokeeps the struct that it is given as its cached schema, as SPARK-61962 made it, so itsschemaholds copies of the user's UDT objects, as PySpark's.4. A cast renders the
simpleString()of its type object (coreCastTarget::Shown,Column::cast_shown; pyspark-rscast_column). PySpark'sCastExpression.__repr__showssimpleString().upper()of the type object. The core rendered the simple string of its core type, so:simpleString()showedUDT, where PySpark showsCAST(x AS SCALE2)andSTRUCT<A:SCALE2,B:SCALE10>;CAST(id AS BOOLEAN)where PySpark showsCAST(id AS SUB)), a known difference that this removes.The core has a hidden cast target that sends the type and renders a given string. The drop-in uses it with the object's
simpleString()when the type holds a UDT or a type of a Python subclass. A cast to a type that does not lower now is made, renders itssimpleString(), and raises at each request, as PySpark's.Why are the changes needed?
So that UDT objects behave as in pyspark-client 4.3.0:
Does this PR introduce any user-facing change?
Yes. With
u = PtUDT(); u.marker = 1, adfofcreateDataFrameof a schema ofu, aScaleUDT(2)whosesimpleString()isscale2, and anArgUDTwhose__init__takes a scale:ArrayType(UserDefinedType())ArrayType(UserDefinedType(), True); itsjson()raises[NOT_IMPLEMENTED] module() is not implemented.df.schema["p"].dataType is uFalse, a newPtUDT()without the attributeTruedf.schemaof a schema ofArgUDT(2)TypeError: ... missing 1 required positional argument: 'scale'ArgUDT(2)F.col("x").cast(ScaleUDT(2))Column<'CAST(x AS SCALE2)'>Column<'CAST(x AS UDT)'>F.col("x").cast(pyspark.mllib.linalg.VectorUDT())Column<'CAST(x AS VECTOR)'>Column<'CAST(x AS UDT)'>How was this patch tested?
Offline tests, of official PySpark 4.3.0's strings and errors. Their bodies pass on pyspark-client 4.3.0, but for the requests to the mock server, and they fail on master:
test_casts_to_the_linalg_types_show_their_simple_strings_as_pyspark(test_dropin_offline.py);test_types_of_a_user_defined_type_of_no_json_keep_it_as_pysparks(test_dropin_types_columns_offline.py), of three such UDTs and seven containers: reprs,simpleString,needConversion,==, hash, copies, pickles,json(),toInternal, the tree string,add,toNullableand the setters;test_casts_show_the_simple_strings_of_their_type_objects_as_pysparks(test_dropin_types_columns_offline.py).E2E tests against a Spark 4.3.0 server. They pass on pyspark-client 4.3.0 and fail on master:
test_uses_of_the_json_of_a_type_of_a_user_defined_type_of_no_json_raise_its_error;test_schemas_of_the_server_make_new_user_defined_type_objects_as_pyspark.test_maps_with_user_defined_keys_collect_their_deserialized_keys_as_pysparkdefines its UDT at module level, whichfromJsonimports, so that it passes on pyspark-client 4.3.0 too.Rust tests:
simple_string_of_the_ml_udts_matches_pyspark(types_behavior.rs);cast_shown_renders_its_shown_type_and_sends_its_type(corecolumn.rs).Probes against the same server, of both clients, compared line by line. All match, but for
createDataFrameof local rows or a pandas DataFrame of a UDT of no JSON, which is left for later:The gates below ran on the branch merged with master at 7ddb02d, with the commit of the stacked GC PR on top of this one's.
The Rust gates:
cargo fmt --all --check;cargo +1.100.0 clippy -p apache-spark-connect-core -p apache-spark-connect -p pyspark-rs --all-targets -- -D warnings;RUSTDOCFLAGS="-D warnings" cargo doc --no-deps -p apache-spark-connect -p apache-spark-connect-core;cargo test -p pyspark-rs --libasrust.ymlruns it: 15 passed;cargo test -p apache-spark-connect -p apache-spark-connect-core --releaseasrust.ymlruns it: 1706 passed, 18 ignored.The Python gates:
checkandformat --check,test_ruff_excludes.pyandscripts/audit_no_stubs.sh;gen_type_stubs.py --pyspark-src <v4.3.0>/python --checkandgen_fn_dispatch.py --check;test_type_stubs_offline.pywith mypy 2.4.1: 12 passed;The offline suite of
coverage.yml: 6446 passed and 77 skipped. The skips are those of master: 68 for "not a public constructor", 8 that needSPARK_REMOTE, andtest_type_stubs_offline.py, which needs mypy and ran above.test_dropin_e2e.pyagainst a Spark 4.3.0 server: 328 passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.