Skip to content

feat(spark): support map/array input and an options argument in to_json - #395

Open
cj-zhukov wants to merge 7 commits into
apache:mainfrom
cj-zhukov:cj-zhukov/to_json-support-map-array-input
Open

cj-zhukov wants to merge 7 commits into
apache:mainfrom
cj-zhukov:cj-zhukov/to_json-support-map-array-input

Conversation

@cj-zhukov

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Spark's to_json accepts a map or an array as the value to serialize, and an optional map(...) of options as a second argument. Both were rejected at planning time: to_json(map('a', 1)) failed with "expects a struct argument", and a second argument failed with "but received 2".

What changes are included in this PR?

  • to_json accepts a struct, a map, any Arrow list type, or an untyped NULL. The encoder already serialized maps and lists when nested, so only the planning check changed; the root type gets the same validation as nested types.
  • to_json(expr, options): the second argument is a constant map of strings, read as Spark's JSONOptions does (case-insensitive names, true/false in any case).
  • The supported options are an enum, ToJsonOption, named after Spark's JSON data source options
    (https://spark.apache.org/docs/latest/sql-data-sources-json.html#data-source-option). This PR implements one, ignoreNullFields. Any other option is an error that lists the supported ones.
  • The private helper struct_to_json is renamed to_json, matching feat: add to_json scalar function datafusion#27518.
  • The function reference is updated.

Are these changes tested?

Yes.

  • Unit tests in to_json.rs for the planning check, map/list input as a column and as a scalar (NULL rows, sliced input, LargeList, FixedSizeList), option parsing, and ignoreNullFields.
  • New records in json/to_json.slt, written without [...] literals so that Spark can parse them.
  • Expected values were checked against Spark 4.2.0 with the PySpark runner in crates/sqllogictest/scripts. It found one difference, to_json(NULL), which Spark answers with NULL; this PR matches that.

Known differences from Spark, marked # xfail spark: intended in the SLT:

  • an option that is not supported is an error, while Spark applies it or ignores a name it does not know;
  • an interval element and an array map key are rejected, as they already are inside a struct.

Are there any user-facing changes?

  • to_json accepts map and array input and the ignoreNullFields option.
  • to_json(NULL) returns NULL instead of a planning error.
  • The rejection message now reads "to_json expects a struct, map or array
    argument".

No change to the public API: expr_fn::to_json(input) still takes one argument.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant