Repository navigation
[SPARK-60101][SQL] Support native data sources implemented in Rust or C++ - #59313
HyukjinKwon wants to merge 15 commits into
Conversation
…(Rust, C++) data sources This adds the columnar data source API, a simpler way to plug a data source into Spark than implementing the Data Source V2 interfaces directly. It follows the Python Data Source API, but data is exchanged only as columnar batches: readers return ColumnarBatches, and writers receive Arrow-backed ColumnarBatches. Spark adapts a data source to Data Source V2, with batch and micro-batch reads, batch and streaming writes, and predicate, column and limit pushdown. A data source is implemented either on the JVM, with the interfaces of org.apache.spark.sql.datasource and a DataSourceProvider, or in native code such as Rust or C++, as a shared library that implements the native methods of NativeBridge as JNI functions and exchanges data through the Arrow C data interface. Spark loads each library into its own copy of NativeBridge, so libraries exporting the same functions can be loaded side by side, and Spark itself ships no native code. A native data source is distributed as a native data source package, a zip file with the extension .sparkpkg that contains a manifest and the library for each platform. Packages are added with spark.addArtifact, in Scala and Python, classic and Spark Connect, or found under spark.sql.dataSource.native.paths; executors get them through the artifacts of the session. The change also adds the spark.sql.dataSource.native.enabled and spark.sql.dataSource.native.paths configurations, the native data source error conditions, the arrow-c-data dependency, and a documentation page with examples in Scala, Rust and C++. Co-authored-by: Isaac <no-reply@databricks.com>
Downstream data sources are only native, so the JVM interfaces of the columnar data source API (DataSource, DataSourceReader, DataSourceWriter, DataSourceStreamReader, DataSourceStreamWriter and DataSourceProvider) are not public anymore. They become the internal ColumnarDataSource traits that the Data Source V2 adapter and the native data sources share, and the Javadoc of NativeBridge now specifies the whole interface on its own. The documentation page is renamed to "Native Data Sources". Co-authored-by: Isaac <no-reply@databricks.com>
A table created with CREATE TABLE ... USING ... OPTIONS (...) passes its options to TableProvider.getTable, while its scans and writes only get the options of their query. The columnar table now merges the options of the table under the options of each scan and write, so that SELECT and INSERT INTO on such a table use them. Co-authored-by: Isaac <no-reply@databricks.com>
…cally Like the Python data sources installed in the Python path, the native data source packages in the native-datasources directory of SPARK_HOME are now found without any configuration. The packages of a session, added with spark.addArtifact or under spark.sql.dataSource.native.paths, take precedence over the installed ones, like the Python data sources registered at runtime take precedence over the installed ones. Co-authored-by: Isaac <no-reply@databricks.com>
- Exclude the `org.apache.spark.sql.datasource` package, which has no
counterpart in the Spark Connect client, from the client compatibility
check, like the other packages specific to classic Spark.
- Escape the Rust and C++ examples of the native data sources page from
Liquid, which failed to parse `{{0, end / 2}, ...}` in the C++ example.
Co-authored-by: Isaac <no-reply@databricks.com>
Instead of the native-datasources directory of SPARK_HOME, find the library of a native data source where native libraries are usually installed, by its file name: the library spark_datasource_<name>, such as libspark_datasource_<name>.so on Linux, implements the data source <name>. Spark looks for it in the native library path: the directories of java.library.path, which include LD_LIBRARY_PATH on Linux, and then the lib directory of each installation prefix in PATH, such as /usr/local/lib for /usr/local/bin. The executors load the library from the same path as the driver, or else find it in their own native library path. The packages of a session still take precedence. Also: - Keep the error of a library that is loaded but rejected, because it cannot be loaded again in another class loader: using it again failed with "already loaded in another classloader". - Add NativeDataSourceConnectSuite, which tests native data sources with a Spark Connect client and server, and move the helpers that build the test library to NativeDataSourceTestUtils. Co-authored-by: Isaac <no-reply@databricks.com>
The Scala linter checks the format of the Spark Connect modules with scalafmt. Co-authored-by: Isaac <no-reply@databricks.com>
|
Just out of curiosity, was the FFM API considered instead of JNI? |
|
@pan3793 Good question. Yes, but FFM cannot be the only bridge yet because of the Java baseline. Spark compiles with FFM would fit this design well, though, and the design leaves room for it:
|
I suppose we can raise our JDK baseline in the upcoming 5.0.0 (early 2027)? And this can also be a 25-specific feature before that happens, with the decoupled Connect client-server architecture, upgrading server-side JDK version should be much easier.
Yeah, this is the biggest advantage in my mind, non-JVM developers see a clean C header file without needing to learn JNI stuff. |
|
I took a look about FFM (and did prototype). I think it's sort of good and bad. For example, we have to release header file together whereas JNI one does not require it. BTW, we will release 4.4.0 in Dec. and 4.5.0 around Mar. since we will do the quarterly release. I think we can keep JNI version, and do FFM one later around Spark 5.0.0. |
for JNI, we define the interface via a Java class ( I suppose, from the perspective of the native datasource connector developer, the latter should be more friendly (hygienic, and save one step to generate it from Java). If we choose to define a header file as the interface directly, we can:
and once we raise our JDK baseline to 25+, we can get rid of the JNI glue code, the ABI keeps no change from the beginning. |
|
Yeah. Let's actually do that. We can deprecate this in 4.5.0, and drop it in 5.0.0 with replacing it to FFM. I am thinking about bringing more native code into Spark 5 in fact .. |
There was a problem hiding this comment.
Thank you for this proposal, @HyukjinKwon. This is great! I reviewed the lookup, distribution and lifecycle of the native data sources, and left 10 inline comments, summarized below in the same order.
Main issues
- One invalid package breaks every lookup miss (
NativeDataSourceRegistry.scalaL65) - Streaming tasks look up the package under the wrong session UUID (
NativeDataSourceRegistry.scalaL168)
Other issues
- Classic
addArtifact(..., file=True)doesn't register the package (session.pyL2299) - Spark comparison semantics (NaN, collation) for pushed predicates (
NativePredicates.scalaL79) - Artifact dedup by file name can ship a different package (
NativeDataSourceRegistry.scalaL152) - The stream writer handle is never closed explicitly (
NativeDataSource.scalaL294) - A failed
commitDataWriterskips the task abort (NativeDataSource.scalaL484) - The registry lookup runs twice for each resolution (
NativeDataSourceV2.scalaL46) - Installed native libraries now block Python data source registration (
DataSource.scalaL686) - The scan description shows the class name instead of the data source name (
ColumnarTable.scalaL148)
The first two items seem to be the most important: one invalid package breaks unrelated lookups, and streaming queries on a cluster with isolated sessions would not find the package on the executors.
…d lifecycle - Skip and log invalid packages during the lookup, unless the manifest of the invalid package lists the requested data source. - Distribute a package when its reader or writer is created, in the active session, which is the cloned session of a streaming query; executors also check the copies in the files of the other sessions by checksum. - Route .sparkpkg files to the JVM in classic addArtifacts regardless of the flags, also in mixed lists, as Spark Connect does. - Do not push predicates on columns with non-binary collated strings, and document the Spark SQL semantics (NaN, -0.0, nulls) of fully evaluated predicates in NativeBridge. - Report a clear conflict when a different package with the same file name is already an artifact of the session. - Close the previous stream writer of a streaming query when it creates a new one, and document the stream writer lifecycle. - Keep the data writer handle when commitDataWriter fails, so that Spark aborts it with abortDataWriter. - Reuse the lookup of DataSource.lookupDataSource in NativeDataSourceV2. - Skip native data sources when registering a Python data source, which takes precedence over them. - Describe the scan by the name of the data source. Co-authored-by: Isaac <no-reply@databricks.com>
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you for addressing all the comments, @HyukjinKwon. I checked 9dd4f88 against each thread, and all 10 points are handled. I left 4 more minor, non-blocking comments:
- An invalid package that lists the name hides a valid one (
NativeDataSourceRegistry.scalaL88) - The thread-local can keep a stale lookup on pooled threads (
NativeDataSourceRegistry.scalaL70) - The package cache is never evicted (
NativeDataSourcePackage.scalaL102) - No end-to-end test for closing the previous stream writer (
NativeDataSource.scalaL565)
sarutak
left a comment
There was a problem hiding this comment.
I went over the lookup/distribution and the reader/writer lifecycle again on 9dd4f88. The fixes for the earlier round all check out in the code, and the overall design (per-library BridgeClassLoader for side-by-side loading, NativeHandle backed by AtomicLong plus Cleaner, Arrow struct release/close on every path, tryWithSafeFinally around the native calls) is solid.
I left four inline comments, none blocking:
distributecollapses "no active session" and "local mode" into one branch, so a missing active session would silently skip distribution rather than fail clearly. I could not find a planning path that actually has no active session, so this is defensive hardening.- The batch writer's
commitcloses the handle outsidetryWithSafeFinally, unlikeabort; iflibrary.committhrows, the driver-side handle falls back to theCleaner. One-line symmetry fix. spark.sql.dataSource.native.enableddefaults totrue: a note on whether opt-in is preferable for the first release of a feature that loads native code (nit).NativeLibrary.loadusessetAccessibleacross class loaders: a comment on why would help future readers (nit).
The last two are nits. Thanks for the very thorough design and docs on this.
…m writers - Prefer a usable package or an installed library over an invalid package or a package of an unsupported interface version that lists the same data source, and only report those when nothing else provides it. - Clear the lookup kept for the provider at the start of every DataSource.lookupDataSource, so a provider never takes an earlier lookup. - Bound the cache of read packages, and forget the packages of a session when its artifacts are cleaned up. - Record writer closes in the test library and check end to end that each micro-batch closes the stream writer of the previous one, and that each micro-batch creates its stream writer with the ID of the query. Co-authored-by: Isaac <no-reply@databricks.com>
…batching - Resolve the provider of classic DataFrameWriter commands and of DataStreamWriter sinks with the session of the Dataset active, so that the lookups of Python and native data sources use that session. - Distribute a package that is not an artifact of the session as a copy named after its file name and checksum, which the lookups do not count as a package of the session: a configured package can be replaced in place, and packages with the same file name no longer collide. Remove the then unreachable NATIVE_DATA_SOURCE_PACKAGE_FILE_NAME_CONFLICT error. - Log a warning when a package is distributed without an active session. - Skip, and log, a configured directory that cannot be read. - Compare the data with the read schema as it round-trips through Arrow, which does not carry collations, interval fields and user-defined types. - Also end the write batches at spark.sql.execution.arrow.maxBytesPerBatch. - Document that a failed batch commit is aborted, which closes the writer, why NativeBridge.load is called reflectively, and the security of discovering libraries by file name. Co-authored-by: Isaac <no-reply@databricks.com>
|
One small thing before merge: could you update the PR description to mention
Since 9ac5a49, batches are also bounded by |
|
BTW, I plan to leave this open a while. I want others to take a look too :-). |
|
@sarutak Thanks, updated. The Internals section now says writes convert rows to Arrow batches "that end at While there, I went through the rest of the description against 9ac5a49 and updated the other parts that changed during the review:
|
LuciferYang
left a comment
There was a problem hiding this comment.
I went through 9ac5a49 again. The fixes from the earlier rounds hold up in the code. Four more inline comments, each checked on 9ac5a49 with a small test suite: Arrow Java for the first two, and a JVM ColumnarDataSource for the last two.
The first one is the one I'd like fixed before merge: when a library returns a sliced Arrow array, Spark reads the wrong rows without any error. The other three are places where checkSchema lets through a layout that ArrowColumnVector can't read, or where NativeBridge promises something Spark doesn't do (calling abort when a write fails, and calling pushLimit at most once).
…h limits once - Read the Arrow stream of a native library array by array, and reject an array with a non-zero offset at any level, which Arrow Java would read from the start of its buffers, with a NATIVE_DATA_SOURCE_ERROR that names the column. - Name the fields of the entries of maps `key` and `value` before creating the vectors, as ArrowColumnVector looks them up by name, so that maps built by arrow-rs can be read; reject decimal256, and report any other column that ArrowColumnVector cannot read with its name. - Create the writer of a batch or streaming write with its writer factory, after the stages that the write depends on ran, so that Spark commits or aborts each writer it creates, also when such a stage fails with adaptive execution. - Ask the reader for a limit only once, as NativeBridge documents, although Spark pushes the limit of LIMIT ... OFFSET twice. - Document these in NativeBridge and the native data sources page. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
@cloud-fan Thanks for the thorough review. I addressed all the inline comments in aa0f3ba (and e3ff80c, a test), and replied to each. On the findings of the summary: Six
Five guards without tests, all added:
PR description: both sentences are updated. The logging one is accurate now, since Spark also logs the packages of an unsupported version. Should version 1 be a C ABI? That is #59424, which I opened for comparison. It is the same feature with a C header as the interface: Could a failed load of Arrow's JNI library look like an unsupported operation? Yes. In Arrow 19, |
|
@LuciferYang Thanks for running the suites again. On memory, aa0f3ba adds a Memory section to the docs page. It says that the memory of a library, the Arrow buffers that it exports and the batches that Spark writes are not accounted by Spark, and that this memory comes out of @tokoko Thanks, that's an interesting direction for the C ABI of #59424. Version 1 needs a few things that ADBC does not have today: streaming reads with offsets, the protocol of distributed writes (task commits with messages, aborts, and epochs for streaming writes), and the pushdown semantics of Spark. With the ingest API that you are working on (apache/arrow-adbc#4317), building on ADBC with extensions where needed could be worth exploring for that interface, so that one driver serves several engines. @gengliangwang @gerashegalov Thanks, all addressed in aa0f3ba, with replies inline. |
Mirrors aa0f3ba and e3ff80c of apache#59313, in the terms of spark_datasource.h: - Spark evaluates all the pushed predicates again after reading. The header states the rule for skipping data, and the jar contains predicate-cases.json, generated by a test from Spark's own evaluation. The JSON of the predicates is ASCII. - spark_ds_create_stream_writer gets the ID of the streaming query. - Offsets must be compact JSON on a single line. - An installed library replaced in place fails with INVALID_NATIVE_DATA_SOURCE_LIBRARY.REPLACED, and the caches identify files with their file key too. - Concurrent distributions of a configured package are serialized, a scan planned again after a failure fails with the same error, unsupported Arrow types and the link errors of Arrow are reported as NATIVE_DATA_SOURCE_ERROR, the column vectors of a read are not freeable between batches, packages of an unsupported version are logged, and PySpark addArtifact only passes file: URIs as they are. - Docs: the writer lifecycle, close functions, Arrow variants, a Memory section, broadcasts with AQE, replaced libraries, and the addArtifact Scaladoc. NativeLibrary already rejected NULL handles and missing offsets, so their tests use its errors. Co-authored-by: Isaac <no-reply@databricks.com>
|
Pushdown semantics has come up quite a few times (e.g. @SamArch27 evaluating how to do pushdown for DuckDB-ADBC). I think it would require something like Substrait so that semantics are consistent across engines and drivers, with each driver supporting a limited subset (e.g. table scans with filters) and translating to SQL if needed. ADBC does do streaming reads, and some drivers can even expose a read as a set of partitions that can be distributed across workers. As for reads with offsets, that could perhaps be done via Substrait as well, if we go that direction. |
|
Also note that @CurtHagenlocher has been thinking about how to support pluggable filesystems and file type drivers (e.g. Excel files over S3) in the ADBC API, which may be relevant here as well, as it would reuse the ADBC ABI. |
|
(let me turn this as a draft, and make #59424 non-draft. per the review comments, I think we should go that way) |
|
@lidavidm Thanks for the context. You're right that ADBC has streaming result sets and partitioned execution ( Substrait for the pushdown would also address the semantics concerns that @tdcmeehan raised here, with each driver supporting a declared subset. For now, Spark evaluates every pushed predicate again after reading, so a library only uses them to skip data. The pluggable filesystems and file type drivers that @CurtHagenlocher is looking at sound relevant too, for a C ABI such as the one in #59424. |
|
Thanks for this, it's a really nice design. Since the PR is still a draft, I'd like to raise a use case from Apache Gluten, the native execution engine for Spark built on Velox. Other native engines such as Comet (DataFusion) are in the same position. Why it matters. Native data sources and native execution engines fit together well. The library reads the data, and the engine runs the rest of the query natively. Today, a query on a native data source with Gluten enabled looks like this: That's two extra copies, so turning on a native engine can make these queries slower. With a small hook, the data could stay columnar in native memory from the source to the shuffle: No C ABI change is needed for this. The JNI Two small requests:
Optional, could be a follow-up: a columnar write entry point (the internal If this direction sounds reasonable, I'm happy to help review. |
|
|
@LuciferYang Thanks, moved to #59424 |
|
I have two suggestions on the DSv2 adapter:
|
|
@HyukjinKwon I'm comparing the implementation vs. what we have done in Gluten. Claude reported a bug when it work with a pyarrow UDF. FYI, @viirya import pyarrow as pa
from pyspark.sql import SparkSession
from pyspark.sql.functions import arrow_udf, col, assert_true, concat_ws, lit
from pyspark.sql.types import LongType
@arrow_udf(LongType())
def ident(v: pa.Array) -> pa.Array:
return v
spark = (SparkSession.builder.master("local[1]")
.config("spark.driver.extraLibraryPath", "/tmp/nds")
.getOrCreate())
def df(end, batch_size):
return (spark.read.format("native_range")
.option("end", str(end)).option("partitions", "1")
.option("batch_size", str(batch_size)).load()
.select("id")
.withColumn("uid", ident(col("id"))))
# Case A: crash. Small input: the reader is closed while outputs are still pending.
try:
df(200, 100).collect()
except Exception as e:
print("A:", "IndexOutOfBoundsException" in str(e))
# Case B: wrong results. Large input: the reused vectors are overwritten by read-ahead.
check = df(2_000_000, 1000).select(
assert_true(col("id") == col("uid"),
concat_ws(" ", lit("MISMATCH id"), col("id"), lit("uid"), col("uid"))))
try:
check.collect() # use collect(), not count(): count() prunes the check away
except Exception as e:
print("B:", next(l for l in str(e).splitlines() if "MISMATCH" in l))
# Control: same query on the row path, correct
spark.conf.set("spark.sql.execution.arrow.pythonUDF.columnarInput.enabled", "false")
print("control mismatches:", sum(r.id != r.uid for r in df(200, 100).collect()))reproduce: A: True There are two separate causes:
I'd suggest to use ColumnarBatch types in SPARK-57468 to solve the batch type conversions elegantly. |
|
Glad to see Spark community started to adding native support. We are thrilled to contribute and improve. I quickly implemented a velox datasource PoC with int column only: https://github.com/FelixYBW/velox/tree/spark-native-datasource-poc/velox/experimental/spark_datasource Here are some benchmark results. Single Parquet file with zstd compression, 553 MiB file (73,420,230 rows, 5 row groups), 11 columns all are INT Benchmark is quite simple, scan all 11 columns then sum. parquet is on local disk, single core on a m7i EC2. Most of the time is actually spent on zstd decompression. Here is a quick benchmark:
@HyukjinKwon We need to disable arrow check ( Gluten scan only performance is too bad due to the VeloxC2R operator. Looks like we need to use Spark's default one instead. Velox datasource query plan is like this: All done with Claude |
|
@FelixYBW Thank you for the report and the reproduction. The bug is in the Arrow columnar input path of Python UDFs (SPARK-56350): it kept the pass-through columns of a batch until the result of the batch came back from Python, after the source had refilled or released their vectors. @sarutak reproduced it without native data sources too, with |
Do you mean you'll support FFM only for the native implementation? Looks like the PR introduced a full copy of every pass-through column of every batch. To Python UDF, the extra copy isn't a big issue. But we should be able to find way to avoid it. Reusing data is one of the main considerations we introduced with the Arrow data format. We spent a lot of effort in Gluten to avoid any extra memcpy |
Yeah that's currently what I'm thinking. |
We also have plan to move Gluten to FFM API. But it's still too early as all customers still use jdk8-17. |
|
Thanks @HyukjinKwon, this is really exciting. Wondering if there are future plans to offload more operators to native libraries beyond the table scan, such as filters, projections, aggregations and joins, up to a native boundary at the exchange. Gluten of course helps with this, but a native solution to help run plans in Velox would be amazing. If so, I think @zhouyuan's request for a threading contract on the stream makes sense. The docs say Spark calls the stream from the task's thread, but a native engine would want to consume it directly from its own threads. |
Compared to more operators, the other missing parts may be more important and key to production:
and many others. |
Mirrors aa0f3ba and e3ff80c of apache#59313, in the terms of spark_datasource.h: - Spark evaluates all the pushed predicates again after reading. The header states the rule for skipping data, and the jar contains predicate-cases.json, generated by a test from Spark's own evaluation. The JSON of the predicates is ASCII. - spark_ds_create_stream_writer gets the ID of the streaming query. - Offsets must be compact JSON on a single line. - An installed library replaced in place fails with INVALID_NATIVE_DATA_SOURCE_LIBRARY.REPLACED, and the caches identify files with their file key too. - Concurrent distributions of a configured package are serialized, a scan planned again after a failure fails with the same error, unsupported Arrow types and the link errors of Arrow are reported as NATIVE_DATA_SOURCE_ERROR, the column vectors of a read are not freeable between batches, packages of an unsupported version are logged, and PySpark addArtifact only passes file: URIs as they are. - Docs: the writer lifecycle, close functions, Arrow variants, a Memory section, broadcasts with AQE, replaced libraries, and the addArtifact Scaladoc. NativeLibrary already rejected NULL handles and missing offsets, so their tests use its errors. Co-authored-by: Isaac <no-reply@databricks.com>
… C++ with the FFM API ### What changes were proposed in this pull request? This is the FFM version of #59313, which it replaces, following the review of #59313: the same feature, with the binary interface as a plain C ABI called through the [Foreign Function and Memory (FFM) API](https://openjdk.org/jeps/454) instead of JNI. It includes all the changes of #59313, up to e3ff80c, changes how Spark calls the libraries, and, following the review of this PR, gives the interface a shape that can grow without new exported functions: a function table, structs that Spark passes to the library, and capabilities for each data source (see **Shape of the interface** below). Native data sources then require Java 22 or later, where the FFM API is final; with Java 17 and 21, Spark compiles and runs as before, and a query that uses a native data source fails with `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`. Like #59313, this PR adds native data sources: data sources implemented in native code, such as Rust or C++, without any JVM code. A native data source is a shared library that implements a binary interface. Spark finds it automatically when it is installed like other native libraries, or in a package file added to a session, and loads it when a query uses it. Spark adapts it to Data Source V2, with batch and micro-batch streaming reads, batch and streaming writes, and column and limit pushdown. Data is exchanged only as columnar batches, through the Arrow C data interface. **Predicate pushdown** is left for a follow-up: unlike #59313, Spark does not pass the filters of a query to a native data source, and filters the rows after reading them. The encoding of the predicates, and the rules that a library must follow to skip data with them, such as how NaN, nulls and strings compare, deserve a review of their own. Since the function table can grow within version 1 of the interface, the follow-up can add `push_predicates` without a new version. **Differences from #59313**: | | #59313 (JNI) | This PR (FFM) | |---|---|---| | Interface | `org.apache.spark.sql.datasource.NativeBridge`, a Java class whose `native` methods the library implements | `spark_datasource.h`, a C header (`sql/core/src/main/resources/org/apache/spark/sql/datasource/`, also in the `spark-sql` JAR) | | What a library exports | `Java_org_apache_spark_sql_datasource_NativeBridge_createDataSource`, ... (29 functions) | `spark_ds_init`, which returns a table of function pointers, `SparkDsLibrary` | | What a library needs | `jni.h`, or the `jni` crate | nothing but the header, or `#[repr(C)]` declarations of its structs | | Errors | throw a Java exception (`ThrowNew`) | return non-zero, with the message from the `last_error` function of the table | | Returned byte arrays (partitions, state, offsets, commit messages) | Java `byte[]`, `byte[][]`, `String` built by the library | passed to a `SparkDsSink` callback, which copies them | | Side-by-side libraries | a copy of `NativeBridge` per library, in its own class loader | each library has its own table (`SymbolLookup.libraryLookup` finds `spark_ds_init` in each) | | Supported operations | the JNI functions that the library implements | the capabilities that `create` returns for each data source | | Java versions | 17, 21, 25 | 22 and later (25 among the supported LTS versions); a clear error otherwise | **Public API**: a single C header, `spark_datasource.h`, whose comments are the specification of the interface, with the same content as the Javadoc of `NativeBridge` in #59313: the lifecycle, the handles, the ownership of the Arrow structs and of the memory that Spark passes, errors (non-zero return values, with the message of `last_error`, reported as `NATIVE_DATA_SOURCE_ERROR`), the threads, and the compatibility rules. Version 1 of the interface, `SPARK_DS_ABI_VERSION`. The `org.apache.spark.sql.datasource.NativeBridge` class of #59313 is removed: there is no JVM API for native data sources. **Shape of the interface**, following tdcmeehan's review, so that version 1 can grow without new exported functions or signatures: - `spark_ds_init(const SparkDsHost* host, const SparkDsLibrary** library)` is the only function that a library exports. Spark calls it once in each JVM, when it loads the library, and it returns the table of the functions of the library, `SparkDsLibrary`. A library sets the functions of the operations it supports, and leaves the others NULL. - The table, and the structs that Spark passes, start with their size. Within an ABI version, fields are only appended: Spark only reads the functions that fit in the size of a table, so a library built against an earlier header keeps working with the functions added later as NULL, and a library only reads the fields that fit in the size of a struct, so that it also runs on earlier releases of Spark. `host->abi_version` is the newest version that Spark supports, the table declares the version that it implements, and Spark rejects a table of a version that it does not support (`UNSUPPORTED_ABI_VERSION`). - From Spark to the library: `SparkDsHost`, valid until the JVM exits, with the version of Spark and a `log` function that writes to the log of the driver or the executor, with the path of the library; `SparkDsCreateInfo`, passed to `create`, with the name of the data source and the options of the user, so that later releases can pass more, such as references to credentials instead of the credentials themselves in the options; `SparkDsTaskContext`, passed to `read` and `create_data_writer` and valid until the stream is released or the data writer is closed, with the IDs of the stage, partition, attempt and task attempt, and `is_interrupted`, which tells a read or a write that its task was killed, since Spark cannot interrupt native code; and `SparkDsWriteInfo`, passed to `create_writer`, which replaces `spark_ds_create_writer` and `spark_ds_create_stream_writer`, with the mode of the write, `SPARK_DS_WRITE_APPEND` or `SPARK_DS_WRITE_TRUNCATE`, and the ID of the query of a streaming write. - Capabilities: `create` returns the capabilities of each data source, which may depend on its name and options: `SPARK_DS_BATCH_READ`, `SPARK_DS_MICRO_BATCH_READ`, `SPARK_DS_BATCH_WRITE`, `SPARK_DS_STREAMING_WRITE` and `SPARK_DS_TRUNCATE`. Spark reports an operation that a data source does not declare as unsupported, without calling the library, and only passes `SPARK_DS_WRITE_TRUNCATE` to a data source with `SPARK_DS_TRUNCATE`. The Data Source V2 table of a data source reports `BATCH_WRITE` and `TRUNCATE` as the data source declares them with the options of the table, so Data Source V2 rejects a batch write or overwrite that it does not support while it analyzes the query (`UNSUPPORTED_FEATURE.TABLE_OPERATION`), before it computes the data to write. The table always reports the reads and the streaming writes, which Data Source V2 would otherwise hand to Data Source V1, whose errors do not say why: they fail with `DATA_SOURCE_*_NOT_SUPPORTED` when Spark plans them. Each operation checks the capabilities again with its own options, which can override the options of a table in a catalog. If a declared capability needs a function that the table leaves NULL, the operation fails on the driver with `NATIVE_DATA_SOURCE_ERROR` and the name of the function, instead of being silently unsupported. - Handles: each handle is released by the close function of its kind, `close`, `close_reader`, `close_stream_reader`, `close_writer` or `close_data_writer`, and by no other function. `commit_data_writer` no longer releases the data writer: Spark calls `close_data_writer` after the commit or the abort. - Threads: the callbacks of the stream that `read` initializes can be called from any thread, but never concurrently, like the functions of `SparkDsHost` and `SparkDsTaskContext`, so that an engine that reads the partitions natively can call them from its own threads. - Types: the header lists the Spark types of version 1 with their Arrow types: boolean, the integer types, float, double, decimal as decimal128, string, binary, date, timestamp, timestamp_ntz, array, map and struct. Spark fails an operation whose schema has another type, such as an interval, a variant or a time, with `NATIVE_DATA_SOURCE_ERROR` before it passes the schema to the library, so that a later version can add types for the data sources that declare a capability for them. | Capability | Required functions | Optional functions | |-----------|--------------------|--------------------| | All | `create`, `close` | `last_error`, `schema` | | `SPARK_DS_BATCH_READ` | `create_reader`, `partitions`, `read`, `close_reader` | `push_limit`, `prune_columns`, `serialize_reader` | | `SPARK_DS_MICRO_BATCH_READ` | `create_stream_reader`, `initial_offset`, `latest_offset`, `stream_partitions`, `read`, `close_stream_reader` | `serialize_stream_reader`, `commit_offset` | | `SPARK_DS_BATCH_WRITE`, `SPARK_DS_STREAMING_WRITE` | `create_writer`, `commit`, `abort`, `close_writer`, `create_data_writer`, `write`, `commit_data_writer`, `abort_data_writer`, `close_data_writer` | `serialize_writer` | **How Spark calls the libraries** (`execution.datasources.v2.ffi.NativeLibrary`): Spark still compiles for Java 17, where the FFM API does not exist, so `NativeLibrary` reaches `java.lang.foreign` only through method handles, looked up once when the first library is loaded, and only on Java 22 or later. Spark calls `spark_ds_init` once for each library in a JVM, and binds each function of the table that is not NULL, within the size of the table, to a downcall handle adapted to take and return primitive values only, with pointers as `long` addresses, so each call is a plain `invokeExact` without reflection, which the JIT compiles like a direct FFM call. The memory that Spark passes to a function is allocated with `Platform`, at the offsets of the structs of the header, and freed after the call, except `SparkDsHost`, kept until the JVM exits, and `SparkDsTaskContext`, freed once the stream is released or the data writer is closed. The upcall stubs, `SparkDsSink`, which copies the byte arrays that the library returns, `SparkDsTaskContext.is_interrupted`, and `SparkDsHost.log`, one for each library, never throw, since an exception thrown by an upcall would crash the JVM. Spark already runs with `--enable-native-access=ALL-UNNAMED`, which the restricted methods of the FFM API need to avoid warnings; a library that cannot be loaded, including when native access is denied, is reported as `INVALID_NATIVE_DATA_SOURCE_LIBRARY.CANNOT_LOAD` or `INVALID_NATIVE_DATA_SOURCE_PACKAGE.INVALID_LIBRARY`, as before. Everything above `NativeLibrary` is the same as in #59313, except for the capabilities, the task contexts, the write modes and the planning of batch scans (see **Other changes**), and predicate pushdown, which this PR leaves out (see **Predicate pushdown**). The Scala classes catch `NativeLibrary.MissingFunctionException` and `NativeLibrary.LoadException` instead of `UnsatisfiedLinkError`. **Arrow data**: Spark still uses `arrow-c-data`, as in #59313, to export and import the schemas and the arrays as Arrow C structs, without copying their buffers. `arrow-c-data` calls the callbacks of the C structs through the JNI library that it bundles, for Linux and macOS on x86_64 and aarch64 and Windows on x86_64: the `get_schema`, `get_next`, `get_last_error` and `release` callbacks of the streams that the libraries export, the `release` callbacks of the schemas and arrays that Spark imports, and the `release` callbacks of the schemas and arrays that Spark exports, which are functions of that JNI library. So Spark calls the libraries with FFM, and the Arrow callbacks go through JNI, inside `arrow-c-data`, once for each batch or schema rather than for each row. Library authors only see the C interface either way. An FFM implementation of the Arrow C data interface, for example in Arrow Java, could replace this JNI library later. **Finding native data sources**: when no Java or Python data source has the requested name, Spark looks for a native data source with that name, in two places: - **Installed libraries, automatically**: like the Python data sources installed in the Python path, a library installed like other native libraries is found without any configuration, by its file name. The library of the data source `rust_range` is `spark_datasource_rust_range`: `libspark_datasource_rust_range.so` on Linux, `libspark_datasource_rust_range.dylib` on macOS and `spark_datasource_rust_range.dll` on Windows. Spark checks whether the file exists in the native library path, which consists of: 1. the directories of `java.library.path`, where the JVM itself finds native libraries. They include `LD_LIBRARY_PATH` on Linux, `DYLD_LIBRARY_PATH` on macOS and `PATH` on Windows, and so `spark.driver.extraLibraryPath` and `spark.executor.extraLibraryPath`; 2. the `lib` directory of each installation prefix in `PATH`: `<prefix>/lib` for each `<prefix>/bin`. So libraries are found where native libraries are usually installed: `/usr/local/lib` (`make install`, CMake, cargo-c, Homebrew on Intel macs), `/opt/homebrew/lib`, `$CONDA_PREFIX/lib` and `$VIRTUAL_ENV/lib` in an activated environment, and `/usr/lib`. A library that implements several data sources is installed under each of their names, for example with symbolic links. The executors load the library from the same path as the driver, or else find it in their own native library path, and only load the same build as the driver, which they check with the SHA-256 checksum of the file. - **Packages of a session**: a native data source package is a zip file with the extension `.sparkpkg`, with a manifest, `spark-native-datasource.json`, that lists the ABI version, the names of the data sources, and the library for each platform such as `linux-x86_64` or `osx-aarch64` (`linux` is Linux with glibc, and Linux with musl is `linux-<arch>-musl`). A session adds packages with `spark.addArtifact` (Scala and Python, classic and Spark Connect: `.sparkpkg` files are added as file artifacts), or with `spark.sql.dataSource.native.paths`. They take precedence over the installed libraries, and the package that a session added last over its other packages and the configured ones, as a Python data source registered again replaces the previous one: a session cannot remove an artifact, so it upgrades a package by adding the new version under another file name. Two configured packages that provide the same data source fail with `NATIVE_DATA_SOURCE_PACKAGE_CONFLICT`. The executors get them through the artifacts of the session, except in local mode, where they read them in place: a configured package is copied there under a name with its checksum, so it can be replaced in place. Spark ignores, and logs, a package that it cannot read or whose ABI version it does not support, unless its manifest lists the requested data source and no other package or installed library provides it. **Short names**: no JVM class is needed for each data source. After the Java data sources (`DataSourceRegister`) and the Python ones, `DataSource.lookupDataSource` looks the name up in the packages of the session and in the native library path, case-insensitively, and resolves it to a single provider class, `NativeDataSourceV2`. The caller then gives it the requested name, the way `PythonDataSourceV2` already works: a small `NamedTableProvider` trait generalizes how `PythonDataSourceV2` gets its short name, so the native provider shares the same call sites. The provider reuses the result of the lookup that found its class, instead of looking the name up again. A Python data source can still be registered with the name of a native data source, and takes precedence over it. Lookups only read manifests and check whether files exist; Spark loads a library when a query uses it. Native data sources work with `spark.read`/`write`, `readStream`/`writeStream`, and SQL tables (`CREATE TABLE ... USING rust_range OPTIONS (...)`, `SELECT` and `INSERT INTO`). **Other changes**, besides those of #59313: - Error conditions: `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION` (`0A000`), raised when a query uses a native data source on Java 21 or earlier, and `NATIVE_DATA_SOURCE_UNSUPPORTED_PLATFORM` (`0A000`), raised where the JNI library of the C data interface of Arrow Java does not load: Arrow Java ships it for Linux with glibc and macOS on x86_64 and aarch64, and for Windows on x86_64. The lookups still find the data source, so the errors say why it cannot be used rather than that it does not exist. `DATA_SOURCE_CONTINUOUS_WRITE_NOT_SUPPORTED` (`0A000`), raised when a continuous query writes to a native data source, before Spark creates a writer, and named after `DATA_SOURCE_CONTINUOUS_SCAN_NOT_SUPPORTED` of Data Source V2 scans. A continuous query that reads a native data source already fails when it starts, since its table does not report `CONTINUOUS_READ`. - A library whose `read` returns success without initializing the stream is reported as an error: `arrow-c-data` calls the callbacks of a stream without checking them, and calling NULL would crash the JVM. - The test library `test_native_datasource.cc` implements the function table, and the tests compile it with `spark_datasource.h` instead of the JNI headers. Compiled with defines, it also provides the libraries that the tests need: a read-only library, a table that ends before `read`, a table of version 2, a failing `spark_ds_init`, no table, and no `spark_ds_init`. - The `CheckConnectJvmClientCompatibility` exclusion of `org.apache.spark.sql.datasource`, added by #59313 for `NativeBridge`, is not needed. - CI: the `Java 25 build with Maven` job of a pull request also runs `NativeDataSourceSuite`, `NativeDataSourceClusterSuite` and `NativeDataSourceConnectSuite`, with the C++ compiler of the runner, since the other jobs, on Java 17, skip the tests that load a library. - Checks of the data: before importing a batch, Spark also checks that the batch and each array have as many children as their type has fields, with non-NULL pointers, and after importing it, that the data agrees with the nullability of the schema to read, at every level. Option keys and values, and column names, that contain U+0000 fail with `NATIVE_DATA_SOURCE_ERROR`. - Packages: the `abiVersion` of a manifest is the oldest version of the interface that the library implements. Spark uses a package whose version is at most the newest that it supports, and ignores the fields of a manifest that it does not know. - Arrow UDFs: Python UDFs over the batches of a native data source need SPARK-60294 (#59551), which disables the Arrow columnar input of Python UDFs (SPARK-56350) by default: it kept references to the pass-through columns of a batch after the source refilled or released their vectors, as the reader of a native data source does. The reader wraps its vectors in `org.apache.spark.sql.execution.vectorized.ReusedArrowColumnVector`, added by SPARK-60306 (#59581), whose `closeIfFreeable` does nothing, so that the columnar input path recognizes them once it is enabled again with a marker for the vectors that their producer reuses (SPARK-60325, #59582). - Batch scans: the Data Source V2 adapter, `execution.datasources.v2.columnar`, plans a batch scan with a driver-only `ColumnarBatchPlanner`, which gets the pushdown, and whose `plan` returns the partitions with a serializable reader factory. For a native data source, the planner holds the reader of the library, and the reader factory only has the location of the library, the name of the data source, the read schema and the state that `serialize_reader` returned. A scan builder builds its scan once, and the scan takes over the planner, or creates one if the operations were not pushed down, so a streaming scan never creates one. A batch is still planned once, when Spark first asks for its partitions, and a planning failure is kept and thrown again. The reader is released once the scan is planned, or by a `Cleaner` if the scan is never planned, as before. - `commit_data_writer` returns its message through the sink. Spark never aborts a data writer once the library returns 0, even if it then rejects the message, and it always calls `close_data_writer` afterwards, which is the only function that releases a data writer. - `arrow-c-data`: SPARK-59718 added it as a provided dependency of `sql/core` for in-process Python UDFs. This PR keeps it a compile dependency, in the `jars` directory of a distribution (`dev/deps`, `LICENSE-binary`), since native data sources need it at run time, so the docs of in-process Python UDFs and the README of their benchmark no longer ask users to supply its JAR. - Publishing the header: `dev/make-distribution.sh` copies `spark_datasource.h` to the `include` directory of a binary distribution, as a JDK ships `include/jni.h`, and the `spark-sql` JAR contains it too. The header only depends on the C standard library, so a library can also copy it into its source tree and build without Spark, as for the Arrow C data interface and ADBC's `adbc.h`; the docs say so, with the URL of the header at the tag of a release. The other changes are those of #59313: the configurations `spark.sql.dataSource.native.enabled` and `spark.sql.dataSource.native.paths`, the error conditions, the `arrow-c-data` dependency, the "Native Data Sources" documentation page (with the examples below), and: - From the latest review round of #59313 (aa0f3ba, e3ff80c), in the terms of the C interface: - `create_writer` gets the ID of the streaming query, `SparkDsWriteInfo.query_id`, which stays the same when the query restarts from its checkpoint. - Offsets must be compact JSON on a single line, as the offset log stores them. - An installed library replaced in place fails with `INVALID_NATIVE_DATA_SOURCE_LIBRARY.REPLACED`, and the caches identify files by their file key too. - Concurrent distributions of a configured package in a session are serialized, a scan planned again after a failure fails with the same error, the unsupported Arrow types and the link errors of Arrow's own JNI library are reported as `NATIVE_DATA_SOURCE_ERROR`, the column vectors of a read are not freeable between batches (which broke struct columns), packages of an unsupported version are logged, PySpark `addArtifact` only passes `file:` URIs as they are, and the docs describe the lifecycle of writers and the memory of native data sources. - A NULL handle was already reported as an error here, by `NativeLibrary`, and a missing offset as well, since the library returns it through the sink. - Existing classes: - `DataSource.lookupDataSource` looks up the native data sources after the Python ones, and `DataSourceRegistration` skips them when it registers a Python data source. - `ArtifactManager` lists the native data source packages of a session, and forgets the cached packages read from the artifacts of a session when it cleans them up. - The classic `DataFrameWriter` builds its commands, and `DataStreamWriter` resolves its sink, with the session of the Dataset active, as the analysis of a query does: the lookups of the Python and native data sources use the active session. - PySpark `addArtifacts` adds `.sparkpkg` files to the session whatever the flags: in classic mode through the JVM session, and with Spark Connect as file artifacts. ### Examples Both examples implement a read-only data source that returns the numbers `[0, end)` as a column `id`, in two partitions, with the option `end` (10 by default). They declare the capability `SPARK_DS_BATCH_READ`, and only implement the functions needed for batch reads, plus `last_error`, leaving the others NULL in their table. <details> <summary><b>Rust</b> (with arrow-rs only)</summary> `Cargo.toml`: ```toml [package] name = "rust_range" version = "0.1.0" edition = "2021" [lib] name = "spark_datasource_rust_range" crate-type = ["cdylib"] [dependencies] arrow-array = { version = "58", features = ["ffi"] } arrow-schema = { version = "58", features = ["ffi"] } ``` `src/lib.rs`: ```rust //! A native data source for Apache Spark, written in Rust. It reads the numbers [0, end) as a //! column `id`, in two partitions. The option `end` defaults to 10. use std::cell::RefCell; use std::ffi::{c_char, c_int, c_void, CStr, CString}; use std::panic::{catch_unwind, AssertUnwindSafe}; use std::sync::Arc; use arrow_array::ffi_stream::FFI_ArrowArrayStream; use arrow_array::{Int64Array, RecordBatch, RecordBatchIterator}; use arrow_schema::ffi::FFI_ArrowSchema; use arrow_schema::{DataType, Field, Schema, SchemaRef}; type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>; /// `SparkDsSink`: receives a byte array returned to Spark, and copies it. type Sink = unsafe extern "C" fn(sink: *mut c_void, data: *const u8, length: i64); /// A function of `SparkDsLibrary` that this data source does not implement, left NULL. type Unused = Option<unsafe extern "C" fn()>; /// `SparkDsCreateInfo` of `spark_datasource.h`: the name and the options of a data source. #[repr(C)] pub struct SparkDsCreateInfo { size: i64, name: *const c_char, option_keys: *const *const c_char, option_values: *const *const c_char, num_options: i64, } /// `SparkDsLibrary` of `spark_datasource.h`, up to `read`, the last function that this data /// source implements: Spark only reads the functions that fit in `size`. #[repr(C)] pub struct SparkDsLibrary { size: i64, abi_version: i32, last_error: unsafe extern "C" fn() -> *const c_char, create: unsafe extern "C" fn( info: *const SparkDsCreateInfo, capabilities: *mut u64, source: *mut *mut c_void, ) -> c_int, schema: unsafe extern "C" fn(source: *mut c_void, out: *mut FFI_ArrowSchema) -> c_int, create_reader: unsafe extern "C" fn( source: *mut c_void, schema: *mut FFI_ArrowSchema, reader: *mut *mut c_void, ) -> c_int, create_stream_reader: Unused, create_writer: Unused, close: unsafe extern "C" fn(source: *mut c_void), push_limit: Unused, prune_columns: Unused, partitions: unsafe extern "C" fn(reader: *mut c_void, add: Sink, sink: *mut c_void) -> c_int, serialize_reader: Unused, close_reader: unsafe extern "C" fn(reader: *mut c_void), initial_offset: Unused, latest_offset: Unused, stream_partitions: Unused, serialize_stream_reader: Unused, commit_offset: Unused, close_stream_reader: Unused, read: unsafe extern "C" fn( task: *const c_void, reader_state: *const u8, reader_state_length: i64, partition: *const u8, partition_length: i64, out: *mut FFI_ArrowArrayStream, ) -> c_int, } const SPARK_DS_BATCH_READ: u64 = 1; static LIBRARY: SparkDsLibrary = SparkDsLibrary { size: std::mem::size_of::<SparkDsLibrary>() as i64, abi_version: 1, last_error, create, schema, create_reader, create_stream_reader: None, create_writer: None, close, push_limit: None, prune_columns: None, partitions, serialize_reader: None, close_reader, initial_offset: None, latest_offset: None, stream_partitions: None, serialize_stream_reader: None, commit_offset: None, close_stream_reader: None, read, }; /// The function that Spark calls when it loads the library. #[no_mangle] pub unsafe extern "C" fn spark_ds_init( _host: *const c_void, library: *mut *const SparkDsLibrary, ) -> c_int { *library = &LIBRARY; 0 } struct RangeSource { end: i64, } struct RangeReader { end: i64, } thread_local! { static LAST_ERROR: RefCell<CString> = RefCell::new(CString::default()); } fn arrow_schema() -> SchemaRef { Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])) } /// Runs `f`, and reports its error or panic to Spark: neither may cross into the JVM. fn call(f: impl FnOnce() -> Result<()>) -> c_int { let message = match catch_unwind(AssertUnwindSafe(f)) { Ok(Ok(())) => return 0, Ok(Err(e)) => e.to_string(), Err(_) => "the data source panicked".to_string(), }; LAST_ERROR.with(|error| *error.borrow_mut() = CString::new(message).unwrap_or_default()); 1 } unsafe extern "C" fn last_error() -> *const c_char { LAST_ERROR.with(|error| error.borrow().as_ptr()) } unsafe extern "C" fn create( info: *const SparkDsCreateInfo, capabilities: *mut u64, source: *mut *mut c_void, ) -> c_int { call(|| { let info = &*info; let mut range = RangeSource { end: 10 }; for i in 0..info.num_options as usize { if CStr::from_ptr(*info.option_keys.add(i)).to_str()? == "end" { range.end = CStr::from_ptr(*info.option_values.add(i)).to_str()?.parse()?; } } *capabilities = SPARK_DS_BATCH_READ; *source = Box::into_raw(Box::new(range)) as *mut c_void; Ok(()) }) } unsafe extern "C" fn schema(_source: *mut c_void, out: *mut FFI_ArrowSchema) -> c_int { call(|| { std::ptr::write(out, FFI_ArrowSchema::try_from(arrow_schema().as_ref())?); Ok(()) }) } unsafe extern "C" fn create_reader( source: *mut c_void, _schema: *mut FFI_ArrowSchema, reader: *mut *mut c_void, ) -> c_int { call(|| { let end = (*(source as *const RangeSource)).end; *reader = Box::into_raw(Box::new(RangeReader { end })) as *mut c_void; Ok(()) }) } unsafe extern "C" fn close(source: *mut c_void) { drop(Box::from_raw(source as *mut RangeSource)); } unsafe extern "C" fn partitions(reader: *mut c_void, add: Sink, sink: *mut c_void) -> c_int { call(|| { let end = (*(reader as *const RangeReader)).end; for (start, end) in [(0, end / 2), (end / 2, end)] { let bytes = [start.to_le_bytes(), end.to_le_bytes()].concat(); add(sink, bytes.as_ptr(), bytes.len() as i64); } Ok(()) }) } unsafe extern "C" fn close_reader(reader: *mut c_void) { drop(Box::from_raw(reader as *mut RangeReader)); } unsafe extern "C" fn read( _task: *const c_void, _reader_state: *const u8, _reader_state_length: i64, partition: *const u8, partition_length: i64, out: *mut FFI_ArrowArrayStream, ) -> c_int { call(|| { let bytes = std::slice::from_raw_parts(partition, partition_length as usize); let start = i64::from_le_bytes(bytes[0..8].try_into()?); let end = i64::from_le_bytes(bytes[8..16].try_into()?); let ids = Int64Array::from_iter_values(start..end); let batch = RecordBatch::try_new(arrow_schema(), vec![Arc::new(ids)])?; let batches = RecordBatchIterator::new(vec![Ok(batch)], arrow_schema()); std::ptr::write(out, FFI_ArrowArrayStream::new(Box::new(batches))); Ok(()) }) } ``` Build and install, here on macOS: ```bash cargo build --release cp target/release/libspark_datasource_rust_range.dylib /usr/local/lib/ ``` Or package, to add it to a session with `spark.addArtifact`: ```bash mkdir -p package/osx-aarch64 cp target/release/libspark_datasource_rust_range.dylib package/osx-aarch64/ cat > package/spark-native-datasource.json <<EOF { "abiVersion": 1, "dataSources": ["rust_range"], "libraries": {"osx-aarch64": "osx-aarch64/libspark_datasource_rust_range.dylib"} } EOF (cd package && zip -r ../rust_range.sparkpkg .) ``` </details> <details> <summary><b>C++</b> (with Arrow C++)</summary> `cpp_range.cc`: ```cpp // A native data source for Apache Spark, written in C++ with Arrow C++. It reads the numbers // [0, end) as a column `id`, in two partitions. The option `end` defaults to 10. #include <cstdint> #include <cstring> #include <memory> #include <stdexcept> #include <string> #include <arrow/api.h> #include <arrow/c/bridge.h> #include "spark_datasource.h" namespace { struct RangeSource { int64_t end = 10; }; struct RangeReader { int64_t end; }; thread_local std::string last_error; std::shared_ptr<arrow::Schema> Schema() { return arrow::schema({arrow::field("id", arrow::int64(), /*nullable=*/false)}); } // Runs `f`, and reports its error to Spark: no C++ exception may cross into the JVM. template <typename F> int Call(F&& f) { try { f(); return 0; } catch (const std::exception& e) { last_error = e.what(); return 1; } catch (...) { last_error = "unknown error"; return 1; } } void Check(const arrow::Status& status) { if (!status.ok()) throw std::runtime_error(status.ToString()); } const char* LastError() { return last_error.c_str(); } int Create(const SparkDsCreateInfo* info, uint64_t* capabilities, void** source) { return Call([&] { auto range = std::make_unique<RangeSource>(); for (int64_t i = 0; i < info->num_options; i++) { if (std::string(info->option_keys[i]) == "end") { range->end = std::stoll(info->option_values[i]); } } *capabilities = SPARK_DS_BATCH_READ; *source = range.release(); }); } int GetSchema(void*, struct ArrowSchema* out) { return Call([&] { Check(arrow::ExportSchema(*Schema(), out)); }); } int CreateReader(void* source, struct ArrowSchema*, void** reader) { return Call([&] { *reader = new RangeReader{static_cast<RangeSource*>(source)->end}; }); } void Close(void* source) { delete static_cast<RangeSource*>(source); } int Partitions(void* reader, SparkDsSink add_partition, void* sink) { return Call([&] { int64_t end = static_cast<RangeReader*>(reader)->end; int64_t bounds[2][2] = {{0, end / 2}, {end / 2, end}}; for (const auto& partition : bounds) { add_partition(sink, reinterpret_cast<const uint8_t*>(partition), sizeof(partition)); } }); } void CloseReader(void* reader) { delete static_cast<RangeReader*>(reader); } int Read(const SparkDsTaskContext*, const uint8_t*, int64_t, const uint8_t* partition, int64_t, struct ArrowArrayStream* out) { return Call([&] { int64_t bounds[2]; std::memcpy(bounds, partition, sizeof(bounds)); arrow::Int64Builder ids; for (int64_t id = bounds[0]; id < bounds[1]; id++) Check(ids.Append(id)); std::shared_ptr<arrow::Array> array; Check(ids.Finish(&array)); auto batch = arrow::RecordBatch::Make(Schema(), array->length(), {array}); auto batches = arrow::RecordBatchReader::Make({batch}, Schema()).ValueOrDie(); Check(arrow::ExportRecordBatchReader(batches, out)); }); } // The functions of the data source. The ones that it does not implement are NULL. const SparkDsLibrary kLibrary = { .size = sizeof(SparkDsLibrary), .abi_version = SPARK_DS_ABI_VERSION, .last_error = LastError, .create = Create, .schema = GetSchema, .create_reader = CreateReader, .close = Close, .partitions = Partitions, .close_reader = CloseReader, .read = Read, }; } // namespace extern "C" SPARK_DS_EXPORT int spark_ds_init(const SparkDsHost*, const SparkDsLibrary** library) { *library = &kLibrary; return 0; } ``` Get `spark_datasource.h`, which is in the `spark-sql` JAR, then build (Arrow C++ 24 requires C++20) and install, or package as above: ```bash unzip -j "$SPARK_HOME"/jars/spark-sql_*.jar org/apache/spark/sql/datasource/spark_datasource.h c++ -std=c++20 -O2 -shared -fPIC -I. -I"$ARROW_HOME/include" -L"$ARROW_HOME/lib" -larrow \ -o libspark_datasource_cpp_range.dylib cpp_range.cc cp libspark_datasource_cpp_range.dylib /usr/local/lib/ ``` </details> Usage is the same as in #59313, on Java 22 or later, in classic PySpark or with Spark Connect. Once installed, the data sources are found automatically, without `spark.addArtifact` or any configuration: ```python >>> spark.read.format("rust_range").option("end", 5).load().show() +---+ | id| +---+ | 0| | 1| | 2| | 3| | 4| +---+ >>> spark.read.format("cpp_range").option("end", 5).load().filter("id >= 3").collect() [Row(id=3), Row(id=4)] >>> spark.sql("CREATE TABLE t USING rust_range OPTIONS (end 3)") >>> spark.sql("SELECT * FROM t").collect() [Row(id=0), Row(id=1), Row(id=2)] ``` Or added to a session as a package: ```python >>> spark.addArtifact("rust_range/rust_range.sparkpkg") >>> spark.read.format("rust_range").option("end", 5).load().count() 5 ``` ### Why are the changes needed? Some data systems only have a client in native code, such as an in-house storage system written in C++ or the C SDK of a vendor, and Spark only needs to read and write them. Using them from Spark today requires writing a JVM Data Source V2 connector with its own JNI glue. With this change, such a source implements a stable binary interface once, in Rust, C++ or any language that can export C functions, is installed like any other native library or distributed as a single file, keeps working with the later releases of Spark that support its version of the interface, and works with every Spark API, including Spark Connect clients in any language, without any JVM code. A source that needs what version 1 does not cover yet, such as row-level operations, or extensions outside of Data Source V2, such as table functions and SQL commands, is better served by a JVM connector, which can still read and write through a native library with the Arrow C stream interface, as lance-spark does with Lance. The docs say so in a new "When to Use a Native Data Source" section. With FFM, the interface is also a plain C ABI, so a library needs no JVM headers or bindings, and other hosts than Spark could load the same libraries. ### Does this PR introduce _any_ user-facing change? Yes. It adds native data sources, with the `spark_datasource.h` interface, the configurations and the error conditions above, on Java 22 or later. Spark finds the native data source libraries installed in the native library path, and `spark.addArtifact` now accepts native data source packages (`.sparkpkg` files). Also, in classic mode, writing with a session that is not the active one now finds the data sources of that session, such as its Python data sources, because `DataFrameWriter` and `DataStreamWriter` resolve the data source with that session active. ### How was this patch tested? All the tests of #59313, ported to the C interface: - `NativeDataSourceSuite`, with the dependency-free C++ test library (`sql/core/src/test/resources/native-datasource/test_native_datasource.cc`), now implementing `spark_datasource.h`, which the tests compile with the C++ compiler of the machine against the header from the classpath. It covers inferred and user-specified schemas, partitions and batches, the supported types, column and limit pushdown, batch writes (append and overwrite), aborted writes, streaming reads and writes over several micro-batches with offset commits, SQL tables with `OPTIONS` (`SELECT` and `INSERT INTO`), errors reported by native code, missing functions, several libraries exporting the same functions side by side, `spark.addArtifact`, packages in directories, package conflicts, the precedence of Java data sources, invalid packages, installed libraries and invalid installed libraries, plus the unit tests for the manifest, the platform names and the native library path. - Tests added during the review, in `NativeDataSourceSuite` unless noted: - Packages: invalid and unsupported packages are ignored unless no other package or installed library provides the data source; a configured package replaced in place; the copies distributed to the executors are named by checksum and are not packages of the session; the cached packages of a session are forgotten when its artifacts are cleaned up; configured directories that cannot be read; distributing a package without an active session. - Lookups: a single lookup for each resolution, which the next lookup on the thread clears; a Python data source registered with the name of an installed native one; batch and streaming writes with a session that is not the active one. - Writes: a failed `commitDataWriter` aborts the data writer; a failed `commit` aborts the write, then closes the writer; the test library records when a writer is closed, which checks that a batch write closes its writer, and that each micro-batch of a streaming write closes the writer of the previous one; `ColumnarDataSourceSuite` checks that each micro-batch creates its stream writer with the ID of the query, and that batches end at `spark.sql.execution.arrow.maxBytesPerBatch`. - Cluster mode: new `NativeDataSourceClusterSuite` runs in `local-cluster[2,1,1024]`, where the executors are separate JVMs: batch reads, writes and streaming with a package added with `spark.addArtifact`, and a configured package that the executors can only load from its distributed copy (named by checksum, under the UUID of the session), which they check by its checksum. - Installed libraries: an executor does not load a library with another checksum than on the driver, at the driver's path or in its native library path. - `spark.addArtifact` in classic PySpark accepts `file://` URIs and paths relative to the working directory of Python. - Reads: the columns of the batches are the `ReusedArrowColumnVector`s of Spark (`read structs in several batches`); read schemas with collations, and the scan described by the name of the data source; sliced arrays, arrays shorter than their parent (a column, a struct field or the elements of a map), `decimal256` and list views are rejected with the name of the column, maps whose entry fields are named `keys` and `values` (as arrow-rs names them) are read, and `limit(...).offset(...)` asks the library for the limit once. - Writes whose upstream stages fail with adaptive execution create no writer (also in `ColumnarDataSourceSuite`). - From the latest review round of #59313: no limit is pushed with a filter; the columns that a query filters on are read; a struct column read in several batches through whole-stage codegen and a nested field; large lists and dictionary encoding rejected with the name of the column; a reader planned again after a failure; offsets that are missing, multi-line or not JSON; the query ID across a restart; a failed micro-batch aborted with its epoch ID; `stop()` closing the stream reader; NULL handles from each of the five create functions; concurrent distributions of a configured package; a valid package with another checksum; the logging of packages of an unsupported version; an installed library replaced in place; link errors; and `addArtifact` with a file name that contains a colon. - `NativeDataSourceConnectSuite` (Spark Connect), `NativeDataSourceClusterSuite` (`local-cluster`, executors in separate JVMs) and `ColumnarDataSourceSuite`. New tests: - `native data sources require Java 22 or later`: on Java 21 or earlier, a query that uses an installed native data source fails with `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`, before Spark loads the library. It runs only on Java 21 or earlier, so it runs in the default CI on Java 17. - `report the errors of native code`: a library without `last_error` reports `create failed with the error code 1.`, and a library whose `read` returns success without initializing the stream fails with `The library did not initialize the stream, or a callback of the stream is NULL.` New tests for the shape of the interface, in `NativeDataSourceSuite`: - `the bindings match the layout of spark_datasource.h`: compiles a program that prints the size of each struct of the header and the offset of each field, and compares them with the offsets that `NativeLibrary` uses. - `operations that a data source does not support`: a read-only library, whose data sources only declare `SPARK_DS_BATCH_READ`, fails batch writes while Spark analyzes the query, with `UNSUPPORTED_FEATURE.TABLE_OPERATION`, before a shuffle stage of the data to write fails, and streaming reads and streaming writes with `DATA_SOURCE_*_NOT_SUPPORTED`; a data source whose library implements batch reads but that only declares streaming reads fails batch reads; a declared capability with a NULL function fails with the name of the function; an overwrite without `SPARK_DS_TRUNCATE` fails, in batch while Spark analyzes the query, and in streaming with the output mode `complete`; and a write to a table in a catalog whose options remove a capability of the table fails when Spark plans it, with `DATA_SOURCE_BATCH_WRITE_NOT_SUPPORTED`, or `NATIVE_DATA_SOURCE_ERROR` for an overwrite. - `ColumnarDataSourceSuite`: `the table reports the capabilities of the data source`, where a batch write or overwrite that the data source does not support fails while Spark analyzes it, without a writer, and `build and plan a batch scan once`, where a scan builder builds one scan, whose batch is planned once, and a failed planning is not retried but fails again with the same error. - `reject the data that has nulls where the schema to read has none`; `reject the types and the strings that a library cannot get`, with an interval in a read, an array of variants in a write, which fails before `create_writer`, and an option with U+0000; `reject the Arrow data that Spark cannot read`, with batches without their columns; and `parse manifests`, with an `abiVersion` that is not positive. - From the review of gerashegalov: `continuous processing fails with a native data source`, for writes and reads; `the package that a session added last replaces the other packages`; `fail clearly where the C data interface of Arrow Java does not load`; `reject the Arrow data that Spark cannot read`, with batches whose struct marks rows as null, counted or not, and arrays with a `null_count` of -1 but no validity buffer, which Arrow Java cannot import; and `platform names`, with musl. - `Arrow UDFs on the columns of a native data source`: a pandas UDF over one batch, two and 2,000, which checks the pass-through columns of each row against the result of the UDF. Locally, it fails without #59551 and passes with it. The CI of pull requests skips it, since the job that runs the native data source suites has no pandas. - `read only the functions that fit in the table of a library`: a table that ends before `read`, like the table of a library built against an earlier header, makes `read` unsupported rather than read past its end. - `pass the version of Spark, and log with the log of Spark`: the library checks `SparkDsHost`, and its messages reach the log of Spark at their level, with the path of the library. - `pass the task to reads and data writers`: each read and each data writer gets the IDs of its task. - `tell a read that its task was killed`: a read that waits until `is_interrupted` returns true ends once its job is canceled. - `reject the libraries that spark_ds_init does not initialize`: a failing `spark_ds_init`, which logs why once, a library that returns no table, and one that does not export `spark_ds_init`, with the same error when they are used again, since Spark initializes a library only once. - `invalid installed libraries` and `invalid packages` check a table of version 2, which Spark rejects with `UNSUPPORTED_ABI_VERSION`. The tests that load a library are skipped before Java 22, like they are skipped without a C++ compiler. So the jobs on Java 17 only run the tests that do not load a library, and the `Java 25 build with Maven` job of a pull request runs the native data source suites, as do the scheduled Java 25 builds. `do not abort a data writer that committed with an invalid message` checks that a data writer released by its commit is not aborted. Locally on macOS (osx-aarch64): - After leaving predicate pushdown out (1f40484): `NativeDataSourceSuite` and `ColumnarDataSourceSuite` pass on Java 26 (Homebrew OpenJDK 26.0.1), with Spark built with SBT on the same JDK. - Java 26 (Homebrew OpenJDK 26.0.1; Spark built with Java 17): `NativeDataSourceSuite`, `NativeDataSourceClusterSuite` (with the executors on Java 26 too), `ColumnarDataSourceSuite` and `NativeDataSourceConnectSuite` pass. With the executors of `local-cluster` on Java 17, the tasks fail with `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`, as expected. - Java 17: `NativeDataSourceSuite` passes, with the tests that load a library skipped, and the Java version test run. - The Rust and C++ examples above were built with the commands above (Rust with arrow-rs 58, C++ with Arrow C++ 24 and `-std=c++20`; both export only `spark_ds_init`), installed in a directory of `java.library.path`, and read by Spark on Java 26: `spark.read.format("rust_range").option("end", 100).load()` and the same with `cpp_range` return the 100 rows in two partitions, and `option("end", "x")` fails with `NATIVE_DATA_SOURCE_ERROR` and the message of the library (`invalid digit found in string` and `stoll: no conversion`). - The documented steps of the C++ example work with the header in the `include` directory of a distribution: the header copied as `dev/make-distribution.sh` copies it, then the documented `c++` command with `-I"$SPARK_HOME/include"`. - The C++ test library builds without warnings with Apple clang (`-Wall -Wextra`), in all its variants. - `SparkThrowableSuite`, for the new error conditions. - `checkstyle` and `scalastyle` of the changed modules. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. Closes #59424 from HyukjinKwon/SPARK-60101-ffm. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>
… C++ with the FFM API ### What changes were proposed in this pull request? This is the FFM version of #59313, which it replaces, following the review of #59313: the same feature, with the binary interface as a plain C ABI called through the [Foreign Function and Memory (FFM) API](https://openjdk.org/jeps/454) instead of JNI. It includes all the changes of #59313, up to e3ff80c, changes how Spark calls the libraries, and, following the review of this PR, gives the interface a shape that can grow without new exported functions: a function table, structs that Spark passes to the library, and capabilities for each data source (see **Shape of the interface** below). Native data sources then require Java 22 or later, where the FFM API is final; with Java 17 and 21, Spark compiles and runs as before, and a query that uses a native data source fails with `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`. Like #59313, this PR adds native data sources: data sources implemented in native code, such as Rust or C++, without any JVM code. A native data source is a shared library that implements a binary interface. Spark finds it automatically when it is installed like other native libraries, or in a package file added to a session, and loads it when a query uses it. Spark adapts it to Data Source V2, with batch and micro-batch streaming reads, batch and streaming writes, and column and limit pushdown. Data is exchanged only as columnar batches, through the Arrow C data interface. **Predicate pushdown** is left for a follow-up: unlike #59313, Spark does not pass the filters of a query to a native data source, and filters the rows after reading them. The encoding of the predicates, and the rules that a library must follow to skip data with them, such as how NaN, nulls and strings compare, deserve a review of their own. Since the function table can grow within version 1 of the interface, the follow-up can add `push_predicates` without a new version. **Differences from #59313**: | | #59313 (JNI) | This PR (FFM) | |---|---|---| | Interface | `org.apache.spark.sql.datasource.NativeBridge`, a Java class whose `native` methods the library implements | `spark_datasource.h`, a C header (`sql/core/src/main/resources/org/apache/spark/sql/datasource/`, also in the `spark-sql` JAR) | | What a library exports | `Java_org_apache_spark_sql_datasource_NativeBridge_createDataSource`, ... (29 functions) | `spark_ds_init`, which returns a table of function pointers, `SparkDsLibrary` | | What a library needs | `jni.h`, or the `jni` crate | nothing but the header, or `#[repr(C)]` declarations of its structs | | Errors | throw a Java exception (`ThrowNew`) | return non-zero, with the message from the `last_error` function of the table | | Returned byte arrays (partitions, state, offsets, commit messages) | Java `byte[]`, `byte[][]`, `String` built by the library | passed to a `SparkDsSink` callback, which copies them | | Side-by-side libraries | a copy of `NativeBridge` per library, in its own class loader | each library has its own table (`SymbolLookup.libraryLookup` finds `spark_ds_init` in each) | | Supported operations | the JNI functions that the library implements | the capabilities that `create` returns for each data source | | Java versions | 17, 21, 25 | 22 and later (25 among the supported LTS versions); a clear error otherwise | **Public API**: a single C header, `spark_datasource.h`, whose comments are the specification of the interface, with the same content as the Javadoc of `NativeBridge` in #59313: the lifecycle, the handles, the ownership of the Arrow structs and of the memory that Spark passes, errors (non-zero return values, with the message of `last_error`, reported as `NATIVE_DATA_SOURCE_ERROR`), the threads, and the compatibility rules. Version 1 of the interface, `SPARK_DS_ABI_VERSION`. The `org.apache.spark.sql.datasource.NativeBridge` class of #59313 is removed: there is no JVM API for native data sources. **Shape of the interface**, following tdcmeehan's review, so that version 1 can grow without new exported functions or signatures: - `spark_ds_init(const SparkDsHost* host, const SparkDsLibrary** library)` is the only function that a library exports. Spark calls it once in each JVM, when it loads the library, and it returns the table of the functions of the library, `SparkDsLibrary`. A library sets the functions of the operations it supports, and leaves the others NULL. - The table, and the structs that Spark passes, start with their size. Within an ABI version, fields are only appended: Spark only reads the functions that fit in the size of a table, so a library built against an earlier header keeps working with the functions added later as NULL, and a library only reads the fields that fit in the size of a struct, so that it also runs on earlier releases of Spark. `host->abi_version` is the newest version that Spark supports, the table declares the version that it implements, and Spark rejects a table of a version that it does not support (`UNSUPPORTED_ABI_VERSION`). - From Spark to the library: `SparkDsHost`, valid until the JVM exits, with the version of Spark and a `log` function that writes to the log of the driver or the executor, with the path of the library; `SparkDsCreateInfo`, passed to `create`, with the name of the data source and the options of the user, so that later releases can pass more, such as references to credentials instead of the credentials themselves in the options; `SparkDsTaskContext`, passed to `read` and `create_data_writer` and valid until the stream is released or the data writer is closed, with the IDs of the stage, partition, attempt and task attempt, and `is_interrupted`, which tells a read or a write that its task was killed, since Spark cannot interrupt native code; and `SparkDsWriteInfo`, passed to `create_writer`, which replaces `spark_ds_create_writer` and `spark_ds_create_stream_writer`, with the mode of the write, `SPARK_DS_WRITE_APPEND` or `SPARK_DS_WRITE_TRUNCATE`, and the ID of the query of a streaming write. - Capabilities: `create` returns the capabilities of each data source, which may depend on its name and options: `SPARK_DS_BATCH_READ`, `SPARK_DS_MICRO_BATCH_READ`, `SPARK_DS_BATCH_WRITE`, `SPARK_DS_STREAMING_WRITE` and `SPARK_DS_TRUNCATE`. Spark reports an operation that a data source does not declare as unsupported, without calling the library, and only passes `SPARK_DS_WRITE_TRUNCATE` to a data source with `SPARK_DS_TRUNCATE`. The Data Source V2 table of a data source reports `BATCH_WRITE` and `TRUNCATE` as the data source declares them with the options of the table, so Data Source V2 rejects a batch write or overwrite that it does not support while it analyzes the query (`UNSUPPORTED_FEATURE.TABLE_OPERATION`), before it computes the data to write. The table always reports the reads and the streaming writes, which Data Source V2 would otherwise hand to Data Source V1, whose errors do not say why: they fail with `DATA_SOURCE_*_NOT_SUPPORTED` when Spark plans them. Each operation checks the capabilities again with its own options, which can override the options of a table in a catalog. If a declared capability needs a function that the table leaves NULL, the operation fails on the driver with `NATIVE_DATA_SOURCE_ERROR` and the name of the function, instead of being silently unsupported. - Handles: each handle is released by the close function of its kind, `close`, `close_reader`, `close_stream_reader`, `close_writer` or `close_data_writer`, and by no other function. `commit_data_writer` no longer releases the data writer: Spark calls `close_data_writer` after the commit or the abort. - Threads: the callbacks of the stream that `read` initializes can be called from any thread, but never concurrently, like the functions of `SparkDsHost` and `SparkDsTaskContext`, so that an engine that reads the partitions natively can call them from its own threads. - Types: the header lists the Spark types of version 1 with their Arrow types: boolean, the integer types, float, double, decimal as decimal128, string, binary, date, timestamp, timestamp_ntz, array, map and struct. Spark fails an operation whose schema has another type, such as an interval, a variant or a time, with `NATIVE_DATA_SOURCE_ERROR` before it passes the schema to the library, so that a later version can add types for the data sources that declare a capability for them. | Capability | Required functions | Optional functions | |-----------|--------------------|--------------------| | All | `create`, `close` | `last_error`, `schema` | | `SPARK_DS_BATCH_READ` | `create_reader`, `partitions`, `read`, `close_reader` | `push_limit`, `prune_columns`, `serialize_reader` | | `SPARK_DS_MICRO_BATCH_READ` | `create_stream_reader`, `initial_offset`, `latest_offset`, `stream_partitions`, `read`, `close_stream_reader` | `serialize_stream_reader`, `commit_offset` | | `SPARK_DS_BATCH_WRITE`, `SPARK_DS_STREAMING_WRITE` | `create_writer`, `commit`, `abort`, `close_writer`, `create_data_writer`, `write`, `commit_data_writer`, `abort_data_writer`, `close_data_writer` | `serialize_writer` | **How Spark calls the libraries** (`execution.datasources.v2.ffi.NativeLibrary`): Spark still compiles for Java 17, where the FFM API does not exist, so `NativeLibrary` reaches `java.lang.foreign` only through method handles, looked up once when the first library is loaded, and only on Java 22 or later. Spark calls `spark_ds_init` once for each library in a JVM, and binds each function of the table that is not NULL, within the size of the table, to a downcall handle adapted to take and return primitive values only, with pointers as `long` addresses, so each call is a plain `invokeExact` without reflection, which the JIT compiles like a direct FFM call. The memory that Spark passes to a function is allocated with `Platform`, at the offsets of the structs of the header, and freed after the call, except `SparkDsHost`, kept until the JVM exits, and `SparkDsTaskContext`, freed once the stream is released or the data writer is closed. The upcall stubs, `SparkDsSink`, which copies the byte arrays that the library returns, `SparkDsTaskContext.is_interrupted`, and `SparkDsHost.log`, one for each library, never throw, since an exception thrown by an upcall would crash the JVM. Spark already runs with `--enable-native-access=ALL-UNNAMED`, which the restricted methods of the FFM API need to avoid warnings; a library that cannot be loaded, including when native access is denied, is reported as `INVALID_NATIVE_DATA_SOURCE_LIBRARY.CANNOT_LOAD` or `INVALID_NATIVE_DATA_SOURCE_PACKAGE.INVALID_LIBRARY`, as before. Everything above `NativeLibrary` is the same as in #59313, except for the capabilities, the task contexts, the write modes and the planning of batch scans (see **Other changes**), and predicate pushdown, which this PR leaves out (see **Predicate pushdown**). The Scala classes catch `NativeLibrary.MissingFunctionException` and `NativeLibrary.LoadException` instead of `UnsatisfiedLinkError`. **Arrow data**: Spark still uses `arrow-c-data`, as in #59313, to export and import the schemas and the arrays as Arrow C structs, without copying their buffers. `arrow-c-data` calls the callbacks of the C structs through the JNI library that it bundles, for Linux and macOS on x86_64 and aarch64 and Windows on x86_64: the `get_schema`, `get_next`, `get_last_error` and `release` callbacks of the streams that the libraries export, the `release` callbacks of the schemas and arrays that Spark imports, and the `release` callbacks of the schemas and arrays that Spark exports, which are functions of that JNI library. So Spark calls the libraries with FFM, and the Arrow callbacks go through JNI, inside `arrow-c-data`, once for each batch or schema rather than for each row. Library authors only see the C interface either way. An FFM implementation of the Arrow C data interface, for example in Arrow Java, could replace this JNI library later. **Finding native data sources**: when no Java or Python data source has the requested name, Spark looks for a native data source with that name, in two places: - **Installed libraries, automatically**: like the Python data sources installed in the Python path, a library installed like other native libraries is found without any configuration, by its file name. The library of the data source `rust_range` is `spark_datasource_rust_range`: `libspark_datasource_rust_range.so` on Linux, `libspark_datasource_rust_range.dylib` on macOS and `spark_datasource_rust_range.dll` on Windows. Spark checks whether the file exists in the native library path, which consists of: 1. the directories of `java.library.path`, where the JVM itself finds native libraries. They include `LD_LIBRARY_PATH` on Linux, `DYLD_LIBRARY_PATH` on macOS and `PATH` on Windows, and so `spark.driver.extraLibraryPath` and `spark.executor.extraLibraryPath`; 2. the `lib` directory of each installation prefix in `PATH`: `<prefix>/lib` for each `<prefix>/bin`. So libraries are found where native libraries are usually installed: `/usr/local/lib` (`make install`, CMake, cargo-c, Homebrew on Intel macs), `/opt/homebrew/lib`, `$CONDA_PREFIX/lib` and `$VIRTUAL_ENV/lib` in an activated environment, and `/usr/lib`. A library that implements several data sources is installed under each of their names, for example with symbolic links. The executors load the library from the same path as the driver, or else find it in their own native library path, and only load the same build as the driver, which they check with the SHA-256 checksum of the file. - **Packages of a session**: a native data source package is a zip file with the extension `.sparkpkg`, with a manifest, `spark-native-datasource.json`, that lists the ABI version, the names of the data sources, and the library for each platform such as `linux-x86_64` or `osx-aarch64` (`linux` is Linux with glibc, and Linux with musl is `linux-<arch>-musl`). A session adds packages with `spark.addArtifact` (Scala and Python, classic and Spark Connect: `.sparkpkg` files are added as file artifacts), or with `spark.sql.dataSource.native.paths`. They take precedence over the installed libraries, and the package that a session added last over its other packages and the configured ones, as a Python data source registered again replaces the previous one: a session cannot remove an artifact, so it upgrades a package by adding the new version under another file name. Two configured packages that provide the same data source fail with `NATIVE_DATA_SOURCE_PACKAGE_CONFLICT`. The executors get them through the artifacts of the session, except in local mode, where they read them in place: a configured package is copied there under a name with its checksum, so it can be replaced in place. Spark ignores, and logs, a package that it cannot read or whose ABI version it does not support, unless its manifest lists the requested data source and no other package or installed library provides it. **Short names**: no JVM class is needed for each data source. After the Java data sources (`DataSourceRegister`) and the Python ones, `DataSource.lookupDataSource` looks the name up in the packages of the session and in the native library path, case-insensitively, and resolves it to a single provider class, `NativeDataSourceV2`. The caller then gives it the requested name, the way `PythonDataSourceV2` already works: a small `NamedTableProvider` trait generalizes how `PythonDataSourceV2` gets its short name, so the native provider shares the same call sites. The provider reuses the result of the lookup that found its class, instead of looking the name up again. A Python data source can still be registered with the name of a native data source, and takes precedence over it. Lookups only read manifests and check whether files exist; Spark loads a library when a query uses it. Native data sources work with `spark.read`/`write`, `readStream`/`writeStream`, and SQL tables (`CREATE TABLE ... USING rust_range OPTIONS (...)`, `SELECT` and `INSERT INTO`). **Other changes**, besides those of #59313: - Error conditions: `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION` (`0A000`), raised when a query uses a native data source on Java 21 or earlier, and `NATIVE_DATA_SOURCE_UNSUPPORTED_PLATFORM` (`0A000`), raised where the JNI library of the C data interface of Arrow Java does not load: Arrow Java ships it for Linux with glibc and macOS on x86_64 and aarch64, and for Windows on x86_64. The lookups still find the data source, so the errors say why it cannot be used rather than that it does not exist. `DATA_SOURCE_CONTINUOUS_WRITE_NOT_SUPPORTED` (`0A000`), raised when a continuous query writes to a native data source, before Spark creates a writer, and named after `DATA_SOURCE_CONTINUOUS_SCAN_NOT_SUPPORTED` of Data Source V2 scans. A continuous query that reads a native data source already fails when it starts, since its table does not report `CONTINUOUS_READ`. - A library whose `read` returns success without initializing the stream is reported as an error: `arrow-c-data` calls the callbacks of a stream without checking them, and calling NULL would crash the JVM. - The test library `test_native_datasource.cc` implements the function table, and the tests compile it with `spark_datasource.h` instead of the JNI headers. Compiled with defines, it also provides the libraries that the tests need: a read-only library, a table that ends before `read`, a table of version 2, a failing `spark_ds_init`, no table, and no `spark_ds_init`. - The `CheckConnectJvmClientCompatibility` exclusion of `org.apache.spark.sql.datasource`, added by #59313 for `NativeBridge`, is not needed. - CI: the `Java 25 build with Maven` job of a pull request also runs `NativeDataSourceSuite`, `NativeDataSourceClusterSuite` and `NativeDataSourceConnectSuite`, with the C++ compiler of the runner, since the other jobs, on Java 17, skip the tests that load a library. - Checks of the data: before importing a batch, Spark also checks that the batch and each array have as many children as their type has fields, with non-NULL pointers, and after importing it, that the data agrees with the nullability of the schema to read, at every level. Option keys and values, and column names, that contain U+0000 fail with `NATIVE_DATA_SOURCE_ERROR`. - Packages: the `abiVersion` of a manifest is the oldest version of the interface that the library implements. Spark uses a package whose version is at most the newest that it supports, and ignores the fields of a manifest that it does not know. - Arrow UDFs: Python UDFs over the batches of a native data source need SPARK-60294 (#59551), which disables the Arrow columnar input of Python UDFs (SPARK-56350) by default: it kept references to the pass-through columns of a batch after the source refilled or released their vectors, as the reader of a native data source does. The reader wraps its vectors in `org.apache.spark.sql.execution.vectorized.ReusedArrowColumnVector`, added by SPARK-60306 (#59581), whose `closeIfFreeable` does nothing, so that the columnar input path recognizes them once it is enabled again with a marker for the vectors that their producer reuses (SPARK-60325, #59582). - Batch scans: the Data Source V2 adapter, `execution.datasources.v2.columnar`, plans a batch scan with a driver-only `ColumnarBatchPlanner`, which gets the pushdown, and whose `plan` returns the partitions with a serializable reader factory. For a native data source, the planner holds the reader of the library, and the reader factory only has the location of the library, the name of the data source, the read schema and the state that `serialize_reader` returned. A scan builder builds its scan once, and the scan takes over the planner, or creates one if the operations were not pushed down, so a streaming scan never creates one. A batch is still planned once, when Spark first asks for its partitions, and a planning failure is kept and thrown again. The reader is released once the scan is planned, or by a `Cleaner` if the scan is never planned, as before. - `commit_data_writer` returns its message through the sink. Spark never aborts a data writer once the library returns 0, even if it then rejects the message, and it always calls `close_data_writer` afterwards, which is the only function that releases a data writer. - `arrow-c-data`: SPARK-59718 added it as a provided dependency of `sql/core` for in-process Python UDFs. This PR keeps it a compile dependency, in the `jars` directory of a distribution (`dev/deps`, `LICENSE-binary`), since native data sources need it at run time, so the docs of in-process Python UDFs and the README of their benchmark no longer ask users to supply its JAR. - Publishing the header: `dev/make-distribution.sh` copies `spark_datasource.h` to the `include` directory of a binary distribution, as a JDK ships `include/jni.h`, and the `spark-sql` JAR contains it too. The header only depends on the C standard library, so a library can also copy it into its source tree and build without Spark, as for the Arrow C data interface and ADBC's `adbc.h`; the docs say so, with the URL of the header at the tag of a release. The other changes are those of #59313: the configurations `spark.sql.dataSource.native.enabled` and `spark.sql.dataSource.native.paths`, the error conditions, the `arrow-c-data` dependency, the "Native Data Sources" documentation page (with the examples below), and: - From the latest review round of #59313 (aa0f3ba, e3ff80c), in the terms of the C interface: - `create_writer` gets the ID of the streaming query, `SparkDsWriteInfo.query_id`, which stays the same when the query restarts from its checkpoint. - Offsets must be compact JSON on a single line, as the offset log stores them. - An installed library replaced in place fails with `INVALID_NATIVE_DATA_SOURCE_LIBRARY.REPLACED`, and the caches identify files by their file key too. - Concurrent distributions of a configured package in a session are serialized, a scan planned again after a failure fails with the same error, the unsupported Arrow types and the link errors of Arrow's own JNI library are reported as `NATIVE_DATA_SOURCE_ERROR`, the column vectors of a read are not freeable between batches (which broke struct columns), packages of an unsupported version are logged, PySpark `addArtifact` only passes `file:` URIs as they are, and the docs describe the lifecycle of writers and the memory of native data sources. - A NULL handle was already reported as an error here, by `NativeLibrary`, and a missing offset as well, since the library returns it through the sink. - Existing classes: - `DataSource.lookupDataSource` looks up the native data sources after the Python ones, and `DataSourceRegistration` skips them when it registers a Python data source. - `ArtifactManager` lists the native data source packages of a session, and forgets the cached packages read from the artifacts of a session when it cleans them up. - The classic `DataFrameWriter` builds its commands, and `DataStreamWriter` resolves its sink, with the session of the Dataset active, as the analysis of a query does: the lookups of the Python and native data sources use the active session. - PySpark `addArtifacts` adds `.sparkpkg` files to the session whatever the flags: in classic mode through the JVM session, and with Spark Connect as file artifacts. ### Examples Both examples implement a read-only data source that returns the numbers `[0, end)` as a column `id`, in two partitions, with the option `end` (10 by default). They declare the capability `SPARK_DS_BATCH_READ`, and only implement the functions needed for batch reads, plus `last_error`, leaving the others NULL in their table. <details> <summary><b>Rust</b> (with arrow-rs only)</summary> `Cargo.toml`: ```toml [package] name = "rust_range" version = "0.1.0" edition = "2021" [lib] name = "spark_datasource_rust_range" crate-type = ["cdylib"] [dependencies] arrow-array = { version = "58", features = ["ffi"] } arrow-schema = { version = "58", features = ["ffi"] } ``` `src/lib.rs`: ```rust //! A native data source for Apache Spark, written in Rust. It reads the numbers [0, end) as a //! column `id`, in two partitions. The option `end` defaults to 10. use std::cell::RefCell; use std::ffi::{c_char, c_int, c_void, CStr, CString}; use std::panic::{catch_unwind, AssertUnwindSafe}; use std::sync::Arc; use arrow_array::ffi_stream::FFI_ArrowArrayStream; use arrow_array::{Int64Array, RecordBatch, RecordBatchIterator}; use arrow_schema::ffi::FFI_ArrowSchema; use arrow_schema::{DataType, Field, Schema, SchemaRef}; type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>; /// `SparkDsSink`: receives a byte array returned to Spark, and copies it. type Sink = unsafe extern "C" fn(sink: *mut c_void, data: *const u8, length: i64); /// A function of `SparkDsLibrary` that this data source does not implement, left NULL. type Unused = Option<unsafe extern "C" fn()>; /// `SparkDsCreateInfo` of `spark_datasource.h`: the name and the options of a data source. #[repr(C)] pub struct SparkDsCreateInfo { size: i64, name: *const c_char, option_keys: *const *const c_char, option_values: *const *const c_char, num_options: i64, } /// `SparkDsLibrary` of `spark_datasource.h`, up to `read`, the last function that this data /// source implements: Spark only reads the functions that fit in `size`. #[repr(C)] pub struct SparkDsLibrary { size: i64, abi_version: i32, last_error: unsafe extern "C" fn() -> *const c_char, create: unsafe extern "C" fn( info: *const SparkDsCreateInfo, capabilities: *mut u64, source: *mut *mut c_void, ) -> c_int, schema: unsafe extern "C" fn(source: *mut c_void, out: *mut FFI_ArrowSchema) -> c_int, create_reader: unsafe extern "C" fn( source: *mut c_void, schema: *mut FFI_ArrowSchema, reader: *mut *mut c_void, ) -> c_int, create_stream_reader: Unused, create_writer: Unused, close: unsafe extern "C" fn(source: *mut c_void), push_limit: Unused, prune_columns: Unused, partitions: unsafe extern "C" fn(reader: *mut c_void, add: Sink, sink: *mut c_void) -> c_int, serialize_reader: Unused, close_reader: unsafe extern "C" fn(reader: *mut c_void), initial_offset: Unused, latest_offset: Unused, stream_partitions: Unused, serialize_stream_reader: Unused, commit_offset: Unused, close_stream_reader: Unused, read: unsafe extern "C" fn( task: *const c_void, reader_state: *const u8, reader_state_length: i64, partition: *const u8, partition_length: i64, out: *mut FFI_ArrowArrayStream, ) -> c_int, } const SPARK_DS_BATCH_READ: u64 = 1; static LIBRARY: SparkDsLibrary = SparkDsLibrary { size: std::mem::size_of::<SparkDsLibrary>() as i64, abi_version: 1, last_error, create, schema, create_reader, create_stream_reader: None, create_writer: None, close, push_limit: None, prune_columns: None, partitions, serialize_reader: None, close_reader, initial_offset: None, latest_offset: None, stream_partitions: None, serialize_stream_reader: None, commit_offset: None, close_stream_reader: None, read, }; /// The function that Spark calls when it loads the library. #[no_mangle] pub unsafe extern "C" fn spark_ds_init( _host: *const c_void, library: *mut *const SparkDsLibrary, ) -> c_int { *library = &LIBRARY; 0 } struct RangeSource { end: i64, } struct RangeReader { end: i64, } thread_local! { static LAST_ERROR: RefCell<CString> = RefCell::new(CString::default()); } fn arrow_schema() -> SchemaRef { Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])) } /// Runs `f`, and reports its error or panic to Spark: neither may cross into the JVM. fn call(f: impl FnOnce() -> Result<()>) -> c_int { let message = match catch_unwind(AssertUnwindSafe(f)) { Ok(Ok(())) => return 0, Ok(Err(e)) => e.to_string(), Err(_) => "the data source panicked".to_string(), }; LAST_ERROR.with(|error| *error.borrow_mut() = CString::new(message).unwrap_or_default()); 1 } unsafe extern "C" fn last_error() -> *const c_char { LAST_ERROR.with(|error| error.borrow().as_ptr()) } unsafe extern "C" fn create( info: *const SparkDsCreateInfo, capabilities: *mut u64, source: *mut *mut c_void, ) -> c_int { call(|| { let info = &*info; let mut range = RangeSource { end: 10 }; for i in 0..info.num_options as usize { if CStr::from_ptr(*info.option_keys.add(i)).to_str()? == "end" { range.end = CStr::from_ptr(*info.option_values.add(i)).to_str()?.parse()?; } } *capabilities = SPARK_DS_BATCH_READ; *source = Box::into_raw(Box::new(range)) as *mut c_void; Ok(()) }) } unsafe extern "C" fn schema(_source: *mut c_void, out: *mut FFI_ArrowSchema) -> c_int { call(|| { std::ptr::write(out, FFI_ArrowSchema::try_from(arrow_schema().as_ref())?); Ok(()) }) } unsafe extern "C" fn create_reader( source: *mut c_void, _schema: *mut FFI_ArrowSchema, reader: *mut *mut c_void, ) -> c_int { call(|| { let end = (*(source as *const RangeSource)).end; *reader = Box::into_raw(Box::new(RangeReader { end })) as *mut c_void; Ok(()) }) } unsafe extern "C" fn close(source: *mut c_void) { drop(Box::from_raw(source as *mut RangeSource)); } unsafe extern "C" fn partitions(reader: *mut c_void, add: Sink, sink: *mut c_void) -> c_int { call(|| { let end = (*(reader as *const RangeReader)).end; for (start, end) in [(0, end / 2), (end / 2, end)] { let bytes = [start.to_le_bytes(), end.to_le_bytes()].concat(); add(sink, bytes.as_ptr(), bytes.len() as i64); } Ok(()) }) } unsafe extern "C" fn close_reader(reader: *mut c_void) { drop(Box::from_raw(reader as *mut RangeReader)); } unsafe extern "C" fn read( _task: *const c_void, _reader_state: *const u8, _reader_state_length: i64, partition: *const u8, partition_length: i64, out: *mut FFI_ArrowArrayStream, ) -> c_int { call(|| { let bytes = std::slice::from_raw_parts(partition, partition_length as usize); let start = i64::from_le_bytes(bytes[0..8].try_into()?); let end = i64::from_le_bytes(bytes[8..16].try_into()?); let ids = Int64Array::from_iter_values(start..end); let batch = RecordBatch::try_new(arrow_schema(), vec![Arc::new(ids)])?; let batches = RecordBatchIterator::new(vec![Ok(batch)], arrow_schema()); std::ptr::write(out, FFI_ArrowArrayStream::new(Box::new(batches))); Ok(()) }) } ``` Build and install, here on macOS: ```bash cargo build --release cp target/release/libspark_datasource_rust_range.dylib /usr/local/lib/ ``` Or package, to add it to a session with `spark.addArtifact`: ```bash mkdir -p package/osx-aarch64 cp target/release/libspark_datasource_rust_range.dylib package/osx-aarch64/ cat > package/spark-native-datasource.json <<EOF { "abiVersion": 1, "dataSources": ["rust_range"], "libraries": {"osx-aarch64": "osx-aarch64/libspark_datasource_rust_range.dylib"} } EOF (cd package && zip -r ../rust_range.sparkpkg .) ``` </details> <details> <summary><b>C++</b> (with Arrow C++)</summary> `cpp_range.cc`: ```cpp // A native data source for Apache Spark, written in C++ with Arrow C++. It reads the numbers // [0, end) as a column `id`, in two partitions. The option `end` defaults to 10. #include <cstdint> #include <cstring> #include <memory> #include <stdexcept> #include <string> #include <arrow/api.h> #include <arrow/c/bridge.h> #include "spark_datasource.h" namespace { struct RangeSource { int64_t end = 10; }; struct RangeReader { int64_t end; }; thread_local std::string last_error; std::shared_ptr<arrow::Schema> Schema() { return arrow::schema({arrow::field("id", arrow::int64(), /*nullable=*/false)}); } // Runs `f`, and reports its error to Spark: no C++ exception may cross into the JVM. template <typename F> int Call(F&& f) { try { f(); return 0; } catch (const std::exception& e) { last_error = e.what(); return 1; } catch (...) { last_error = "unknown error"; return 1; } } void Check(const arrow::Status& status) { if (!status.ok()) throw std::runtime_error(status.ToString()); } const char* LastError() { return last_error.c_str(); } int Create(const SparkDsCreateInfo* info, uint64_t* capabilities, void** source) { return Call([&] { auto range = std::make_unique<RangeSource>(); for (int64_t i = 0; i < info->num_options; i++) { if (std::string(info->option_keys[i]) == "end") { range->end = std::stoll(info->option_values[i]); } } *capabilities = SPARK_DS_BATCH_READ; *source = range.release(); }); } int GetSchema(void*, struct ArrowSchema* out) { return Call([&] { Check(arrow::ExportSchema(*Schema(), out)); }); } int CreateReader(void* source, struct ArrowSchema*, void** reader) { return Call([&] { *reader = new RangeReader{static_cast<RangeSource*>(source)->end}; }); } void Close(void* source) { delete static_cast<RangeSource*>(source); } int Partitions(void* reader, SparkDsSink add_partition, void* sink) { return Call([&] { int64_t end = static_cast<RangeReader*>(reader)->end; int64_t bounds[2][2] = {{0, end / 2}, {end / 2, end}}; for (const auto& partition : bounds) { add_partition(sink, reinterpret_cast<const uint8_t*>(partition), sizeof(partition)); } }); } void CloseReader(void* reader) { delete static_cast<RangeReader*>(reader); } int Read(const SparkDsTaskContext*, const uint8_t*, int64_t, const uint8_t* partition, int64_t, struct ArrowArrayStream* out) { return Call([&] { int64_t bounds[2]; std::memcpy(bounds, partition, sizeof(bounds)); arrow::Int64Builder ids; for (int64_t id = bounds[0]; id < bounds[1]; id++) Check(ids.Append(id)); std::shared_ptr<arrow::Array> array; Check(ids.Finish(&array)); auto batch = arrow::RecordBatch::Make(Schema(), array->length(), {array}); auto batches = arrow::RecordBatchReader::Make({batch}, Schema()).ValueOrDie(); Check(arrow::ExportRecordBatchReader(batches, out)); }); } // The functions of the data source. The ones that it does not implement are NULL. const SparkDsLibrary kLibrary = { .size = sizeof(SparkDsLibrary), .abi_version = SPARK_DS_ABI_VERSION, .last_error = LastError, .create = Create, .schema = GetSchema, .create_reader = CreateReader, .close = Close, .partitions = Partitions, .close_reader = CloseReader, .read = Read, }; } // namespace extern "C" SPARK_DS_EXPORT int spark_ds_init(const SparkDsHost*, const SparkDsLibrary** library) { *library = &kLibrary; return 0; } ``` Get `spark_datasource.h`, which is in the `spark-sql` JAR, then build (Arrow C++ 24 requires C++20) and install, or package as above: ```bash unzip -j "$SPARK_HOME"/jars/spark-sql_*.jar org/apache/spark/sql/datasource/spark_datasource.h c++ -std=c++20 -O2 -shared -fPIC -I. -I"$ARROW_HOME/include" -L"$ARROW_HOME/lib" -larrow \ -o libspark_datasource_cpp_range.dylib cpp_range.cc cp libspark_datasource_cpp_range.dylib /usr/local/lib/ ``` </details> Usage is the same as in #59313, on Java 22 or later, in classic PySpark or with Spark Connect. Once installed, the data sources are found automatically, without `spark.addArtifact` or any configuration: ```python >>> spark.read.format("rust_range").option("end", 5).load().show() +---+ | id| +---+ | 0| | 1| | 2| | 3| | 4| +---+ >>> spark.read.format("cpp_range").option("end", 5).load().filter("id >= 3").collect() [Row(id=3), Row(id=4)] >>> spark.sql("CREATE TABLE t USING rust_range OPTIONS (end 3)") >>> spark.sql("SELECT * FROM t").collect() [Row(id=0), Row(id=1), Row(id=2)] ``` Or added to a session as a package: ```python >>> spark.addArtifact("rust_range/rust_range.sparkpkg") >>> spark.read.format("rust_range").option("end", 5).load().count() 5 ``` ### Why are the changes needed? Some data systems only have a client in native code, such as an in-house storage system written in C++ or the C SDK of a vendor, and Spark only needs to read and write them. Using them from Spark today requires writing a JVM Data Source V2 connector with its own JNI glue. With this change, such a source implements a stable binary interface once, in Rust, C++ or any language that can export C functions, is installed like any other native library or distributed as a single file, keeps working with the later releases of Spark that support its version of the interface, and works with every Spark API, including Spark Connect clients in any language, without any JVM code. A source that needs what version 1 does not cover yet, such as row-level operations, or extensions outside of Data Source V2, such as table functions and SQL commands, is better served by a JVM connector, which can still read and write through a native library with the Arrow C stream interface, as lance-spark does with Lance. The docs say so in a new "When to Use a Native Data Source" section. With FFM, the interface is also a plain C ABI, so a library needs no JVM headers or bindings, and other hosts than Spark could load the same libraries. ### Does this PR introduce _any_ user-facing change? Yes. It adds native data sources, with the `spark_datasource.h` interface, the configurations and the error conditions above, on Java 22 or later. Spark finds the native data source libraries installed in the native library path, and `spark.addArtifact` now accepts native data source packages (`.sparkpkg` files). Also, in classic mode, writing with a session that is not the active one now finds the data sources of that session, such as its Python data sources, because `DataFrameWriter` and `DataStreamWriter` resolve the data source with that session active. ### How was this patch tested? All the tests of #59313, ported to the C interface: - `NativeDataSourceSuite`, with the dependency-free C++ test library (`sql/core/src/test/resources/native-datasource/test_native_datasource.cc`), now implementing `spark_datasource.h`, which the tests compile with the C++ compiler of the machine against the header from the classpath. It covers inferred and user-specified schemas, partitions and batches, the supported types, column and limit pushdown, batch writes (append and overwrite), aborted writes, streaming reads and writes over several micro-batches with offset commits, SQL tables with `OPTIONS` (`SELECT` and `INSERT INTO`), errors reported by native code, missing functions, several libraries exporting the same functions side by side, `spark.addArtifact`, packages in directories, package conflicts, the precedence of Java data sources, invalid packages, installed libraries and invalid installed libraries, plus the unit tests for the manifest, the platform names and the native library path. - Tests added during the review, in `NativeDataSourceSuite` unless noted: - Packages: invalid and unsupported packages are ignored unless no other package or installed library provides the data source; a configured package replaced in place; the copies distributed to the executors are named by checksum and are not packages of the session; the cached packages of a session are forgotten when its artifacts are cleaned up; configured directories that cannot be read; distributing a package without an active session. - Lookups: a single lookup for each resolution, which the next lookup on the thread clears; a Python data source registered with the name of an installed native one; batch and streaming writes with a session that is not the active one. - Writes: a failed `commitDataWriter` aborts the data writer; a failed `commit` aborts the write, then closes the writer; the test library records when a writer is closed, which checks that a batch write closes its writer, and that each micro-batch of a streaming write closes the writer of the previous one; `ColumnarDataSourceSuite` checks that each micro-batch creates its stream writer with the ID of the query, and that batches end at `spark.sql.execution.arrow.maxBytesPerBatch`. - Cluster mode: new `NativeDataSourceClusterSuite` runs in `local-cluster[2,1,1024]`, where the executors are separate JVMs: batch reads, writes and streaming with a package added with `spark.addArtifact`, and a configured package that the executors can only load from its distributed copy (named by checksum, under the UUID of the session), which they check by its checksum. - Installed libraries: an executor does not load a library with another checksum than on the driver, at the driver's path or in its native library path. - `spark.addArtifact` in classic PySpark accepts `file://` URIs and paths relative to the working directory of Python. - Reads: the columns of the batches are the `ReusedArrowColumnVector`s of Spark (`read structs in several batches`); read schemas with collations, and the scan described by the name of the data source; sliced arrays, arrays shorter than their parent (a column, a struct field or the elements of a map), `decimal256` and list views are rejected with the name of the column, maps whose entry fields are named `keys` and `values` (as arrow-rs names them) are read, and `limit(...).offset(...)` asks the library for the limit once. - Writes whose upstream stages fail with adaptive execution create no writer (also in `ColumnarDataSourceSuite`). - From the latest review round of #59313: no limit is pushed with a filter; the columns that a query filters on are read; a struct column read in several batches through whole-stage codegen and a nested field; large lists and dictionary encoding rejected with the name of the column; a reader planned again after a failure; offsets that are missing, multi-line or not JSON; the query ID across a restart; a failed micro-batch aborted with its epoch ID; `stop()` closing the stream reader; NULL handles from each of the five create functions; concurrent distributions of a configured package; a valid package with another checksum; the logging of packages of an unsupported version; an installed library replaced in place; link errors; and `addArtifact` with a file name that contains a colon. - `NativeDataSourceConnectSuite` (Spark Connect), `NativeDataSourceClusterSuite` (`local-cluster`, executors in separate JVMs) and `ColumnarDataSourceSuite`. New tests: - `native data sources require Java 22 or later`: on Java 21 or earlier, a query that uses an installed native data source fails with `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`, before Spark loads the library. It runs only on Java 21 or earlier, so it runs in the default CI on Java 17. - `report the errors of native code`: a library without `last_error` reports `create failed with the error code 1.`, and a library whose `read` returns success without initializing the stream fails with `The library did not initialize the stream, or a callback of the stream is NULL.` New tests for the shape of the interface, in `NativeDataSourceSuite`: - `the bindings match the layout of spark_datasource.h`: compiles a program that prints the size of each struct of the header and the offset of each field, and compares them with the offsets that `NativeLibrary` uses. - `operations that a data source does not support`: a read-only library, whose data sources only declare `SPARK_DS_BATCH_READ`, fails batch writes while Spark analyzes the query, with `UNSUPPORTED_FEATURE.TABLE_OPERATION`, before a shuffle stage of the data to write fails, and streaming reads and streaming writes with `DATA_SOURCE_*_NOT_SUPPORTED`; a data source whose library implements batch reads but that only declares streaming reads fails batch reads; a declared capability with a NULL function fails with the name of the function; an overwrite without `SPARK_DS_TRUNCATE` fails, in batch while Spark analyzes the query, and in streaming with the output mode `complete`; and a write to a table in a catalog whose options remove a capability of the table fails when Spark plans it, with `DATA_SOURCE_BATCH_WRITE_NOT_SUPPORTED`, or `NATIVE_DATA_SOURCE_ERROR` for an overwrite. - `ColumnarDataSourceSuite`: `the table reports the capabilities of the data source`, where a batch write or overwrite that the data source does not support fails while Spark analyzes it, without a writer, and `build and plan a batch scan once`, where a scan builder builds one scan, whose batch is planned once, and a failed planning is not retried but fails again with the same error. - `reject the data that has nulls where the schema to read has none`; `reject the types and the strings that a library cannot get`, with an interval in a read, an array of variants in a write, which fails before `create_writer`, and an option with U+0000; `reject the Arrow data that Spark cannot read`, with batches without their columns; and `parse manifests`, with an `abiVersion` that is not positive. - From the review of gerashegalov: `continuous processing fails with a native data source`, for writes and reads; `the package that a session added last replaces the other packages`; `fail clearly where the C data interface of Arrow Java does not load`; `reject the Arrow data that Spark cannot read`, with batches whose struct marks rows as null, counted or not, and arrays with a `null_count` of -1 but no validity buffer, which Arrow Java cannot import; and `platform names`, with musl. - `Arrow UDFs on the columns of a native data source`: a pandas UDF over one batch, two and 2,000, which checks the pass-through columns of each row against the result of the UDF. Locally, it fails without #59551 and passes with it. The CI of pull requests skips it, since the job that runs the native data source suites has no pandas. - `read only the functions that fit in the table of a library`: a table that ends before `read`, like the table of a library built against an earlier header, makes `read` unsupported rather than read past its end. - `pass the version of Spark, and log with the log of Spark`: the library checks `SparkDsHost`, and its messages reach the log of Spark at their level, with the path of the library. - `pass the task to reads and data writers`: each read and each data writer gets the IDs of its task. - `tell a read that its task was killed`: a read that waits until `is_interrupted` returns true ends once its job is canceled. - `reject the libraries that spark_ds_init does not initialize`: a failing `spark_ds_init`, which logs why once, a library that returns no table, and one that does not export `spark_ds_init`, with the same error when they are used again, since Spark initializes a library only once. - `invalid installed libraries` and `invalid packages` check a table of version 2, which Spark rejects with `UNSUPPORTED_ABI_VERSION`. The tests that load a library are skipped before Java 22, like they are skipped without a C++ compiler. So the jobs on Java 17 only run the tests that do not load a library, and the `Java 25 build with Maven` job of a pull request runs the native data source suites, as do the scheduled Java 25 builds. `do not abort a data writer that committed with an invalid message` checks that a data writer released by its commit is not aborted. Locally on macOS (osx-aarch64): - After leaving predicate pushdown out (1f40484): `NativeDataSourceSuite` and `ColumnarDataSourceSuite` pass on Java 26 (Homebrew OpenJDK 26.0.1), with Spark built with SBT on the same JDK. - Java 26 (Homebrew OpenJDK 26.0.1; Spark built with Java 17): `NativeDataSourceSuite`, `NativeDataSourceClusterSuite` (with the executors on Java 26 too), `ColumnarDataSourceSuite` and `NativeDataSourceConnectSuite` pass. With the executors of `local-cluster` on Java 17, the tasks fail with `NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`, as expected. - Java 17: `NativeDataSourceSuite` passes, with the tests that load a library skipped, and the Java version test run. - The Rust and C++ examples above were built with the commands above (Rust with arrow-rs 58, C++ with Arrow C++ 24 and `-std=c++20`; both export only `spark_ds_init`), installed in a directory of `java.library.path`, and read by Spark on Java 26: `spark.read.format("rust_range").option("end", 100).load()` and the same with `cpp_range` return the 100 rows in two partitions, and `option("end", "x")` fails with `NATIVE_DATA_SOURCE_ERROR` and the message of the library (`invalid digit found in string` and `stoll: no conversion`). - The documented steps of the C++ example work with the header in the `include` directory of a distribution: the header copied as `dev/make-distribution.sh` copies it, then the documented `c++` command with `-I"$SPARK_HOME/include"`. - The C++ test library builds without warnings with Apple clang (`-Wall -Wextra`), in all its variants. - `SparkThrowableSuite`, for the new error conditions. - `checkstyle` and `scalastyle` of the changed modules. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) This pull request and its description were written by Isaac. Closes #59424 from HyukjinKwon/SPARK-60101-ffm. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com> (cherry picked from commit 99a1bf9) Signed-off-by: Hyukjin Kwon <hyukjin.kwon@databricks.com>

What changes were proposed in this pull request?
This PR adds native data sources: data sources implemented in native code, such as Rust or C++, without any JVM code. A native data source is a shared library that implements a binary interface. Spark finds it automatically when it is installed like other native libraries, or in a package file added to a session, and loads it when a query uses it. Spark adapts it to Data Source V2, with batch and micro-batch streaming reads, batch and streaming writes, and predicate, column and limit pushdown. Data is exchanged only as columnar batches, through the Arrow C data interface.
Public API: a single class,
org.apache.spark.sql.datasource.NativeBridge(@Evolving, since 4.4.0). A library implements itsnativemethods as JNI functions, such asJava_org_apache_spark_sql_datasource_NativeBridge_createDataSource, and its Javadoc is the specification of the interface: the lifecycle, the handles, the ownership of the Arrow structs, errors (Java exceptions, reported asNATIVE_DATA_SOURCE_ERROR), the JSON format of the pushed predicates (a fixed set of comparison,IN, null, string and boolean predicates), which Spark evaluates again after reading, so that a library only uses them to skip data and must never skip a row for which they are true under the semantics of Spark SQL, for example for NaN and-0.0(the jar of Spark SQL contains test cases for libraries,org/apache/spark/sql/datasource/predicate-cases.json), the encoding of strings (Java strings, which JNI converts to modified UTF-8; the JSON of the predicates is ASCII), and the compatibility rules: within an ABI version, functions are only added as optional functions, and the signature and semantics of an existing function never change. Spark loads each library into its own copy of the class, with a separate class loader, so any number of libraries exporting the same functions can be loaded side by side. Spark itself ships no native code for native data sources, besides the JNI library that the newarrow-c-datadependency bundles. The interface uses JNI rather than the FFM API, which is final only from Java 22 while Spark supports Java 17 and 21; the interface is versioned, so that a later version can be a plain C ABI called through FFM.abiVersion,createDataSource,closeDataSourceschemacreateReader,partitions,read,closeReaderpushPredicates,pushLimit,pruneColumns,serializeReadercreateStreamReader,initialOffset,latestOffset,streamPartitions,read,closeStreamReaderserializeStreamReader,commitOffsetcreateWriter,commit,abort,closeWriter,createDataWriter,write,commitDataWriter,abortDataWriterserializeWritercreateStreamWriter,commit,abort,closeWriter,createDataWriter,write,commitDataWriter,abortDataWriterserializeWriterA library only exports the functions of the operations it supports. Spark reports a missing required function as
DATA_SOURCE_*_NOT_SUPPORTED.Finding native data sources: when no Java or Python data source has the requested name, Spark looks for a native data source with that name, in two places:
Installed libraries, automatically: like the Python data sources installed in the Python path, a library installed like other native libraries is found without any configuration, by its file name. The library of the data source
rust_rangeisspark_datasource_rust_range:libspark_datasource_rust_range.soon Linux,libspark_datasource_rust_range.dylibon macOS andspark_datasource_rust_range.dllon Windows. Spark checks whether the file exists in the native library path, which consists of:java.library.path, where the JVM itself finds native libraries. They includeLD_LIBRARY_PATHon Linux,DYLD_LIBRARY_PATHon macOS andPATHon Windows, and sospark.driver.extraLibraryPathandspark.executor.extraLibraryPath;libdirectory of each installation prefix inPATH:<prefix>/libfor each<prefix>/bin.So libraries are found where native libraries are usually installed:
/usr/local/lib(make install, CMake, cargo-c, Homebrew on Intel macs),/opt/homebrew/lib,$CONDA_PREFIX/liband$VIRTUAL_ENV/libin an activated environment, and/usr/lib. A library that implements several data sources is installed under each of their names, for example with symbolic links. The executors load the library from the same path as the driver, or else find it in their own native library path, and only load the same build as the driver, which they check with the SHA-256 checksum of the file. A JVM loads a library from a path only once, so after a new build is installed at the same path, the queries of an application that loaded the previous build fail withINVALID_NATIVE_DATA_SOURCE_LIBRARY.REPLACED, which asks to restart it, instead of running mixed builds.Packages of a session: a native data source package is a zip file with the extension
.sparkpkg, with a manifest,spark-native-datasource.json, that lists the ABI version, the names of the data sources, and the library for each platform such aslinux-x86_64orosx-aarch64. A session adds packages withspark.addArtifact(Scala and Python, classic and Spark Connect:.sparkpkgfiles are added as file artifacts), or withspark.sql.dataSource.native.paths. They take precedence over the installed libraries, and the executors get them through the artifacts of the session, except in local mode, where they read them in place: a configured package is copied there under a name with its checksum, so it can be replaced in place. Spark ignores, and logs, a package that it cannot read or whose ABI version it does not support, unless its manifest lists the requested data source and no other package or installed library provides it.Short names: no JVM class is needed for each data source. After the Java data sources (
DataSourceRegister) and the Python ones,DataSource.lookupDataSourcelooks the name up in the packages of the session and in the native library path, case-insensitively, and resolves it to a single provider class,NativeDataSourceV2. The caller then gives it the requested name, the wayPythonDataSourceV2already works: a smallNamedTableProvidertrait generalizes howPythonDataSourceV2gets its short name, so the native provider shares the same call sites. The provider reuses the result of the lookup that found its class, instead of looking the name up again. A Python data source can still be registered with the name of a native data source, and takes precedence over it. Lookups only read manifests and check whether files exist; Spark loads a library when a query uses it. Native data sources work withspark.read/write,readStream/writeStream, and SQL tables (CREATE TABLE ... USING rust_range OPTIONS (...),SELECTandINSERT INTO).Internals:
execution.datasources.v2.columnar: an internal columnar data source interface, adapted to Data Source V2. Writes convert rows to Arrow batches that end atspark.sql.execution.arrow.maxRecordsPerBatchrows or once they holdspark.sql.execution.arrow.maxBytesPerBatchbytes, whichever is reached first, as for Python UDFs. The writer of a write is created right before its tasks start, after the shuffle stages that adaptive query execution runs first, so that each writer is committed or aborted: the last stage of the data runs in the tasks of the write, so a failure there aborts the writer. Spark evaluates all the pushed predicates again after reading, and only shows the predicates that a library evaluates completely in the plan. A micro-batch streaming write gets a new stream writer for each micro-batch, and the previous one of the query is closed when the next one is created.execution.datasources.v2.ffi: the discovery of packages and installed libraries, package parsing, the distribution of packages to the executors, library loading, the implementation of the internal interface on top ofNativeBridgewitharrow-c-data, which reads the Arrow streams array by array to reject arrays with a non-zero offset (Arrow Java ignores offsets) or with fewer values than their parent refers to (Arrow Java pads them with nulls), the JSON encoding of pushed predicates, limited to the predicates listed inNativeBridge, and the checks of what a library returns: non-zero handles and offsets that are JSON on a single line. The reader imports the batches of a partition into the same Arrow vectors, which are not freeable between batches.Other changes:
spark.sql.dataSource.native.enabled(static, defaulttrue) andspark.sql.dataSource.native.paths.NATIVE_DATA_SOURCE_ERROR,INVALID_NATIVE_DATA_SOURCE_LIBRARY(CANNOT_LOAD,REPLACED,UNSUPPORTED_ABI_VERSION),INVALID_NATIVE_DATA_SOURCE_PACKAGE(INVALID_MANIFEST,INVALID_LIBRARY,UNSUPPORTED_ABI_VERSION,UNSUPPORTED_PLATFORM),NATIVE_DATA_SOURCE_LIBRARY_NOT_FOUND,NATIVE_DATA_SOURCE_PACKAGE_CONFLICTandNATIVE_DATA_SOURCE_PACKAGE_NOT_FOUND.org.apache.arrow:arrow-c-data, with the same version asarrow-vector. It bundles its own JNI library for Linux, macOS and Windows.SparkSession.addArtifactinsql/api, which now lists.sparkpkgfiles.org/apache/spark/sql/datasource/predicate-cases.jsoninsql/core, the predicate test cases for libraries, generated byNativeDataSourceSuitefrom the evaluation of Spark.DataSource.lookupDataSourcelooks up the native data sources after the Python ones, andDataSourceRegistrationskips them when it registers a Python data source.ArtifactManagerlists the native data source packages of a session, and forgets the cached packages read from the artifacts of a session when it cleans them up.DataFrameWriterbuilds its commands, andDataStreamWriterresolves its sink, with the session of the Dataset active, as the analysis of a query does: the lookups of the Python and native data sources use the active session.addArtifactsadds.sparkpkgfiles to the session whatever the flags: in classic mode through the JVM session (passingfile:URIs as they are and resolving the other paths in Python), and with Spark Connect as file artifacts.CheckConnectJvmClientCompatibilityexcludes the neworg.apache.spark.sql.datasourcepackage, which the Spark Connect client does not have.Examples
Both examples implement a read-only data source that returns the numbers
[0, end)as a columnid, in two partitions, with the optionend(10 by default). They only export the 8 functions needed for batch reads.Rust (with the
jnicrate and arrow-rs)Cargo.toml:src/lib.rs:Build and install, here on macOS:
Or package, to add it to a session with
spark.addArtifact:C++ (with Arrow C++)
cpp_range.cc:Build (Arrow C++ 24 requires C++20) and install, or package as above:
Usage, in classic PySpark or with Spark Connect. Once installed, the data sources are found automatically, without
spark.addArtifactor any configuration:Or added to a session as a package:
Why are the changes needed?
Many data systems have their best client libraries in native code, such as arrow-rs, DataFusion, delta-rs, iceberg-rust and Lance in Rust, or C++ libraries. Using them from Spark today requires writing a JVM Data Source V2 connector with its own JNI glue. With this change, a native data source implements a stable binary interface once, is installed like any other native library or distributed as a single file, and works with every Spark API, including Spark Connect clients in any language, without any JVM code.
Does this PR introduce any user-facing change?
Yes. It adds native data sources, with the
NativeBridgeinterface, the configurations and the error conditions above. Spark finds the native data source libraries installed in the native library path, andspark.addArtifactnow accepts native data source packages (.sparkpkgfiles). Also, in classic mode, writing with a session that is not the active one now finds the data sources of that session, such as its Python data sources, becauseDataFrameWriterandDataStreamWriterresolve the data source with that session active.How was this patch tested?
NativeDataSourceSuite, with a dependency-free C++ test library (sql/core/src/test/resources/native-datasource/test_native_datasource.cc) compiled by the tests with the C++ compiler of the machine; the tests that load it are skipped if there is no compiler. It covers inferred and user-specified schemas, partitions and batches, the supported types, predicate, column and limit pushdown, batch writes (append and overwrite), aborted writes, streaming reads and writes over several micro-batches with offset commits, SQL tables withOPTIONS(SELECTandINSERT INTO), errors reported by native code, missing functions, several libraries exporting the same functions side by side,spark.addArtifact, packages in directories, package conflicts, the precedence of Java data sources, and invalid packages (manifest, ABI version, platform and library). It also has unit tests for the manifest, the platform names, the native library path and the JSON encoding of predicates.NativeDataSourceSuitealso covers installed libraries: found automatically in the native library path with case-insensitive names, a library installed under several names, writes, the precedence of the packages of a session, the lookup by name on executors (NATIVE_DATA_SOURCE_LIBRARY_NOT_FOUND), and invalid installed libraries (CANNOT_LOAD, andUNSUPPORTED_ABI_VERSION, with the same error when the library is used again).NativeDataSourceConnectSuite, with a Spark Connect client and server:spark.addArtifactof a package, reads, writes, SQL tables, the isolation of the packages of different sessions, and installed libraries.ColumnarDataSourceSuite, which tests the internal Data Source V2 adapter with a data source implemented on the JVM..sparkpkgartifacts in classic mode, whatever the flags and in lists with other files, and with Spark Connect.NativeDataSourceSuiteunless noted:commitDataWriteraborts the data writer; a failedcommitaborts the write, then closes the writer; the test library records when a writer is closed, which checks that a batch write closes its writer, and that each micro-batch of a streaming write closes the writer of the previous one;ColumnarDataSourceSuitechecks that each micro-batch creates its stream writer with the ID of the query, and that batches end atspark.sql.execution.arrow.maxBytesPerBatch.NativeDataSourceClusterSuiteruns inlocal-cluster[2,1,1024], where the executors are separate JVMs: batch reads, writes and streaming with a package added withspark.addArtifact, and a configured package that the executors can only load from its distributed copy (named by checksum, under the UUID of the session), which they check by its checksum.NativeBridgeare pushed down, not arithmetic or other functions.spark.addArtifactin classic PySpark acceptsfile://URIs and paths relative to the working directory of Python.decimal256and list views are rejected with the name of the column, maps whose entry fields are namedkeysandvalues(as arrow-rs names them) are read, andlimit(...).offset(...)asks the library for the limit once.ColumnarDataSourceSuite).stop()closes the stream reader.createfunctions are rejected.REPLACED, and works again once the loaded build is installed again.addArtifactwith a relative file name that contains a colon.libdirectory of an installation prefix inPATHwithout any configuration (and not found without it), found throughjava.library.path, added withspark.addArtifact, and as SQL tables.-Wall -Wextra), andNativeDataSourceSuiteruns on Linux with g++ in CI.ResolvedDataSourceSuite,ArtifactManagerSuite,PythonDataSourceSuite,PythonStreamingDataSourceSuite,DataStreamReaderWriterSuite,DataFrameReaderWriterSuite,DataFrameWriterV2Suite,DataSourceSuite,DDLSourceLoadSuite,DataSourceV2Suite,StreamingDataSourceV2Suite,SQLConfSuite,SparkThrowableSuiteandSparkConfigBindingPolicySuite.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.