Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 8 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,13 +145,14 @@ those modules (of protoc 3.20 and later, which build their classes with it) get
classes, whose fields take ours as they are: their files are added to the runtime's pool at run
time (`pbmsg::pool::register`; see `docs/compatibility.md`).
`DataFrame(plan, session)`, `DataFrame._plan` and `Column._expr` are the other half of
that bridge: a `LogicalPlan` of Python sends its `plan(session)` relation as it is, of the shallow
placeholder that `DataFrame._plan.plan(session)` gives of the relation of a DataFrame, which the
core replaces wherever a plan, a command or an analysis built in Python holds a relation: every
input of every relation (`plan::relation_inputs_of`), the relation of each command that holds one,
the inputs of the plans of `pyspark.ml`, and the messages of the extensions of a plugin's relations
and commands (`plan::resolve_relation`, `resolve_command`, `resolve_analysis`), but not in an
expression's extension: "Plans of a DataFrame's relation" in `dev/parity/known-differences.md`.
that bridge: a `LogicalPlan` of Python sends its `plan(session)` relation as it is, of the
placeholder that `DataFrame._plan.plan(session)` gives, of one level, which holds the relation of
the DataFrame encoded, built at the call, and which the core replaces wherever a plan, a command or
an analysis built in Python holds a relation: every input of every relation
(`plan::relation_inputs_of`), the relation of each command that holds one, the inputs of the plans
of `pyspark.ml`, and the messages of the extensions of a plugin's relations and commands
(`plan::resolve_relation`, `resolve_command`, `resolve_analysis`), but not in an expression's
extension: "Plans of a DataFrame's relation" in `dev/parity/known-differences.md`.

## 3. Close API gaps (three layers)

Expand Down
16 changes: 12 additions & 4 deletions crates/pyspark-rs/src/dataframe.rs
Original file line number Diff line number Diff line change
Expand Up @@ -967,11 +967,19 @@ impl PyDataFrame {
}

/// The bytes of the placeholder of the relation of this DataFrame, which the plan of `_plan`
/// sends in its place: shallow, whatever the depth of the DataFrame, and replaced by the
/// core with that relation when the plan reaches it.
fn _relation_placeholder<'py>(&self, py: Python<'py>) -> Bound<'py, pyo3::types::PyBytes> {
/// sends in its place: one extension that holds the relation encoded, which the core builds
/// now, as PySpark's `plan(session)` builds the relation, with its errors, and replaces the
/// placeholder with when the plan reaches it, also after this DataFrame is gone.
fn _relation_placeholder<'py>(
&self,
py: Python<'py>,
) -> PyResult<Bound<'py, pyo3::types::PyBytes>> {
use prost::Message;
pyo3::types::PyBytes::new(py, &self.dataframe.placeholder_relation().encode_to_vec())
let placeholder = self
.dataframe
.standalone_placeholder_relation()
.to_pyerr()?;
Ok(pyo3::types::PyBytes::new(py, &placeholder.encode_to_vec()))
}

/// A DataFrame pickles, as PySpark Connect's, of its session, which gives back the active
Expand Down
3 changes: 3 additions & 0 deletions crates/spark-connect/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ base64 = "0.22"
futures.workspace = true
# The gRPC servers of the tests.
tonic = { workspace = true, features = ["server", "router"] }
# The tests decode the relations of the placeholders of deep DataFrames, as the drop-in does,
# whose `pyspark-rs` lifts prost's limit of 100 nested messages: so do the tests.
prost = { workspace = true, features = ["no-recursion-limit"] }

[features]
# Run Rust UDFs on Spark via WebAssembly (the `wasm_udf` module). Off by
Expand Down
16 changes: 13 additions & 3 deletions crates/spark-connect/src/dataframe.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1771,14 +1771,24 @@ impl DataFrame {

/// A relation that stands for the relation of this DataFrame, `to_relation`, in a relation
/// built outside the core, such as a plan of `pyspark.ml`, which the core replaces by that
/// relation when the relation reaches it (`from_relation`, `execute_command_proto`). It
/// is shallow, so it is cheap to build and to decode whatever the depth of this DataFrame.
/// Not a stable API.
/// relation when the relation reaches it (`from_relation`, `execute_command_proto`), while
/// this DataFrame is alive. It is shallow, so it is cheap to build and to decode whatever the
/// depth of this DataFrame. Not a stable API.
#[doc(hidden)]
pub fn placeholder_relation(&self) -> proto::Relation {
crate::plan::placeholder_relation(&self.plan, self.session.plan_ids_shared())
}

/// As [`DataFrame::placeholder_relation`], a placeholder that holds the relation of this
/// DataFrame, as it is built now, encoded in the value of its extension, with the errors of
/// the build: it stands for that relation after this DataFrame is gone, and its encoding
/// too, as the relation that PySpark's `plan(session)` gives does, and it is of one level
/// for a decoder of protobuf outside the core. Not a stable API.
#[doc(hidden)]
pub fn standalone_placeholder_relation(&self) -> Result<proto::Relation> {
crate::plan::standalone_placeholder_relation(&self.plan, self.session.plan_ids_shared())
}

/// Get the schema of this DataFrame.
///
/// The first successful lookup is cached and shared with exact clones.
Expand Down
Loading
Loading