Repository navigation
docs(spark): generate function reference from #[user_doc] - #115
Conversation
Add a datafusion-spark-docgen crate with a print_functions_docs binary, dev/update_function_docs.sh to write docs/functions.md, the generated file, and a CI check that keeps it up to date. Part of apache#55 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce-generator # Conflicts: # .github/workflows/lint.yml
There was a problem hiding this comment.
@andygrove, thanks for this change. Generating the reference from existing UDF metadata is appropriate and keeps documentation aligned with the registered functions.
Approving.
The remaining suggestions concern CI coverage, regeneration safety, and wording accuracy. None are blocking.
| - run: dev/update_function_docs.sh | ||
| - name: Check docs/functions.md is up to date | ||
| run: | | ||
| if ! git diff --exit-code -- docs/functions.md; then |
There was a problem hiding this comment.
Could we use git status --porcelain -- docs/functions.md instead? git diff --exit-code misses untracked files, so CI passes if the reference is deleted and regenerated.
There was a problem hiding this comment.
Good catch. Switched to git status --porcelain -- docs/functions.md, which also catches deleted and untracked files, and the step now prints the status and diff on failure. Fixed in 7233f36.
| differs from the DataFusion built-in of the same name, if there is one. | ||
| HEADER | ||
| cargo run --quiet -p datafusion-spark-docgen --bin print_functions_docs | ||
| } > "${TARGET_FILE}" |
There was a problem hiding this comment.
A failed cargo run leaves the reference truncated to its header. Could we generate into a temporary file, replace the target only on success, and clean up with a trap?
There was a problem hiding this comment.
Done in 7233f36. The script generates into a mktemp file and moves it over docs/functions.md only on success. A trap removes the temp file on any exit.
|
|
||
| Not every Spark function is implemented yet. Progress is tracked in the | ||
| [`datafusion-spark` epic]; see the [API documentation] for the functions available today. | ||
| [`datafusion-spark` epic]. See the [function reference] for the functions available today, with |
There was a problem hiding this comment.
The reference excludes registered functions without documentation(). Could we say "documented functions" here and in README.md:52 and crates/spark/src/lib.rs:49 to avoid overstating coverage?
There was a problem hiding this comment.
Agreed, 10 registered functions still lack documentation(). Changed to "documented functions" in crates/spark/README.md, README.md and crates/spark/src/lib.rs in 7233f36.
- Generate into a temporary file so a failed build leaves docs/functions.md intact. - Use git status --porcelain in CI so a deleted or untracked reference is caught. - Say "documented functions", since functions without documentation() are omitted.
docs/functions.md had grown to about 2,500 lines and would reach roughly 10,000 with full Spark coverage. The generator now writes docs/functions/README.md, an index of categories and of every function, and one page per function type and doc section, such as scalar/math.md. Links between related functions resolve across pages.
|
@kosiew thanks for the review. Since your approval I've split the function reference into one page per category, because the single page was already about 2,500 lines and would reach roughly 10,000 with full Spark coverage. |
Part of #55
Rationale
The recent
#[user_doc]PRs (#58, #103–#107) attach user documentation to most functions, but nothing in the repo renders it.cargo docdoesn't show it either, since#[user_doc]builds a runtimeDocumentationvalue rather than rustdoc, and those PRs cut the rustdoc down to a one-line summary. So the docs can't be read anywhere yet.Changes
crates/docgen(datafusion-spark-docgen,publish = false) with aprint_functions_docsbinary. It is based on DataFusion'sprint_functions_docs. It writes Markdown for all scalar, aggregate and window functions returned byall_default_*_functions(), with arguments, examples, alternative syntax and aliases.docs/functions, with one page per function type and doc section (for examplescalar/math.md) and aREADME.mdindex. The index links each section with its function count, then lists every function and alias alphabetically. A single page was about 2,500 lines already and would reach roughly 10,000 with full Spark coverage.JSON Functions,URL Functions) come after, sorted by label.array_contains,map_from_arrays,map_from_entries,mod,parse_url,pmod,spark_cast,str_to_map,try_parse_url,xxhash64).dev/update_function_docs.sh, which generates into a temporary directory and replacesdocs/functionsonly on success, and commit the generated pages.function docs up to datejob tolint.ymlthat regenerates the pages and fails ifgit status --porcelain -- docs/functionsreports any change, including deleted or untracked files.CONTRIBUTING.mdandAGENTS.md, and link the reference from the READMEs and crate docs.The published
datafusion-sparkcrate is unchanged apart from a doc comment link.Testing
cargo fmt --all -- --check,cargo clippy --all-targets --all-features --workspace -- -D warnings,cargo test -p datafusion-sparkandcargo test -p datafusion-spark-sqllogictestpass.dev/update_function_docs.shand checked the rendered pages. The split pages contain the same functions and aliases as the earlier single page, and a second run leaves the tree clean.cargofail during generation and checked that the existing pages were left untouched.