diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..6c540cd --- /dev/null +++ b/.gitignore @@ -0,0 +1,44 @@ +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# Virtual environments +venv/ +env/ +ENV/ +.venv + +# IDEs +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Test results +results/ +output/ +*.log \ No newline at end of file diff --git a/README.md b/README.md index c12bb06..03d9afa 100644 --- a/README.md +++ b/README.md @@ -1,276 +1,253 @@ -# Python Packaging with Deephaven +# Deephaven Python packaging examples -This example demonstrates how to create and deploy Python packages that use Deephaven. It shows you how to package both command-line tools and reusable libraries using modern Python packaging standards. +This repository shows how to package Python code that uses [Deephaven Community Core](https://deephaven.io/community/) so that it can be installed with `pip`. It contains three small, self-contained example packages. Each example demonstrates exactly one packaging pattern: -This example accompanies the [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) guide in the Deephaven documentation. +| Example | Pattern | Installing it provides | +|---|---|---| +| [`my_dh_library/`](my_dh_library/) | Library only | Functions to import in Python code | +| [`my_dh_cli/`](my_dh_cli/) | Command line tool only | A `my-dh-query` terminal command | +| [`my_dh_toolkit/`](my_dh_toolkit/) | Library and command line tools combined | Importable functions plus `my-dh-toolkit-query` and `my-dh-toolkit-process` commands | -## What you'll learn +`my_dh_toolkit` is the other two patterns merged into a single package: its library modules play the same role as `my_dh_library`, and its commands play the same role as `my_dh_cli`. -This example shows you how to: +All three examples follow the [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) conventions: a `pyproject.toml` file for metadata, dependencies, and entry points, and the src-layout for source code. This repository accompanies the [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) guide, which explains the underlying concepts in depth. -- Create installable Python packages with Deephaven dependencies -- Build command-line tools that process data with Deephaven -- Package reusable library code for other projects -- Manage dependencies with `pyproject.toml` -- Distribute packages as wheel archives +## Choose an example -## Project structure +- Start from **`my_dh_library`** to share reusable functions that other projects import. There is no command line interface. +- Start from **`my_dh_cli`** to ship a tool that users run from a terminal. No library code is exposed. +- Start from **`my_dh_toolkit`** to provide both: importable functions for Python users and commands for terminal users. -The example includes three complete packaging scenarios: +## Prerequisites -### 1. Library-only package (`my_dh_library/`) +- Python 3.9 or later. +- pip. +- Java 17 or later (required by `deephaven-server`, which each example installs as a dependency). -A reusable library with Deephaven query functions that other projects can import. +## Get the examples -``` -my_dh_library/ -├── src/ -│ └── my_dh_library/ -│ ├── __init__.py -│ ├── queries.py -│ └── utils.py -├── pyproject.toml -└── README.md +Clone the repository and work from its root directory. All commands below are run from the repository root. + +```shell +git clone https://github.com/deephaven-examples/deephaven-python-packaging.git +cd deephaven-python-packaging ``` -### 2. CLI-only package (`my_dh_cli/`) +## Sample data -Command-line tools for processing data with Deephaven. +The examples read the CSV files in the `data/` directory: -``` -my_dh_cli/ -├── src/ -│ └── my_dh_package/ -│ ├── __init__.py -│ ├── __main__.py -│ ├── cli.py -│ └── processor.py -├── pyproject.toml -├── data/ -│ └── sample.csv -└── README.md -``` +- `data/sample.csv` — a single 10-row file with `Name`, `Score`, `Value`, and `Category` columns. It is the input for the single-file examples: the library snippets, `my-dh-query`, and `my-dh-toolkit-query`. +- `data/batch/` — three smaller files (`file1.csv`, `file2.csv`, and `file3.csv`) with the same columns but different rows. It is the input for `my-dh-toolkit-process`, which processes every CSV file in a directory. -### 3. Combined package (`my_dh_toolkit/`) +## Example 1: `my_dh_library` — a library -Both reusable library code and command-line tools in one package. +**The story:** package reusable Deephaven query functions so that other projects can `pip install` the package and import the functions. ``` -my_dh_toolkit/ +my_dh_library/ ├── src/ -│ └── my_dh_package/ -│ ├── __init__.py -│ ├── __main__.py -│ ├── cli.py -│ ├── queries.py -│ └── utils.py -├── pyproject.toml +│ └── my_dh_library/ +│ ├── __init__.py # Exports the public API +│ ├── queries.py # Query functions: filter, compute, summarize +│ └── utils.py # Table validation helpers +├── pyproject.toml # Declares metadata and the deephaven-server dependency └── README.md ``` -## Prerequisites - -- Python 3.8 or later -- pip (Python package installer) -- Basic familiarity with Python packaging +There is no `[project.scripts]` section in `pyproject.toml` and no `__main__.py` — this package is only ever imported. -## Quick start +### Try it -Clone the repository: +Install the package and start Python: ```shell -git clone https://github.com/deephaven-examples/python-packaging.git -cd python-packaging -``` - -Choose an example to try: - -### Try the CLI package - -```shell -cd my_dh_cli -pip install -e . -my-dh-query data/sample.csv --verbose -my-dh-process data/ --output results/ -``` - -### Try the library package - -```shell -cd my_dh_library -pip install -e . +pip install -e ./my_dh_library python ``` -Then in Python: +A library that uses Deephaven needs a running server in the same process, so start one before importing `deephaven` modules: ```python +# A Deephaven server must be running before deephaven modules are imported. +from deephaven_server import Server +Server(port=10000, jvm_args=["-Xmx4g"]).start() + +# Import and use the installed library. from my_dh_library.queries import filter_by_threshold from deephaven import read_csv -data = read_csv("../data/sample.csv") +data = read_csv("data/sample.csv") filtered = filter_by_threshold(data, "Score", 75.0) -print(f"Filtered to {filtered.size} rows") -``` - -### Try the combined package - -```shell -cd my_dh_toolkit -pip install -e . - -# Use as a library -python -c "from my_dh_toolkit.queries import filter_by_threshold; print('Library imported successfully')" - -# Use as CLI tools -my-dh-query ../data/sample.csv -my-dh-process ../data/ --output results/ +print(f"{filtered.size} of {data.size} rows have Score > 75") ``` -## What's included - -### Command-line tools +### What to study -The CLI examples demonstrate: +- [`pyproject.toml`](my_dh_library/pyproject.toml) — the `dependencies` list installs `deephaven-server` automatically, and `[tool.setuptools.packages.find]` points setuptools at `src/`. +- [`queries.py`](my_dh_library/src/my_dh_library/queries.py) — plain functions that take and return Deephaven tables. +- [`__init__.py`](my_dh_library/src/my_dh_library/__init__.py) — re-exports the public functions. -- **Entry point scripts** - Commands installed to your PATH -- **Module execution** - Running with `python -m package_name` -- **Argument parsing** - Using Click for robust CLI interfaces -- **Multiple commands** - Single package with multiple tools -- **Verbose output** - Optional detailed logging +## Example 2: `my_dh_cli` — a command line tool -### Library modules +**The story:** package a Deephaven script as a terminal command. `pip install` creates a `my-dh-query` command that users run without writing any Python. -The library examples show: - -- **Reusable query functions** - Common Deephaven operations -- **Type hints** - Proper function signatures -- **Public API exports** - Clean import patterns -- **Documentation** - Docstrings for all functions - -### Configuration +``` +my_dh_cli/ +├── src/ +│ └── my_dh_cli/ +│ ├── __init__.py +│ ├── __main__.py # Enables `python -m my_dh_cli` during development +│ └── cli.py # The command implementation +├── pyproject.toml # Declares the my-dh-query entry point +└── README.md +``` -All examples include: +The command comes from one line in `pyproject.toml`: -- **`pyproject.toml`** - Modern Python packaging configuration -- **Dependency management** - Automatic installation of Deephaven and other requirements -- **Version constraints** - Ensuring compatible package versions -- **Entry points** - Mapping command names to Python functions +```toml +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" +``` -## Building and distributing +### Try it -Each example can be built into a distributable wheel: +Install the package, then run the command on the sample data: ```shell -cd my_dh_cli # or any example directory -pip install build -python -m build +pip install -e ./my_dh_cli +my-dh-query data/sample.csv --verbose ``` -This creates a `.whl` file in the `dist/` directory that can be: +The command starts its own Deephaven server, reads the CSV file, adds a computed column, and reports the row count. No separate setup is needed. -- Installed locally: `pip install dist/my_dh_cli-0.1.0-py3-none-any.whl` -- Distributed to others -- Published to PyPI: `python -m twine upload dist/*` +### What to study -## Running the examples +- [`pyproject.toml`](my_dh_cli/pyproject.toml) — the `[project.scripts]` section maps the command name to a function. +- [`cli.py`](my_dh_cli/src/my_dh_cli/cli.py) — a [Click](https://click.palletsprojects.com/) command that starts the Deephaven server itself, so it works as a standalone tool. +- [`__main__.py`](my_dh_cli/src/my_dh_cli/__main__.py) — allows `python -m my_dh_cli data/sample.csv` as an alternative during development. -### Development mode +## Example 3: `my_dh_toolkit` — a library and command line tools in one package -Install in editable mode to make changes without reinstalling: +**The story:** one package that provides both interfaces. Python users import its query functions, just as in `my_dh_library`; terminal users run its installed commands, just as in `my_dh_cli`. The library modules reuse the `my_dh_library` code, and the command modules reuse the `my_dh_cli` code plus a second command that shows one package installing multiple commands. -```shell -pip install -e . +``` +my_dh_toolkit/ +├── src/ +│ └── my_dh_toolkit/ +│ ├── __init__.py # Intentionally contains no imports — see "What to study" +│ ├── __main__.py +│ ├── cli.py # Implements my-dh-toolkit-query (same code as my_dh_cli) +│ ├── processor.py # Implements my-dh-toolkit-process +│ ├── queries.py # Library query functions (same code as my_dh_library) +│ └── utils.py # Library table helpers (same code as my_dh_library) +├── pyproject.toml # Declares both commands +└── README.md ``` -### Regular installation +### Try the commands -Install from the built wheel: +Install the package, then run each command: ```shell -pip install dist/package_name-0.1.0-py3-none-any.whl +pip install -e ./my_dh_toolkit +my-dh-toolkit-query data/sample.csv --verbose +my-dh-toolkit-process data/batch --output output --verbose ``` -### Without installation +`my-dh-toolkit-query` processes one CSV file. `my-dh-toolkit-process` processes every CSV file in a directory and writes one result file per input to the output directory. Like `my-dh-query` in the previous example, each command starts its own Deephaven server. -Run directly from source using module execution: +### Try the library -```shell -python -m my_dh_package input_data.csv -``` +The same installation also provides the library. In a Python session, start a Deephaven server, then import and use the query functions: -## Sample data +```python +# A Deephaven server must be running before deephaven modules are imported. +from deephaven_server import Server +Server(port=10000, jvm_args=["-Xmx4g"]).start() -The `data/` directory contains sample CSV files for testing: +# Import and use the installed library. +from my_dh_toolkit.queries import filter_by_threshold +from deephaven import read_csv -- `sample.csv` - Small dataset with Name, Age, and Score columns -- `batch/` - Multiple CSV files for batch processing examples +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +print(f"{filtered.size} of {data.size} rows have Score > 75") +``` + +### What to study -You can use your own CSV files with these examples. +- [`pyproject.toml`](my_dh_toolkit/pyproject.toml) — a single `[project.scripts]` section defines both commands. +- [`__init__.py`](my_dh_toolkit/src/my_dh_toolkit/__init__.py) — contains no imports, and that is deliberate. Importing any `deephaven` module fails unless a Deephaven server is already running in the process. When a command such as `my-dh-toolkit-query` starts, Python imports the `my_dh_toolkit` package before the command has started its server. If `__init__.py` imported the query functions, that import chain would reach `deephaven` and every command would fail at startup. Keeping `__init__.py` empty and importing the library from its submodules (`my_dh_toolkit.queries`, `my_dh_toolkit.utils`) avoids the problem. `my_dh_library` can safely re-export its functions from `__init__.py` because it has no commands: it is only ever imported after a server is running. -## Key concepts +## Adapt an example for your own project -### Entry point scripts vs module execution +Each example is a template. To turn one into your own package: -The examples demonstrate two ways to run Python packages: +1. **Copy the example** that matches your scenario: -1. **Entry point scripts** - Commands defined in `[project.scripts]` that become available after installation ```shell - my-dh-query data.csv + cp -r my_dh_cli my_tool + cd my_tool ``` -2. **Module execution** - Running packages with `python -m` without installation +2. **Rename the import package** — the directory under `src/` is the name used in `import` statements: + ```shell - python -m my_dh_package data.csv + mv src/my_dh_cli src/my_tool ``` -See the [Execution patterns](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/#execution-patterns) section of the guide for when to use each method. - -### Package structure +3. **Update `pyproject.toml`** — set your own `name`, `version`, and `description`, and point any `[project.scripts]` entries at the new package: -All examples use the **src-layout**, which is the recommended structure for Python packages. This keeps source code separate from tests and configuration files. + ```toml + [project] + name = "my_tool" -### Dependencies + [project.scripts] + my-tool = "my_tool.cli:app" + ``` -The examples show how to: +4. **Update internal imports** to the new package name (for example, `from my_tool.cli import app` in `__main__.py`). -- Specify required packages (like `deephaven-server`) -- Set version constraints -- Define optional dependencies for features like visualization or testing +5. **Replace the example logic** with your own code, and add any packages it needs to `dependencies` in `pyproject.toml`. Keep `deephaven-server` in the list so it installs automatically. -## Related documentation +6. **Reinstall and test:** -- [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) - Complete guide -- [Install and use Python packages](https://deephaven.io/core/docs/how-to-guides/install-and-use-python-packages/) -- [Use the Deephaven Python package](https://deephaven.io/core/docs/how-to-guides/deephaven-python-package/) -- [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) -- [Click documentation](https://click.palletsprojects.com/) + ```shell + pip install -e . + my-tool --help + ``` -## Troubleshooting +Three names must stay in sync: the package directory under `src/`, the module paths in `[project.scripts]`, and the package name in `import` statements. -### Command not found after installation +## Install and distribute -If your command isn't found after installation: +The examples above use editable installs (`pip install -e ./my_dh_cli`), which pick up source edits without reinstalling — ideal while developing. The other common options: -- Ensure the installation completed without errors -- Check that the installation directory is in your PATH -- Try reinstalling: `pip install --force-reinstall .` +- **Regular install from source:** `pip install ./my_dh_cli` +- **Build and install a wheel** — the format to use when distributing a package to other machines or publishing to a package index: -### Import errors + ```shell + pip install build + python -m build my_dh_cli + pip install my_dh_cli/dist/my_dh_cli-0.1.0-py3-none-any.whl + ``` -If you encounter import errors: + Wheels can be shared directly or published to PyPI with [`twine`](https://twine.readthedocs.io/). -- Verify all dependencies are installed: `pip list` -- Check that you're using Python 3.8 or later -- Ensure Deephaven is installed: `pip install deephaven-server` +## Troubleshooting -### Module not found errors +- **Command not found after installation** — confirm the install succeeded (`pip show my_dh_cli`) and that the Python scripts directory is on `PATH`. Installing inside an activated virtual environment avoids most `PATH` issues. +- **`deephaven` import errors** — the Deephaven server must be started (as shown in the library examples) before `deephaven` modules are imported, and Java 17 or later must be available. +- **Module not found after renaming** — check that the directory under `src/`, the `[project.scripts]` module paths, and the `import` statements all use the new package name, then reinstall with `pip install -e .`. -If Python can't find your modules: +## Related documentation -- Verify `__init__.py` files exist in all package directories -- Check that package names in `[project.scripts]` match your directory structure -- Try reinstalling in editable mode: `pip install -e .` +- [Packaging custom code and dependencies](https://deephaven.io/core/docs/how-to-guides/sysadmin/setuptools-deployment/) — the guide this repository accompanies. +- [Install and use Python packages](https://deephaven.io/core/docs/how-to-guides/install-and-use-python-packages/) +- [Use the Deephaven Python package](https://deephaven.io/core/docs/how-to-guides/deephaven-python-package/) +- [Python Packaging User Guide](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) +- [Click documentation](https://click.palletsprojects.com/) ## Note diff --git a/data/batch/file1.csv b/data/batch/file1.csv new file mode 100644 index 0000000..954231a --- /dev/null +++ b/data/batch/file1.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Kara,81,130,A +Liam,94,175,B +Mona,77,100,A diff --git a/data/batch/file2.csv b/data/batch/file2.csv new file mode 100644 index 0000000..6975da9 --- /dev/null +++ b/data/batch/file2.csv @@ -0,0 +1,4 @@ +Name,Score,Value,Category +Nina,86,115,C +Omar,90,155,B +Pria,74,80,A diff --git a/data/batch/file3.csv b/data/batch/file3.csv new file mode 100644 index 0000000..733a416 --- /dev/null +++ b/data/batch/file3.csv @@ -0,0 +1,5 @@ +Name,Score,Value,Category +Quinn,93,165,C +Rosa,84,125,B +Sam,79,95,A +Tara,88,145,C diff --git a/data/sample.csv b/data/sample.csv new file mode 100644 index 0000000..1d1b856 --- /dev/null +++ b/data/sample.csv @@ -0,0 +1,11 @@ +Name,Score,Value,Category +Alice,85,120,A +Bob,92,150,B +Charlie,78,95,A +Diana,88,110,C +Eve,95,180,B +Frank,72,85,A +Grace,91,160,C +Henry,83,105,B +Iris,89,140,A +Jack,76,90,C diff --git a/my_dh_cli/README.md b/my_dh_cli/README.md new file mode 100644 index 0000000..004a4d8 --- /dev/null +++ b/my_dh_cli/README.md @@ -0,0 +1,61 @@ +# My Deephaven CLI + +An example of packaging a Deephaven script as a command line tool. Installing this package creates one terminal command, `my-dh-query`. No library code is exposed — users of this package never write Python. + +The command is defined by the `[project.scripts]` entry point in [`pyproject.toml`](pyproject.toml): + +```toml +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" +``` + +## Installation + +From the repository root: + +```shell +pip install ./my_dh_cli +``` + +Or in editable mode for development: + +```shell +pip install -e ./my_dh_cli +``` + +## Usage + +Run the installed command on a CSV file. The command starts its own Deephaven server, so no separate setup is needed: + +```shell +my-dh-query data/sample.csv --verbose +``` + +It reads the file, adds a `DoubleScore` computed column, and reports the number of rows processed. + +During development, the package also runs without an entry point via [`__main__.py`](src/my_dh_cli/__main__.py): + +```shell +python -m my_dh_cli data/sample.csv --verbose +``` + +## Command reference + +### my-dh-query + +Process a CSV file with Deephaven. The file must contain a `Score` column. + +**Arguments:** + +- `input_file` - Path to the CSV file to process. + +**Options:** + +- `--verbose, -v` - Enable verbose output. + +## Requirements + +- Python 3.9 or later +- Java 17 or later +- Deephaven Server 0.35.0 or later +- Click 8.0.0 or later diff --git a/my_dh_cli/pyproject.toml b/my_dh_cli/pyproject.toml new file mode 100644 index 0000000..7890660 --- /dev/null +++ b/my_dh_cli/pyproject.toml @@ -0,0 +1,22 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_cli" +version = "0.1.0" +description = "Command line tool for data processing" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", + # click implements the command line interface. + "click>=8.0.0", +] + +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_cli/src/my_dh_cli/__init__.py b/my_dh_cli/src/my_dh_cli/__init__.py new file mode 100644 index 0000000..cd9785f --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/__init__.py @@ -0,0 +1,3 @@ +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" diff --git a/my_dh_cli/src/my_dh_cli/__main__.py b/my_dh_cli/src/my_dh_cli/__main__.py new file mode 100644 index 0000000..ff7364e --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/__main__.py @@ -0,0 +1,4 @@ +from my_dh_cli.cli import app + +if __name__ == "__main__": + app() diff --git a/my_dh_cli/src/my_dh_cli/cli.py b/my_dh_cli/src/my_dh_cli/cli.py new file mode 100644 index 0000000..640248e --- /dev/null +++ b/my_dh_cli/src/my_dh_cli/cli.py @@ -0,0 +1,53 @@ +import click + + +def my_dh_query(input_file: str, verbose: bool = False): + """Read a CSV file and perform a simple query operation on the data.""" + from deephaven import read_csv + from pathlib import Path + + input_path = Path(input_file) + + if not input_path.exists(): + raise click.ClickException(f"Input file does not exist: '{input_path}'") + if not input_path.is_file(): + raise click.ClickException(f"Input path is not a file: '{input_path}'") + + if verbose: + click.echo(f"Processing {input_file}...") + + try: + source = read_csv(input_file) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") + + column_names = [col.name for col in source.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{input_path.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + result = source.update(formulas=["DoubleScore = Score * 2"]) + + if verbose: + click.echo(f"Processed {result.size} rows") + + return result + + +@click.command() +@click.argument("input_file", type=click.Path(exists=True)) +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def app(input_file: str, verbose: bool) -> None: + """Process data with Deephaven.""" + from deephaven_server import Server + + Server(port=10000, jvm_args=["-Xmx4g"]).start() + + result = my_dh_query(input_file, verbose) + click.echo("Processing complete!") + + +if __name__ == "__main__": + app() diff --git a/my_dh_library/README.md b/my_dh_library/README.md new file mode 100644 index 0000000..ff17d0e --- /dev/null +++ b/my_dh_library/README.md @@ -0,0 +1,57 @@ +# My Deephaven Library + +An example of packaging reusable Deephaven query functions as a library. Installing this package makes its functions importable from any Python code. There are no command line tools — this package is only ever imported. + +## Installation + +From the repository root: + +```shell +pip install ./my_dh_library +``` + +Or in editable mode for development: + +```shell +pip install -e ./my_dh_library +``` + +## Usage + +> [!NOTE] +> All Deephaven functionality requires a running server in the same Python process. Start the server before importing `deephaven` modules. + +From the repository root, start Python and use the library: + +```python +# A Deephaven server must be running before deephaven modules are imported. +from deephaven_server import Server +Server(port=10000, jvm_args=["-Xmx4g"]).start() + +# Import and use the installed library. +from my_dh_library.queries import filter_by_threshold, add_computed_columns +from deephaven import read_csv + +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +enhanced = add_computed_columns(filtered) +``` + +## Available functions + +### Query functions (`my_dh_library.queries`) + +- `filter_by_threshold(table, column, threshold)` - Filter table rows where the column value exceeds the threshold. +- `add_computed_columns(table)` - Add commonly used computed columns to a table. +- `summarize_by_group(table, group_col, value_col)` - Create summary statistics grouped by a column. + +### Utility functions (`my_dh_library.utils`) + +- `validate_columns(table, required_columns)` - Check if a table has all required columns. +- `get_table_info(table)` - Get basic information about a table. + +## Requirements + +- Python 3.9 or later +- Java 17 or later +- Deephaven Server 0.35.0 or later diff --git a/my_dh_library/pyproject.toml b/my_dh_library/pyproject.toml new file mode 100644 index 0000000..0e1f738 --- /dev/null +++ b/my_dh_library/pyproject.toml @@ -0,0 +1,17 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_library" +version = "0.1.0" +description = "Reusable Deephaven query functions" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", +] + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_library/src/my_dh_library/__init__.py b/my_dh_library/src/my_dh_library/__init__.py new file mode 100644 index 0000000..6e191c0 --- /dev/null +++ b/my_dh_library/src/my_dh_library/__init__.py @@ -0,0 +1,7 @@ +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" + +from my_dh_library.queries import filter_by_threshold, add_computed_columns, summarize_by_group + +__all__ = ["filter_by_threshold", "add_computed_columns", "summarize_by_group"] diff --git a/my_dh_library/src/my_dh_library/queries.py b/my_dh_library/src/my_dh_library/queries.py new file mode 100644 index 0000000..eb0adfc --- /dev/null +++ b/my_dh_library/src/my_dh_library/queries.py @@ -0,0 +1,35 @@ +"""Reusable Deephaven query functions.""" + +from deephaven.table import Table +from deephaven import agg +from .utils import validate_columns + + +def filter_by_threshold(table: Table, column: str, threshold: float) -> Table: + """Filter table rows where column value exceeds threshold.""" + validate_columns(table, [column], raise_error=True) + return table.where(f"{column} > {threshold}") + + +def add_computed_columns(table: Table) -> Table: + """Add commonly used computed columns to a table.""" + validate_columns(table, ["Value"], raise_error=True) + return table.update( + [ + "DoubleValue = Value * 2", + "IsHigh = Value > 100", + ] + ) + + +def summarize_by_group(table: Table, group_col: str, value_col: str) -> Table: + """Create summary statistics grouped by a column.""" + validate_columns(table, [group_col, value_col], raise_error=True) + return table.agg_by( + [ + agg.sum_(f"Sum = {value_col}"), + agg.avg(f"Avg = {value_col}"), + agg.count_("Count"), + ], + by=[group_col], + ) diff --git a/my_dh_library/src/my_dh_library/utils.py b/my_dh_library/src/my_dh_library/utils.py new file mode 100644 index 0000000..c4d1abb --- /dev/null +++ b/my_dh_library/src/my_dh_library/utils.py @@ -0,0 +1,41 @@ +"""Utility functions for working with Deephaven tables.""" + +from __future__ import annotations + +from deephaven.table import Table + + +def validate_columns(table: Table, required_columns: list[str], raise_error: bool = False) -> bool: + """Check if table has all required columns. + + Args: + table: The table to validate + required_columns: List of column names that must be present + raise_error: If True, raises ValueError when columns are missing + + Returns: + True if all columns are present, False otherwise + + Raises: + ValueError: If raise_error is True and columns are missing + """ + table_columns = [col.name for col in table.columns] + missing = [col for col in required_columns if col not in table_columns] + + if missing: + if raise_error: + raise ValueError( + f"Column(s) {missing} not found in table. " + f"Available columns: {', '.join(table_columns)}" + ) + return False + return True + + +def get_table_info(table: Table) -> dict: + """Get basic information about a table.""" + return { + "num_rows": table.size, + "num_columns": len(table.columns), + "columns": [col.name for col in table.columns], + } diff --git a/my_dh_toolkit/README.md b/my_dh_toolkit/README.md new file mode 100644 index 0000000..3b14798 --- /dev/null +++ b/my_dh_toolkit/README.md @@ -0,0 +1,107 @@ +# My Deephaven Toolkit + +An example of one package with two interfaces: + +- **A library** — importable query functions, matching the [`my_dh_library`](../my_dh_library/) example. +- **Command line tools** — two terminal commands, following the same pattern as the [`my_dh_cli`](../my_dh_cli/) example. + +The commands are defined by the `[project.scripts]` entry points in [`pyproject.toml`](pyproject.toml): + +```toml +[project.scripts] +my-dh-toolkit-query = "my_dh_toolkit.cli:app" +my-dh-toolkit-process = "my_dh_toolkit.processor:process" +``` + +## Installation + +From the repository root: + +```shell +pip install ./my_dh_toolkit +``` + +Or in editable mode for development: + +```shell +pip install -e ./my_dh_toolkit +``` + +## Usage as command line tools + +Run the installed commands on the sample data. Each command starts its own Deephaven server, so no separate setup is needed: + +```shell +my-dh-toolkit-query data/sample.csv --verbose +my-dh-toolkit-process data/batch --output output --verbose +``` + +`my-dh-toolkit-query` processes a single CSV file. `my-dh-toolkit-process` processes every CSV file in a directory and writes the results to the output directory. + +## Usage as a library + +> [!NOTE] +> All Deephaven functionality requires a running server in the same Python process. Start the server before importing `deephaven` modules. + +From the repository root, start Python and use the library: + +```python +# A Deephaven server must be running before deephaven modules are imported. +from deephaven_server import Server +Server(port=10000, jvm_args=["-Xmx4g"]).start() + +# Import and use the installed library. +from my_dh_toolkit.queries import filter_by_threshold, add_computed_columns +from deephaven import read_csv + +data = read_csv("data/sample.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +enhanced = add_computed_columns(filtered) +``` + +## Command reference + +### my-dh-toolkit-query + +Process a single CSV file with Deephaven. The file must contain a `Score` column. + +**Arguments:** + +- `input_file` - Path to the CSV file to process. + +**Options:** + +- `--verbose, -v` - Enable verbose output. + +### my-dh-toolkit-process + +Batch process every CSV file in a directory. Each file must contain a `Score` column. + +**Arguments:** + +- `directory` - Directory containing CSV files to process. + +**Options:** + +- `--output, -o` - Output directory (default: `./output`). +- `--verbose, -v` - Enable verbose output. + +## Available functions + +### Query functions (`my_dh_toolkit.queries`) + +- `filter_by_threshold(table, column, threshold)` - Filter table rows where the column value exceeds the threshold. +- `add_computed_columns(table)` - Add commonly used computed columns to a table. +- `summarize_by_group(table, group_col, value_col)` - Create summary statistics grouped by a column. + +### Utility functions (`my_dh_toolkit.utils`) + +- `validate_columns(table, required_columns)` - Check if a table has all required columns. +- `get_table_info(table)` - Get basic information about a table. + +## Requirements + +- Python 3.9 or later +- Java 17 or later +- Deephaven Server 0.35.0 or later +- Click 8.0.0 or later diff --git a/my_dh_toolkit/pyproject.toml b/my_dh_toolkit/pyproject.toml new file mode 100644 index 0000000..ebdd348 --- /dev/null +++ b/my_dh_toolkit/pyproject.toml @@ -0,0 +1,23 @@ +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_toolkit" +version = "0.1.0" +description = "Deephaven library and CLI tools" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", + # click implements the command line interfaces. + "click>=8.0.0", +] + +[project.scripts] +my-dh-toolkit-query = "my_dh_toolkit.cli:app" +my-dh-toolkit-process = "my_dh_toolkit.processor:process" + +[tool.setuptools.packages.find] +where = ["src"] diff --git a/my_dh_toolkit/src/my_dh_toolkit/__init__.py b/my_dh_toolkit/src/my_dh_toolkit/__init__.py new file mode 100644 index 0000000..f0c1ff3 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/__init__.py @@ -0,0 +1,9 @@ +"""My Deephaven package for data processing. + +This __init__ deliberately imports nothing that requires Deephaven: the CLI +entry points import this package before a Deephaven server is running, so the +package must be importable without one. The library API lives in the +`my_dh_toolkit.queries` and `my_dh_toolkit.utils` submodules. +""" + +__version__ = "0.1.0" diff --git a/my_dh_toolkit/src/my_dh_toolkit/__main__.py b/my_dh_toolkit/src/my_dh_toolkit/__main__.py new file mode 100644 index 0000000..c7407a9 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/__main__.py @@ -0,0 +1,4 @@ +from my_dh_toolkit.cli import app + +if __name__ == "__main__": + app() diff --git a/my_dh_toolkit/src/my_dh_toolkit/cli.py b/my_dh_toolkit/src/my_dh_toolkit/cli.py new file mode 100644 index 0000000..640248e --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/cli.py @@ -0,0 +1,53 @@ +import click + + +def my_dh_query(input_file: str, verbose: bool = False): + """Read a CSV file and perform a simple query operation on the data.""" + from deephaven import read_csv + from pathlib import Path + + input_path = Path(input_file) + + if not input_path.exists(): + raise click.ClickException(f"Input file does not exist: '{input_path}'") + if not input_path.is_file(): + raise click.ClickException(f"Input path is not a file: '{input_path}'") + + if verbose: + click.echo(f"Processing {input_file}...") + + try: + source = read_csv(input_file) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") + + column_names = [col.name for col in source.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{input_path.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + result = source.update(formulas=["DoubleScore = Score * 2"]) + + if verbose: + click.echo(f"Processed {result.size} rows") + + return result + + +@click.command() +@click.argument("input_file", type=click.Path(exists=True)) +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def app(input_file: str, verbose: bool) -> None: + """Process data with Deephaven.""" + from deephaven_server import Server + + Server(port=10000, jvm_args=["-Xmx4g"]).start() + + result = my_dh_query(input_file, verbose) + click.echo("Processing complete!") + + +if __name__ == "__main__": + app() diff --git a/my_dh_toolkit/src/my_dh_toolkit/processor.py b/my_dh_toolkit/src/my_dh_toolkit/processor.py new file mode 100644 index 0000000..459b053 --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/processor.py @@ -0,0 +1,77 @@ +import click +from pathlib import Path + + +def batch_process(directory: str, output_dir: str, verbose: bool = False) -> None: + """Process multiple CSV files from a directory.""" + input_path = Path(directory) + output_path = Path(output_dir) + + if not input_path.exists(): + raise click.ClickException(f"Input directory does not exist: '{input_path}'") + if not input_path.is_dir(): + raise click.ClickException(f"Input path is not a directory: '{input_path}'") + + try: + output_path.mkdir(parents=True, exist_ok=True) + except PermissionError: + raise click.ClickException(f"Permission denied: Cannot create output directory '{output_path}'") + except OSError as e: + raise click.ClickException(f"Failed to create output directory '{output_path}': {e}") + + if input_path.resolve() == output_path.resolve(): + raise click.ClickException( + f"Input and output directories must be different: '{input_path}'" + ) + + from deephaven import read_csv, write_csv + + csv_files = list(input_path.glob("*.csv")) + + if verbose: + click.echo(f"Found {len(csv_files)} CSV files to process") + + for csv_file in csv_files: + if verbose: + click.echo(f"Processing {csv_file.name}...") + + try: + table = read_csv(str(csv_file)) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{csv_file}': {e}") + + column_names = [col.name for col in table.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{csv_file.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + processed = table.update(formulas=["ProcessedScore = Score * 2"]) + + output_file = output_path / f"processed_{csv_file.name}" + try: + write_csv(processed, str(output_file)) + except Exception as e: + raise click.ClickException(f"Failed to write output file '{output_file}': {e}") + + if verbose: + click.echo(f" Processed {processed.size} rows -> {output_file.name}") + + +@click.command() +@click.argument("directory", type=click.Path(exists=True, file_okay=False)) +@click.option("--output", "-o", default="./output", help="Output directory") +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def process(directory: str, output: str, verbose: bool) -> None: + """Batch process CSV files with Deephaven.""" + from deephaven_server import Server + + Server(port=10000, jvm_args=["-Xmx4g"]).start() + + batch_process(directory, output, verbose) + click.echo("Batch processing complete!") + + +if __name__ == "__main__": + process() diff --git a/my_dh_toolkit/src/my_dh_toolkit/queries.py b/my_dh_toolkit/src/my_dh_toolkit/queries.py new file mode 100644 index 0000000..eb0adfc --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/queries.py @@ -0,0 +1,35 @@ +"""Reusable Deephaven query functions.""" + +from deephaven.table import Table +from deephaven import agg +from .utils import validate_columns + + +def filter_by_threshold(table: Table, column: str, threshold: float) -> Table: + """Filter table rows where column value exceeds threshold.""" + validate_columns(table, [column], raise_error=True) + return table.where(f"{column} > {threshold}") + + +def add_computed_columns(table: Table) -> Table: + """Add commonly used computed columns to a table.""" + validate_columns(table, ["Value"], raise_error=True) + return table.update( + [ + "DoubleValue = Value * 2", + "IsHigh = Value > 100", + ] + ) + + +def summarize_by_group(table: Table, group_col: str, value_col: str) -> Table: + """Create summary statistics grouped by a column.""" + validate_columns(table, [group_col, value_col], raise_error=True) + return table.agg_by( + [ + agg.sum_(f"Sum = {value_col}"), + agg.avg(f"Avg = {value_col}"), + agg.count_("Count"), + ], + by=[group_col], + ) diff --git a/my_dh_toolkit/src/my_dh_toolkit/utils.py b/my_dh_toolkit/src/my_dh_toolkit/utils.py new file mode 100644 index 0000000..c4d1abb --- /dev/null +++ b/my_dh_toolkit/src/my_dh_toolkit/utils.py @@ -0,0 +1,41 @@ +"""Utility functions for working with Deephaven tables.""" + +from __future__ import annotations + +from deephaven.table import Table + + +def validate_columns(table: Table, required_columns: list[str], raise_error: bool = False) -> bool: + """Check if table has all required columns. + + Args: + table: The table to validate + required_columns: List of column names that must be present + raise_error: If True, raises ValueError when columns are missing + + Returns: + True if all columns are present, False otherwise + + Raises: + ValueError: If raise_error is True and columns are missing + """ + table_columns = [col.name for col in table.columns] + missing = [col for col in required_columns if col not in table_columns] + + if missing: + if raise_error: + raise ValueError( + f"Column(s) {missing} not found in table. " + f"Available columns: {', '.join(table_columns)}" + ) + return False + return True + + +def get_table_info(table: Table) -> dict: + """Get basic information about a table.""" + return { + "num_rows": table.size, + "num_columns": len(table.columns), + "columns": [col.name for col in table.columns], + } diff --git a/setuptools-deployment.md b/setuptools-deployment.md new file mode 100644 index 0000000..a41da7d --- /dev/null +++ b/setuptools-deployment.md @@ -0,0 +1,801 @@ +--- +title: Packaging custom code and dependencies +sidebar_label: Python packaging +--- + +[Python packaging](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) enables you to create distributable packages containing custom code, command line tools, and managed dependencies. Deephaven's pip-installable packages integrate seamlessly with modern Python packaging tools, allowing you to build reusable libraries and executable scripts that leverage Deephaven's query engine. This guide walks through the concepts and patterns for packaging Deephaven-based Python projects. + +Python packaging with [`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) provides: + +- **Reusable libraries** - Package query functions and utilities for import by other projects. +- **Command line tools** - Build executable scripts with entry point definitions. +- **Dependency management** - Automatically install Deephaven and required packages. +- **Distribution** - Share code as wheel archives via PyPI or direct distribution. +- **Version control** - Specify compatible dependency versions for reproducible installations. + +## Example repository + +The examples in this guide use the [deephaven-python-packaging](https://github.com/deephaven-examples/deephaven-python-packaging) repository. It demonstrates three complete packaging scenarios with working code, sample data, and comprehensive documentation. + +To explore the examples, clone the repository: + +```bash +git clone https://github.com/deephaven-examples/deephaven-python-packaging.git +cd deephaven-python-packaging +``` + +The repository contains three example packages: + +- `my_dh_library/` - Library-only package with reusable query functions. +- `my_dh_cli/` - CLI-only package with command line tools. +- `my_dh_toolkit/` - Combined package with both library and CLI functionality. + +## Package structure + +The example packages in this guide use the **src-layout** described in the Python Packaging Authority's [src layout vs flat layout discussion](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/). This layout keeps source code separate from tests and configuration files: + +``` +my_dh_project/ +├── src/ +│ └── my_dh_package/ +│ ├── __init__.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +### Key components + +- **`src/`** - Source directory containing the package code. +- **`my_dh_package/`** - The Python package (directory name used in imports). +- **`__init__.py`** - Makes the directory importable and can export public API. +- **`pyproject.toml`** - Defines package metadata, dependencies, and entry points. +- **Module files** - Python files containing your functions and classes. + +The package name under `src/` determines how users import your code. For example, with `src/my_dh_library/`, users import via `from my_dh_library import ...`. + +## Server initialization + +Deephaven requires a running server before using any Deephaven functionality. The server must be initialized in the same Python process that uses Deephaven: + +```python +from deephaven_server import Server + +# Initialize and start the server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Now you can import and use Deephaven +from deephaven import read_csv +data = read_csv("data.csv") +``` + +### Key points + +- Each Python process has its own JVM. +- Starting a server in one terminal doesn't help another terminal. +- Entry-point CLI commands should start their own server internally (see [Use CLI functions](#use-cli-functions)) so they work standalone; only functions imported directly need an already-running session. +- The examples size the JVM to 4 GB with `jvm_args=["-Xmx4g"]`; adjust this value to fit the workload. + +## Packaging scenarios + +Different projects have different needs. The example repository demonstrates three common scenarios. The Python usage snippets below assume a running Deephaven server, as shown in [Server initialization](#server-initialization). + +### Library-only package + +Package reusable code without CLI tools. Other projects import your modules. + +**Structure:** + +``` +my_dh_library/ +├── src/ +│ └── my_dh_library/ +│ ├── __init__.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```python +from my_dh_library.queries import filter_by_threshold, add_computed_columns +from deephaven import read_csv + +data = read_csv("data.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +``` + +**Use when:** + +- Creating reusable utilities for other projects. +- You don't need a command line interface. +- Code will be imported, not executed directly. + +### CLI-only package + +Package executable command line tools without exposing library code. + +**Structure:** + +``` +my_dh_cli/ +├── src/ +│ └── my_dh_cli/ +│ ├── __init__.py +│ ├── __main__.py +│ └── cli.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```bash +my-dh-query data.csv --verbose +``` + +**Use when:** + +- Building command line tools for data processing +- You need clean function interfaces +- You don't need to expose library code to other projects + +### Combined package + +Package both reusable library code and command line tools. + +**Structure:** + +``` +my_dh_toolkit/ +├── src/ +│ └── my_dh_toolkit/ +│ ├── __init__.py +│ ├── __main__.py +│ ├── cli.py +│ ├── processor.py +│ ├── queries.py +│ └── utils.py +├── pyproject.toml +└── README.md +``` + +**Usage:** + +```python +# As a library +from my_dh_toolkit.queries import filter_by_threshold +from deephaven import read_csv + +data = read_csv("data.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +``` + +```bash +# As CLI commands +my-dh-toolkit-query data.csv --verbose +my-dh-toolkit-process data/batch/ --output results/ --verbose +``` + +**Use when:** + +- You need both library and CLI functionality +- You want to provide multiple interfaces to the same code +- Library functions are useful independently + +## Create a new package + +
+Step-by-step instructions for creating packages from scratch + +This section walks through creating each type of package from scratch. + +### Create a library-only package + +Create the directory structure: + +```bash +mkdir -p my_dh_library/src/my_dh_library +cd my_dh_library +``` + +Create `pyproject.toml`: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_library" +version = "0.1.0" +description = "Reusable Deephaven query functions" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", +] + +[tool.setuptools.packages.find] +where = ["src"] +``` + +Create `src/my_dh_library/__init__.py`: + +```python +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" + +from my_dh_library.queries import filter_by_threshold, add_computed_columns, summarize_by_group + +__all__ = ["filter_by_threshold", "add_computed_columns", "summarize_by_group"] +``` + +Create `src/my_dh_library/utils.py`: + +```python +"""Utility functions for working with Deephaven tables.""" + +from __future__ import annotations + +from deephaven.table import Table + + +def validate_columns(table: Table, required_columns: list[str], raise_error: bool = False) -> bool: + """Check if table has all required columns. + + Args: + table: The table to validate + required_columns: List of column names that must be present + raise_error: If True, raises ValueError when columns are missing + + Returns: + True if all columns are present, False otherwise + + Raises: + ValueError: If raise_error is True and columns are missing + """ + table_columns = [col.name for col in table.columns] + missing = [col for col in required_columns if col not in table_columns] + + if missing: + if raise_error: + raise ValueError( + f"Column(s) {missing} not found in table. " + f"Available columns: {', '.join(table_columns)}" + ) + return False + return True + + +def get_table_info(table: Table) -> dict: + """Get basic information about a table.""" + return { + "num_rows": table.size, + "num_columns": len(table.columns), + "columns": [col.name for col in table.columns], + } +``` + +Create `src/my_dh_library/queries.py`: + +```python +"""Reusable Deephaven query functions.""" + +from deephaven.table import Table +from deephaven import agg +from .utils import validate_columns + + +def filter_by_threshold(table: Table, column: str, threshold: float) -> Table: + """Filter table rows where column value exceeds threshold.""" + validate_columns(table, [column], raise_error=True) + return table.where(f"{column} > {threshold}") + + +def add_computed_columns(table: Table) -> Table: + """Add commonly used computed columns to a table.""" + validate_columns(table, ["Value"], raise_error=True) + return table.update( + [ + "DoubleValue = Value * 2", + "IsHigh = Value > 100", + ] + ) + + +def summarize_by_group(table: Table, group_col: str, value_col: str) -> Table: + """Create summary statistics grouped by a column.""" + validate_columns(table, [group_col, value_col], raise_error=True) + return table.agg_by( + [ + agg.sum_(f"Sum = {value_col}"), + agg.avg(f"Avg = {value_col}"), + agg.count_("Count"), + ], + by=[group_col], + ) +``` + +Create `README.md` with installation and usage instructions. + +### Create a CLI-only package + +Create the directory structure: + +```bash +mkdir -p my_dh_cli/src/my_dh_cli +cd my_dh_cli +``` + +Create `pyproject.toml`: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_cli" +version = "0.1.0" +description = "Command line tool for data processing" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", + # click implements the command line interface. + "click>=8.0.0", +] + +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" + +[tool.setuptools.packages.find] +where = ["src"] +``` + +Create `src/my_dh_cli/__init__.py`: + +```python +"""My Deephaven package for data processing.""" + +__version__ = "0.1.0" +``` + +Create `src/my_dh_cli/__main__.py`: + +```python +from my_dh_cli.cli import app + +if __name__ == "__main__": + app() +``` + +Create `src/my_dh_cli/cli.py`: + +```python +import click + + +def my_dh_query(input_file: str, verbose: bool = False): + """Read a CSV file and perform a simple query operation on the data.""" + from deephaven import read_csv + from pathlib import Path + + input_path = Path(input_file) + + if not input_path.exists(): + raise click.ClickException(f"Input file does not exist: '{input_path}'") + if not input_path.is_file(): + raise click.ClickException(f"Input path is not a file: '{input_path}'") + + if verbose: + click.echo(f"Processing {input_file}...") + + try: + source = read_csv(input_file) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") + + column_names = [col.name for col in source.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{input_path.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + result = source.update(formulas=["DoubleScore = Score * 2"]) + + if verbose: + click.echo(f"Processed {result.size} rows") + + return result + + +@click.command() +@click.argument("input_file", type=click.Path(exists=True)) +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def app(input_file: str, verbose: bool) -> None: + """Process data with Deephaven.""" + from deephaven_server import Server + + Server(port=10000, jvm_args=["-Xmx4g"]).start() + + result = my_dh_query(input_file, verbose) + click.echo("Processing complete!") + + +if __name__ == "__main__": + app() +``` + +Create `README.md` with installation and usage instructions. + +### Create a combined package + +Create the directory structure: + +```bash +mkdir -p my_dh_toolkit/src/my_dh_toolkit +cd my_dh_toolkit +``` + +Create `pyproject.toml`: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_toolkit" +version = "0.1.0" +description = "Deephaven library and CLI tools" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", + # click implements the command line interfaces. + "click>=8.0.0", +] + +[project.scripts] +my-dh-toolkit-query = "my_dh_toolkit.cli:app" +my-dh-toolkit-process = "my_dh_toolkit.processor:process" + +[tool.setuptools.packages.find] +where = ["src"] +``` + +Create `src/my_dh_toolkit/__init__.py`: + +```python +"""My Deephaven package for data processing. + +This __init__ deliberately imports nothing that requires Deephaven: the CLI +entry points import this package before a Deephaven server is running, so the +package must be importable without one. The library API lives in the +`my_dh_toolkit.queries` and `my_dh_toolkit.utils` submodules. +""" + +__version__ = "0.1.0" +``` + +The empty `__init__.py` is the key structural difference from a library-only package: if it imported the query functions, the import chain would reach `deephaven` and both commands would fail before they could start their server. + +Create `src/my_dh_toolkit/__main__.py`: + +```python +from my_dh_toolkit.cli import app + +if __name__ == "__main__": + app() +``` + +Create the library modules (`queries.py`, `utils.py`) using the same code as the library-only package. + +Create `src/my_dh_toolkit/cli.py` using the same code as the CLI-only package. + +Create `src/my_dh_toolkit/processor.py`: + +```python +import click +from pathlib import Path + + +def batch_process(directory: str, output_dir: str, verbose: bool = False) -> None: + """Process multiple CSV files from a directory.""" + input_path = Path(directory) + output_path = Path(output_dir) + + if not input_path.exists(): + raise click.ClickException(f"Input directory does not exist: '{input_path}'") + if not input_path.is_dir(): + raise click.ClickException(f"Input path is not a directory: '{input_path}'") + + try: + output_path.mkdir(parents=True, exist_ok=True) + except PermissionError: + raise click.ClickException(f"Permission denied: Cannot create output directory '{output_path}'") + except OSError as e: + raise click.ClickException(f"Failed to create output directory '{output_path}': {e}") + + if input_path.resolve() == output_path.resolve(): + raise click.ClickException( + f"Input and output directories must be different: '{input_path}'" + ) + + from deephaven import read_csv, write_csv + + csv_files = list(input_path.glob("*.csv")) + + if verbose: + click.echo(f"Found {len(csv_files)} CSV files to process") + + for csv_file in csv_files: + if verbose: + click.echo(f"Processing {csv_file.name}...") + + try: + table = read_csv(str(csv_file)) + except Exception as e: + raise click.ClickException(f"Failed to read CSV file '{csv_file}': {e}") + + column_names = [col.name for col in table.columns] + if "Score" not in column_names: + raise click.ClickException( + f"File '{csv_file.name}' is missing required column 'Score'. " + f"Available columns: {', '.join(column_names)}" + ) + + processed = table.update(formulas=["ProcessedScore = Score * 2"]) + + output_file = output_path / f"processed_{csv_file.name}" + try: + write_csv(processed, str(output_file)) + except Exception as e: + raise click.ClickException(f"Failed to write output file '{output_file}': {e}") + + if verbose: + click.echo(f" Processed {processed.size} rows -> {output_file.name}") + + +@click.command() +@click.argument("directory", type=click.Path(exists=True, file_okay=False)) +@click.option("--output", "-o", default="./output", help="Output directory") +@click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") +def process(directory: str, output: str, verbose: bool) -> None: + """Batch process CSV files with Deephaven.""" + from deephaven_server import Server + + Server(port=10000, jvm_args=["-Xmx4g"]).start() + + batch_process(directory, output, verbose) + click.echo("Batch processing complete!") + + +if __name__ == "__main__": + process() +``` + +Create `README.md` with installation and usage instructions. + +
+ +## Configure `pyproject.toml` + +The [`pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) file defines your package configuration. + +### Configuration options + +Here's a detailed breakdown of `pyproject.toml` for a library-only package: + +```toml +[build-system] +requires = ["setuptools>=61.0", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "my_dh_library" +version = "0.1.0" +description = "Reusable Deephaven query functions" +readme = "README.md" +requires-python = ">=3.9" +dependencies = [ + # deephaven-server also provides the deephaven module (through its deephaven-core dependency). + "deephaven-server>=0.35.0", +] + +[tool.setuptools.packages.find] +where = ["src"] +``` + +### Key sections + +- **`[build-system]`** - Specifies setuptools as the build backend +- **`[project]`** - Package metadata and dependencies +- **`name`** - Project name (used for `pip install`) +- **`dependencies`** - Required packages, installed automatically +- **`[tool.setuptools.packages.find]`** - Tells setuptools to find packages in `src/` + +For CLI packages, add a `[project.scripts]` section: + +```toml +[project.scripts] +my-dh-query = "my_dh_cli.cli:app" +``` + +This creates a command line entry point that calls the `app` function from `my_dh_cli.cli`. + +## Managing dependencies + +Dependencies are specified in the `dependencies` field: + +```toml +[project] +dependencies = [ + "deephaven-server>=0.35.0", + "click>=8.0.0", + "pandas>=2.0.0", +] +``` + +Declaring `deephaven-server` is sufficient for Deephaven: it depends on a matching version of `deephaven-core`, which provides the `deephaven` module that packages import. + +### Version constraints + +Use version specifiers to control which versions are acceptable: + +- `>=0.35.0` - Minimum version (0.35.0 or higher) +- `>=2.0.0,<3.0.0` - Version range (2.x only) +- `~=1.24.0` - Compatible release (>=1.24.0, <1.25.0) +- `==1.0.0` - Exact version (not recommended for libraries) + +### Optional dependencies + +Define optional feature sets that users can install separately: + +```toml +[project.optional-dependencies] +visualization = [ + "matplotlib>=3.7.0", + "seaborn>=0.12.0", +] +dev = [ + "pytest>=7.0.0", + "black>=23.0.0", +] +``` + +Users can install optional dependencies: + +```bash +pip install my_dh_library[visualization] +pip install my_dh_library[visualization,dev] +``` + +## Installation and usage + +### Install a package + +Install from source in editable mode for development: + +```bash +cd my_dh_library +pip install -e . +``` + +Or install normally: + +```bash +pip install . +``` + +### Use a library package + +After installation, import and use the library functions: + +```python +# Start the Deephaven server +from deephaven_server import Server +server = Server(port=10000, jvm_args=["-Xmx4g"]) +server.start() + +# Import and use library functions +from my_dh_library.queries import filter_by_threshold +from deephaven import read_csv + +data = read_csv("data.csv") +filtered = filter_by_threshold(data, "Score", 75.0) +``` + +> [!NOTE] +> All Deephaven functionality requires a running server. Start the server before importing Deephaven modules. + +### Use CLI functions + +Entry-point commands like `my-dh-query` start their own Deephaven server, so they run as standalone terminal commands: + +```bash +my-dh-query data.csv --verbose +``` + +For programmatic use, install a library package (or the combined package) and import its functions as shown in [Use a library package](#use-a-library-package). + +## Building and distributing + +Build a distributable wheel: + +```bash +cd my_dh_library +pip install build +python -m build +``` + +This creates a `.whl` file in `dist/` that can be: + +- Installed locally: `pip install dist/my_dh_library-0.1.0-py3-none-any.whl` +- Distributed to others +- Published to PyPI: `python -m twine upload dist/*` + +## Best practices + +### Package structure + +- Prefer the src-layout for packages like these examples +- Keep package names lowercase with underscores +- Match the package directory name to the import name +- Include `__init__.py` in all package directories +- In packages that define entry-point commands, keep `__init__.py` free of imports that require a running Deephaven server + +### Dependencies + +- Specify minimum versions for Deephaven and critical dependencies +- Use version ranges for flexibility +- Group related optional dependencies +- Document any system-level dependencies + +### Documentation + +- Include a comprehensive README.md +- Document all public functions and classes +- Provide usage examples +- Explain server initialization requirements + +### Testing + +- Write tests for all public functions +- Test with different Deephaven versions +- Include sample data for testing +- Document how to run tests + +## Next steps + +The [deephaven-python-packaging](https://github.com/deephaven-examples/deephaven-python-packaging) repository provides complete, working examples of all three packaging scenarios. Clone the repository and explore the examples to see how to structure your own Deephaven packages. + +Each example includes: + +- Complete source code +- Configured `pyproject.toml` +- Comprehensive README +- Usage examples + +The repository also provides shared sample data in its `data/` directory for trying the examples. + +## Related documentation + +- [Install and use Python packages](https://deephaven.io/core/docs/how-to-guides/install-and-use-python-packages/) +- [Use the Deephaven Python package](https://deephaven.io/core/docs/how-to-guides/deephaven-python-package/) +- [Writing your `pyproject.toml`](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/) +- [src layout vs flat layout](https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/) +- [Creating and packaging command-line tools](https://packaging.python.org/en/latest/guides/creating-command-line-tools/) +- [Setuptools documentation](https://setuptools.pypa.io/) +- [Click documentation](https://click.palletsprojects.com/)