diff --git a/CHANGES.md b/CHANGES.md index d34dca2..309ee05 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,26 +1,44 @@ # XRLint Change History +## Version 0.7.0 (in development) + +### Documentation + +- Completed the configuration guide, replacing all "Coming soon" sections with + explanations and examples for file filters, opening options, rule settings, + presets, custom plugins, rules, and processors. +- Updated the getting-started, CLI, Python API, example, and contribution guides + to match current behavior, including plugin discovery, configuration precedence, + exit status, and dataset-tree traversal. +- Enhanced the generated rule reference with qualified rule identifiers, preset + names, option schemas, and a legend for rule categories and severities. +- Corrected CLI help, API docstrings, and rule descriptions, and documented the + ACDD 1.1 and strict-recommended preset limitations, including a working override + for the latter. + ## Version 0.6.0 (from 2026-09-11) ### Adjustments and Enhancements -- Core rule 'time-coordinates' now support ms, µs and ns. (#66) +- Core rule `time-coordinate` now supports ms, µs, and ns datetime precision. (#66) - Implemented an initial set of - [Attribute Conventions Data Discovery (ACCD)](https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery) + [Attribute Convention for Data Discovery (ACDD)](https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery) rules adapted from the [IOOS Compliance Checker](https://github.com/ioos/compliance-checker/) library (many thanks to @abkfenris): - Configs for ACDD 1.0, 1.1, and 1.3, and with selectable levels of severity. - The recommended set uses ACDD 1.3 with the highly recomended rules as errors. - - Global attribute existance rules. + The recommended set uses ACDD 1.3 with conventions and highly recommended + attribute checks as errors; other checks are warnings. + - Global attribute existence rules. - Checks that ACDD is in the conventions attribute. - Makes sure the date attributes are ISO formatted. - Metadata links are URLs. - - The ID attribute should not be blank. + - The `id` attribute should not contain spaces. - Load plugins from entry points allowing plugins to be discovered from installed libraries. - - Automatically generate rule documentation removing the manual need to run `mkruleref.py`. + - Automatically generate rule documentation during the MkDocs build, + removing the need to run `mkruleref.py` manually. - Fixed metadata rules for datatrees: global/common attributes defined on parent groups are now considered by the relevant core rules. (#63) @@ -60,7 +78,7 @@ - Rule `no-empty-chunks` has been taken off the `"recommended"` settings as there is no easy/efficient way to tell whether a dataset has - been written using `write_emtpy_chunks` option or not. + been written using the `write_empty_chunks` option or not. The rule message itself has been fixed. (#45) - Adjusted messages of rules `var-units` and `time-coordinate` @@ -121,7 +139,7 @@ ## Version 0.4.0 (from 2025-01-27) -- Fixed and enhanced core rule `time-coordinate`. `(#33) +- Fixed and enhanced core rule `time-coordinate`. (#33) - New xcube rule `no-chunked-coords`. (#29) - New xcube multi-level dataset rules: - `ml-dataset-meta`: verifies that a meta info file exists and is consistent; diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cceac9b..8c7fe62 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,99 +1,141 @@ -# How to contribute - -The XRLint project welcomes contributions of any form -as long as you respect our [code of conduct](CODE_OF_CONDUCT.md) and stay -in line with the following instructions and guidelines. - -If you have suggestions, ideas, feature requests, or if you have identified -a malfunction or error, then please -[post an issue](https://github.com/bcdev/xrlint/issues). - -If you'd like to submit code or documentation changes, we ask you to provide a -pull request (PR) -[here](https://github.com/bcdev/xrlint/pulls). -For code and configuration changes, your PR must be linked to a -corresponding issue. - -To ensure that your code contributions are consistent with our project’s -coding guidelines, please make sure all applicable items of the following -checklist are addressed in your PR. - -**PR checklist** - -* Format and check code using [ruff](https://docs.astral.sh/ruff/) with - default settings: `ruff format` and `ruff check`. See also section - [code style](#code-style) below. -* Your change shall not break existing unit tests. - `pytest` must run without errors. -* Add unit tests for any new code not yet covered by tests. -* Make sure test coverage stays close to 100% for any change. - Use `pytest --cov=xrlint --cov-report=html` to verify. -* If your change affects the current project documentation, - please adjust it and include the change in the PR. - Run `mkdocs serve` to verify. +# Contributing to XRLint + +We welcome code, documentation, bug reports, and ideas. Please follow our +[Code of Conduct](CODE_OF_CONDUCT.md). +Use [GitHub issues](https://github.com/bcdev/xrlint/issues) for bugs and proposals, +and submit changes through a [pull request](https://github.com/bcdev/xrlint/pulls). +Code and configuration changes must be linked to a corresponding issue. + +## Development setup + +Use Python 3.10 or newer. From the repository root, install the project in +editable mode with development and documentation dependencies: + +```bash +python -m pip install -e ".[dev,doc]" +``` + +An editable installation also registers the `xrlint` command and built-in plugin +entry points. Merely adding the source directory to `PYTHONPATH` does not register +those entry points. + +If using conda or mamba, create and activate the repository environment, then +install the project and extras to include dependencies declared in +`pyproject.toml`: + +```bash +conda env create -f environment.yml +conda activate xrlint +python -m pip install -e ".[dev,doc]" +``` + +## Project layout + +| Path | Responsibility | +| --- | --- | +| `xrlint/cli/` | Click command, configuration discovery, file traversal, and reporting. | +| `xrlint/linter.py`, `xrlint/_linter/` | Linter configuration, opening, traversal, and rule execution. | +| `xrlint/config.py` | Configuration conversion, plugin discovery, and merging. | +| `xrlint/rule.py`, `node.py`, `plugin.py`, `processor.py` | Extension interfaces and metadata. | +| `xrlint/plugins/` | Core, xcube, and ACDD rules and presets. | +| `xrlint/formatters/` | Text, JSON, and HTML reports. | +| `tests/` | Tests mirroring the package structure. | +| `examples/` | Custom configurations and API examples. | +| `docs/` | MkDocs pages and the rule-reference generator. | + +## Checks before submitting + +Run the checks relevant to your change from the repository root: + +```bash +python -m ruff format --check +python -m ruff check +python -m pytest --cov=xrlint --cov-branch --cov-report=html +python -m mkdocs build --strict +``` + +- Keep existing tests passing and add coverage for new or changed behavior. +- Aim to keep coverage close to 100%; review the report in `htmlcov/index.html`. +- Update documentation and examples when behavior or configuration changes. +- For documentation changes, preview the site with `python -m mkdocs serve` + and verify links, code blocks, and generated references. +- Describe the problem, resulting behavior, and validation in the pull request. ## Code style -The code style of XRLint equals the default settings -of [black](https://black.readthedocs.io/). Since black is -un-opinionated regarding the order of imports, we group and -sort imports statements according to the default settings of -[isort](https://pycqa.github.io/isort/) which boils down to +Use Ruff's formatter and linter with the repository configuration: -0. Future imports -1. Python standard library imports, e.g., `os`, `typing`, etc -2. 3rd-party imports, e.g., `xarray`, `zarr`, etc -3. 1st-party XRLint module imports using absolute paths, - e.g., `from xrlint.a.b.c import d`. -4. 1st-party XRLint module imports from local modules: - Relative imports such as `from .c import d` are ok - while `..c import d` are not ok. +```bash +python -m ruff format +python -m ruff check +``` -Use `typing.TYPE_CHECKING` to resolve forward references -and effectively avoid circular dependencies. +Group imports in this order: future imports, standard library, third-party +packages, absolute XRLint imports, and local relative imports. The repository +configures isort with the Black profile. Same-package imports such as +`from .module import name` are acceptable; avoid parent-relative imports such as +`from ..module import name`. -## Contributing a XRLint Rule +Use `typing.TYPE_CHECKING` for type-only imports to avoid circular dependencies. +Document public APIs with Google-style docstrings. -### Rule Naming +## Contributing a rule -The rule naming conventions for XRLint are based ESLint: +Choose a lowercase, hyphen-separated name describing a single requirement. +Prefix prohibitions with `no-`, as in `no-empty-attrs`. Plugin namespaces are +separated with a slash in configuration, such as `xcube/cube-dims-order`. -* Lower-case only. -* Use dashes between words (kebab-case). -* The rule name should be chosen based on what shall be - achieved, of what shall be regulated. It names a contract. -* If your rule only disallows something, - prefix it with `no-` such as `no-empty-attrs` for disallowing - empty attributes in dataset nodes. -* If your rule is enforcing the inclusion of something, - use a short name without a special prefix. -* Plugins should add a prefix before their rule names - separated by a slash, e.g., `xcube/spatial-dims-order`. +Place the implementation in +`xrlint/plugins//rules/.py`, replacing hyphens with underscores. +Derive from `RuleOp`, register with `plugin.define_rule()`, and implement only +the callbacks needed. Keep the reason for each rule easy to explain. -### Rule Design +Provide a description, version, relevant documentation URL, and a schema for +any options. Schemas currently document options; runtime schema validation is +not implemented. Keep constructor arguments and schema metadata consistent. +Decide explicitly whether the rule belongs in a recommended preset. -* The reasoning behind a rule should be **easy to grasp**. +Place tests in `tests/plugins//rules/test_.py`. +Use `RuleTester` with valid and invalid datasets, including parameter cases when +applicable. See the [rule development examples](docs/examples.md#developing-rules) +and [extension guide](docs/config.md#custom-rules). -* A rule should serve for a **single purpose only**. Try subdividing - complex rule logic into multiple rules with simpler logic. +## Contributing a plugin -* Each rule should be defined in a dedicated module named after the rule, - i.e., `/rules/`. The module name should be the rule's name - with dashes replaced by underscores. +Plugins contribute rules, processors, and named configurations. An importable +plugin module must define `export_plugin()` returning a `Plugin` object. +For automatic discovery, declare an entry point in the plugin package: -* Write a comprehensive test for your rule logic which should be defined - in a dedicated module under `tests`, i.e., `tests/rules/test_`. - Consider using `xrlint.testing.RuleTester` which can save a lot of - time and is used for almost all in-built rules. +```toml +[project.entry-points."xrlint.rules"] +my_plugin = "my_package.xrlint_plugin" +``` -## Contributing an XRLint Plugin +The entry point targets the module, not its factory function. Use a consistent +entry-point name, `PluginMeta.name`, and rule namespace. Install the package to +register the entry point. Discovery makes rules available; users still enable +them through presets or rule configuration. -New plugins should be added to the `xrlint.rules` entry point table, which will cause them to be automatically loaded by XRLint, and to be included in the rule documentation. +Local Python configurations can use a plugin directly without packaging. +See [Custom Plugins](docs/config.md#custom-plugins). -```toml -# pyproject.toml -[project.entry-points."xrlint.rules"] -core = "xrlint.plugins.core" -xcube = "xrlint.plugins.xcube" -acdd = "xrlint.plugins.acdd" -``` \ No newline at end of file +## Building documentation + +```bash +python -m mkdocs build --strict +python -m mkdocs serve +``` + +The build writes `site/`. The preview command serves the site locally and +rebuilds when files change. Edit Markdown pages in `docs/`; update `mkdocs.yml` +when adding navigation entries. + +`docs/mkruleref.py` runs automatically through `mkdocs-gen-files`, generating +`rule-ref.md` from discovered plugin metadata. Do not create or edit that +generated page manually. Update rule descriptions and schemas in the source, +or change the generator for presentation changes. The API page uses +`mkdocstrings` to render source docstrings. + +The plugins installed in the build environment determine the generated rule +reference. Use a project environment without unrelated third-party XRLint +plugins when building the project's published documentation. diff --git a/README.md b/README.md index b6eed57..1af7705 100644 --- a/README.md +++ b/README.md @@ -1,44 +1,92 @@ [![CI](https://github.com/bcdev/xrlint/actions/workflows/tests.yml/badge.svg)](https://github.com/bcdev/xrlint/actions/workflows/tests.yml) -[![codecov](https://codecov.io/gh/bcdev/xrlint/graph/badge.svg?token=GVKuJao97t)](https://codecov.io/gh/bcdev/xrlint) +[![codecov](https://codecov.io/gh/bcdev/xrlint/graph/badge.svg)](https://codecov.io/gh/bcdev/xrlint) [![PyPI Version](https://img.shields.io/pypi/v/xrlint)](https://pypi.org/project/xrlint/) [![Conda Version](https://anaconda.org/conda-forge/xrlint/badges/version.svg)](https://anaconda.org/conda-forge/xrlint) -[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v0.json)](https://github.com/charliermarsh/ruff) -[![GitHub License](https://img.shields.io/github/license/bcdev/xrlint)](https://github.com/bcdev/xrlint) +[![GitHub License](https://img.shields.io/github/license/bcdev/xrlint)](LICENSE) # XRLint - A linter for xarray datasets -XRLint is a [linting](https://en.wikipedia.org/wiki/Lint_(software)) -tool and library for [xarray](https://docs.xarray.dev/) datasets. -Its design is heavily inspired by the awesome [ESLint](https://eslint.org/) tool. +XRLint checks xarray datasets for metadata, structure, and convention issues. +Use it from the command line to check dataset files, or from Python to validate +`xarray.Dataset` and `xarray.DataTree` objects. Its configurable rules and plugin +model are inspired by ESLint. +## Features -## Features +- Configurable rules for dataset metadata, coordinates, variables, and groups. +- YAML, JSON, and Python configurations with file-specific overrides. +- Local datasets and remote sources supported by the installed xarray backends + and fsspec filesystem implementations. +- Text, JSON, and HTML reports, plus notebook rendering of results. +- Custom rules, plugins, processors, and reusable configurations. -- Flexible validation for - [`xarray.Dataset`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html) and - [`xarray.DataTree`](https://docs.xarray.dev/en/stable/generated/xarray.DataTree.html) objects - by configurable rules. -- Available from CLI and Python API. -- Custom plugins providing custom rule sets allow addressing - different dataset conventions. -- Project-specific configurations including configuration of individual - rules and file-specific settings. -- Works with dataset files in the local filesystem or any of the remote - filesystems supported by xarray. +XRLint reports findings and suggestions; it does not automatically modify data. +Its built-in rules cover selected convention requirements, not full compliance +certification. -## Inbuilt Rules +## Quick start -The following plugins provide XRLint's [inbuilt rules](https://bcdev.github.io/xrlint/rule-ref/): +Requires Python 3.10 or newer. Install XRLint and a backend for your data: -- `xrlint.plugins.core`: implementing the rules for - [tidy data](https://tutorial.xarray.dev/intermediate/data_cleaning/05.1_intro.html) - and the - [CF-Conventions](https://cfconventions.org/cf-conventions/cf-conventions.html). -- `xrlint.plugins.xcube`: implementing the rules for - [xcube datasets](https://xcube.readthedocs.io/en/latest/cubespec.html). - Note, this plugin is fully optional. You must manually configure - it to apply its rules. It may be moved into a separate GitHub repo later. -- `xrlint.plugins.acdd`: implements rules for [Attribute Conventions Dataset Discovery](https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery). - Note, this plugin is fully optional. You must manually configure it to apply its rules. +```bash +python -m pip install xrlint netCDF4 +xrlint --init +xrlint data/example.nc +``` +For Zarr datasets, install `zarr` too. The initial configuration enables the core +`recommended` preset. No rules are enabled automatically without configuration. +To validate an in-memory dataset: + +```python +import xarray as xr +from xrlint.linter import new_linter + +dataset = xr.Dataset(attrs={"title": "Example dataset"}) +result = new_linter("recommended").validate(dataset) + +for message in result.messages: + print(message.rule_id, message.node_path, message.message) + +print(f"{result.error_count} errors, {result.warning_count} warnings") +``` + +This deliberately minimal dataset produces metadata warnings. +See [Getting Started](https://bcdev.github.io/xrlint/start/) for a complete +file-based example and installation alternatives. + +## Built-in plugins + +All three plugins are discovered when XRLint is installed. Select their presets +or individual rules to enable checks. + +| Plugin | Scope | Preset | +| --- | --- | --- | +| `core` | General dataset quality and selected CF convention checks | `recommended` | +| `xcube` | xcube dataset structure and multi-level datasets | `xcube/recommended` | +| `acdd` | Attribute Convention for Data Discovery (ACDD) metadata | `acdd/recommended` | + +For example, enable core and ACDD checks in `xrlint-config.yaml`: + +```yaml +- recommended +- acdd/recommended +- rules: + var-units: error +``` + +## Documentation and contributing + +- [Getting Started](https://bcdev.github.io/xrlint/start/) +- [Configuration](https://bcdev.github.io/xrlint/config/) +- [Rule Reference](https://bcdev.github.io/xrlint/rule-ref/) +- [CLI](https://bcdev.github.io/xrlint/cli/) and [Python API](https://bcdev.github.io/xrlint/api/) +- [Examples](https://bcdev.github.io/xrlint/examples/) +- [Change history](CHANGES.md) + +Report bugs or request features through [GitHub issues](https://github.com/bcdev/xrlint/issues). +See [CONTRIBUTING.md](CONTRIBUTING.md) for development and documentation setup, +and follow our [Code of Conduct](CODE_OF_CONDUCT.md). + +XRLint is distributed under the [MIT License](LICENSE). diff --git a/docs/about.md b/docs/about.md index c18c9e4..f0af381 100644 --- a/docs/about.md +++ b/docs/about.md @@ -1,82 +1,54 @@ # About XRLint -## Changelog - -You can find the complete XRLint changelog -[here](https://github.com/bcdev/xrlint/blob/main/CHANGES.md). +XRLint is an open-source linter and Python library for xarray datasets. +Its configurable rules and plugin model are inspired by ESLint. -## Reporting +## Changelog -If you have suggestions, ideas, feature requests, or if you have identified -a malfunction or error, then please -[post an issue](https://github.com/bcdev/xrlint/issues). +See the [change history](https://github.com/bcdev/xrlint/blob/main/CHANGES.md) +for release notes and API changes. -## Contributions +## Reporting and contributions -The XRLint project welcomes contributions of any form -as long as you respect our -[code of conduct](https://github.com/bcdev/xrlint/blob/main/CODE_OF_CONDUCT.md) -and follow our -[contribution guide](https://github.com/bcdev/xrlint/blob/main/CONTRIBUTING.md). +Report bugs and propose features through +[GitHub issues](https://github.com/bcdev/xrlint/issues). Include the XRLint and +Python versions, your configuration and command, the observed output, and a +small reproducible dataset or script when possible. -If you'd like to submit code or documentation changes, we ask you to provide a -pull request (PR) -[here](https://github.com/bcdev/xrlint/pulls). -For code and configuration changes, your PR must be linked to a -corresponding issue. +Contributions should follow the +[Code of Conduct](https://github.com/bcdev/xrlint/blob/main/CODE_OF_CONDUCT.md) +and [contribution guide](https://github.com/bcdev/xrlint/blob/main/CONTRIBUTING.md). +Code and configuration pull requests must reference a corresponding issue. ## Development -To install the XRLint development environment into an existing Python environment +From a repository checkout, install the project and its development extras: ```bash -pip install .[dev,doc] +python -m pip install -e ".[dev,doc]" +python -m ruff check +python -m pytest --cov=xrlint --cov-branch --cov-report=html ``` -or create a new environment using `conda` or `mamba` - -```bash -mamba env create -``` +Editable installation registers the command and plugin entry points needed by +the tests and documentation generator. See the contribution guide for conda +setup, code style, and the project layout. -### Testing and Coverage +## Documentation -XRLint uses [pytest](https://docs.pytest.org/) for unit-level testing -and code coverage analysis. +Build and preview from the repository root: ```bash -pytest --cov=xrlint --cov-report html +python -m mkdocs build --strict +python -m mkdocs serve ``` -### Code Style - -XRLint source code is formatted and quality-controlled using -using [ruff](https://docs.astral.sh/ruff/): - -```bash -ruff format -ruff check -``` - -### Documentation - -XRLint documentation is built using the [mkdocs](https://www.mkdocs.org/) tool. - -With repository root as current working directory: - -```bash -pip install .[doc] - -mkdocs build -mkdocs serve -mkdocs gh-deploy -``` - -The rule reference page is generated by a script called `docs/mkruleref.py` which is called by mkdocs during build. +The generated site is written to `site/`. The rule reference is generated by +`docs/mkruleref.py` during the build, using installed plugin metadata. +The API reference is rendered from Python docstrings by `mkdocstrings`. ## License -XRLint is open source made available under the terms and conditions of the +XRLint is distributed under the [MIT License](https://github.com/bcdev/xrlint/blob/main/LICENSE). - -Copyright © 2025 Brockmann Consult Development +See the license file and source headers for copyright notices. diff --git a/docs/api.md b/docs/api.md index 35ed2cd..08a9c31 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1,6 +1,88 @@ # Python API -This chapter provides a plain reference for the XRLint Python API. +Use `new_linter()` for individual datasets and `XRLint` for file discovery and +reports. The reference below is generated from the public classes and functions. + +## Validate a dataset + +```python +import xarray as xr +from xrlint.linter import new_linter + +linter = new_linter("recommended", rules={"var-units": "error"}) +dataset = xr.Dataset(attrs={"title": "Example dataset"}) +result = linter.validate(dataset, file_path="example.nc") + +for message in result.messages: + print(message.severity, message.rule_id, message.node_path, message.message) + +assert result.fatal_error_count == 0 +``` + +`new_linter()` loads installed plugins, but enables only the rules you configure. +It does not read configuration files. `Linter()` alone starts without plugin +registrations. Both accept configuration objects and named presets; additional +configuration passed to `validate()` is merged after the linter's configuration. + +Pass a source path to open a dataset, for example `linter.validate("example.nc")`. +The default opener closes files it opens. Existing datasets remain under the +caller's control. `file_path` labels in-memory results and also determines which +file-specific configuration objects match. + +Findings are returned in `Result.messages`. Severity `1` means warning and `2` +means error. Inspect `error_count`, `warning_count`, and `fatal_error_count`, and +use `message.suggestions` for any suggested corrections. Configuration or custom +rule errors may raise exceptions; not every failure is converted to a result. + +`result.to_json()` returns Python JSON-compatible values; use `json.dumps()` to +encode them. `result.to_html()` returns HTML, and notebooks display the result +as HTML automatically. This API does not apply the CLI's warning threshold. + +## Validate files and write reports + +```python +from xrlint.cli.engine import XRLint + +engine = XRLint( + no_config_lookup=True, + output_format="json", + output_path="report.json", + max_warnings=0, +) +engine.init_config("recommended") +results = engine.validate_files(["data/"]) +report = engine.format_results(results) +engine.write_report(report) + +failed = engine.result_stats.error_count > 0 or engine.max_warnings_exceeded +print(f"Checked {engine.result_stats.result_count} datasets; failed={failed}") +``` + +`validate_files()` returns an iterator: validation happens as you consume it. +`format_results()` consumes it and updates `result_stats`. Statistics accumulate +on the engine, so create a new engine for an independent run. Engine methods do +not terminate your process; inspect the counts to implement your own exit policy. + +To read a specific configuration, construct `XRLint(config_path="config.yaml")` +and call `init_config()`. With default constructor options, `init_config()` uses +the same working-directory discovery as the CLI. Arguments to `init_config()` +are appended after file and command-line rule configuration. + +## Dataset trees + +Pass an `xr.DataTree` directly to `validate()` to validate grouped data. Tree +traversal visits group nodes and the datasets at leaves. A tree without children +is treated as a dataset. Dataset contents on non-leaf groups are not independently +traversed. The core `conventions` and `content-desc` rules consider inherited +parent-group attributes, with local attributes taking precedence. + +Import tree-specific node types directly: + +```python +from xrlint.node import DataTreeNode, XarrayNode +``` + +These types are not currently re-exported by `xrlint.all`. ## Overview @@ -24,7 +106,7 @@ This chapter provides a plain reference for the XRLint Python API. [RuleContext][xrlint.rule.RuleContext] and [RuleExit][xrlint.rule.RuleExit]. Decorator [define_rule][xrlint.rule.define_rule] allows defining rules. - The `node` module defines the nodes passed to [RuleOp][xrlint.rule.RuleOp]: - base classes [None][xrlint.node.Node], [XarrayNode][xrlint.node.XarrayNode], + base classes [Node][xrlint.node.Node], [XarrayNode][xrlint.node.XarrayNode], and the specific nodes [DataTreeNode][xrlint.node.DataTreeNode], [DatasetNode][xrlint.node.DatasetNode], [VariableNode][xrlint.node.VariableNode], [AttrsNode][xrlint.node.AttrsNode], and [AttrNode][xrlint.node.AttrNode]. @@ -43,8 +125,9 @@ This chapter provides a plain reference for the XRLint Python API. of [RuleTest][xrlint.testing.RuleTest]s. Note: - the `xrlint.all` convenience module exports all of the above from a - single module. + the `xrlint.all` convenience module exports many common API definitions from + one module. Use the direct imports in this reference for definitions it does + not export, including `DataTreeNode`, `XarrayNode`, and `ResultStats`. ## CLI API @@ -82,6 +165,11 @@ Note: ::: xrlint.rule.RuleMeta +::: xrlint.rule.RuleConfig + options: + inherited_members: + - from_value + ::: xrlint.rule.RuleOp ::: xrlint.rule.RuleContext @@ -122,6 +210,23 @@ Note: ::: xrlint.result.Suggestion +::: xrlint.result.ResultStats + +## Formatter API + +The CLI provides `simple`, `json`, and `html` formatters. The formatter interfaces +are also available to applications that need to build their own reports. + +::: xrlint.formatter.Formatter + +::: xrlint.formatter.FormatterMeta + +::: xrlint.formatter.FormatterOp + +::: xrlint.formatter.FormatterContext + +::: xrlint.formatter.FormatterRegistry + ## Testing API ::: xrlint.testing.RuleTester diff --git a/docs/cli.md b/docs/cli.md index 624eb32..a2a261a 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -1,57 +1,100 @@ # Command Line Interface -After installation, the `xrlint` command can be used from the terminal. -The following are the command's usage help including a short description -of its options and arguments: +After installation, use `xrlint` from a terminal: +```text +xrlint [OPTIONS] [FILES]... ``` -Usage: xrlint [OPTIONS] [FILES]... - - Validate the given dataset FILES. - - When executed, XRLint does the following three things: - - (1) Unless options '--no-config-lookup' or '--config' are used it searches - for a default configuration file in the current working directory. Default - configuration files are determined by their filename, namely - 'xrlint_config.py' or 'xrlint-config.', where refers to the - filename extensions 'json', 'yaml', and 'yml'. A Python configuration file - ('*.py'), is expected to provide XRLInt configuration from a function - 'export_config()', which may include custom plugins and rules. - - (2) It then validates each dataset in FILES against the configuration. The - default dataset patters are '**/*.zarr' and '**/.nc'. FILES may comprise - also directories or URLs. The supported URL protocols are the ones supported - by xarray. Using remote protocols may require installing additional packages - such as S3Fs (https://s3fs.readthedocs.io/) for the 's3' protocol. If a - directory is provided that not matched by any file pattern, it will be - traversed recursively. - - (3) The validation result is dumped to standard output if not otherwise - stated by '--output-file'. The output format is 'simple' by default. Other - inbuilt formats are 'json' and 'html' which you can specify using the '-- - format' option. - - Please refer to the documentation (https://bcdev.github.io/xrlint/) for more - information. - -Options: - --no-config-lookup Disable use of default configuration files - -c, --config FILE Use this configuration instead of looking for a - default configuration file - --print-config FILE Print the configuration for the given file - --plugin MODULE Specify plugins. MODULE is the name of Python module - that defines an 'export_plugin()' function. - --rule SPEC Specify rules. SPEC must have format ': - ' (note the space character). - -o, --output-file FILE Specify file to write report to - -f, --format NAME Use a specific output format - default: simple - --color / --no-color Force enabling/disabling of color - --max-warnings COUNT Number of warnings to trigger nonzero exit code - - default: 5 - --init Write initial configuration file 'xrlint- - config.yaml' and exit. - --version Show the version and exit. - --help Show this message and exit. +`FILES` accepts dataset paths, directories, and remote URLs. The CLI finds +configuration, selects matching datasets, validates each one, and writes a +report. Run `xrlint --help` for the installed version's usage text. + +## Common commands + +```bash +xrlint --init +xrlint example.nc +xrlint data/ +xrlint --config configs/production.yaml data/ +xrlint --print-config data/example.nc +xrlint --rule "var-units: error" data/ +xrlint --format html --output-file report.html data/ +xrlint --format json --output-file report.json --max-warnings 0 data/ +``` + +For a one-off check without a configuration file: + +```bash +xrlint --no-config-lookup --rule "no-empty-attrs: warn" example.nc ``` + +## Options + +| Option | Behavior | +| --- | --- | +| `--no-config-lookup` | Disable automatic configuration-file discovery. | +| `-c, --config FILE` | Read this configuration file instead of searching. | +| `--print-config FILE` | Print the computed configuration as JSON and exit, without opening the dataset. | +| `--plugin MODULE` | Load a module exporting `export_plugin()`. Repeat to load multiple plugins. | +| `--rule SPEC` | Append a YAML rule mapping, such as `"var-units: error"`. Repeat for multiple rules; include the space after `:`. | +| `-o, --output-file FILE` | Write the report to a file rather than standard output. | +| `-f, --format NAME` | Select `simple` (default), `json`, or `html`. | +| `--color / --no-color` | Enable or disable styling for simple console output; enabled by default. | +| `--max-warnings COUNT` | Allow this many warnings before failure; default `5`. | +| `--init` | Create `xrlint-config.yaml` in the current directory and exit; refuse to overwrite it. | +| `--version` | Show the installed version and exit. | +| `--help` | Show command help and exit. | + +## Configuration and file selection + +Without `--config` or `--no-config-lookup`, discovery checks the current directory. +See [Configuration File](config.md#configuration-file) for the exact filename +order, including legacy names. The CLI does not enable a preset automatically +when no file is found; it reports `no rules configured` unless rules are supplied +another way. + +By default, dataset discovery selects `**/*.nc` and `**/*.zarr`. Unmatched +directories are walked recursively. A matched dataset directory such as a Zarr +store is processed as a single dataset. Configuration can add file types and +global exclusions; see [File and Ignore Patterns](config.md#file-and-ignore-patterns). + +Pass concrete paths or directories. A wildcard argument is expanded only if +your shell expands it; XRLint does not expand it itself. Use `xrlint .` to scan +the current directory. Running `xrlint` with no file arguments does no work. + +Remote access requires the appropriate filesystem and dataset backends. +For example, S3 Zarr access requires `s3fs` and `zarr`. +Opening options are configured per dataset; they are not forwarded to the +filesystem used for directory listing. See [Remote datasets](examples.md#remote-datasets). + +## Reports + +- `simple` streams readable messages with file paths, node paths, rule identifiers, + and a summary. Reports written to a file are unstyled. +- `json` writes an object with a `results` array. Each result contains a file path + and serialized messages, and may include its computed configuration. +- `html` writes a browser-readable report. + +Messages use severity `1` for warnings and `2` for errors. Fatal messages indicate +problems such as opening failures. A successful write of a report does not mean +validation passed: check the exit status as well. + +Use `--output-file` for machine-readable reports. Configuration notices and +warning-limit messages can also appear on standard output. + +## Exit status + +| Status | Meaning | +| --- | --- | +| `0` | No errors and warnings do not exceed the limit; also successful help, version, initialization, or configuration inspection. | +| `1` | Validation errors, warnings above the limit, or operational errors such as missing configuration or an unknown output format. | +| `2` | Command-line usage errors, such as an unknown option or invalid integer argument. | + +With `--max-warnings 5`, five warnings pass and six fail. Use +`--max-warnings 0` to fail on any warning. Negative values do not disable this +check: the implementation compares the count directly with the supplied limit. + +A run that selects no datasets can exit successfully, and a bare `xrlint` +invocation also exits successfully without validation. In automation, provide +explicit input paths and verify that your file-selection patterns match data. diff --git a/docs/config.md b/docs/config.md index 98df3a6..2e5b516 100644 --- a/docs/config.md +++ b/docs/config.md @@ -1,139 +1,478 @@ # Configure XRLint -_**Note**: this chapter's material is based on the documentation of how to [configure ESLint](https://eslint.org/docs/latest/use/configure/). -Many parts have been copied and adjusted as it applies in many similar ways to XRLint._ +XRLint uses an ordered list of configuration objects and named presets. +The CLI and Python API share this model. Loading a plugin makes its rules +available; a preset or `rules` entry enables them. ## Configuration File -The XRLint configuration file may be named any of the following: +Run `xrlint --init` to create `xrlint-config.yaml` containing `- recommended`. +Without `--config`, the CLI looks in the **current working directory only**, +using the first file it finds in this order: -* YAML format: `xrlint-config.yaml` (or use extension `.yml`) -* JSON format: `xrlint-config.json` -* Python module: `xrlint_config.py` (note the underscore) +1. `xrlint-config.yaml` +2. `xrlint-config.yml` +3. `xrlint-config.json` +4. `xrlint_config.yaml` (legacy name) +5. `xrlint_config.yml` (legacy name) +6. `xrlint_config.json` (legacy name) +7. `xrlint_config.py` -It should be placed in the root directory of your project and export -an array of [configuration objects](#configuration-objects) or -references to [predefined configuration objects](#predefined-configuration-objects). +It does not search parent directories or dataset directories. +Use `xrlint --config path/to/config.yaml data/` for an explicit file, or +`--no-config-lookup` to disable discovery. There is no implicit recommended +configuration: a run that loads no rules fails with `no rules configured`. -Here’s a YAML example: +These configurations are equivalent. The installed `xcube` plugin is +discovered automatically. ```yaml -- files: ["**/*.zarr", "**/*.nc"] -- plugins: - xcube: xrlint.plugins.xcube - recommended - xcube/recommended +- rules: + xcube/grid-mapping-naming: "off" ``` -Same using JSON: - ```json [ - {"files": ["**/*.zarr", "**/*.nc"]}, - { - "plugins": { - "xcube": "xrlint.plugins.xcube" - } - }, "recommended", - "xcube/recommended" + "xcube/recommended", + {"rules": {"xcube/grid-mapping-naming": "off"}} ] ``` -And as Python module: +A Python file must define `export_config()`: ```python def export_config(): return [ - {"files": ["**/*.zarr", "**/*.nc"]}, - {"plugins": {"xcube": "xrlint.plugins.xcube"}}, "recommended", "xcube/recommended", + {"rules": {"xcube/grid-mapping-naming": "off"}}, ] ``` +Python configurations can also contain plugin objects and processor instances. ## Configuration Objects -Each configuration object contains all the information XRLint needs -to execute on a set of files. Each configuration object is made up of -these properties: - -* `name` - A name for the configuration object. - This is used in error messages and config inspector to help identify which - configuration object is being used. -* `files` - A list of glob patterns indicating the files or URLs that the - configuration object should apply to. If not specified, the configuration - object applies to all files matched by any other configuration object. - See section [File and Ignore Patterns](#file-and-ignore-patterns) below. -* `ignores` - A list of glob patterns indicating the files and URLs that the - configuration object should not apply to. If not specified, the configuration - object applies to all files matched by `files`. If ignores is used without any - other keys in the configuration object, then the patterns act as _global ignores_. - See section [File and Ignore Patterns](#file-and-ignore-patterns) below. -* `opener_options` - A dictionary specifying keyword-arguments that are passed - directly to the `xarray.open_dataset()` function. The available options are - dependent on the xarray backend selected by the `engine` option. - See section [Opener Options](#opener-options) below. -* `linter_options` - A dictionary containing settings related to - the linting process. (Currently not used.) - See section [Linter Options](#linter-options) below. -* `settings` - An object containing name-value pairs of information that should - be available to all rules. -* `plugins` - A dictionary containing a name-value mapping of plugin names - to either plugin module names or `Plugin` objects. When `files` is specified, - these plugins are only available to the matching files. - See sections [Configuring Plugins](#configuring-plugins) - and [Custom Plugins](#custom-plugins) below. -* `rules` - An object containing the configured rules. - When `files` or `ignores` are specified, these rule configurations are only - available to the matching files. - See sections [Configuring Rules](#configuring-rules) - and [Custom Rules](#custom-rules) below. -* `processor` - A string indicating the name of a processor inside of a plugin, - i.e., `"/"`. In Python configurations - it can also be an object of type `ProcessorOp` containing - `preprocess()` and `postprocess()` methods. - See sections [Configuring Processors](#custom-processors) - and [Custom Processors](#custom-processors) below. +Each object can contain the following optional properties: + +| Property | Purpose | +| --- | --- | +| `name` | A descriptive label for the object. | +| `files` | Patterns selecting paths to which this object applies. | +| `ignores` | Patterns excluding paths from this object. | +| `opener_options` | Keyword arguments for opening datasets. | +| `linter_options` | Reserved for linting options; currently has no effect. | +| `settings` | Shared values available to rules through `ctx.settings`. | +| `plugins` | Mapping from namespaces to plugin module names, objects, or definitions. | +| `rules` | Mapping from rule identifiers to severities and arguments. | +| `processor` | A `"namespace/processor-name"` reference or Python `ProcessorOp` instance. | + +Objects without file filters apply to every selected dataset. Matching objects +merge from first to last. Later rule severities override earlier ones; different +rule identifiers accumulate. Option dictionaries merge by key, including nested +dictionaries; option lists merge by index. A later plugin under the same +namespace replaces the earlier plugin. A later non-null processor replaces the +earlier processor. + +Rule arguments have a specific merge rule: when severity stays the same, +positional arguments merge by index and keyword arguments merge by key. +When severity changes, the later rule configuration replaces the earlier one, +including its arguments. Repeat arguments you want to retain when changing +severity. + +Inspect the merged configuration with: + +```bash +xrlint --print-config data/example.nc +``` + +This does not open the dataset. It inspects configuration, but does not check +whether the CLI's global file filter will select the path. ## File and Ignore Patterns -_Coming soon_ +The CLI first selects dataset paths using a global file filter, then merges +the configuration objects that apply to each selected path. + +- Default included patterns are `**/*.nc` and `**/*.zarr`. +- An object containing only `files` and/or `ignores` (optionally with a `name`) + contributes to the global filter. Its `files` patterns **add to** the defaults. +- An object that also contains rules, settings, a processor, or other options + filters only that object's contribution. Its `ignores` do not exclude the + dataset from the whole run. +- Directories not selected as datasets are walked recursively. A selected Zarr + store is treated as one dataset. To exclude a subtree, match its descendants. + +For example, add HDF5 files, exclude generated data, and make missing units an +error only in published datasets: + +```yaml +- files: ["**/*.h5"] +- ignores: ["**/generated/**"] +- recommended +- files: ["**/published/**"] + rules: + var-units: error +``` + +Adding an extension does not install an xarray backend. To validate only NetCDF +files, pass those files explicitly or exclude other recognized formats, for +example with `ignores: ["**/*.zarr"]`. + +Patterns match the whole supplied path or URL; they are not rebased to the +configuration file. Paths found during traversal may be absolute. Use `/` +separators and patterns such as `**/published/**` when the leading path is not +fixed, including on Windows. + +| Syntax | Meaning | +| --- | --- | +| `*` | Zero or more characters other than `/`. | +| `**` | Zero or more characters, including `/`. | +| `**/` | Also matches paths without a leading directory. | +| `?` | One character. | +| `#text` | A comment pattern; ignored. | +| `!pattern` | Negation; in `ignores`, can re-include a matching excluded path. | + +This is a small glob implementation, not full minimatch or Git ignore syntax. +Brace expansion and character classes are unsupported. A trailing slash does +not mean "all descendants": use `**/cache/**`, not just `cache/`. +The built-in ignore names `.git` and `node_modules` are exact patterns; use +`**/.git/**` and `**/node_modules/**` for subtree exclusions. + +For a simple ignore exception, place the negation immediately after its exclusion: + +```yaml +- ignores: ["**/scratch/**", "!**/scratch/reference.nc"] +- recommended +``` + +Pass directories or concrete paths as CLI arguments. Glob patterns belong in +configuration; XRLint does not expand wildcard arguments itself. + +The low-level `Linter.validate()` API computes matching configuration directly; +it does not perform CLI discovery or split out global filters. Use +`XRLint.validate_files()` for the CLI's file-selection behavior. ## Opener Options -_Coming soon_ +For a source, XRLint tries `xarray.open_datatree()` first and falls back to +`xarray.open_dataset()` if that fails with a supported opening error. It selects +`engine="zarr"` for `.zarr` paths unless an engine is supplied. +`opener_options` provides keyword arguments to these openers: + +```yaml +- recommended +- files: ["**/*.nc"] + opener_options: + engine: netcdf4 + decode_times: true +``` + +Available options depend on your installed xarray version and backend. Keep +decoding enabled unless your rules are designed for undecoded data. +Opening options have no effect on existing `xr.Dataset` or `xr.DataTree` objects. + +For a public S3 Zarr dataset, opening options can include: + +```yaml +- recommended +- files: ["s3://**/*.zarr"] + opener_options: + engine: zarr + backend_kwargs: + storage_options: + anon: true +``` + +Install `zarr` and `s3fs` for this example. These storage options go to the +dataset opener. CLI directory listing uses a separate fsspec filesystem and +does not receive `opener_options`; for anonymous bucket traversal see +[Examples](examples.md#remote-datasets). + +When a processor is selected, it receives `opener_options` and controls opening. +The default opener closes datasets it opens; processors manage their resources. ## Linter Options -_Coming soon_ +`linter_options` is accepted and merged, but no options are currently consumed. +Omit it. Warning limits, formats, and output paths belong to +[CLI options](cli.md) or the `XRLint` constructor. + +Use `settings` for information shared by custom rules: + +```yaml +- recommended +- settings: + institution: Example Research Institute +``` + +A custom rule reads this as `ctx.settings.get("institution")`. +Settings have no effect unless a rule uses them. ## Configuring Plugins -_Coming soon_ +The CLI and `new_linter()` discover installed plugins from the `xrlint.rules` +entry-point group. An installed XRLint distribution supplies `core`, `xcube`, +and `acdd`. Discovery does not enable their rules. + +Core rule identifiers omit the namespace, as in `var-units`. Internally, the +core plugin is registered as `__core__`, which appears in printed configuration; +`core/` is not its namespace. Other plugins +use identifiers such as `xcube/dataset-title` or `acdd/1.3-conventions`. +Named presets follow the same convention: `recommended` selects core rules; +`acdd/recommended` selects ACDD rules. + +For plugins without an entry point, register an importable module explicitly: + +```yaml +- plugins: + project: my_project.xrlint_plugin +- project/recommended +``` + +The module must export `export_plugin()`. Put its registration before references +to its presets. Python configurations can instead supply a `Plugin` instance +or a dictionary defining a virtual plugin. The mapping key is the namespace +used in rule identifiers; use a consistent namespace in presets too. + +`--plugin MODULE` makes a module's rules available using its `meta.name` as the +namespace. Register custom plugins in the configuration file itself if that file +references their presets: the file is resolved before CLI plugin registrations +are merged. ## Configuring Rules -_Coming soon_ +| Value | Numeric equivalent | Effect | +| --- | --- | --- | +| `"off"` | `0` | Disable the rule. | +| `"warn"` | `1` | Emit warnings; the CLI warning limit determines failure. | +| `"error"` | `2` | Emit errors; any error fails a CLI validation run. | + +Quote `"off"` in YAML to keep it a string rather than a YAML boolean. +A rule's category (`problem`, `suggestion`, or `layout`) does not determine its +severity. See the [Rule Reference](rule-ref.md) for identifiers and options. + +To configure arguments, use a list starting with severity. Remaining items +are positional arguments, except that a final dictionary becomes keyword +arguments to the rule operation's constructor: + +```yaml +- recommended +- rules: + no-empty-attrs: "off" + var-units: error + access-latency: [warn, {threshold: 5.0}] + conventions: [error, {match: "^CF-"}] + var-desc: [warn, {attrs: [long_name]}] + content-desc: [warn, {skip_vars: true}] +``` + +For example, `[warn, 5.0]` also supplies the `access-latency` threshold +positionally. The `content-desc` option is named `skip_vars`. + +Schemas describe supported arguments, but XRLint does not yet validate +arguments against them before constructing a rule. Invalid names or types +may cause an exception or be ignored by the rule implementation. + +CLI `--rule` entries are appended after the file configuration: + +```bash +xrlint --rule "var-units: error" --rule "access-latency: [warn, {threshold: 5.0}]" data/ +``` ## Configuring Processors -_Coming soon_ +A processor opens a source as zero or more datasets, then combines their +validation messages. It applies only to sources, not existing in-memory xarray +objects. Each selected path uses at most one processor. + +The `xcube/recommended` and `xcube/all` presets already recognize `*.levels` +directories and select `xcube/multi-level-dataset`. To select it explicitly: + +```yaml +- files: ["**/*.levels"] +- recommended +- files: ["**/*.levels"] + processor: xcube/multi-level-dataset +``` + +The filter-only object adds the extension to CLI discovery. The processor reads +levels and optional `.zlevels` metadata. Use `xcube/recommended` as well to enable +checks specific to multi-level datasets. ## Predefined Configuration Objects -_Coming soon_ +Presets expand into configuration objects at their position in the list. +Combine presets and place overrides after them. + +| Preset | Contents | +| --- | --- | +| `recommended` | Core checks with warnings and errors; `no-empty-chunks` is disabled. | +| `all` | All core rules at error severity. | +| `xcube/recommended` | xcube checks plus discovery and processing of `*.levels` datasets. | +| `xcube/all` | All xcube rules at error severity, with the same multi-level setup. | +| `acdd/recommended` or `acdd/acdd_1.3` | ACDD 1.3: conventions and highly recommended attributes are errors; other checks are warnings. | +| `acdd/acdd_1.3_strict` | All included ACDD 1.3 checks are errors. | +| `acdd/acdd_1.3_warn` | All included ACDD 1.3 checks are warnings. | +| `acdd/acdd_1.1` | Exported preset, but its rule implementations are missing in 0.6.0; see below. | +| `acdd/acdd_1.0` | ACDD 1.0 attribute presence checks. | + +Plugin presets do not implicitly enable core rules. To combine core and ACDD: + +```yaml +- recommended +- acdd/recommended +``` + +In version 0.6.0, `acdd/acdd_1.1` references three `acdd/1.1-attrs-*` rules that +are not registered. Selecting it produces unknown-rule errors. The 1.0 and 1.3 +presets have registered rule implementations; choose the version appropriate +to your data. + +Version 0.6.0 also exports `acdd/acdd_1.3_strict_recommended`, but it contains an +unprefixed `1.3-attrs-recommended` rule identifier that resolves incorrectly. +Use this working configuration instead: + +```yaml +- acdd/recommended +- rules: + acdd/1.3-attrs-recommended: error +``` ## Custom Plugins -_Coming soon_ +Save this complete Python configuration as `xrlint_config.py`: + +```python +from xrlint.node import DatasetNode +from xrlint.plugin import new_plugin +from xrlint.rule import RuleContext, RuleOp + +plugin = new_plugin(name="project", version="1.0.0") + + +@plugin.define_rule("required-title") +class RequiredTitle(RuleOp): + """Require a specific dataset title.""" + + def __init__(self, title: str = "Example dataset"): + self.title = title + + def validate_dataset(self, ctx: RuleContext, node: DatasetNode): + if node.dataset.attrs.get("title") != self.title: + ctx.report( + f"Expected dataset title {self.title!r}.", + suggestions=[f"Set the title attribute to {self.title!r}."], + ) + + +plugin.define_config( + "recommended", {"rules": {"project/required-title": "error"}} +) + + +def export_plugin(): + return plugin + + +def export_config(): + return [ + {"plugins": {"project": plugin}}, + "recommended", + "project/recommended", + ] +``` + +Run `xrlint --config xrlint_config.py data/`. An explicit path avoids a +previously created YAML file taking precedence over the Python file. + +For a separately packaged plugin, put the plugin and `export_plugin()` in an +importable module, such as `my_project.xrlint_plugin`, and declare: + +```toml +[project.entry-points."xrlint.rules"] +project = "my_project.xrlint_plugin" +``` + +After installation, discovery uses `plugin.meta.name`. Keep the entry-point +name, metadata name, and preset namespace consistent. The entry point targets +a module with `export_plugin()`, not the function itself. + +See [Examples](examples.md#configuration) for a dictionary-based virtual plugin. ## Custom Rules -_Coming soon_ +Derive from `RuleOp` and register the class with +`@plugin.define_rule("rule-name")`, as above, or use `@define_rule(...)` +and add it to a plugin's rules dictionary. + +| Callback | Receives | +| --- | --- | +| `validate_datatree(ctx, node)` | A `DataTreeNode` for a tree group. | +| `validate_dataset(ctx, node)` | A `DatasetNode` exposing `node.dataset`. | +| `validate_variable(ctx, node)` | A `VariableNode` exposing `node.array` and `node.name`. | +| `validate_attrs(ctx, node)` | An `AttrsNode` exposing an attribute mapping. | +| `validate_attr(ctx, node)` | An `AttrNode` exposing an attribute name and value. | + +Dataset traversal visits the dataset, global attributes, coordinate variables, +and data variables, including each variable's attributes. Each rule gets its +own traversal. Tree traversal validates group nodes and visits dataset contents +at leaves; a tree without children is treated as a dataset. Contents attached +to non-leaf groups are not independently traversed as datasets. + +Use `ctx.report()` to emit a message with the current rule identifier, severity, +and node path. Rules should not depend on the configured severity. +`ctx.settings` provides shared values; during dataset callbacks, `ctx.dataset` +is the current dataset. `ctx.access_latency` is measured opening time, or +`None` for in-memory inputs. + +Raise `RuleExit` to stop the current rule's entire traversal. A normal `return` +skips only the current callback. Suggestions are accessible from +`Message.suggestions`; automatic fixes are not implemented. + +See [Developing rules](examples.md#developing-rules) for `RuleTester` examples +and the [Rule API](api.md#rule-api) for the interfaces. ## Custom Processors -_Coming soon_ +Subclass `ProcessorOp` and implement both methods: + +- `preprocess(file_path, opener_options)` returns a list of + `(xr.Dataset | xr.DataTree, path)` pairs. +- `postprocess(messages, file_path)` receives one message list per returned + dataset and returns the final flat message list. + +This processor can be added to the custom plugin above. It loads a dataset into +memory and closes its file before returning: + +```python +import xarray as xr + +from xrlint.processor import ProcessorOp + + +@plugin.define_processor("loaded-dataset") +class LoadedDataset(ProcessorOp): + def preprocess(self, file_path, opener_options): + with xr.open_dataset(file_path, **dict(opener_options)) as dataset: + dataset.load() + return [(dataset, file_path)] + + def postprocess(self, messages, file_path): + return [message for group in messages for message in group] +``` + +Append this object to `export_config()`'s returned list: + +```python +{"files": ["**/*.nc"], "processor": "project/loaded-dataset"} +``` +Loading the entire dataset suits small examples; production processors should +choose resource management appropriate to their data. XRLint does not close +datasets returned by processors. A named processor is constructed without +arguments. Use `opener_options` for opening settings or supply a configured +processor instance directly in a Python configuration. diff --git a/docs/examples.md b/docs/examples.md index 48e95e1..7965b38 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -1,35 +1,192 @@ # Examples +These examples build on [Getting Started](start.md). File-based examples need +the appropriate storage backend; in-memory examples need only XRLint. +The repository's [examples directory](https://github.com/bcdev/xrlint/tree/main/examples) +contains additional executable Python modules. + ## Configuration -::: examples.plugin_config +Combine core and ACDD checks, then tighten one rule for published files: + +```yaml +- recommended +- acdd/recommended +- ignores: ["**/scratch/**"] +- files: ["**/published/**"] + rules: + var-units: error +``` + +Use it from the CLI with `xrlint --config config.yaml data/`. + +For custom Python plugins, the repository provides two equivalent configurations: + +| Example | Approach | +| --- | --- | +| [plugin_config.py](https://github.com/bcdev/xrlint/blob/main/examples/plugin_config.py) | A `Plugin` object with a rule registered by `plugin.define_rule()`. | +| [virtual_plugin_config.py](https://github.com/bcdev/xrlint/blob/main/examples/virtual_plugin_config.py) | A dictionary containing metadata, rule classes, and a named preset. | - options: - members: false +Both define a `hello/good-title` rule that expects the title `Hello World!`, and +combine it with core recommended checks. From an installed repository checkout: -Source code: [`examples/plugin_config.py`](https://github.com/bcdev/xrlint/blob/main/examples/plugin_config.py) +```bash +xrlint --config examples/plugin_config.py example.nc +xrlint --config examples/virtual_plugin_config.py example.nc +``` -::: examples.virtual_plugin_config +The configuration can also be passed to the Python API: - options: - members: false +```python +import xarray as xr -Source code: [`examples/virtual_plugin_config.py`](https://github.com/bcdev/xrlint/blob/main/examples/virtual_plugin_config.py) +from examples.plugin_config import export_config +from xrlint.linter import new_linter + +dataset = xr.Dataset(attrs={"title": "Hello World!"}) +result = new_linter(export_config()).validate(dataset) +assert not any(m.rule_id == "hello/good-title" for m in result.messages) +``` + +Other recommended rules can still report missing metadata. For a complete +standalone plugin definition, see [Custom Plugins](config.md#custom-plugins). ## Developing rules -::: examples.rule_testing +This small rule checks only the dataset title: + +```python +import xarray as xr + +from xrlint.node import DatasetNode +from xrlint.rule import RuleContext, RuleOp, define_rule +from xrlint.testing import RuleTest, RuleTester - options: - members: false -Source code: [`examples/rule_testing.py`](https://github.com/bcdev/xrlint/blob/main/examples/rule_testing.py) +@define_rule("good-title") +class GoodTitle(RuleOp): + """Require a greeting as the dataset title.""" + + def validate_dataset(self, ctx: RuleContext, node: DatasetNode): + if node.dataset.attrs.get("title") != "Hello World!": + ctx.report("Expected title 'Hello World!'.") + + +RuleTester().run( + "good-title", + GoodTitle, + valid=[RuleTest(dataset=xr.Dataset(attrs={"title": "Hello World!"}))], + invalid=[ + RuleTest( + dataset=xr.Dataset(), + expected=["Expected title 'Hello World!'."], + ) + ], +) +``` + +Invalid cases must supply `expected`: either a message count or a list of expected +message texts. Valid cases must omit it. For parameterized rules, pass `args` +or `kwargs` to `RuleTest`. + +Use `RuleTester.define_test()` to generate a `unittest.TestCase` class discoverable +by pytest. The [rule_testing.py example](https://github.com/bcdev/xrlint/blob/main/examples/rule_testing.py) +demonstrates both approaches: + +```bash +python -m examples.rule_testing +python -m pytest tests/test_examples.py +``` ## API usage -::: examples.check_s3_bucket +Validate an in-memory dataset with a focused rule configuration: + +```python +import xarray as xr +from xrlint.linter import new_linter + +dataset = xr.Dataset() +linter = new_linter(rules={"no-empty-attrs": "error"}) +result = linter.validate(dataset, file_path="sample.nc") + +assert result.error_count == 1 +for message in result.messages: + print(message.node_path, message.message) + for suggestion in message.suggestions or []: + print("Suggestion:", suggestion.desc) +``` + +See [Validate files and write reports](api.md#validate-files-and-write-reports) +for directory traversal, JSON output, and aggregate statistics. + +## Remote datasets + +For S3 Zarr data, install the backend and filesystem support: + +```bash +python -m pip install zarr s3fs +``` + +Replace the example bucket and dataset names with your data. Open a public +dataset through the low-level linter: + +```python +from xrlint.linter import new_linter + +linter = new_linter("recommended") +result = linter.validate( + "s3://your-public-bucket/example.zarr", + opener_options={ + "engine": "zarr", + "backend_kwargs": {"storage_options": {"anon": True}}, + }, +) +print(result.error_count, result.warning_count) +``` + +For anonymous discovery, configure the listing filesystem explicitly. CLI +directory listing does not receive dataset `opener_options`: + +```python +import fsspec +from xrlint.linter import new_linter + +filesystem = fsspec.filesystem("s3", anon=True) +linter = new_linter("recommended") +for store in filesystem.glob("your-public-bucket/data/*.zarr"): + result = linter.validate( + filesystem.unstrip_protocol(store), + opener_options={ + "engine": "zarr", + "backend_kwargs": {"storage_options": {"anon": True}}, + }, + ) + print(result.file_path, result.error_count, result.warning_count) +``` + +For authenticated sources, configure the filesystem and backend credentials +through their normal mechanisms. The repository's +[check_s3_bucket.py](https://github.com/bcdev/xrlint/blob/main/examples/check_s3_bucket.py) +demonstrates high-level traversal using the environment's S3 configuration; +it accesses the network and its bucket must be accessible to that environment. + +## Multi-level datasets + +To inspect xcube multi-level datasets, save this as `config.yaml`: + +```yaml +- recommended +- xcube/recommended +``` + +Then run: - options: - members: false +```bash +xrlint --config config.yaml data/example.levels +``` -Source code: [`examples/check_s3_bucket.py`](https://github.com/bcdev/xrlint/blob/main/examples/check_s3_bucket.py) +The xcube preset adds the `*.levels` file pattern, selects the multi-level +processor, and enables the associated rules. See +[Configuring Processors](config.md#configuring-processors) and +[Custom Processors](config.md#custom-processors) for the opening lifecycle. diff --git a/docs/index.md b/docs/index.md index 4cc3cd8..992a458 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,37 +1,47 @@ # XRLint - A linter for xarray datasets - -XRLint is a [linting](https://en.wikipedia.org/wiki/Lint_(software)) -tool and library for [xarray](https://docs.xarray.dev/) datasets. -Its design is heavily inspired by the awesome [ESLint](https://eslint.org/) tool. - - -## Features - -- Flexible validation for - [`xarray.Dataset`](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html) and - [`xarray.DataTree`](https://docs.xarray.dev/en/stable/generated/xarray.DataTree.html) objects - by configurable rules. -- Available from CLI and Python API. -- Custom plugins providing custom rule sets allow addressing - different dataset conventions. -- Project-specific configurations including configuration of individual - rules and file-specific settings. -- Works with dataset files in the local filesystem or any of the remote - filesystems supported by xarray. - - -## Inbuilt Rules - -The following plugins provide XRLint's [inbuilt rules](rule-ref.md): - -- `core`: implementing the rules for - [tidy data](https://tutorial.xarray.dev/intermediate/data_cleaning/05.1_intro.html) - and the - [CF-Conventions](https://cfconventions.org/cf-conventions/cf-conventions.html). -- `xcube`: implementing the rules for - [xcube datasets](https://xcube.readthedocs.io/en/latest/cubespec.html). - Note, this plugin is fully optional. You must manually configure - it to apply its rules. It may be moved into a separate GitHub repo later. -- `acdd`: implements rules for [Attribute Convention for Data Discovery](https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3). - +XRLint checks xarray datasets for metadata, structure, and convention issues. +Use the [CLI](cli.md) for files and directories or the [Python API](api.md) for +`xarray.Dataset` and `xarray.DataTree` objects. Its configurable rules and plugin +model are inspired by ESLint. + +## Features + +- Rules for dataset metadata, coordinates, variables, and tree groups. +- YAML, JSON, or Python configuration, including file-specific settings. +- Local files and remote sources supported by installed xarray backends and + fsspec filesystem implementations. +- Text, JSON, and HTML reports, with rich result display in notebooks. +- Custom rules, plugins, processors, and reusable configurations. + +XRLint reports findings and suggestions; it does not automatically modify +datasets. Its rules check selected convention requirements and do not establish +complete compliance with a convention. + +## Built-in Rules + +The [Rule Reference](rule-ref.md) describes all rules discovered during the +documentation build, including their options and preset membership. + +| Plugin | Scope | Preset | +| --- | --- | --- | +| `core` | General dataset quality and selected CF convention checks | `recommended` | +| `xcube` | xcube dataset structure, including multi-level datasets | `xcube/recommended` | +| `acdd` | Attribute Convention for Data Discovery metadata, with versioned presets | `acdd/recommended` | + +Installed plugins load automatically. Their rules run only when enabled by +configuration; plugin presets can be combined: + +```yaml +- recommended +- xcube/recommended +- acdd/recommended +``` + +## Where to start + +Follow [Getting Started](start.md) to install XRLint and validate a small dataset. +Then use [Configuration](config.md) to select rules for your project. +[Examples](examples.md) covers custom rules, processors, and remote datasets. +See [About](about.md) for contribution and build instructions and +[Development Notes](todo.md) for current limitations and possible future work. diff --git a/docs/mkruleref.py b/docs/mkruleref.py index 4b7f39c..309240d 100644 --- a/docs/mkruleref.py +++ b/docs/mkruleref.py @@ -2,7 +2,10 @@ # This software is distributed under the terms and conditions of the # MIT license (https://mit-license.org/). +import json + from xrlint.config import plugins_from_entry_points +from xrlint.constants import CORE_PLUGIN_NAME from xrlint.plugin import Plugin from xrlint.rule import RuleConfig @@ -21,9 +24,6 @@ "layout": "material-text", } -# read_more_icon = "material-book-open-outline" -read_more_icon = "material-information-variant" - def write_rule_ref_page(): import mkdocs_gen_files @@ -35,13 +35,31 @@ def write_rule_ref_page(): with mkdocs_gen_files.open("rule-ref.md", "w") as stream: stream.write("# Rule Reference\n\n") stream.write( - "This page is auto-generated from XRLint's builtin" - " rules.\n" - "New rules will be added by upcoming XRLint releases.\n\n" + "This page is generated from plugins discovered through the " + "`xrlint.rules` entry-point group in the build environment. " + "Installing a plugin makes its rules available; select a preset " + "or configure individual rules to enable them.\n\n" + "Rule categories: :material-bug: problem, " + ":material-lightbulb: suggestion, :material-text: layout. " + "Preset severities: :material-lightning-bolt: error, " + ":material-alert: warning, :material-circle-off-outline: off.\n\n" + "Use rule identifiers in the `rules` mapping. Options follow the " + "severity, for example `access-latency: [warn, {threshold: 5.0}]`. " + "Schemas below describe option types, defaults, and constraints; " + "runtime schema validation is not yet implemented. See " + "[Configuring Rules](config.md#configuring-rules).\n\n" + "Preset membership below summarizes rules across configuration " + "objects; file-specific filters still determine applicability. " + "See [Predefined Configuration Objects]" + "(config.md#predefined-configuration-objects), including the " + "known ACDD preset limitations.\n\n" ) for plugin_name in sorted(plugins.keys()): plugin = plugins[plugin_name] - stream.write(f"## {plugin.meta.name} Rules\n\n") + display_name = ( + "core" if plugin.meta.name == CORE_PLUGIN_NAME else plugin.meta.name + ) + stream.write(f"## {display_name} Rules\n\n") if plugin.meta.ref: stream.write(f"- `{plugin.meta.ref.removesuffix(':export_plugin')}`\n") if plugin.meta.docs_url: @@ -56,20 +74,41 @@ def write_plugin_rules(stream, plugin: Plugin): stream.write( f"### :{rule_type_icons.get(rule_meta.type)}: `{rule_meta.name}`\n\n" ) + qualified_id = ( + rule_id + if plugin.meta.name == CORE_PLUGIN_NAME + else f"{plugin.meta.name}/{rule_id}" + ) + stream.write(f"Rule identifier: `{qualified_id}`\n\n") stream.write(rule_meta.description or "_No description._") if rule_meta.docs_url: stream.write(f"\n[More...]({rule_meta.docs_url})") stream.write("\n\n") # List the predefined configurations that contain the rule - stream.write("Contained in: ") + memberships = [] for config_id in sorted(config_rules.keys()): rule_configs = config_rules[config_id] - rule_config = rule_configs.get(rule_id) or rule_configs.get( - f"{plugin.meta.name}/{rule_id}" - ) + rule_config = rule_configs.get(qualified_id) + if rule_config is None and plugin.meta.name == CORE_PLUGIN_NAME: + rule_config = rule_configs.get(f"{CORE_PLUGIN_NAME}/{rule_id}") if rule_config is not None: - stream.write(f" `{config_id}`-:{severity_icons[rule_config.severity]}:") - stream.write("\n\n") + preset_id = ( + config_id + if plugin.meta.name == CORE_PLUGIN_NAME + else f"{plugin.meta.name}/{config_id}" + ) + memberships.append( + f"`{preset_id}` :{severity_icons[rule_config.severity]}:" + ) + stream.write( + "Contained in: " + (", ".join(memberships) or "No preset.") + "\n\n" + ) + if rule_meta.schema is not None: + stream.write("**Options schema**\n\n```json\n") + stream.write(json.dumps(rule_meta.schema, indent=2, ensure_ascii=False)) + stream.write("\n```\n\n") + else: + stream.write("No configurable options are declared.\n\n") def get_plugin_rule_configs(plugin: Plugin) -> dict[str, dict[str, RuleConfig]]: diff --git a/docs/start.md b/docs/start.md index 021500b..898a272 100644 --- a/docs/start.md +++ b/docs/start.md @@ -2,118 +2,131 @@ ## Installation +XRLint requires Python 3.10 or newer: + ```bash -pip install xrlint +python -m pip install xrlint ``` -or +Alternatively, install with conda: ```bash conda install -c conda-forge xrlint ``` - -## Command line interface - -Get basic help: +Install a backend for the files you want to open. For the NetCDF example below: ```bash -xrlint --help +python -m pip install netCDF4 ``` -Initializing a new project with +For Zarr files, install `zarr`; for S3 access, install `s3fs` as well. +The base package does not install every storage backend. + +## Command line interface + +Run the following commands in a new working directory: ```bash +xrlint --help xrlint --init ``` -writes a configuration file `xrlint-config.yaml` -into the current working directory: +Initialization creates `xrlint-config.yaml`: ```yaml - recommended ``` -This configuration file tells XRLint to use the predefined configuration -named `recommended`. +This enables the core recommended rules. `--init` refuses to overwrite an +existing file. XRLint discovers configuration in the current working directory, +not next to each dataset. -Create a dataset to test XRLint: +Save this Python script as `make_example.py` and run it with +`python make_example.py`: -```bash -python ->>> import xarray as xr ->>> test_ds = xr.Dataset(attrs={"title": "Test Dataset"}) ->>> test_ds.to_zarr("test.zarr") ->>> exit() +```python +import xarray as xr + +dataset = xr.Dataset(attrs={"title": "Example dataset"}) +dataset.to_netcdf("example.nc", engine="netcdf4") ``` -And run XRLint: +Validate the file: ```bash -xrlint test.zarr +xrlint example.nc ``` -You can now override the predefined settings by adding your custom -rule configurations: +This minimal dataset is intentionally missing some metadata, so warnings are +expected. Reports identify the rule and the affected dataset node. +The CLI exits with status 1 for any error or when warnings exceed the default +limit of five. To fail on any warning, use `--max-warnings 0`. + +Add overrides after the preset in `xrlint-config.yaml`: ```yaml - recommended - rules: - no-empty-attrs: off - var-units-attr: warn + no-empty-attrs: "off" + var-units: warn grid-mappings: error ``` -!!! note inline end "Built in and auto-loading plugins" - - The included plugins (such as `xcube` in the example configs here) and those from external libraries that are findable via [entry points](https://setuptools.pypa.io/en/latest/userguide/entry_point.html) do not need to be explicitly loaded. +Check an entire directory, inspect configuration, or write a report: - Run `xrlint --print-config ` to view the loaded plugins and configured rules. +```bash +xrlint data/ +xrlint --print-config example.nc +xrlint --format json --output-file report.json example.nc +``` - Custom plugins, or those that are not loadable via entry points will need to be explcitly loaded via the plugins object. +Pass a directory explicitly to scan it; running `xrlint` without file arguments +does not scan the current directory. See [CLI](cli.md) for all options and exit +behavior. -You can add rules from plugins as well: +## Add plugin rules -```yaml -- recommended -- plugins: - xcube: xrlint.plugins.xcube -- xcube/recommended -``` - -And customize its rules, if desired: +The installed `core`, `xcube`, and `acdd` plugins are discovered automatically. +Add a preset to enable its checks: ```yaml - recommended -# Explicit loading of included plugins is unneeded, see note -# - plugins: -# xcube: xrlint.plugins.xcube -- xcube/recommended +- xcube/recommended - rules: - xcube/grid-mapping-naming: off + xcube/grid-mapping-naming: "off" xcube/lat-lon-naming: warn ``` -Note the prefix `xcube/` used for the rule names. +Use `acdd/recommended` for ACDD 1.3 metadata checks. Plugin presets do not +implicitly enable core rules, so keep `recommended` when you want both. +See [Configuration](config.md) for all presets and custom plugin registration. ## Python API -The easiest approach to use the Python API is to import `xrlint.all`. -It contains all the public definitions from the `xrlint` package. - -```python -import xrlint.all as xrl -``` - -Start by creating a linter with recommended settings -using the `new_linter()` function . +Use `new_linter()` to load installed plugins and select rules explicitly: ```python import xarray as xr -import xrlint.all as xrl +from xrlint.linter import new_linter -test_ds = xr.Dataset(attrs={"title": "Test Dataset"}) +dataset = xr.Dataset(attrs={"title": "Example dataset"}) +linter = new_linter("recommended") +result = linter.validate(dataset, file_path="example.nc") -linter = xrl.new_linter("recommended") -linter.validate(test_ds) +print(f"{result.error_count} errors, {result.warning_count} warnings") +for message in result.messages: + print(message.rule_id, message.node_path, message.message) ``` + +`file_path` labels the result and controls file-specific configuration matching; +it does not open a file when the input is already an xarray object. +To open a file, call `linter.validate("example.nc")`. + +In a notebook, display `result` as the last expression in a cell for an HTML +report. Many public classes are also available through `import xrlint.all as xrl`; +see [Python API](api.md) for direct imports and tree-specific classes. + +The low-level linter does not search for configuration files. Use the +[high-level API](api.md#validate-files-and-write-reports) for CLI-style discovery +and directory traversal. diff --git a/docs/todo.md b/docs/todo.md index 2826c9e..a87c916 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -1,70 +1,47 @@ -# To Do - -## Required - -- enhance docs - - complete configuration page - - provide guide page - - use mkdocstrings ref syntax in docstrings - - provide configuration examples (use as tests?) - - add `docs_url` to all existing rules - - rule ref should cover rule parameters - -## Desired - -- project logo -- add `xcube` rule that helps to identify chunking issues -- apply rule op args/kwargs validation schema -- allow outputting suggestions, if any, that are emitted by some rules - - add CLI option - - expand/collapse messages with suggestions in Jupyter notebooks -- validate `RuleConfig.args/kwargs` against `RuleMeta.schema` - (see code TODO) - -## Nice to have - -- support `autofix` feature -- support `md` (markdown) output format -- support formatter op args/kwargs and apply validation schema - -# Ideas - -## Allow for different dataset openers - -- introduce `dataset_options` config: - - `opener: OpenerOp` - - `opener_options: dict[str, Any]` - -## Other plugins - -- `sgrid`: https://sgrid.github.io/sgrid/ -- `ugrid`: https://ugrid-conventions.github.io/ugrid-conventions/ - -## Generalize data linting - -Do not limit validations to `xr.Dataset`. -However, this requires new rule sets. - -To allow for other data models, we need to allow -for a specific validator type for a given data type. - -The validator validates specific node types -that are characteristic for a data type. - -To do so a traverser must traverse the elements of the data -and pass each node to the validator. - -Note, this is the [_Visitor Pattern_](https://en.wikipedia.org/wiki/Visitor_pattern), -where the validator is the _Visitor_ and a node refers to _Element_. - -To support the CLI mode, we need different data opener -types that can read the data from a file path. - -1. open data, if given data is a file path: - - find opener for file path - - open data -2. validate data - - find root element type and visitor type for data - - call the root element `accept(validator)` that validates the - root element `validate.root()` and starts traversal of - child elements. +# Development Notes + +This page records limitations and possible future work. It is not a release +schedule or a list of features currently supported. + +## Current limitations + +- Rule argument schemas are published as metadata but are not validated before + rule construction. See the TODO in `xrlint/_linter/apply.py`. +- Automatic dataset fixes are not implemented. Suggestions are available in + result messages, but there is no CLI option for applying them. +- The only built-in report formats are `simple`, `json`, and `html`. + Formatter-specific constructor options cannot currently be configured by CLI. +- `linter_options` is reserved and has no operational effect. +- CLI filesystem discovery does not receive dataset `opener_options`. + Anonymous or specially configured remote listing requires a separately + configured filesystem. +- Tree traversal validates dataset contents at leaves. Dataset contents on + non-leaf groups are not independently traversed, although the core + `conventions` and `content-desc` rules can read parent metadata. +- In 0.6.0, `acdd/acdd_1.1` references unregistered rules, and + `acdd/acdd_1.3_strict_recommended` contains an invalid unprefixed rule name. + See the [preset limitations and override](config.md#predefined-configuration-objects). + +## Potential improvements + +- Validate rule arguments against schemas before execution. +- Add dedicated, consistent presentation of suggestions in console and notebook + reports. +- Support automatic fixes and Markdown reports. +- Expose formatter arguments and validate them against formatter schemas. +- Improve chunking diagnostics and add focused rules where useful. +- Add documentation URLs for rules that currently lack them. +- Add a project logo. + +## Design ideas + +A future opener abstraction could separate source selection and opening from +rule execution. Today, custom [processors](config.md#custom-processors) provide +the extension point for alternate dataset layouts. + +Additional plugins could cover structured and unstructured grid conventions. + +Supporting data models beyond xarray would require explicit decisions about +opening, node types, traversal, and rule compatibility. The current rule +callbacks use a visitor-style traversal over xarray-specific nodes; a generalized +design would need an equivalent traversal contract for each supported model. diff --git a/mkdocs.yml b/mkdocs.yml index ca30b93..d50359e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -13,6 +13,7 @@ nav: - Python API: api.md - Examples: examples.md - About: about.md + - Development Notes: todo.md theme: name: material diff --git a/xrlint/cli/main.py b/xrlint/cli/main.py index 07707e9..5d5d942 100644 --- a/xrlint/cli/main.py +++ b/xrlint/cli/main.py @@ -120,16 +120,16 @@ def main( filename, namely 'xrlint_config.py' or 'xrlint-config.', where refers to the filename extensions 'json', 'yaml', and 'yml'. A Python configuration file ('*.py'), - is expected to provide XRLInt configuration from a function + is expected to provide XRLint configuration from a function 'export_config()', which may include custom plugins and rules. (2) It then validates each dataset in FILES against the configuration. - The default dataset patters are '**/*.zarr' and '**/.nc'. + The default dataset patterns are '**/*.zarr' and '**/*.nc'. FILES may comprise also directories or URLs. The supported URL protocols are the ones supported by xarray. Using remote protocols may require installing additional packages such as S3Fs (https://s3fs.readthedocs.io/) for the 's3' protocol. - If a directory is provided that not matched by any file pattern, + If a directory is provided that is not matched by any file pattern, it will be traversed recursively. (3) The validation result is dumped to standard output if not otherwise diff --git a/xrlint/config.py b/xrlint/config.py index de9da61..c20e569 100644 --- a/xrlint/config.py +++ b/xrlint/config.py @@ -71,7 +71,7 @@ def get_entry_point_plugins() -> "ConfigObject": """Create a configuration object that includes the plugins loaded from entry points. Returns: - A new `Config` object + A new `ConfigObject` containing the discovered plugins. """ return ConfigObject(plugins=plugins_from_entry_points()) @@ -110,24 +110,23 @@ class ConfigObject(MappingConstructible, JsonSerializable): the configuration object applies to all files matched by any other configuration object. - When a configuration object contains only the files property - without accompanying rules or settings, it effectively acts as - a _global file filter_. This means that XRLint will recognize - and process only the files matching these patterns, thereby - limiting its scope to the specified files. The inbuilt - global file filters are `["**/*.zarr", "**/*.nc"]`. + In CLI discovery, an object containing only file and ignore patterns + (optionally with a name) contributes to the global file filter. + Its included patterns add to the defaults `["**/*.zarr", "**/*.nc"]`; + they do not replace them. """ ignores: list[str] | None = None """An array of glob patterns indicating the files that the configuration object should not apply to. If not specified, the configuration object applies to all files matched by `files`. - If `ignores` is used without any other keys in the configuration - object, then the patterns act as _global ignores_. + In CLI discovery, objects containing only file and ignore patterns + (optionally with a name) contribute global exclusions. Otherwise, + these patterns exclude only this object's contribution. """ linter_options: dict[str, Any] | None = None - """A dictionary containing options related to the linting process.""" + """Reserved for linting options. Currently no options are consumed.""" opener_options: dict[str, Any] | None = None """A dictionary containing options that are passed to @@ -347,9 +346,8 @@ def compute_config_object(self, file_path: str) -> ConfigObject | None: file_path: A dataset file path. Returns: - A `Config` object which may be empty, or `None` - if `file_path` is not included by any `files` pattern - or intentionally ignored by global `ignores`. + A merged `ConfigObject`, or `None` if no object matches. + Global filtering is handled separately by CLI discovery. """ config_obj = None diff --git a/xrlint/linter.py b/xrlint/linter.py index 75b482a..7f0dfff 100644 --- a/xrlint/linter.py +++ b/xrlint/linter.py @@ -16,8 +16,8 @@ def new_linter(*configs: ConfigLike, **config_props: Any) -> "Linter": - """Create a new `Linter` with the core plugin included and the - given additional configuration. + """Create a new `Linter` with installed entry-point plugins and the + given configuration. Rules must be enabled explicitly. Args: *configs: Variable number of configuration-like arguments. @@ -39,7 +39,7 @@ class Linter: Using the constructor directly creates an empty linter with no configuration - even without the core plugin and its predefined rule configurations. - If you want a linter with core plugin included use the + If you want a linter with installed plugins included use the `new_linter()` function. Args: diff --git a/xrlint/plugins/core/rules/access_latency.py b/xrlint/plugins/core/rules/access_latency.py index de753e2..0aa9b9f 100644 --- a/xrlint/plugins/core/rules/access_latency.py +++ b/xrlint/plugins/core/rules/access_latency.py @@ -18,7 +18,7 @@ version="1.0.0", description=( "Ensure that the time it takes to open a dataset from its source" - " does a exceed a given `threshold` in seconds." + " does not exceed a given `threshold` in seconds." f" The default threshold is `{DEFAULT_THRESHOLD}`." ), schema=schema( diff --git a/xrlint/plugins/core/rules/content_desc.py b/xrlint/plugins/core/rules/content_desc.py index 33b333b..f85d6ca 100644 --- a/xrlint/plugins/core/rules/content_desc.py +++ b/xrlint/plugins/core/rules/content_desc.py @@ -34,7 +34,7 @@ "- `commons`: list of names of required variable attributes" " that can also be defined globally." f" Defaults to `{DEFAULT_COMMON_ATTRS}`.\n" - "- `no_vars`: do not check variables at all." + "- `skip_vars`: do not check variables at all." f" Defaults to `{DEFAULT_SKIP_VARS}`.\n" "- `ignored_vars`: list of ignored variables (regex patterns)." f" Defaults to `{DEFAULT_IGNORED_VARS}`.\n" diff --git a/xrlint/plugins/xcube/rules/no_chunked_coords.py b/xrlint/plugins/xcube/rules/no_chunked_coords.py index 12b46b8..206a8d0 100644 --- a/xrlint/plugins/xcube/rules/no_chunked_coords.py +++ b/xrlint/plugins/xcube/rules/no_chunked_coords.py @@ -19,7 +19,7 @@ description=( "Coordinate variables should not be chunked." " Can be used to identify performance issues, where chunked coordinates" - " can cause slow opening if datasets due to the many chunk-fetching" + " can cause slow opening of datasets due to the many chunk-fetching" " requests made to (remote) filesystems with low bandwidth." " You can use the `limit` parameter to specify an acceptable number " f" of chunks. Its default is {DEFAULT_LIMIT}." diff --git a/xrlint/result.py b/xrlint/result.py index b01de3c..05e5b56 100644 --- a/xrlint/result.py +++ b/xrlint/result.py @@ -91,10 +91,9 @@ class Result(JsonSerializable): """The aggregated information of linting a dataset.""" file_path: str - """The absolute path to the file of this result. - This is the string "" if the file path is unknown - (when you didn't pass the `file_path` option to the - `xrlint.lint_dataset()` method). + """The path or label associated with this result, which may be relative. + For an in-memory input without a source or explicit `file_path`, + `Linter.validate()` uses "" or "". """ config_object: Union["ConfigObject", None] = None diff --git a/xrlint/rule.py b/xrlint/rule.py index d044080..cbf9584 100644 --- a/xrlint/rule.py +++ b/xrlint/rule.py @@ -159,13 +159,13 @@ class RuleMeta(OperationMeta): """Rule documentation URL.""" schema: dict[str, Any] | list[dict[str, Any]] | bool | None = None - """JSON Schema used to specify and validate the rule operation - options. + """JSON Schema describing the rule operation options. + Runtime validation against this schema is not yet implemented. It can take the following values: - Use `None` (the default) to indicate that the rule operation - as no options at all. + has no declared options. - Use a schema to indicate that the rule operation takes keyword arguments only. The schema's type must be `"object"`. diff --git a/xrlint/version.py b/xrlint/version.py index d7072d8..a5f95a2 100644 --- a/xrlint/version.py +++ b/xrlint/version.py @@ -2,4 +2,4 @@ # This software is distributed under the terms and conditions of the # MIT license (https://mit-license.org/). -version = "0.6.0" +version = "0.7.0.dev0"