Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
tag:
description: Published release tag matching pyproject.toml
required: true
default: v0.1.0
default: v0.1.1
type: string
confirm:
description: Upload this reviewed release to PyPI
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
tag:
description: Release tag matching pyproject.toml
required: true
default: v0.1.0
default: v0.1.1
type: string

permissions:
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ uv tool install nullius-eval
nullius serve
```

A specific version: `uv tool install nullius-eval==0.1.0`.\
A specific version: `uv tool install nullius-eval==0.1.1`.\
The main branch, from source:

```bash
Expand Down
2 changes: 1 addition & 1 deletion app/packaging/PYPI.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Update to the latest version with:
uv tool upgrade nullius-eval
```

To select a specific version, use `uv tool install nullius-eval==0.1.0`.
To select a specific version, use `uv tool install nullius-eval==0.1.1`.
Add `--force` when replacing an existing installation. Explicit version pins
are retained by `uv tool upgrade`; use `uv tool install --force nullius-eval`
to return to the latest version without a pin.
Expand Down
16 changes: 8 additions & 8 deletions app/packaging/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The repository stays private; publication to PyPI makes the uploaded package
and its included source public, independently of GitHub access.

`pyproject.toml` owns the package version. The release tag is that version
prefixed with `v`: version `0.1.0` corresponds to tag `v0.1.0`. Release
prefixed with `v`: version `0.1.1` corresponds to tag `v0.1.1`. Release
notes live in `docs/releases/<tag>.md`.

## Build the downloads locally
Expand All @@ -23,15 +23,15 @@ From the repository root, with Python 3.12 or later and `uv` available:
```bash
python3 app/packaging/release.py build \
--out-dir /tmp/nullius-release \
--tag v0.1.0
--tag v0.1.1
```

The tag must match the package version, and the output directory must be absent
or empty. Existing downloads are never overwritten. The output contains:

```text
nullius_eval-0.1.0-py3-none-any.whl
nullius_eval-0.1.0.tar.gz
nullius_eval-0.1.1-py3-none-any.whl
nullius_eval-0.1.1.tar.gz
SHA256SUMS
```

Expand All @@ -45,7 +45,7 @@ tool replacement. It does not run a benchmark or read personal settings.
To check a wheel independently:

```bash
python3 app/packaging/verify.py /tmp/nullius-release/nullius_eval-0.1.0-py3-none-any.whl
python3 app/packaging/verify.py /tmp/nullius-release/nullius_eval-0.1.1-py3-none-any.whl
```

To validate the complete downloaded set, including checksums and package
Expand All @@ -54,7 +54,7 @@ metadata, without rebuilding it:
```bash
python3 app/packaging/release.py verify \
--dist-dir /tmp/nullius-release \
--tag v0.1.0
--tag v0.1.1
```

The source distribution is a Python build input. Neither archive is a
Expand All @@ -66,7 +66,7 @@ the wheel's `py3-none-any` filename.

1. Review and merge the version, release notes and intended changes into `main`.
2. Open **Actions → draft release → Run workflow**, select `main`, and provide
the matching tag, such as `v0.1.0`.
the matching tag, such as `v0.1.1`.
3. The workflow runs the backend and route suites, builds and verifies the
packages, then creates a draft release for the exact commit it checked
out: a prerelease for an `rc` tag such as `v0.1.0rc2`, the latest release
Expand Down Expand Up @@ -144,6 +144,6 @@ uninstall that tool with `uv tool uninstall nullius` before installing
tool environment and remain in place.

The default command installs the latest version without a pin. To select this
specific version, use `uv tool install nullius-eval==0.1.0`. Add `--force`
specific version, use `uv tool install nullius-eval==0.1.1`. Add `--force`
when replacing an existing installation. `uv tool upgrade nullius-eval`
updates an unpinned installation; it retains an explicit version constraint.
1 change: 1 addition & 0 deletions app/site/mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ nav:
- Metrics:
- Metric field guide: field-guide.md
- Release notes:
- v0.1.1: releases/v0.1.1.md
- v0.1.0: releases/v0.1.0.md
- v0.1.0rc2: releases/v0.1.0rc2.md
- v0.1.0rc1: releases/v0.1.0rc1.md
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Nullius runs AI evaluations on your machines and opens every score into the evid
| [Capabilities](capabilities/supported/README.md) | Every capability, its port, how to add one |
| [The standard](standard.md) | Ports, kits, setups, controls and proofs |
| [CLI](cli/README.md) | Every command, by topic |
| [Release notes](releases/v0.1.0.md) | Highlights, changelog and installation notes by version |
| [Release notes](releases/v0.1.1.md) | Highlights, changelog and installation notes by version |
| [Architecture](how-it-works.md) | Execution paths and files on disk |
| [Metric field guide](field-guide.md) | Every metric and outcome; *Reference → Metrics* on the website |

Expand Down
6 changes: 3 additions & 3 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ uv tool upgrade nullius-eval
```

To select a specific version instead, use `uv tool install
nullius-eval==0.1.0`. If replacing an existing installation, add `--force`.
nullius-eval==0.1.1`. If replacing an existing installation, add `--force`.
An explicit version pin is retained by `uv tool upgrade`; change the pin with
`uv tool install --force nullius-eval==VERSION`, or remove it with
`uv tool install --force nullius-eval`.
Expand All @@ -57,7 +57,7 @@ Wheel downloads are listed under
repository's access permissions. From the folder containing the file:

```bash
uv tool install ./nullius_eval-0.1.0-py3-none-any.whl
uv tool install ./nullius_eval-0.1.1-py3-none-any.whl
nullius serve
```

Expand All @@ -69,7 +69,7 @@ From a checkout, build the wheel, then install the resulting file:

```bash
uv build --wheel --out-dir dist
uv tool install ./dist/nullius_eval-0.1.0-py3-none-any.whl
uv tool install ./dist/nullius_eval-0.1.1-py3-none-any.whl
nullius serve
```

Expand Down
72 changes: 72 additions & 0 deletions docs/releases/v0.1.1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
**v0.1.1 · Hugging Face endpoints, OSWorld and machine constraints**

- **Hugging Face endpoint.** Send a run's Hugging Face downloads to a mirror such as `https://hf-mirror.com` from Settings → Data providers, or with `HF_ENDPOINT`. It reaches drivers, Original setups, MCP servers and runs on hosts; pinned files are still checked by their sha256.
- **Machine constraints.** Set the CPUs, memory, processes, concurrent units and GPUs Nullius may use on each machine. Test checks that each limit is enforced, and standard runs apply them to their containers and processes. Run hosts is now one table.
- **New benchmark: OSWorld1** on the Nullius standard: OSWorld's Ubuntu tasks, each in its own VM, with OSWorld's own agents as the Original setup and a null control. The standard defines the `agent.computer/v1` port it declares, and a locked install fetches the git submodules its lock names.
- **GSM8K-Platinum.** GSM8K offers the cleaned test split as a second dataset, with the same unit numbers as the original.
- **Run listed units.** `nullius standard run ... --units FILE` runs only the units a file names; merge refuses shards whose lists differ.
- **Runs by launch.** A participant run over several shards is one row with the merged numbers, and unfolds into one row per shard. Controls are drawn whatever their state, and the task-check line names a control that did not check every task.
- **Fix.** A model provider's Remove appears and disappears after a save without a reload.

## Contributors

Thank you to everyone who built this release:
@aitianshi, @harenome and @pierreleca.

## Install

Install the latest version with [uv](https://docs.astral.sh/uv/) and Python 3.12+:

```bash
uv tool install nullius-eval
nullius serve
```

For an existing unpinned installation, run `uv tool upgrade nullius-eval`.
Open <http://127.0.0.1:5057>.

<details markdown="1">
<summary>Full changelog</summary>

## What's Changed

- feat(standard): define agent.computer/v1 and install a locked project's git submodules by @pierreleca in [#114](https://github.com/ThinkFlowLab/nullius/pull/114)
- feat(catalogue): add OSWorld1 with OSWorld's own agents and a null control by @pierreleca in [#117](https://github.com/ThinkFlowLab/nullius/pull/117)
- feat(standard): run the units a file lists with --units by @pierreleca in [#120](https://github.com/ThinkFlowLab/nullius/pull/120)
- feat(settings): set and enforce what Nullius may use on each machine by @harenome in [#122](https://github.com/ThinkFlowLab/nullius/pull/122)
- fix(settings): bring a provider card's Remove up to date after a save by @harenome in [#125](https://github.com/ThinkFlowLab/nullius/pull/125)
- fix(runs): draw a launch's controls whatever their state by @harenome in [#127](https://github.com/ThinkFlowLab/nullius/pull/127)
- feat(settings): send Hugging Face downloads to a chosen endpoint by @aitianshi in [#129](https://github.com/ThinkFlowLab/nullius/pull/129)
- feat(catalogue): offer GSM8K-Platinum as a second dataset of GSM8K by @aitianshi in [#132](https://github.com/ThinkFlowLab/nullius/pull/132)
- feat(runs): fold a launch's shards into one row per participant by @harenome in [#137](https://github.com/ThinkFlowLab/nullius/pull/137)

Test and CI changes: [#135](https://github.com/ThinkFlowLab/nullius/pull/135).

## New Contributors

- @pierreleca made their first contribution in [#114](https://github.com/ThinkFlowLab/nullius/pull/114)

**Full Changelog**: [v0.1.0...v0.1.1](https://github.com/ThinkFlowLab/nullius/compare/v0.1.0...v0.1.1)

</details>

<details markdown="1">
<summary>Installation options, downloads and validation</summary>

To pin this version, use `uv tool install nullius-eval==0.1.1`.
Alternatively, install the attached wheel with
`uv tool install ./nullius_eval-0.1.1-py3-none-any.whl`.
Add `--force` when replacing an existing tool installation. Settings,
catalogue additions and runs stay outside the installed package;
`nullius paths` shows their locations.

The downloads contain the wheel, source archive and `SHA256SUMS`.
After downloading all three files into one directory, verify them on Linux
with `sha256sum --check SHA256SUMS`.

Release validation targets Linux with Python 3.12: backend and route tests,
installed commands and resources, and user-storage preservation. It does not
establish macOS or Windows support or validate every live benchmark/provider
combination. These are Python packages with a local web UI.

</details>
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "nullius-eval"
version = "0.1.0"
version = "0.1.1"
description = "An evaluation framework for AI agents. Bring any benchmark, agent or capability, run it on your machines, and inspect the evidence behind every score. Bring anything, run it, prove it."
readme = "app/packaging/PYPI.md"
requires-python = ">=3.12"
Expand Down
Loading