Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Weekly, grouped updates for the dev toolchain (PHPUnit, PHPStan,
# php-cs-fixer) and the workflow actions. The merged PRs are also what
# keep the scheduled CI run alive: GitHub disables schedules in public
# repos after 60 days without activity.
version: 2
updates:
- package-ecosystem: composer
directory: /
schedule:
interval: weekly
day: monday
open-pull-requests-limit: 5
commit-message:
prefix: "chore(deps)"
groups:
composer-minor-patch:
update-types: [minor, patch]

- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
day: monday
commit-message:
prefix: "chore(actions)"
groups:
actions:
patterns: ["*"]
24 changes: 9 additions & 15 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ on:
push:
branches: [main]
pull_request:
# Weekly run against fresh runners and PHP patch releases, so a green
# badge means "green this week". GitHub disables scheduled workflows in
# public repos after 60 days with no repository activity; the weekly
# Dependabot PRs (and merging them) are what keep this schedule alive.
schedule:
- cron: '29 6 * * 1' # Mondays 06:29 UTC
workflow_dispatch:

env:
# Opt every JavaScript-based action into the Node 24 runtime ahead of
Expand All @@ -20,10 +27,7 @@ jobs:
steps:
- uses: actions/checkout@v4

# Pinned to 2.36.0: setup-php 2.37.x stopped installing PHP on the
# macos-14 runners on 2026-06-12 (probe in DisplaceTech/ext-template).
# Unpin when upstream ships a fix.
- uses: shivammathur/setup-php@2.36.0
- uses: shivammathur/setup-php@v2
with:
php-version: '8.4'
coverage: none
Expand All @@ -47,20 +51,10 @@ jobs:
# 8.5 is the current stable; 8.4 is the most-deployed; 8.3 is
# security-only through end of 2026 but still widely installed.
php: ['8.3', '8.4', '8.5']
exclude:
# setup-php cannot currently install PHP 8.5 on the macos-14
# runners (2026-06-12: 2.37.x fails for every PHP version there,
# and 2.36.0 predates 8.5). Re-add this leg when ext-template's
# "macOS probe" workflow passes on @v2 again.
- os: macos-14
php: '8.5'
steps:
- uses: actions/checkout@v4

# Pinned to 2.36.0: setup-php 2.37.x stopped installing PHP on the
# macos-14 runners on 2026-06-12 (probe in DisplaceTech/ext-template).
# Unpin when upstream ships a fix.
- uses: shivammathur/setup-php@2.36.0
- uses: shivammathur/setup-php@v2
with:
php-version: ${{ matrix.php }}
coverage: none
Expand Down
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Changelog

All notable changes to ai-toolkit are documented here. The format is
based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
this project adheres to [Semantic Versioning](https://semver.org/).
Chunk-boundary algorithm changes are always minor releases, even when
signature-compatible. Dates are tag dates.

## [0.1.1] - 2026-10-29

No code changes; chunk boundaries are identical to 0.1.0.

### Changed

- CI runs weekly as well as on push, and tests PHP 8.5 on macOS arm64
again; setup-php is unpinned from 2.36.0, which carried known
advisories. Dependabot keeps the dev toolchain and actions current.
- README: Status section, stack table, links to the write-up; PLAN.md
reframed as design notes; remaining ideas moved to `future` issues.

## [0.1.0] - 2026-06-11

First release.

### Added

- Text chunkers behind one `Chunker` interface, sized in characters
(multibyte-aware) with a hard `mb_strlen($chunk) <= $size` invariant:
`RecursiveCharacterChunker`, `SentenceChunker`, `FixedSizeChunker`.
- `Vector\Packed`: `pack()` / `unpack()`, `dot()`, `cosine()`, `norm()`,
per-vector `normalize()`, and Matryoshka `truncate()` over packed
float32 buffers.

[0.1.1]: https://github.com/DisplaceTech/ai-toolkit/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/DisplaceTech/ai-toolkit/releases/tag/v0.1.0
22 changes: 12 additions & 10 deletions PLAN.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,14 @@
# ai-toolkit — Plan
# ai-toolkit — design notes

Part of the [phpai stack](https://github.com/DisplaceTech). Pure-PHP
utilities every doc example wants; boring on purpose. No models, no
extensions, no framework ambitions.

## v0.1.0 scope (current)
This file holds design notes and decision history, not a to-do list.
What shipped is in [`CHANGELOG.md`](CHANGELOG.md); remaining ideas are
[`future` issues](https://github.com/DisplaceTech/ai-toolkit/issues?q=is%3Aissue+is%3Aopen+label%3Afuture), each with its reasoning.

## v0.1.0 scope (shipped 2026-06-11)

- [x] `Vector\Packed` — pack/unpack, dot, cosine, norm, per-vector
normalize, MRL truncation over packed float32 buffers.
Expand All @@ -19,15 +23,13 @@ extensions, no framework ambitions.
- [x] Rewrite ext-infer / ext-turbovec doc examples on these helpers
where it clarifies — done 2026-06-11 (recipes in both repos).

## Later (unscheduled)
## Remaining work

Tracked as [`future` issues](https://github.com/DisplaceTech/ai-toolkit/issues?q=is%3Aissue+is%3Aopen+label%3Afuture):

- Markdown-aware chunker (split on headings before paragraphs) if doc
examples keep wanting it.
- `Packed::topK()` brute-force search helper — only if a doc example
needs search without ext-turbovec; otherwise it competes with the
extension for no reason.
- Token-count estimation helper — only with a credible accuracy story;
~4 chars/token guidance lives in the README until then.
- [#1](https://github.com/DisplaceTech/ai-toolkit/issues/1) Markdown-aware chunker
- [#2](https://github.com/DisplaceTech/ai-toolkit/issues/2) `Packed::topK()` brute-force helper, only if a doc example needs it
- [#3](https://github.com/DisplaceTech/ai-toolkit/issues/3) Token-count estimation helper, only with a credible accuracy story

## Design notes

Expand Down
34 changes: 34 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,16 @@

---

## Status

Feature-complete for its scope. Maintained: dependency updates and
compatibility fixes ship as patch/minor releases — see the
[CHANGELOG](CHANGELOG.md). Last verified: PHP 8.3/8.4/8.5 on Linux x86_64 and macOS arm64, `ext-mbstring` only, 2026-10-29.

Remaining ideas are tracked as [`future` issues](https://github.com/DisplaceTech/ai-toolkit/issues?q=is%3Aissue+is%3Aopen+label%3Afuture).
**Co-maintainers welcome** — if you run this in production and want a
say in where it goes, open an issue or a PR.

## What is ai-toolkit?

The plumbing every embed-and-search pipeline needs and no one wants to
Expand Down Expand Up @@ -106,6 +116,30 @@ database · **document loaders/parsers** (PDF, HTML, ...) — bring your
own text · **an orchestration framework** — chains and agents belong
to the frameworks.

## Part of a stack

Transcribe → chunk → embed → search → answer, entirely on your hardware:

| Stage | Package | What it does |
| ----- | ------- | ------------ |
| Transcribe | [ext-whisper](https://github.com/DisplaceTech/ext-whisper) | Speech-to-text over whisper.cpp |
| Chunk | **ai-toolkit** (this repo) | Text chunkers and packed-vector math, pure PHP |
| Embed · rerank · answer | [ext-infer](https://github.com/DisplaceTech/ext-infer) | Local LLM chat, embeddings and reranking over llama.cpp |
| Search | [ext-turbovec](https://github.com/DisplaceTech/ext-turbovec) | Compressed in-memory vector search (TurboQuant) |
| Integrate | [ai-contracts](https://github.com/DisplaceTech/ai-contracts) | The interfaces frameworks code against |
| Reference app | [rag-over-markdown](https://github.com/DisplaceTech/rag-over-markdown) | The whole pipeline over a folder of markdown |

Every package links the others; the extensions share one scaffold,
[ext-template](https://github.com/DisplaceTech/ext-template).

## Writing about this

- [ai-contracts & ai-toolkit: The Primitives a PHP AI Ecosystem Needs](https://eric.mann.blog/ai-contracts-ai-toolkit-php-primitives/)
- [RAG Over Markdown: An End-to-End Local Pipeline in Pure PHP](https://eric.mann.blog/rag-over-markdown-local-pipeline-php/)

The stack's launch series ran Sept 22 – Oct 1, 2026; everything else
is at [eric.mann.blog/writing](https://eric.mann.blog/writing/).

## License

[MIT](LICENSE) &copy; 2026 Eric Mann / Displace Technologies
Loading