Repository navigation
feat(multimodal): video input - #60
Merged
Merged
Conversation
…'s video kind
copyMedia accepts { kind: 'video', frames, timestampsMs, timestampIntervalMs?,
notes? } beside the image and audio kinds and owns every frame and note off
the JS thread. liblloyal lays the frames out as llama.cpp's video helper does,
so Qwen-VL fuses adjacent frames in pairs. The addon never decodes a video
file; MTMD_VIDEO stays off.
The image suite's former "frames" assertion is relabelled for what it is:
several still images, which never fuse at this pin (594 == 3 x 198 cells).
The video assertion costs the clip at the helper's own tokenization (758),
lands the quote (775 for the user turn, the prompt tokens llama-server
reports for the same clip and question), answers "right" for the direction of
motion, refuses mixed frame sizes per entry, and rejects a video without
timestamps as a type error.
liblloyal at feature/native-video.
There was a problem hiding this comment.
🟡 Changes recommended
Video ownership coverage and native contract documentation should be completed, alongside the README grammar correction.
3 open findings
What changed in this PR
Adds caller-decoded video frames as a multimodal input backed by liblloyal video support.
Changes:
- Marshals video frames, timestamps, intervals, and notes into native inputs.
- Adds video integration and validation coverage.
- Documents video input usage.
| File | Description |
|---|---|
src/SessionContext.cpp |
Implements native video marshalling and ownership. |
test/integration.ts |
Tests video tokenization, inference, and validation. |
test/audio-integration.ts |
Updates invalid-kind tests for video support. |
README.md |
Documents the video input format. |
liblloyal |
Advances to the video-capable kernel revision. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The relabelled prompt made the CPU runners stop before a token. The prompt's words are not what assertion 8 proves; several still images and text in one prefill are, with the new no-fusion check beside them.
- liblloyal with the video layout review fixes (034ce4a). - The image suite holds frame ownership: every frame a byte-offset view of one backing store, zeroed while measurement and prefill run. - The prefill declaration documents the video input shape. - README grammar.
…a reseed is counted _branchFastForward(parent, child) marshals liblloyal's fast_forward and turns every refusal into a JS error. _branchSamplerChainReseed now reseeds through the branch (reseed_sampler), so a parent reseeded after a fork has changed since it and cannot be fast-forwarded over.
… the binding The reseed goes through liblloyal's reseed_sampler, which counts it as a change: a fork taken before it is refused with nothing moved. A greedy chain has no PRNG to reseed, so it does not refuse. The test fails against the old reseed body that bypassed the branch.
… CHANGELOG matches the tags)
StoreCommitWorker accepts every token, then decodes, and on a failure swaps the sampler, grammar and metrics back. The revision each accept bumped stayed, so a fork taken before a refused commit could no longer fast-forward although its parent was as it had been. The rollback now restores the revision of every branch the failure left intact; a branch the failure poisoned keeps the bump poisoning gave it. Duplicate handles are refused before anything is accepted. The integration suite gains the case, red before the change on SmolLM2: a batch refused with rc 1 leaves its parent fast-forwardable, and a duplicate commit is refused before any accept. A `branch` suite runs the branch-state cases on their own, so they also run on Qwen3.5-4B apart from the rest of the text suite.
The range ">=3.0.0 || >=4.0.0-0 <5.0.0" had no upper bound, so npm could pair this binding with any later SDK major and its native contract. It is now ">=5.0.0-0 <6.0.0": sdk 5, its prereleases, nothing else. The peer test checks what the range refuses as well as what it admits, red before the change.
… into - README: audio has been admitted since 3.3.0 when the loaded projector has an audio encoder, and refused before decoding otherwise; the multimodal prefill takes images, audio and video. - SessionContext.hpp: MtmdSource yields text and media segments; a video expands into image and text chunks. - liblloyal at 37bbc39 (docs: retainOnly, seq_id ownership, partial and poisoned, fast_forward's in-flight rule).
Contributor
Author
|
This push:
|
The peer range names SDK 5 only, but the dev dependency still asked for ^3.0.0, so the integration suites, which use the SDK's Branch and BranchStore through this package's re-exports, ran against a major the binding no longer supports. The dev dependency is ^5.0.0 and the lockfile resolves 5.0.0; its root now mirrors the manifest. The platform packages stay out of the lockfile, as the release workflow expects. Verified on SDK 5 with the local build: the text suite on SmolLM2, 321 passing; the branch suite on Qwen3.5-4B, 44 passing; the unit checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds video as a multimodal input kind:
copyMediacopies it into liblloyal'sMediaInput::Kind::Video.OwnedMedia::Videoowns the frames, times and notes for the length of the call, and the existing_storePrefillMultimodaland_cellsMultimodalcarry it. There's no new method.MTMD_VIDEOstays off, so decoded frames come from the caller.liblloyal: the submodule moves to
feature/native-video(dde39da), see lloyal-ai/liblloyal.Tests: run with
LLOYAL_LOCAL=1, against the local build.test/integration.ts), Qwen3.5-4B with its projector: new assertion 8b. A 16-frame video costs 758 bare and lands 775 in a turn, matching llama-server. It answers "right" for the square's direction. Assertion 8 is relabelled as multi-image.test/audio-integration.ts), Qwen3-ASR: 7/7.'video'is now a valid kind, and an unknown kind is still refused.The README's multimodal section now covers video.