Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion quest/m2/ts-model/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,4 @@ Public API: the translator's output for moq-net's model.

## Required

- [hashbrown as the runtime's map](/quest/m2/ts-model/hashbrown.md) - hashlink's `hashbrown` tables translate as `rt.HashMap`, with std's capacity
- [A crate's own hasher](/quest/m2/ts-model/own-hasher.md) - hashlink's default hasher, a crate's own `Hasher`, hashes keys as native's does
24 changes: 0 additions & 24 deletions quest/m2/ts-model/hashbrown.md

This file was deleted.

25 changes: 25 additions & 0 deletions quest/m2/ts-model/own-hasher.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# [M] A crate's own hasher

## Goal

[rust2-ts](/ts/README.md) translates hashing through a crate's own `Hasher` and
`BuildHasher`, as `hashlink`'s default `DefaultHashBuilder` wraps hashbrown's: std's
provided `BuildHasher::hash_one`, and std's `Hash` impls writing into that hasher with the
calls std makes, proven by differential tests against native Rust.

## Plan

Found by [#398](https://github.com/moq-dev/rust2/pull/398), which maps hashbrown's
`DefaultHashBuilder` and `DefaultHasher` to std's: a `LinkedHashMap::new()` then refuses
with "unsupported call to `<hashlink::DefaultHashBuilder as
std::hash::BuildHasher>::hash_one::<&u16>`". moq's model reaches it through hashlink's
default hasher, the census's last refusals in `hashlink/src/lib.rs`.

- `hash_one` is `build_hasher`, then `Hash::hash`, then `finish`, each the crate's.
- std's `Hash` of an integer, `str`, slice, tuple, and the rest calls the hasher's
`write_*` methods in std's order, length prefixes and `str`'s `0xff` included, so a
hasher that reads its input, not only hashlink's forwarding one, sees what native's does.
- Today only std's `DefaultHasher` takes std's `Hash` impls, written by the runtime's
functions.

Public API: the translator's output.
1 change: 1 addition & 0 deletions quest/m3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,5 @@ generic std gap is planned here even when moq never calls it.
- [Type errors at the Rust that made them](/quest/m3/ts-span-precision.md) - a `tsc` error on output names the tail expression or call argument that made it
- [An empty iterator of mutable references](/quest/m3/ts-empty-places.md) - `iter::empty::<&mut T>()` given to `zip` or `chain` translates
- [An Option of numbers in the result slot](/quest/m3/ts-option-slot.md) - `varint_opt`'s `Option` comes back through the slot with no object per call
- [Dropping a crate's generic type inside another](/quest/m3/ts-nested-generic-drop.md) - an `LruCache` holding a `LinkedHashMap` drops as rustc's glue does
- [Fresh writes through a zip, a copy, and a repeat](/quest/m3/ts-fresh-shapes.md) - zipping a `Vec<[T; N]>`, reading a `T` out of `Zip::next`, and `copied` or `vec![x; n]` over a `Copy` type translate
20 changes: 20 additions & 0 deletions quest/m3/ts-nested-generic-drop.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# [S] Dropping a crate's generic type inside another

## Goal

[rust2-ts](/ts/README.md) drops a crate's generic type held in a field of another of the
crate's generic types, as `hashlink`'s `LruCache<K, V, S>` holds a `LinkedHashMap<K, V,
S>`, in the order rustc's glue does, proven by differential tests against native Rust.

## Plan

Found by [hashbrown as the runtime's map](https://github.com/moq-dev/rust2/pull/398): a
fixture dropping an `LruCache` refuses with "dropping `hashlink::LinkedHashMap<K, V, S>`,
which is not supported yet" at the end of the function holding it, from
`drop_glue_call`, which takes the generic field's glue only in a census. A
`LinkedHashMap` dropped by itself translates, so the outer glue's shim is what lowers the
field over its type parameters rather than the instance. `LinkedHashSet` is expected to
refuse the same way. No consumer calls them yet; they are planned here as part of
covering stock crates.

Public API: the translator's output.
28 changes: 28 additions & 0 deletions ts/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 34 additions & 4 deletions ts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,8 @@ crate translation reaches is refused, and so are two versions of one crate it re
| `bytes::Bytes` | `rt.ByteSlice`, as `&[u8]` is: its clones and slices share bytes that never change |
| `bytes::BytesMut` | `rt.BytesMut`: a window of storage, which what it splits off and freezes share without overlapping |
| `rand::rngs::ThreadRng` | `rt.ThreadRng`, which holds no state |
| `hashbrown::HashMap`, `HashSet` | `rt.HashMap<K, V>` and `rt.HashSet<K>`, as std's, which are hashbrown's |
| `hashbrown::HashTable<T>` | `rt.HashTable<T>`: an `rt.HashMap` of nothing to each item, under the hash its caller gives; its entries are `rt.TableEntry<T>`, the place of the item an occupied one holds |

`Vec<u8>` storage and `[x; N]` byte arrays up to 1 KiB share 8 KiB `ArrayBuffer`s, as Node's
`Buffer` pool does, so the `buffer` behind a view holds other bytes too: read it through
Expand Down Expand Up @@ -527,6 +529,24 @@ of them:
The rest of `bytes`, such as the other provided methods of the crate's own `Buf` and the
impls forwarding through a `&mut` or a `Box`, translates from its MIR, refused where
that is.
- `hashbrown`'s `HashMap` and `HashSet` are std's, which wrap them, and map as std's do,
hashbrown's own methods included: `allocation_size` is a slot and a control byte for each
entry the capacity holds, and its `Entry` is std's. Its `HashTable` keeps each item
under the hash the caller gives, a `u64` past 2^53 included, chained by the caller's
equality, which the runtime calls on each item under that hash, rather than on those
hashbrown's tag bytes pick. It never calls a method's `hasher`, as it never moves an
item. `find`, `find_mut`, `find_entry`, `entry`, `insert_unique`, `retain`, `iter`,
`iter_mut`, `drain`, `clear`, `len`, `clone`, `Debug`, and what reads and sets capacity
map, and so do its entries' `insert`, `or_insert`, `or_insert_with`, `and_modify`,
`get`, `get_mut`, `into_mut`, `remove`, and `into_table`. What reads hashbrown's
buckets, such as `find_bucket_index` and `iter_hash`, is refused, as are
`get_many_mut`, `extract_if`, and `try_reserve`. hashbrown's `DefaultHashBuilder` and
`DefaultHasher` are std's `RandomState` and `DefaultHasher`: foldhash's state is seeded
at random as std's is, so which hash a key gets is never observable. A crate's own
`Hasher`, such as `hashlink`'s default over hashbrown's, is refused
([a crate's own hasher](/quest/m2/ts-model/own-hasher.md)). Nothing of hashbrown
translates, so `hashlink`, which keeps its entries in a `HashTable`, translates over the
runtime's.
- `rand::rng()` is an `rt.ThreadRng`, whose `random` of an integer or `bool` and
`random_range` of an integer range map to `rt/rand.ts`. Each value comes from
`crypto.getRandomValues`, uniform as `rand`'s are, though no seed reproduces them, as
Expand Down Expand Up @@ -742,7 +762,13 @@ they need.
it takes 0.5-0.8x the time two sorted arrays did, from 16 keys to 4096, in Node, Bun,
and Chrome; 64 keys a node measured best of 16 to 128. A hash map's JS primitive keys
are a JS `Map`'s, the engine's own hash table; its other keys go into a second `Map`
by their translated hash, chained by `Eq`. `VecDeque` is a ring buffer over an array
by their translated hash, chained by `Eq`. A hash map, set, or table holds only what std
guarantees of its capacity, at least its length and what `with_capacity` and `reserve`
ask for, doubling as inserts grow it, and keeps it as it holds fewer or clears, as a
table keeps its allocation; hashbrown's exact bucket counts are not followed (decided
with the maintainer on 2026-11-01). It holds one, as `cap`, only where an operation
sets it or a removal reads it, so an insert never reads or writes one, and reading it
doubles it from 3 while short of the length. `VecDeque` is a ring buffer over an array
whose length is a power of two. A `Vec`, `VecDeque`, and `BinaryHeap` hold what std
guarantees of their capacity: at least their length and what `with_capacity`,
`reserve`, and `reserve_exact` ask for, and std's exactly where its doubling fixes it,
Expand Down Expand Up @@ -810,7 +836,7 @@ an iterator adaptor, and values that drop in some iterators ([drop order](/quest
function without a mapping of its own that makes or reads an `Option` whose `None` is
`rt.None`, as `Iterator::scan` of `Option`s does, an `Option` of an `Option` of `()` or another
`Option`, an instance of a generic type whose field is an `Option` of a type parameter
that can be `undefined`, a `String`'s or hash collection's `capacity`, looking up a key by
that can be `undefined`, a `String`'s `capacity`, looking up a key by
another type it borrows as through the crate's own `Borrow`, a `&mut` to a value JS
copies that std makes, other than a map's, or hands a closure, as `iter_mut` and
`and_modify` do, one into a `Result`'s payload passed on or kept, `Hasher::write` of bytes, the adaptors
Expand Down Expand Up @@ -888,13 +914,17 @@ built natively for wasm32 and run beside its translation on the same inputs by
impls, beside values that drop inside std's types. Its caller is hand-written TS (`tests/task.ts`): a FIFO executor that runs a
script of spawns, sends, polls, and drops, and logs every poll, wake, and drop, as the
fixture's native executor does. The logs must match byte for byte.
- `stock` calls `bytes`, `rand`, and `tracing` with no config, against native Rust, whose
- `stock` calls `bytes`, `hashbrown`, `rand`, and `tracing` with no config, and `hashlink`
over `hashbrown`, against native Rust, whose
`rand::rng()` reads the fixture's own getrandom backend on wasm32. It checks `Bytes`'s
slices and panics, how each is made and printed, that no event evaluates a field, and
that each random value is in its range, every value of a small range drawn. It splits,
freezes, grows, and writes a `BytesMut` by ops its input picks, ending with every
piece's bytes, and reads and writes through `Buf` and `BufMut` generically, a buffer of
its own included. `caller.ts` names the inputs that reserve at the edges of
its own included. It changes hashbrown's maps and sets, keyed by JS primitives and by
tuples, and `HashTable`s of tuples and of numbers, whose places it writes, under hashes
that collide and that pass 2^53, checks their capacity by std's guarantees, and writes a
`hashlink::LinkedHashMap` in its own order. `caller.ts` names the inputs that reserve at the edges of
`BytesMut`'s 512 MiB limit.
- `bind` binds `buf::Blob`, a renamed dependency's type with `unsafe` inside, and a function and a type of its own that read through raw
pointers, the type dropping an `Arc` where Rust drops it, to the TS in `bind/`. It keys
Expand Down
22 changes: 22 additions & 0 deletions ts/bench/hashed.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// The `hash/` workloads' hand-written route: `bench.rs`'s op 14 over a `Map`, as TypeScript
// keeps a map of numbers.

/** Run op 14, as `bench::run` does, for the count `data` starts with: with `selector` 1, inserts alone. */
export function hashed(selector: number, data: Uint8Array): Uint8Array {
const padded = new Uint8Array(8);
padded.set(data.subarray(0, 8));
const count = Math.min(Number(new DataView(padded.buffer).getBigUint64(0)), 1 << 16);
const key = (i: number) => (i * 40503) % (1 << 16);
const map = new Map<number, number>();
for (let i = 0; i < count; i++) map.set(key(i), i);
let sum = 0;
if (selector !== 1) {
for (let i = 0; i < count; i++) sum += map.get(key(count - 1 - i)) ?? 0;
for (let i = 0; i < count; i++) map.delete(key(i));
}
const out = new Uint8Array(16);
const view = new DataView(out.buffer);
view.setBigUint64(0, BigInt(sum));
view.setBigUint64(8, BigInt(map.size));
return out;
}
2 changes: 1 addition & 1 deletion ts/bench/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,7 @@ function render(): string {
"Rust at moq-wasm's size-optimized profile, run with its input already in memory;",
"copying the input in and the output out are timed apart. `hand-written` is moq's",
"`js/net`, called as its own benchmarks call it, for `iter/` the same chain with ES",
"iterator helpers (`bench/iter.ts`), for `btree/` a sorted array (`bench/sorted.ts`), and",
"iterator helpers (`bench/iter.ts`), for `btree/` a sorted array (`bench/sorted.ts`), for `hash/` a `Map` (`bench/hashed.ts`), and",
"for `time/` conversions `bigint`, as moq's `js/hang` converts timescales (`bench/rescale.ts`),",
"and for `time/checked-add` and `time/ord` moq's own `Timestamp` (`bench/timestamp.ts`).",
"Each JS route and `wasm` runs alone in a worker of its own, a fresh one each round. `A/A` is the translation again: its",
Expand Down
3 changes: 3 additions & 0 deletions ts/bench/moq.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import * as Varint from "moq/js/net/src/varint.ts";

import { iterate } from "./iter.ts";
import { order } from "./order.ts";
import { hashed } from "./hashed.ts";
import { push } from "./push.ts";
import { rescale } from "./rescale.ts";
import { ordered } from "./sorted.ts";
Expand Down Expand Up @@ -65,6 +66,8 @@ export function run(input: Uint8Array): Uint8Array | Promise<Uint8Array> {
return order(selector, data);
case 13:
return push(data);
case 14:
return hashed(selector, data);
default:
throw new Error(`unknown workload ${op}`);
}
Expand Down
6 changes: 6 additions & 0 deletions ts/bench/workloads.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,12 @@ export function workloads(encode: (input: Uint8Array) => Uint8Array): Workload[]
for (const count of [16, 256, 4096]) {
out.push({ name: `btree/${count}`, op: "map", items: count, input: Uint8Array.of(8, 0, ...u64(BigInt(count))) });
}
// A `HashMap` of each size, through `bench.rs`'s op 14: each key inserted, looked up, and
// removed, and each key inserted alone.
for (const count of [16, 256, 4096]) {
out.push({ name: `hash/${count}`, op: "map", items: count, input: Uint8Array.of(14, 0, ...u64(BigInt(count))) });
out.push({ name: `hash/insert/${count}`, op: "map", items: count, input: Uint8Array.of(14, 1, ...u64(BigInt(count))) });
}

// Microseconds since moq's 2020 epoch, a frame apart at 30 fps, to 90 kHz and back:
// each product passes 2^53 and each quotient is back under it. `bench.rs`'s op 9 converts
Expand Down
47 changes: 47 additions & 0 deletions ts/fixtures/codec/expected/bench.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ export function run(input: rt.ByteSlice): rt.ByteVec {
order(selector, new rt.ByteSlice(input.data, input.start + 2, input.length - 2), out);
} else if (op === 13) {
push(selector, new rt.ByteSlice(input.data, input.start + 2, input.length - 2), out);
} else if (op === 14) {
hashed(selector, new rt.ByteSlice(input.data, input.start + 2, input.length - 2), out);
} else {
decoded(decodeGroup(new rt.ByteSlice(input.data, input.start + 2, input.length - 2), liteVersion(selector), out), out);
}
Expand Down Expand Up @@ -318,6 +320,51 @@ export function ordered(data: rt.ByteSlice, out: rt.ByteVec): void {
harness_putU64(out, sum);
}

/**
* `count` keys, scattered as `ordered`'s, inserted into a `HashMap`, and unless `selector`
* is 1, each looked up and removed: the sum of what the lookups found, and the length left.
*/
export function hashed(selector: rt.u8, data: rt.ByteSlice, out: rt.ByteVec): void {
let _16!: rt.u32;
let sum!: rt.u64;
let _30!: rt.u32;
let _51!: rt.u32;
const _11: (a0: rt.u32) => rt.u32 = (i4: rt.u32): rt.u32 => {
return rt.u32Rem(rt.u32Mul(i4, 40503), 1 << 16);
};
const count2: rt.u32 = rt.u64Low(rt.u64Min(takeU64(data)[0], 65536));
const map: rt.HashMap<rt.u32, rt.u32> = rt.hashNew() as rt.HashMap<rt.u32, rt.u32>;
let iterStart: rt.u32 = 0;
const iterEnd: rt.u32 = count2;
while ((_16 = iterStart++) < iterEnd) {
const i: rt.u32 = _16;
const _22: rt.u32 = _11(i);
rt.hashInsert(map, _22, i);
}
b2: {
sum = 0;
if (selector !== 1) {
let iterStart2: rt.u32 = 0;
const iterEnd2: rt.u32 = count2;
while ((_30 = iterStart2++) < iterEnd2) {
const _40: rt.u32 = _11(rt.u32Sub(rt.u32Sub(count2, 1), _30));
sum = rt.u64Add(sum, option_unwrapOr$11c0we2(rt.hashGet(map, _40), 0) as rt.u64);
}
let iterStart3: rt.u32 = 0;
const iterEnd3: rt.u32 = count2;
while (true) {
if (!((_51 = iterStart3++) < iterEnd3)) {
break b2;
}
const _58: rt.u32 = _11(_51);
rt.hashRemove(map, _58);
}
}
}
harness_putU64(out, sum);
putUsize(out, rt.hashLen(map));
}

/**
* `count` integers sorted, sorted by a key, searched for the greatest and least, pushed
* onto a `BinaryHeap` and popped off, or collected into a `BTreeMap` or `BTreeSet`, as
Expand Down
Loading