diff --git a/NANOGPT_PERFORMANCE.md b/NANOGPT_PERFORMANCE.md new file mode 100644 index 00000000..4a2c25be --- /dev/null +++ b/NANOGPT_PERFORMANCE.md @@ -0,0 +1,294 @@ +# NanoGPT SQL execution and training performance + +Measured November 1, 2026, on Apple M5 Ultra (30 CPU cores, 64 GPU cores, +256 GiB unified memory), macOS 27.0.1. GPU game contention was off for the +final runs. This report covers the implementation and the measurements; +ratios from different workloads must not be multiplied together. + +[nanoGPT](https://github.com/karpathy/nanoGPT) is a small GPT training +implementation. This work keeps the model and optimizer expressed in SQL: +[DataFusion](https://datafusion.apache.org/) plans the SQL, +[ddx](https://github.com/xqlsystems/ddx) differentiates it, +[EinFold](https://github.com/xqlsystems/einfold) lowers numerical operations +to compact array programs, and [MLX](https://github.com/ml-explore/mlx) +executes float32 arithmetic on Apple's GPU. CPU execution uses NumPy +float64. There is no replacement model or handwritten backward pass. + +## Setup, plan reuse and CPU overhead + +| Change | Reason and observed improvement | +|---|---| +| Count Dataset chunk partitions arithmetically | Avoid constructing a Cartesian grid merely to count it. Registration of a 4,096×16 Dataset with one-cell chunks fell from 31.372 to 0.405 ms, **77.44×**. This applies to that fine-chunk case, not all registration. | +| Register ordinary in-memory Arrow batches directly | Remove the temporary catalog table used during registration. Included in the preceding combined measurement; no isolated ratio. | +| Retain prepared input buffers and update their contents | Repeated update-and-sum of 1,048,576 values fell from 2.769 to 1.027 ms, **2.70×**, versus re-registering every call. | +| Cache forward logical plans and ddx derivative programs | Avoid parsing and planning identical SQL on each step. A repeated-CTE DataFusion product fell from 1.709 to 1.307 ms, **1.31×**. Prepared grad/VJP/JVP calls reuse derivative programs and invalidate them when the catalog/settings change. | +| Preserve Context metadata when installing extensions | Keep Dataset reconstruction and prepared invalidation correct when adding execution extensions. Correctness improvement; no isolated speed claim. | +| Expose `Context(threads=...)` and benchmark `--threads` | Target-partition sweep from 1 to 30 reduced DataFusion attention from 6,797 to 795 ms, **8.55×**, and EinFold attention from 25.1 to 9.3 ms, **2.70×**. The default already used the 30 physical cores. | + +An independent 30→60 partition comparison gave DataFusion 790.149→747.087 ms +(1.06×), but EinFold 8.737→9.076 ms (3.9% slower). Higher partition counts +are explicit workload choices, not a universal improvement or a promise of +that many OS threads. BLAS has separate thread controls. + +Preparing the tiny original training example improved warm DataFusion time +from 1,227.4 to 1,103.2 ms (1.11×), but its five-step total increased from +6.504 to 7.036 s because of setup/first-use costs. Warm performance alone +does not imply faster startup. See the [CPU/kernel report](benchmarks/apple/README.md) +and [five-step report](benchmarks/apple/nanogpt-5-steps.md). + +## Shared CTE execution and kernel dispatch + +The explicit shared executor evaluates each reachable common table +expression (CTE) once per invocation, retains reusable stage programs, and +gives ddx the original view definitions. Fresh inputs and physical join +state prevent reuse of old numerical values. Native SQL parsing and +verified ordered Arrow providers remove Python parsing and unnecessary +sorts. Function overrides invalidate permission to use equivalent array +primitives. + +The session-local EinFold kernel hook dispatches validated contractions +and fused causal attention to NumPy or MLX. SQL checks of keys, duplicates, +NULLs and row domains precede dispatch; unsupported cases keep their SQL +execution. GPU precision is explicitly float32. Automatic thresholds keep +tiny work on CPU. + +| Five-step original NanoGPT workload, warm loss + update | Before sharing | After sharing | Improvement | +|---|---:|---:|---:| +| Tiny 4,416 parameters, CPU kernel host | 314 ms | 206 ms | 1.53× | +| Tiny 4,416 parameters, GPU kernel host | 371 ms | 224 ms | 1.66× | +| 213,504 parameters, CPU | 551 ms | 431 ms | 1.28× | +| 213,504 parameters, GPU | 601 ms | 482 ms | 1.25× | + +The tiny shared GPU run used 160 matrix and 5 attention kernels rather +than 1,015 and 200. At this stage, shared CPU took 206 ms versus DuckDB +306 ms and plain DataFusion 1,260 ms. These are tiny-model controls. +Shared GPU remained about as slow as shared CPU for the 123.6M model: +163.65 versus 164.03 s per update pair. GPU dispatch alone left the +backward SQL/Arrow bottlenecks intact. + +Isolated kernel results include 256×512×256 CPU matmul 8.698→4.422 ms +(1.97×), and 1024³ original CPU→GPU matmul 64.131→35.174 ms (1.82×). +The new NumPy CPU implementation took 33.745 ms on the latter, slightly +faster than GPU. Attention at 16,384 tokens/context 64 improved +95.044→71.292 ms (1.33×). Tiny forced-GPU training regressed; it is not +presented as an acceleration. See [shared execution measurements](benchmarks/apple/nanogpt-shared.md). + +## Compact execution of forward and backward SQL + +Profiling identified coordinate expansion, sorting, Python/Arrow +registration, repeated backward stages, and device readbacks. In one +earlier embedding-gradient stage, materializing about 926 MB of scalar +rows took 18.58 s. A coordinate-expansion/sort profile consumed about +3.3 s. These profiles diagnose bottlenecks, not isolated before/after +speedups. + +`DenseExecutor` retains each axis domain once and stores numeric values +separately. Joins lower to gathers or contractions; window reductions +broadcast; UNION branches retain bag semantics and accumulate gradients. +Row presence and SQL NULL validity are distinct masks. Ordering compact +axes before Arrow expansion avoids expanded-coordinate sorting when the +domains prove that order. A bounded computed-value LRU limits retained +intermediates. Sources are imported afresh for ordinary query calls; +public results remain independent Arrow snapshots. + +MLX provides matmul, reductions, gather/scatter and compiled elementwise +expressions with explicit numeric arguments. Integer/domain guards refuse +unsupported behavior. Unsupported SQL falls back before caller-visible +mutation; strict benchmarks disable fallback to expose omissions. + +| Parameters / context / batch | Shared CPU → compact CPU | CPU improvement | Shared GPU → compact GPU | GPU improvement | +|---|---:|---:|---:|---:| +| 4,416 / 16 / 4 | 193 → 52 ms | 3.7× | 219 → 78 ms | 2.8× | +| 213,504 / 64 / 1 | 431 → 61 ms | 7.1× | 482 → 83 ms | 5.8× | +| 10,745,088 / 256 / 1 | 29.257 → 1.406 s | 20.8× | 29.937 → 0.589 s | 50.8× | +| 123,563,520 / 16 / 1 | 164.034 → 4.037 s | 40.6× | 163.646 → 2.434 s | 67.2× | + +All rows are **training from scratch**, five sequential loss/backward/SGD +updates. Warm means exclude the first update pair. Every loss and weight +update was checked against a float64 JAX reference outside the timer. +These original examples use tanh GELU, SGD, Shakespeare character IDs +padded to a vocabulary of 50,257 for GPT-2 sizes, and no dropout. They are +not the Shakespeare AdamW recipe below or a GPT-2 language-model quality +benchmark. The shared tiny baseline was rerun, so its value differs from +the earlier sharing experiment. + +The 123.6M/context-16 compact GPU first pair took 3.750 s; warm pairs +took 2.434 s, with peak sampled RSS 17.91 GiB versus earlier shared GPU +40.0 GiB. CPU first/warm pairs took 5.193/4.037 s, with RSS 22.90 GiB +versus 38.7 GiB. At context 1,024, the 124,337,664-parameter GPU model +completed five training updates: first pair 7.496 s, warm 5.575 s, peak +RSS 28.72 GiB. CPU full-context trials did not complete and give no speedup. + +An 8→32 GiB computed cache experiment reduced full-context GPU warm time +14.242→5.835 s (2.44×); a subsequent final run measured 5.575 s. A cache +budget is not process RSS. The tiny compact CPU/GPU controls took +52/78 ms versus DuckDB 274 ms; there is no matching full-size DuckDB +training result. See [compact execution evidence](benchmarks/apple/dense-execution.md) +and the [framework comparison](benchmarks/apple/nanogpt-frameworks.md). + +## Complete GPU-resident Shakespeare training + +`benchmarks/ml/03_shakespeare.py` matches the character-training recipe +from nanoGPT: batch 12, context 256, 6 layers/heads, embedding width 384, +65-character vocabulary, **10,745,088 parameters**, exact GELU, dropout +0.2, tied embeddings, no biases, global clipping at 1, and SQL AdamW +(betas 0.9/0.99, epsilon 1e-8, weight decay 0.1 excluding layer norms). +Learning rate is 1e-3→1e-4 with warmup 100 and decay through iteration +2,000; evaluation every 200 uses 20 batches each for train/validation. +The local native warmup formula is `lr * (iteration + 1)/(warmup + 1)`. + +The recipe uses grouped MAX/join softmax normalization. Its common shift +is marked `ddx_stop_gradient`: shifting all logits by the same scalar +cancels analytically in softmax, avoiding an unnecessary max/backward +window subgraph. Optimizer moments occupy separate SQL tables with complete +coordinate domains. These choices preserve the checked gradients; their +individual timing contributions were not isolated. Checkpoints store +weights, moments, next iteration, RNG state and recipe metadata in NPZ, +and resume tests check the subsequent updates. + +`TrainingSession` owns private rectangular, finite, nonnullable state +arrays, including weights and AdamW moments. A prepared step evaluates +ddx gradients, clipping and all SQL updates in one program. The new ddx +`Prepared.values` metadata identifies the primal loss already computed +by differentiation, eliminating the extra forward query. State remains +on GPU between steps and is published atomically after validation. +Failed updates retain the old state and dropout counter. + +Whole-step `mx.compile` accepts weights, moments, batch values and optimizer +scalars as explicit arguments. Coordinate layouts, schemas, views and +session settings guard reuse. Bounded token domains permit GPU index joins +and embedding-gradient scatter; actual bound values are checked before +upload. Structural uniqueness checks are retained or proven from exact +coordinate/count bounds; unsupported dynamic host decisions decline +compilation. GPU dropout uses explicit seed/step keys and resumes from +checkpointed RNG state. Numerical host conversion during tracing is refused. + +| Same 10.7M-character model, batch 12 | First step | Warm mean | Warm sample | +|---|---:|---:|---| +| Arrow compact CPU, float64, 32 GiB cache | 10.362 s | 9.747 s | Updates 2–5; all checked | +| Arrow compact MLX GPU, float32 | 2.523 s | 2.024 s | Updates 2–5; all checked | +| Resident GPU, uncompiled, CPU masks | 1.840 s | 1.356 s | Updates 2–5; all checked | +| Compiled resident GPU, CPU masks | 5.618 s | 120.7 ms | Updates 2–5; all checked | +| Compiled resident GPU, GPU dropout | 1.380 s | 145.9 ms | Updates 2–5; all checked | +| Compiled resident GPU, GPU dropout | 1.468 s | **101.3 ms** | Updates 2–100; no per-step oracle | +| Native nanoGPT / PyTorch MPS, float32, compile=False | See raw record | **27.0 ms** | Updates 2–100 | + +The delivered 100-step path is **19.98× faster** than the earlier Arrow +GPU path and **3.76× slower** than native PyTorch MPS on this recipe. +The fully checked five-step run improved **13.87×** over the Arrow path. +The 100-step comparison uses a longer warm window than the five-step +baseline and includes GPU dropout inside the step. These are observations +from individual serial runs, not confidence intervals or isolated gains +attributable to each component. CPU-mask and GPU-dropout trajectories differ. + +First-step times include preparation/compilation within that API call, +but exclude process startup and the preceding 40 evaluation calls. Shader +caches may be warm from earlier trials; the 1.380–5.618 s variation must +not be described as a cold-start improvement. + +Timed SQL steps include the complete forward/backward/clipping/AdamW +update, state validation/publication and reported loss scalar. Dataset +loading, token sampling/upload, evaluation, correctness exports and +checkpoint I/O are outside the timer. Native synchronized timing includes +next-batch prefetch. Native uses its own PyTorch RNG trajectory; recipe +equivalence is separate from the identical-input oracle checks. + +All 100 timed resident steps exported **zero numerical arrays** and read +back **zero array bytes**; each read one validation flag and one loss +scalar. There was one whole-step compilation. Batch input uploads occur +outside this timer. Scalar synchronization, CPU planning/setup and final +checkpoint exports remain; this is not a claim of zero host interaction. +Byte counters describe logical array transfers, not measured physical +bus traffic on Apple's unified-memory hardware. + +Process RSS was 2,639,101,952 bytes after step 1 and 2,644,819,968 after +step 100, a 5.7 MB increase. Last-step MLX active/cache/peak counters were +532,354,042 / 5,081,660,545 / 4,428,988,178 bytes. These counters overlap +with unified-memory/RSS accounting and must not be added together. The +100-step observation supports stable retention over that run, not an +unbounded-memory guarantee. Earlier Arrow runs did not record comparable +RSS, so no exact memory-reduction ratio is claimed for this recipe. + +## Correctness, reproduction and limitations + +The final xarray-sql non-integration suite passed **714 tests**, with +277 optional skips and 102 integration tests deselected. The suite covers +SQL values/coordinates, NULL/duplicate behavior, live inputs, cache +invalidation, numerical precision, state ownership, failed-step atomicity, +changing compiled token/LR inputs, snapshots and checkpoint/resume. + +This was repeated against wheels rebuilt from the persistent ddx/EinFold +checkouts, then again after integrating upstream's newer EinFold runtime: +[final publication test log](benchmarks/apple/resident-final-merged-correctness.log). +ddx passed its Rust workspace suite (535 passed, 10 ignored). EinFold's +Rust suite had 72 passes, 4 ignored and one NaN-sign behavior assertion +failure, reproduced unchanged on its clean pinned base on this macOS host. +Both passed Clippy, formatting and warning-free Rust docs; EinFold also +passed REUSE license checks. The [final combined Python binding suite](benchmarks/apple/publication-upstream-python.log) +passed 128 tests with 6 optional skips. + +The five-step resident recipe checked every loss, raw gradient, weight and +both optimizer moments against original nanoGPT `model.py` and PyTorch +AdamW, using identical arrays, batches and explicit dropout masks outside +the timer (float32 tolerance 3e-5). The float64 Arrow recipe used 1e-8. +The 100-step performance run intentionally omitted per-step oracle exports; +all losses were finite. Neither run establishes convergence or general SQL +support for every compiled program. + +Build the pinned optional dependencies using the +[EinFold instructions](accelerators/einfold_mlx/README.md) and +[ddx instructions](accelerators/ddx_math/README.md), then from this checkout: + +```sh +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python benchmarks/ml/03_shakespeare.py \ + --compile-training --gpu-dropout --steps 5 \ + --verify-nanogpt /path/to/nanoGPT --out-dir out-shakespeare-verified + +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python benchmarks/ml/03_shakespeare.py \ + --compile-training --gpu-dropout --steps 100 --profile-memory \ + --out-dir out-shakespeare-resident + +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/pytest -q -m 'not integration' +``` + +Use `--data-dir /path/to/nanoGPT/data/shakespeare_char` if needed, after +running nanoGPT's data preparation. Omit `--steps` to follow the full native +iteration-0-through-2000 recipe (2,001 updates). Each ordinary launch +initializes once from scratch; `--resume` loads a checkpoint. + +The [ML benchmark README](benchmarks/ml/README.md) documents model-size +flags, all execution modes, evaluation and resume. Native comparison is +reproducible with [compare_shakespeare_mps.py](benchmarks/apple/compare_shakespeare_mps.py). +Recorded results are in [resident-summary.json](benchmarks/apple/resident-summary.json), +the [five-step checked metrics](benchmarks/apple/shakespeare-resident-verified-gpu/metrics.jsonl), +the [100-step metrics](benchmarks/apple/shakespeare-resident-steady-gpu/metrics.jsonl), +and the [native 100-step record](benchmarks/apple/shakespeare-resident-torch-mps-100.json). +Run `python benchmarks/apple/summarize_resident.py` to recompute ratios. + +The pinned local integration builds used ddxdb `0.4.6+xqlmath.3`, EinFold +`0.2.0+xqlmlx.6`, DataFusion Python 55.0.0, MLX 0.32.4, NumPy 2.5.4, +PyArrow 26.0.0, Python 3.12.14 and PyTorch 2.15.0. Upstream contributions +retain normal package versions; local suffixes identify benchmark builds. +The ddx and EinFold repositories contain their own `OPTIMIZATION.md` reports. + +## Review and dependency order + +| Repository / change | PR | Depends on | +|---|---|---| +| ddx: exact error-function derivatives | [256](https://github.com/xqlsystems/ddx/pull/256) | Independent | +| ddx: prepared primal-output metadata | [257](https://github.com/xqlsystems/ddx/pull/257) | ddx 256 | +| EinFold: session-local numerical backends | [156](https://github.com/xqlsystems/einfold/pull/156) | Independent | +| EinFold: compact logical-plan arrays | [157](https://github.com/xqlsystems/einfold/pull/157) | EinFold 156 | +| xarray-sql: setup and plan reuse | [301](https://github.com/xqlsystems/xarray-sql/pull/301) | Independent | +| xarray-sql: shared stages and numerical kernels | [302](https://github.com/xqlsystems/xarray-sql/pull/302) | xarray-sql 301, EinFold 156 | +| xarray-sql: compact array integration | [303](https://github.com/xqlsystems/xarray-sql/pull/303) | xarray-sql 302, EinFold 157 | +| xarray-sql: resident SQL training | [305](https://github.com/xqlsystems/xarray-sql/pull/305) | xarray-sql 303, ddx 257 | +| xarray-sql: reports, harnesses and evidence | [304](https://github.com/xqlsystems/xarray-sql/pull/304) | Independent documentation branch; describes the complete stack | + +Branches are published through the `cromwellian` forks because that account +cannot push upstream branches. GitHub requires those PRs to target an +upstream branch, so dependent PR default diffs include their predecessors +until those merge. Each description links the incremental comparison for +its own change. No PR is merged by this work. The reports retain benchmark +provenance separately from later upstream runtime integration and packaging +changes; see [publication sources](benchmarks/apple/publication-sources.json). diff --git a/README.md b/README.md index 42fe05a7..58708530 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,8 @@ # xarray-sql +See [NanoGPT performance and optimization measurements](NANOGPT_PERFORMANCE.md) +for SQL training features, startup costs, per-step timings and memory. + _Query [Xarray](https://xarray.dev/) with SQL_ ![PyPI Version](https://img.shields.io/pypi/v/xarray-sql?color=green) diff --git a/benchmarks/apple/.gitignore b/benchmarks/apple/.gitignore new file mode 100644 index 00000000..84ba7e94 --- /dev/null +++ b/benchmarks/apple/.gitignore @@ -0,0 +1,9 @@ +# Benchmark scratch output stays local. Reports explicitly retain their +# referenced text evidence; new runs must not silently stage checkpoints. +*.log +*.csv +*.json +*.jsonl +*.txt +*.pstats +*.npz diff --git a/benchmarks/apple/README.md b/benchmarks/apple/README.md new file mode 100644 index 00000000..e755bd45 --- /dev/null +++ b/benchmarks/apple/README.md @@ -0,0 +1,272 @@ +# Apple M5 Ultra: measured SQL tensor acceleration + +Implemented directly in the local xarray-sql checkout. The optional +[EinFold/MLX bridge](../../accelerators/einfold_mlx/README.md) connects +[EinFold](https://github.com/xqlsystems/einfold), a tensor optimizer for SQL, +to [MLX](https://github.com/ml-explore/mlx), Apple's array framework. +[ddx](https://github.com/xqlsystems/ddx) continues to differentiate SQL; +MLX evaluates arithmetic inside the physical plan, including arithmetic +in ddx's forward and backward programs. + +## Measurement conditions + +Mac Studio, Apple M5 Ultra: 30 CPU cores, 64 GPU cores, 256 GiB memory; +macOS 27.0.1, Python 3.12.14. Python packages: DataFusion 55.0.0, +EinFold 0.2.0+xqlmlx.1, ddxdb 0.4.6, MLX 0.32.4, NumPy 2.5.4, +PyArrow 26.0.0. The EinFold native library uses DataFusion 55.2. +`MLX_ENABLE_TF32=0` throughout final GPU runs. + +Each number is the median of **seven measured executions after one warmup**. +Registration and NumPy reference construction are excluded. SQL planning, +execution, native/Python copies, sorting and Arrow materialization are +included. GPU results synchronize before their Arrow buffers are returned. +The same SQL and seeded inputs run on each host within a run. No compilation +or separate benchmark process ran concurrently with the final measurements. +Raw files contain every sample, maximum absolute error, environment and +completed-kernel counts. Variation between repeated runs was visible at the +millisecond level; small differences should not be treated as robust gains. + +The EinFold baseline uses the **same pinned native library with no backend +callback installed**. Its existing tensor detection is responsible for most +of the speedup over plain DataFusion. Incremental kernel speedups are reported +separately below. No results from another chip or published benchmark are +presented as local measurements. + +## Default dispatch: end-to-end milliseconds + +`MLX(precision="float32")` permits GPU arithmetic but uses float64 CPU +matrix multiplication or original CPU attention below the measured GPU +crossover. These particular workloads all select CPU arithmetic. + +| Workload | Plain DataFusion | EinFold CPU | Option B default dispatch | vs DataFusion | +|---|---:|---:|---:|---:| +| (256×512) · (512×256) | 36.510 | 8.698 | 4.775 | 7.65× | +| (1024×512) · (512×256) | 108.929 | 14.434 | 10.593 | 10.28× | +| Causal attention, 1,024 tokens, 64 features | 181.283 | 6.126 | 6.058 | 29.93× | +| Causal attention, 4,096 tokens, 64 features | 3,985.345 | 17.381 | 16.593 | 240.18× | + +The float64 CPU matrix backend alone reduced the 256-row product from +**8.698 ms to 4.422 ms (1.97×)**, without reducing precision. + +## Incremental GPU measurements + +This run explicitly sets `min_gpu_ops=0` to compare direct GPU arithmetic +with both CPU implementations. The default threshold still selects GPU +for the 8,192- and 16,384-token attention cases. It selects CPU for this +matrix product. Each GPU case below completed eight GPU kernels, including +the warmup, as verified by `statistics()`. + +| Workload | Original EinFold CPU | Float64 CPU backend | MLX GPU float32 | GPU vs original EinFold | +|---|---:|---:|---:|---:| +| (1024×1024) · (1024×1024) | 64.131 | 33.745 | 35.174 | 1.82× | +| Causal attention, 8,192 tokens, 64 features | 36.204 | 38.158 | 33.864 | 1.07× | +| Causal attention, 16,384 tokens, 64 features | 95.044 | 97.978 | 71.292 | 1.33× | + +Float64 CPU matrix multiplication is slightly faster than the GPU for this +matrix. GPU acceleration is useful for large attention, and is not a +universal replacement for CPU execution. The calibration run at the earlier +`2**28` GPU threshold made 4,096-token attention about 6% slower than original +EinFold. The final `2**33` threshold keeps it on the CPU. The raw calibration +file is retained rather than hiding this regression. + +## Repeated ddx gradients + +The query computes `Y = A·B`, `L = SUM(Y²)/2`, and `grad(L, B)`; +`A` has 64 features and `B` has 32 outputs. `xarray_sql.ad.prepare` retains +ddx's programs, checks catalog/schema/settings changes, and materializes +final results. Dataset writes use the existing validation and read current +values; regular preparation also supports replacing tables by name. + +| Rows in A | Option B SQL every call | Option B prepared | Repeated-step speedup | +|---|---:|---:|---:| +| 256 | 14.479 | 9.424 | 1.54× | +| 2,048 | 19.543 | 14.049 | 1.39× | + +These small gradients select CPU arithmetic. At 2,048 rows, plain DataFusion +prepared execution remains faster (11.635 ms) than Option B. +GPU kernels cannot eliminate the planning and materialization cost of a +small SQL gradient program. Actual GPU gradient execution is covered by the +forced-offload correctness tests, including attention's backward pass. + +## CPU profile and implementation decisions + +The initial [cProfile output](cpu-profile.txt) covers attention at 16 and 64 +tokens with DataFusion and released EinFold 0.1.0. Of 2.192 seconds in the +demo's main function, 0.977 seconds accumulated under ddx rewriting and +execution across 24 gradient calls. There were 336 intermediate +materializations, taking 0.678 seconds **inside** that time, not in addition +to it. DataFusion Arrow collection accounted for 0.939 seconds across 589 +calls. These include native work; cProfile does not separate Rust/Metal +kernel instructions. The profiled initial timings are not mixed with the +unprofiled final tables. + +Changes guided by this evidence and the kernel comparisons: + +- Add `ad.prepare` to reuse differentiation programs and avoid repeated + objective planning, program keying and view checks. +- Give EinFold a per-session native executor hook, after its dense input + validation. The existing row/domain/NULL reconstruction stays intact. +- Replace sufficiently large dense scalar accumulation with NumPy float64 + matrix multiplication on CPU, or explicitly permitted MLX float32 GPU + arithmetic. Native thresholds avoid Python buffer copies for tiny work. +- Bound attention query tiles, trim keys outside allowed intervals, and use + MLX's causal or mask-free kernels only when normalized ranges match. +- Retain CPU fallback for special values and unsupported relational shapes; + never cache input values across queries. +- Preserve xarray Context methods, Dataset metadata and prepared-query + invalidation when installing DataFusion extension bundles. + +## Training overheads and CPU parallelism + +The additional overhead pass covers all six optimization directions: + +| Direction | Implementation and evidence | +|---|---| +| Reduce planning and repeated CTE expansion | Forward `Context.prepare` retains the optimized logical plan, bypassing repeated SQL parsing and CTE expansion. ddx preparation retains differentiation programs. The repeated-CTE benchmark below checks current results. | +| Reuse plans across training steps | `Context.prepare` now accepts grad/VJP/JVP SQL as well as forward queries. It delegates gradient caching and catalog/settings checks to `ad.prepare`; multiple parameter updates still use `ad.prepare` to share a loss. | +| Reduce Python/Arrow registration | Uniform in-memory Datasets register Arrow batches directly, eliminating the temporary catalog table. Partition counts use arithmetic rather than enumerating the Cartesian chunk grid. Prepared writes retain shared value buffers instead of deregistering and repivoting tables. | +| Fuse numerical operators through EinFold | EinFold's physical planner recognizes contractions and fused attention. Attention scores, shifted exponentials, normalization and weighted sums use its fused pipeline, with original SQL fallbacks. | +| Execute fused operators with MLX | The session-local native hook dispatches eligible validated fused attention to MLX. The GPU measurements above separate its contribution from EinFold's existing rewrites. | +| Increase CPU concurrency | `Context(threads=...)` and the main benchmark's `--threads` expose the partition target. The default already uses 30 CPU cores; a 1/10/20/30/60 sweep and a 30/60 confirmation measure oversubscription rather than assume it helps. | + +The new measurements use nine timed repetitions after one warmup. The +fine-chunk baseline was recorded before replacing chunk-grid enumeration; +the final result uses the arithmetic count and direct Arrow registration. +All sums, products, gradients and repeated-CTE query results were checked. + +| Overhead case | Baseline | Improved | Speedup | +|---|---:|---:|---:| +| Register a 4,096×16 Dataset with one-cell chunks | 31.372 ms | 0.405 ms | 77.44× | +| Update and sum 1,048,576 values: register each step vs prepared writes | 2.769 ms | 1.027 ms | 2.70× | +| Repeated-CTE product, plain DataFusion: SQL each call vs prepared | 1.709 ms | 1.307 ms | 1.31× | + +Registration still scales with Arrow coordinate construction for large +ordinary tables. The dramatic fine-chunk improvement removes 65,536 +unneeded chunk-slice constructions; it does not imply that every Dataset +registers 77 times faster. Preparing is also not beneficial for every query. + +The independent concurrency confirmation, on identical 2,048-token causal +attention inputs within that run, gave: + +| Engine | 30 target partitions | 60 target partitions | Change | +|---|---:|---:|---:| +| datafusion | 790.149 ms | 747.087 ms | 1.06× faster | +| einfold | 8.737 ms | 9.076 ms | 3.9% slower | + +The earlier full sweep found 1 → 30 partitions reduced plain DataFusion +attention from 6,797 ms to 795 ms and EinFold attention from 25.1 ms to +9.3 ms. The 30/60 confirmation supports leaving the default at the physical +CPU count and making higher concurrency explicit per workload. The setting +controls target parallel partitions, not a promise of exactly that many OS +threads or a change to NumPy's separate BLAS thread controls. + +```python +ctx = Context(threads=60) # explicit workload-specific concurrency +ctx.from_dataset("weights", initial_weights) +step = ctx.prepare(gradient_sql) +for values in training_batches: + rows = step(weights=values) # no table re-registration +``` + +Limits on plan/CTE reuse are explicit: the installed DataFusion 55 Python +API has no `ExecutionPlan.reset_state`. Forward preparation reuses the +optimized logical plan and uses physical copies only when supported and +cheaper; ddx preparation retains its programs while planning their execution +steps as needed. This does **not** provide a general physical-plan cache or +compute an arbitrary repeated CTE once at runtime. SQL `AS MATERIALIZED` is +rejected by this installed parser. No automatic CTE rewrite was added that +could change volatile expressions or introduce plan nodes ddx cannot +serialize through Substrait. + +Raw data: [registration baseline](fine-chunks-before.json), +[initial overhead baseline](training-overheads-before.json), +[final overheads](training-overheads-final.json), +[full concurrency sweep](training-overheads.json), and +[30/60 confirmation](thread-confirmation.json). + +```sh +.venv/bin/python -m benchmarks.apple.overheads --skip-thread-sweep \ + --output benchmarks/apple/training-overheads-final.json +.venv/bin/python -m benchmarks.apple.overheads --only-thread-sweep \ + --threads 30 60 --thread-tokens 2048 --output benchmarks/apple/thread-confirmation.json +``` + +## Correctness and limits + +- Full Python unit suite after the registration, preparation and concurrency + changes: **608 passed, 277 skipped, 102 integration tests deselected**, + including **all 38 accelerator tests**. Coverage includes bags/duplicates, NULLs, sparse and empty + inputs, NaN/infinity/signed zero/extreme magnitudes, partitioned batches, + current-value reads, xarray metadata, prepared writes, grad/VJP/JVP, + stop-gradient, typed ddx refusals and attention backward derivatives. +- EinFold native suite: **72 passed, 4 ignored**, with one existing + plain-DataFusion assumption excluded: `nan_compares_by_its_sign`. It + expects the NaN from `inf - inf` to be negative. Plain DataFusion on this + ARM Mac produces a positive NaN, without the accelerator installed. The + original test remains unchanged in the patch. Running the full native + suite without the exclusion exposes this failure. +- Ruff formatting/lint and patch application checks passed. Focused mypy + checks of `mlx.py` and `ad.py` passed. Repository-wide mypy reports the + same five existing errors as an unmodified HEAD copy (in `df.py`, + `backends/pyarrow.py`, `sql.py`, and `roundtrip.py`), with no added errors. +- Default float64 CPU products matched the NumPy reference exactly in these + runs. Final GPU maximum absolute errors were **0.000216** for the + matrix product and **6.08e-07** for attention. All benchmark executions + checked results; GPU tolerances were `rtol=atol=3e-4`, while accelerator + unit tests use `3e-5`. Float32 GPU mode explicitly changes numerical + precision; it does **not** promise float64 rounding/error bounds. +- MLX's default M5 matrix mode initially failed those tolerances. Final + tests and measurements disable it using `MLX_ENABLE_TF32=0` at process + launch, as required by the accelerator. See + [MLX numerical precision](https://ml-explore.github.io/mlx/build/html/usage/precision.html). +- This is an optional integration requiring the included, pinned EinFold + patch; installing the normal EinFold wheel alone does not provide the + hook. The patched wheel is installed in this checkout's `.venv`. +- The 512 MiB backend estimate bounds requested working buffers, not all + driver allocations or caches. Peak process/GPU memory was not measured; + no peak-memory speedup or hard memory-cap claim is made. + +## Reproduce and inspect + +[Build and usage instructions](../../accelerators/einfold_mlx/README.md). +Final [default timings](m5-ultra-results.json), [large GPU timings](m5-ultra-large-results.json), +[threshold calibration](m5-ultra-crossover-results.json), +[initial profiled timings](initial-attention.csv), and [CPU profile](cpu-profile.txt). + +```sh +MLX_ENABLE_TF32=0 .venv/bin/python -m benchmarks.apple.benchmark \ + --matrix-rows 256 1024 --features 512 --outputs 256 \ + --attention-tokens 1024 4096 --gradient-rows 256 2048 \ + --reps 7 --output benchmarks/apple/m5-ultra-results.json + +MLX_ENABLE_TF32=0 .venv/bin/python -m benchmarks.apple.benchmark \ + --hosts einfold einfold_blas einfold_mlx \ + --matrix-rows 1024 --features 1024 --outputs 1024 \ + --attention-tokens 8192 16384 --gradient-rows 256 \ + --min-gpu-ops 0 --reps 7 --output benchmarks/apple/m5-ultra-large-results.json + +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python -m pytest -m 'not integration' +``` + +## nanoGPT model comparison + +Benchmark 2 was also run with `--steps 5` across the original hosts and +explicit prepared CPU, automatic GPU and forced-kernel controls. See the +[full nanoGPT comparison](nanogpt-5-steps.md) for all eight variants, cold +versus warm timings, numerical checks and actual GPU kernel counts. GPU +microbenchmark gains do not imply gains for this small end-to-end model. + +The subsequent [shared-CTE comparison](nanogpt-shared.md) combines shared +common table expression execution, cached plans and MLX GPU arithmetic, +with five-step training and larger nanoGPT configurations. It records +successful checks separately from memory-limited full-vocabulary runs. + +The [native JAX and nanoGPT comparison](nanogpt-frameworks.md) measures the +same full-parameter model directly in JAX and the upstream PyTorch model +on this M5 Ultra. It checks every update and shows the remaining large +performance gap between SQL execution and native array training. + +The later [complete performance report](../../NANOGPT_PERFORMANCE.md) +tracks compact execution and the resident Shakespeare trainer, with +startup, per-step and memory measurements. Its [resident summary](resident-summary.json) +records the 100-step result and the updated native PyTorch comparison. diff --git a/benchmarks/apple/benchmark.py b/benchmarks/apple/benchmark.py new file mode 100644 index 00000000..05c258de --- /dev/null +++ b/benchmarks/apple/benchmark.py @@ -0,0 +1,224 @@ +"""Measure SQL -> DataFusion -> EinFold -> MLX, including Arrow results. + +Run from the repository root with: + .venv/bin/python -m benchmarks.apple.benchmark --output benchmarks/apple/results.json + +Every host executes identical SQL. GPU kernels synchronize before returning +Arrow buffers. Registration is excluded; SQL planning, native/Python buffer +copies, execution, sorting and Arrow materialization are included. Warmups +are excluded from medians. ddx gradients are checked against closed-form +NumPy derivatives; attention and products against float64 NumPy results. +""" + +from __future__ import annotations + +import argparse +import cProfile +import importlib.metadata +import json +import os +import platform +import statistics +import subprocess +import time +from pathlib import Path + +import mlx.core as mx +import numpy as np +import xarray as xr +from einfold.datafusion import EinFold + +from xarray_sql import Context, ad +from xarray_sql.mlx import MLX + +PRODUCT = """SELECT a.i, b.k, SUM(a.v*b.v) AS v FROM a JOIN b ON a.j=b.j +GROUP BY a.i,b.k ORDER BY a.i,b.k""" +LOSS = """WITH pred AS ( +SELECT a.i,b.k,SUM(a.v*b.v) AS p FROM a JOIN b ON a.j=b.j GROUP BY a.i,b.k), +loss AS (SELECT SUM(p*p)/2 AS l FROM pred) +SELECT j,k,grad(loss,v) AS v FROM b ORDER BY j,k""" +ATTENTION = """WITH scores AS ( +SELECT q.i,k.j,SUM(q.v*k.v)/8.0 AS s FROM q JOIN k ON q.d=k.d +WHERE q.i>=k.j GROUP BY q.i,k.j), +weights AS (SELECT i,j,exp(s-MAX(s) OVER (PARTITION BY i)) AS w FROM scores), +norm AS (SELECT i,SUM(w) AS z FROM weights GROUP BY i) +SELECT w.i,v.d,SUM(w.w/n.z*v.v) AS v FROM weights w +JOIN v ON w.j=v.j JOIN norm n ON w.i=n.i GROUP BY w.i,v.d ORDER BY w.i,v.d""" + + +def _dataset(a, dims): + return xr.Dataset( + {"v": (dims, a)}, + coords={d: np.arange(n) for d, n in zip(dims, a.shape)}, + ) + + +def _context(host, min_gpu_ops, threads=None): + ctx = Context(threads=threads) + extension = None + if host == "einfold": + extension = EinFold() + elif host in ("einfold_blas", "einfold_mlx"): + extension = MLX( + precision="float32" if host == "einfold_mlx" else "float64", + min_gpu_ops=min_gpu_ops, + ) + if extension is not None: + ctx = ctx.with_extensions(extension) + return ctx, extension + + +def _measure(run, want, reps, tolerance): + run() # warm up compilation and the runtime + times = [] + error = 0.0 + for _ in range(reps): + start = time.perf_counter() + rows = run() + times.append(time.perf_counter() - start) + got = rows["v"].to_numpy().reshape(want.shape) + np.testing.assert_allclose(got, want, rtol=tolerance, atol=tolerance) + error = max(error, float(np.max(np.abs(got - want), initial=0))) + return { + "seconds": statistics.median(times), + "samples": times, + "max_abs_error": error, + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--hosts", + nargs="+", + choices=["datafusion", "einfold", "einfold_blas", "einfold_mlx"], + default=["datafusion", "einfold", "einfold_blas", "einfold_mlx"], + ) + parser.add_argument( + "--matrix-rows", nargs="+", type=int, default=[256, 2048] + ) + parser.add_argument( + "--attention-tokens", nargs="+", type=int, default=[256, 1024] + ) + parser.add_argument( + "--gradient-rows", nargs="+", type=int, default=[256, 2048] + ) + parser.add_argument( + "--threads", + type=int, + help="DataFusion target partitions; default is CPU count", + ) + parser.add_argument("--min-gpu-ops", type=int, default=1 << 33) + parser.add_argument("--features", type=int, default=64) + parser.add_argument("--outputs", type=int, default=128) + parser.add_argument("--reps", type=int, default=5) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--profile", type=Path) + args = parser.parse_args() + rng = np.random.default_rng(20261031) + metadata = { + "platform": platform.platform(), + "python": platform.python_version(), + "hardware": subprocess.check_output( + ["sysctl", "-n", "machdep.cpu.brand_string"], text=True + ).strip(), + "gpu": mx.device_info(), + "versions": { + p: importlib.metadata.version(p) + for p in [ + "datafusion", + "einfold", + "ddxdb", + "mlx", + "numpy", + "pyarrow", + ] + }, + "min_gpu_ops": args.min_gpu_ops, + "threads": args.threads or os.cpu_count(), + "matrix_features": args.features, + "matrix_outputs": args.outputs, + "reps": args.reps, + "warmups": 1, + "MLX_ENABLE_TF32": os.environ.get("MLX_ENABLE_TF32"), + } + report = {"environment": metadata, "results": []} + profiler = cProfile.Profile() if args.profile else None + if profiler: + profiler.enable() + problems = [] + for n in args.matrix_rows: + a, b = ( + rng.normal(size=(n, args.features)), + rng.normal(size=(args.features, args.outputs)), + ) + problems.append( + ( + "matmul", + n, + [("a", a, ("i", "j")), ("b", b, ("j", "k"))], + PRODUCT, + a @ b, + ) + ) + for n in args.attention_tokens: + q, k, v = (rng.normal(size=(n, 64)) for _ in range(3)) + score = q @ k.T / 8.0 + score[np.triu_indices(n, 1)] = -np.inf + weights = np.exp(score - np.max(score, axis=1, keepdims=True)) + want = weights / weights.sum(axis=1, keepdims=True) @ v + problems.append( + ( + "attention", + n, + [ + ("q", q, ("i", "d")), + ("k", k, ("j", "d")), + ("v", v, ("j", "d")), + ], + ATTENTION, + want, + ) + ) + for n in args.gradient_rows: + a, b = rng.normal(size=(n, 64)), rng.normal(size=(64, 32)) + problems.append( + ( + "gradient", + n, + [("a", a, ("i", "j")), ("b", b, ("j", "k"))], + LOSS, + a.T @ (a @ b), + ) + ) + for task, n, tables, sql, want in problems: + for host in args.hosts: + ctx, extension = _context(host, args.min_gpu_ops, args.threads) + for name, a, dims in tables: + ctx.from_dataset(name, _dataset(a, dims)) + tolerance = 3e-4 if host == "einfold_mlx" else 1e-9 + modes = ["sql", "prepared"] if task == "gradient" else ["sql"] + for mode in modes: + if mode == "prepared": + prepared = ad.prepare(ctx, [sql]) + run = lambda prepared=prepared: prepared()[0] + else: + run = lambda ctx=ctx, sql=sql: ctx.sql(sql).to_arrow_table() + result = _measure(run, want, args.reps, tolerance) + result.update(task=task, size=n, host=host, mode=mode) + if isinstance(extension, MLX): + result["kernels"] = extension.statistics() + report["results"].append(result) + print( + f"{task:10} {n:5} {host:14} {mode:8} {result['seconds'] * 1000:9.3f} ms error={result['max_abs_error']:.3g}", + flush=True, + ) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n") + if profiler: + profiler.disable() + profiler.dump_stats(args.profile) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/apple/compare_nanogpt_frameworks.py b/benchmarks/apple/compare_nanogpt_frameworks.py new file mode 100644 index 00000000..683ce6c5 --- /dev/null +++ b/benchmarks/apple/compare_nanogpt_frameworks.py @@ -0,0 +1,338 @@ +"""Check and time native array frameworks on the SQL nanoGPT workload. + +Uses the same model, initial weights, five Shakespeare batches and SGD as +``benchmarks/ml/02_nanogpt.py``. JAX (https://github.com/jax-ml/jax) runs +on its installed device. PyTorch (https://pytorch.org) imports the original +nanoGPT model (https://github.com/karpathy/nanoGPT), with matching dimensions +and tanh GELU. CPU uses float64; Apple's MPS GPU uses float32. Each timed +step returns the loss and updated parameters, checked against the SQL +demo's float64 JAX reference outside the timer. Device synchronization is +included. Source downloads and package installation are separate. +""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.util +import json +import platform +import statistics +import subprocess +import sys +import time +from pathlib import Path + +import jax +import jax.numpy as jnp +import numpy as np + +try: + import torch +except ImportError: + torch = None + +_ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(_ROOT / "benchmarks/ml")) + + +def _module(name, path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +def _torch_load(model, params): + with torch.no_grad(): + for name, target in ( + ("wte", model.transformer.wte.weight), + ("wpe", model.transformer.wpe.weight), + ("ln_f", model.transformer.ln_f.weight), + ): + target.copy_( + torch.as_tensor( + params[name], dtype=target.dtype, device=target.device + ) + ) + for layer, block in enumerate(model.transformer.h): + packed = np.concatenate( + [ + params[name][layer].reshape(768, 768).T + for name in ("wq", "wk", "wv") + ] + ) + for target, source in ( + (block.attn.c_attn.weight, packed), + ( + block.attn.c_proj.weight, + params["wo"][layer].reshape(768, 768).T, + ), + (block.mlp.c_fc.weight, params["w_fc"][layer].T), + (block.mlp.c_proj.weight, params["w_proj"][layer].T), + (block.ln_1.weight, params["ln_1"][layer]), + (block.ln_2.weight, params["ln_2"][layer]), + ): + target.copy_( + torch.as_tensor( + source, dtype=target.dtype, device=target.device + ) + ) + # The SQL demo uses GPT-2's tanh approximation; upstream's + # default nn.GELU uses erf. Keep the benchmark math identical. + block.mlp.gelu = torch.nn.GELU(approximate="tanh") + + +def _torch_arrays(model): + def array(tensor): + return tensor.detach().cpu().numpy() + + yield "wte", array(model.transformer.wte.weight) + yield "wpe", array(model.transformer.wpe.weight) + yield "ln_f", array(model.transformer.ln_f.weight) + for name in ("wq", "wk", "wv", "wo", "w_fc", "w_proj", "ln_1", "ln_2"): + layers = [] + for block in model.transformer.h: + if name in ("wq", "wk", "wv"): + index = ("wq", "wk", "wv").index(name) + layers.append( + array( + block.attn.c_attn.weight[ + index * 768 : (index + 1) * 768 + ] + ).T.reshape(768, 12, 64) + ) + elif name == "wo": + layers.append( + array(block.attn.c_proj.weight).T.reshape(12, 64, 768) + ) + elif name == "w_fc": + layers.append(array(block.mlp.c_fc.weight).T) + elif name == "w_proj": + layers.append(array(block.mlp.c_proj.weight).T) + else: + layers.append(array(getattr(block, name).weight)) + yield name, np.stack(layers) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--backend", + required=True, + choices=[ + "jax_eager", + "jax_separate", + "jax_jit", + "torch_cpu", + "torch_mps", + ], + ) + parser.add_argument("--context", type=int, default=16) + parser.add_argument("--steps", type=int, default=5) + parser.add_argument("--forward-only", action="store_true") + parser.add_argument("--nanogpt-source", type=Path) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + if min(args.context, args.steps) <= 0: + parser.error("context and steps must be positive") + n = _module("xql_nanogpt_reference", _ROOT / "benchmarks/ml/02_nanogpt.py") + n.LAYERS, n.EMBED, n.HEADS, n.HEAD, n.HIDDEN, n.CONTEXT = ( + 12, + 768, + 12, + 64, + 3072, + args.context, + ) + init = n.init_params(50257, np.random.default_rng(0)) + data = list( + zip( + range(args.steps), + n.batches(n.shakespeare(), 1, np.random.default_rng(1)), + ) + ) + reference = dict(init) + value_grad = jax.value_and_grad(n.jax_loss) + lr = 0.01 + gpu = args.backend == "torch_mps" + tolerance = 3e-5 if gpu else 1e-9 + result = { + "backend": args.backend, + "context": args.context, + "parameters": sum(w.size for w in init.values()), + "parameter_shapes": {name: list(w.shape) for name, w in init.items()}, + "parameter_counts": {name: w.size for name, w in init.items()}, + "steps": args.steps, + "batch": 1, + "learning_rate": lr, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": args.forward_only, + "rtol": tolerance, + "atol": tolerance, + "python": platform.python_version(), + "jax_version": jax.__version__, + "reference_sha256": hashlib.sha256( + (_ROOT / "benchmarks/ml/02_nanogpt.py").read_bytes() + ).hexdigest(), + "rows": [], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + } + started = time.perf_counter() + if args.backend.startswith("jax"): + params = jax.tree.map(lambda w: jnp.array(w, copy=True), init) + jax.block_until_ready(params) + result["devices"] = [str(d) for d in jax.devices()] + + def train_step(p, tok, nxt): + if args.backend == "jax_separate": + loss = n.jax_loss(p, tok, nxt) + jax.block_until_ready(loss) + gradients = jax.grad(n.jax_loss)(p, tok, nxt) + else: + loss, gradients = value_grad(p, tok, nxt) + return { + name: w - lr * gradients[name] for name, w in p.items() + }, loss + + function = n.jax_loss if args.forward_only else train_step + if args.backend == "jax_jit": + begin = time.perf_counter() + tok, nxt = data[0][1]["tok"].values, data[0][1]["next"].values + function = jax.jit( + function, donate_argnums=() if args.forward_only else (0,) + ) + function = function.lower(params, tok, nxt).compile() + result["compile_seconds"] = time.perf_counter() - begin + else: + if torch is None or args.nanogpt_source is None: + parser.error( + "PyTorch and --nanogpt-source are required for torch backends" + ) + torch.set_num_threads(30) + if gpu and not torch.backends.mps.is_available(): + parser.error("MPS GPU is unavailable") + upstream = _module( + "xql_upstream_nanogpt", args.nanogpt_source / "model.py" + ) + result["nanogpt_revision"] = subprocess.check_output( + ["git", "-C", str(args.nanogpt_source), "rev-parse", "HEAD"], + text=True, + ).strip() + result["nanogpt_model_sha256"] = hashlib.sha256( + (args.nanogpt_source / "model.py").read_bytes() + ).hexdigest() + result["torch_version"] = torch.__version__ + device, dtype = ( + ("mps", torch.float32) if gpu else ("cpu", torch.float64) + ) + old = torch.get_default_dtype() + try: + torch.set_default_dtype(dtype) + model = upstream.GPT( + upstream.GPTConfig( + block_size=args.context, + vocab_size=50257, + n_layer=12, + n_head=12, + n_embd=768, + dropout=0, + bias=False, + ) + ).to(device) + finally: + torch.set_default_dtype(old) + _torch_load(model, init) + model.train(not args.forward_only) + optimizer = torch.optim.SGD(model.parameters(), lr=lr) + result["devices"] = [device] + result["torch_threads"] = torch.get_num_threads() + assert ( + sum(p.numel() for p in model.parameters()) == result["parameters"] + ) + if gpu: + torch.mps.synchronize() + result["setup_seconds_including_compile"] = time.perf_counter() - started + for step, batch in data: + tok, nxt = batch["tok"].values, batch["next"].values + if args.backend.startswith("jax"): + tokens, targets = jax.device_put(tok), jax.device_put(nxt) + jax.block_until_ready((tokens, targets)) + begin = time.perf_counter() + if args.forward_only: + loss = function(params, tokens, targets) + else: + params, loss = function(params, tokens, targets) + jax.block_until_ready((params, loss)) + elapsed = time.perf_counter() - begin + else: + tokens = torch.as_tensor(tok, device=device) + targets = torch.as_tensor(nxt, device=device) + if gpu: + torch.mps.synchronize() + begin = time.perf_counter() + with torch.set_grad_enabled(not args.forward_only): + if not args.forward_only: + optimizer.zero_grad(set_to_none=True) + _, loss = model(tokens, targets) + if not args.forward_only: + loss.backward() + optimizer.step() + if gpu: + torch.mps.synchronize() + elapsed = time.perf_counter() - begin + loss = loss.detach().cpu().item() + if args.forward_only: + want = n.jax_loss(reference, tok, nxt) + else: + want, gradients = value_grad(reference, tok, nxt) + reference = { + name: w - lr * gradients[name] for name, w in reference.items() + } + jax.block_until_ready((reference, want)) + errors = [ + n.assert_close("loss", loss, want, rtol=tolerance, atol=tolerance) + ] + if not args.forward_only: + actual = ( + params.items() + if args.backend.startswith("jax") + else _torch_arrays(model) + ) + for name, got in actual: + errors.append( + n.assert_close( + name, + got, + reference[name], + rtol=tolerance, + atol=tolerance, + ) + ) + row = { + "step": step, + "seconds": elapsed, + "loss": float(want), + "check": "ok", + "max_scaled_error": max(errors), + } + result["rows"].append(row) + print(args.backend, row, flush=True) + args.output.write_text(json.dumps(result, indent=2) + "\n") + result["warm_seconds"] = ( + statistics.mean(r["seconds"] for r in result["rows"][1:]) + if args.steps > 1 + else result["rows"][0]["seconds"] + ) + result["five_step_seconds"] = sum(r["seconds"] for r in result["rows"]) + result["five_steps_with_setup_compile_seconds"] = ( + result["five_step_seconds"] + result["setup_seconds_including_compile"] + ) + args.output.write_text(json.dumps(result, indent=2) + "\n") + + +if __name__ == "__main__": + main() diff --git a/benchmarks/apple/compare_shakespeare_mps.py b/benchmarks/apple/compare_shakespeare_mps.py new file mode 100644 index 00000000..cbef8e36 --- /dev/null +++ b/benchmarks/apple/compare_shakespeare_mps.py @@ -0,0 +1,225 @@ +"""Time the original nanoGPT training loop on Apple's PyTorch MPS backend. + +nanoGPT (https://github.com/karpathy/nanoGPT) supplies train.py and its +Shakespeare character recipe. An AST instrumentation adds synchronized +timers around each native forward/backward/AdamW update, leaving the +model, dropout, optimizer and data pipeline unchanged. Source files in +the nanoGPT checkout are only read. +""" + +from __future__ import annotations + +import argparse +import ast +import hashlib +import json +import math +import os +import platform +import statistics +import subprocess +import sys +import time +from pathlib import Path + +import torch + + +def _call(statement, owner, attribute): + return ( + isinstance(statement, ast.Expr) + and isinstance(statement.value, ast.Call) + and isinstance(statement.value.func, ast.Attribute) + and isinstance(statement.value.func.value, ast.Name) + and statement.value.func.value.id == owner + and statement.value.func.attr == attribute + ) + + +def _instrument(source): + tree = ast.parse(source) + candidates = [node for node in tree.body if isinstance(node, ast.While)] + if len(candidates) != 1: + raise ValueError("expected one top-level nanoGPT training loop") + loop = candidates[0] + starts = [ + i + for i, node in enumerate(loop.body) + if isinstance(node, ast.For) + and isinstance(node.target, ast.Name) + and node.target.id == "micro_step" + ] + ends = [ + i + for i, node in enumerate(loop.body) + if _call(node, "optimizer", "zero_grad") + ] + if len(starts) != 1 or len(ends) != 1 or starts[0] >= ends[0]: + raise ValueError( + "cannot identify nanoGPT's forward/backward/update interval" + ) + begin = ast.parse( + "torch.mps.synchronize(); _xql_started = _xql_clock()" + ).body + end = ast.parse("""torch.mps.synchronize() +_xql_elapsed = _xql_clock() - _xql_started +_xql_record(iter_num, lr, loss, _xql_elapsed) +""").body + # Insert from the end so the original statement indexes remain valid. + loop.body[ends[0] + 1 : ends[0] + 1] = end + loop.body[starts[0] : starts[0]] = begin + return ast.fix_missing_locations(tree) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--nanogpt-source", type=Path, required=True) + parser.add_argument("--steps", type=int, default=5) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument( + "--sql-summary", + type=Path, + default=Path(__file__).with_name("shakespeare-summary.json"), + ) + args = parser.parse_args() + if args.steps < 2: + parser.error("at least two updates are required for a warm mean") + if not torch.backends.mps.is_available(): + parser.error("MPS GPU is unavailable") + source_dir, output = args.nanogpt_source.resolve(), args.output.resolve() + source = (source_dir / "train.py").read_text() + tree = _instrument(source) + arguments = [ + "train.py", + "config/train_shakespeare_char.py", + "--device=mps", + "--dtype=float32", + "--compile=False", + "--batch_size=12", + "--block_size=256", + "--n_layer=6", + "--n_head=6", + "--n_embd=384", + "--gradient_accumulation_steps=1", + f"--max_iters={args.steps - 1}", + "--lr_decay_iters=2000", + "--warmup_iters=100", + "--eval_interval=200", + "--eval_iters=20", + "--log_interval=1", + "--wandb_log=False", + f"--out_dir={output.parent / (output.stem + '-out')}", + ] + records = [] + + def record(iteration, lr, loss, seconds): + value = float(loss.detach().cpu()) + if not math.isfinite(value) or seconds <= 0: + raise ValueError("invalid native training loss or duration") + records.append( + { + "iteration": iteration, + "lr": lr, + "loss": value, + "seconds": seconds, + "tokens_per_second": 3072 / seconds, + } + ) + + namespace = { + "__name__": "__main__", + "__file__": str(source_dir / "train.py"), + "_xql_clock": time.perf_counter, + "_xql_record": record, + } + previous_dir, previous_argv = Path.cwd(), sys.argv + sys.path.insert(0, str(source_dir)) + try: + os.chdir(source_dir) + sys.argv = arguments + # Run the explicitly selected local trainer, with only timing hooks. + exec(compile(tree, str(source_dir / "train.py"), "exec"), namespace) # noqa: S102 + finally: + sys.argv = previous_argv + os.chdir(previous_dir) + sys.path.remove(str(source_dir)) + if [r["iteration"] for r in records] != list(range(args.steps)): + raise ValueError( + "native train.py did not execute the requested update count" + ) + for row in records: + expected = 1e-3 * (row["iteration"] + 1) / 101 + if not math.isclose(row["lr"], expected, rel_tol=1e-12): + raise ValueError( + "native learning-rate warmup differs from the SQL recipe" + ) + config = namespace["config"] + assert config["dropout"] == 0.2 and config["bias"] is False + assert config["learning_rate"] == 1e-3 and config["min_lr"] == 1e-4 + assert config["beta1"] == 0.9 and config["beta2"] == 0.99 + assert config["weight_decay"] == 0.1 and config["grad_clip"] == 1.0 + assert namespace["meta_vocab_size"] == 65 + parameters = sum(p.numel() for p in namespace["model"].parameters()) + assert parameters == 10745088 + warm = statistics.mean(row["seconds"] for row in records[1:]) + result = { + "backend": "torch_mps", + "date": "2026-11-01", + "python": platform.python_version(), + "platform": platform.platform(), + "torch_version": torch.__version__, + "steps": args.steps, + "parameters": parameters, + "batch_tokens": 3072, + "config": config, + "nanogpt_revision": subprocess.check_output( + ["git", "-C", str(source_dir), "rev-parse", "HEAD"], text=True + ).strip(), + "source_sha256": { + name: hashlib.sha256((source_dir / name).read_bytes()).hexdigest() + for name in ( + "train.py", + "model.py", + "config/train_shakespeare_char.py", + ) + }, + "instrumentation_sha256": hashlib.sha256( + Path(__file__).read_bytes() + ).hexdigest(), + "native_argv": arguments, + "rows": records, + "warm_mean_seconds": warm, + "warm_tokens_per_second": 3072 / warm, + "attention": "original model.py scaled_dot_product_attention selection", + "scope": "Synchronized native forward, backward, gradient clipping, AdamW, zero_grad and next-batch prefetch. Evaluation, initialization, logging and checkpoint I/O excluded. Weights and optimizer state remain on MPS.", + "matching": "Same model, data split, optimizer, dropout and LR schedule as the SQL recipe; stock PyTorch RNGs, rather than SQL NumPy RNGs. max_iters=steps-1 stops native's inclusive loop after exactly steps updates. compile=False.", + } + sql = json.loads(args.sql_summary.read_text()) + result["sql_comparison"] = [ + { + "backend": run["backend"], + "seconds": run["warm_mean_seconds"], + "sql_over_torch_ratio": run["warm_mean_seconds"] / warm, + } + for run in sql["runs"] + ] + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(json.dumps(result, indent=2, default=str) + "\n") + print( + json.dumps( + { + key: result[key] + for key in ( + "backend", + "warm_mean_seconds", + "warm_tokens_per_second", + "sql_comparison", + ) + } + ), + flush=True, + ) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/apple/cpu-profile.txt b/benchmarks/apple/cpu-profile.txt new file mode 100644 index 00000000..ffcf58bf --- /dev/null +++ b/benchmarks/apple/cpu-profile.txt @@ -0,0 +1,43 @@ +Sat Oct 31 23:31:20 2026 /private/tmp/xql-baseline-attention.prof + + 2716130 function calls (2664128 primitive calls) in 2.758 seconds + + Ordered by: cumulative time + List reduced from 8756 to 35 due to restriction <35> + + ncalls tottime percall cumtime percall filename:lineno(function) + 1829/1 0.018 0.000 2.766 2.766 {built-in method builtins.exec} + 1 0.000 0.000 2.765 2.765 01_attention.py:1() + 182 0.002 0.000 2.223 0.012 __init__.py:1() + 1 0.001 0.001 2.192 2.192 01_attention.py:214(main) + 61 0.001 0.000 1.180 0.019 _hosts.py:89(query) + 18 0.000 0.000 1.159 0.064 _bench.py:85(time) + 8 0.000 0.000 1.026 0.128 _bench.py:110(attempt) + 383/83 0.001 0.000 0.995 0.012 sql.py:245(sql) + 24 0.000 0.000 0.983 0.041 ad.py:99(sql) + 24 0.000 0.000 0.983 0.041 ad.py:750(sql) + 24 0.000 0.000 0.983 0.041 ad.py:777(sql_all) + 24 0.000 0.000 0.977 0.041 __init__.py:211(rewrite) + 24 0.032 0.001 0.977 0.041 {built-in method ddxdb._ddxdb._rewrite} + 589 0.000 0.000 0.939 0.002 dataframe.py:1739(to_arrow_table) + 589 0.935 0.002 0.939 0.002 {method 'to_arrow_table' of 'datafusion.DataFrame' objects} + 2193/13 0.003 0.000 0.852 0.066 :1349(_find_and_load) + 1717/13 0.002 0.000 0.852 0.066 :1304(_find_and_load_unlocked) + 1507/9 0.002 0.000 0.849 0.094 :911(_load_unlocked) + 1300/9 0.001 0.000 0.849 0.094 :993(exec_module) + 3815/22 0.002 0.000 0.848 0.039 :480(_call_with_frames_removed) + 838/91 0.000 0.000 0.747 0.008 {built-in method builtins.__import__} + 2805/619 0.001 0.000 0.739 0.001 :1390(_handle_fromlist) + 336 0.008 0.000 0.678 0.002 ad.py:392(materialize) + 431/379 0.000 0.000 0.666 0.002 traceback_util.py:190(reraise_with_filtered_traceback) + 76 0.000 0.000 0.591 0.008 pjit.py:1119(_pjit_call_impl_python) + 370/294 0.001 0.000 0.565 0.002 profiler.py:585(wrapper) + 76 0.000 0.000 0.520 0.007 pxla.py:1229(compile) + 71 0.001 0.000 0.520 0.007 pxla.py:1706(from_hlo) + 71 0.000 0.000 0.514 0.007 pxla.py:1519(_cached_compilation) + 60/58 0.000 0.000 0.514 0.009 pjit.py:263(cache_miss) + 71 0.000 0.000 0.508 0.007 compiler.py:436(compile_or_get_cached) + 71 0.000 0.000 0.498 0.007 compiler.py:771(_compile_and_write_cache) + 71 0.497 0.007 0.497 0.007 compiler.py:341(backend_compile_and_load) + 437/167 0.001 0.000 0.494 0.003 core.py:687(bind) + 494/167 0.001 0.000 0.493 0.003 core.py:736(bind_with_trace) diff --git a/benchmarks/apple/dense-before-stages-summary.json b/benchmarks/apple/dense-before-stages-summary.json new file mode 100644 index 00000000..dabfea5f --- /dev/null +++ b/benchmarks/apple/dense-before-stages-summary.json @@ -0,0 +1,5644 @@ +{ + "host": "shared_cpu", + "context": 16, + "instrumented": true, + "stages": [ + { + "name": "__ddx_77668c50_1_saved_0", + "seconds": 0.0830829170299694, + "compile_seconds": 0.0029816250316798687, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_1", + "seconds": 0.053183124982751906, + "compile_seconds": 0.003785667009651661, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_2", + "seconds": 0.04624150006566197, + "compile_seconds": 0.0064265410182997584, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_3", + "seconds": 0.046304875053465366, + "compile_seconds": 0.00633874989580363, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_4", + "seconds": 0.0009725000709295273, + "compile_seconds": 0.0026871670270338655, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_5", + "seconds": 0.04911620810162276, + "compile_seconds": 0.00551541696768254, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_6", + "seconds": 0.0007793749682605267, + "compile_seconds": 0.00254620797932148, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_7", + "seconds": 0.001510916044935584, + "compile_seconds": 0.00476062495727092, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_8", + "seconds": 0.005846834043040872, + "compile_seconds": 0.0025149999419227242, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_9", + "seconds": 0.049393208930268884, + "compile_seconds": 0.003804708016104996, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_10", + "seconds": 0.04457054194062948, + "compile_seconds": 0.0043811670038849115, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_11", + "seconds": 0.057055250043049455, + "compile_seconds": 0.006856917054392397, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_12", + "seconds": 0.015947000007145107, + "compile_seconds": 0.0031031660037115216, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_13", + "seconds": 0.04988808301277459, + "compile_seconds": 0.004246958065778017, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_14", + "seconds": 0.049163166899234056, + "compile_seconds": 0.005276625044643879, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_15", + "seconds": 0.0474832080071792, + "compile_seconds": 0.008207167033106089, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_16", + "seconds": 0.05072245793417096, + "compile_seconds": 0.007719833985902369, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_17", + "seconds": 0.0008639999432489276, + "compile_seconds": 0.002766416990198195, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_18", + "seconds": 0.04841679101809859, + "compile_seconds": 0.00754600006621331, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_19", + "seconds": 0.0007794169941917062, + "compile_seconds": 0.002552666002884507, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_20", + "seconds": 0.0015972499968484044, + "compile_seconds": 0.003862750018015504, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_21", + "seconds": 0.005822625011205673, + "compile_seconds": 0.002553416066803038, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_22", + "seconds": 0.04901312501169741, + "compile_seconds": 0.004817166016437113, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_23", + "seconds": 0.05005024990532547, + "compile_seconds": 0.006006999989040196, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_24", + "seconds": 0.06101837498135865, + "compile_seconds": 0.00925937492866069, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_25", + "seconds": 0.02461233304347843, + "compile_seconds": 0.0033361250534653664, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_26", + "seconds": 0.054812459042295814, + "compile_seconds": 0.006639666971750557, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_27", + "seconds": 0.04935095796827227, + "compile_seconds": 0.00809108407702297, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_28", + "seconds": 0.049880500067956746, + "compile_seconds": 0.009469458949752152, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_29", + "seconds": 0.052384582930244505, + "compile_seconds": 0.00950675003696233, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_30", + "seconds": 0.000992208020761609, + "compile_seconds": 0.002896833000704646, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_31", + "seconds": 0.05212916596792638, + "compile_seconds": 0.010434709023684263, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_32", + "seconds": 0.0008572089718654752, + "compile_seconds": 0.002699458971619606, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_33", + "seconds": 0.0017859999788925052, + "compile_seconds": 0.00412662501912564, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_34", + "seconds": 0.006042041932232678, + "compile_seconds": 0.002640417078509927, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_35", + "seconds": 0.05019187496509403, + "compile_seconds": 0.006481750053353608, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_36", + "seconds": 0.04953099996782839, + "compile_seconds": 0.007229958078823984, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_37", + "seconds": 0.06315358402207494, + "compile_seconds": 0.009937666007317603, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_38", + "seconds": 0.01671562495175749, + "compile_seconds": 0.0030676660826429725, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_39", + "seconds": 0.060394749976694584, + "compile_seconds": 0.007317624986171722, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_40", + "seconds": 0.06937808298971504, + "compile_seconds": 0.009406125056557357, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_41", + "seconds": 0.05446574999950826, + "compile_seconds": 0.012234875001013279, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_42", + "seconds": 0.05196700000669807, + "compile_seconds": 0.011604541912674904, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_43", + "seconds": 0.001343625015579164, + "compile_seconds": 0.003093707957305014, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_44", + "seconds": 0.05275612499099225, + "compile_seconds": 0.012359707965515554, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_45", + "seconds": 0.0007840830367058516, + "compile_seconds": 0.0026588329346850514, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_46", + "seconds": 0.0016745839966461062, + "compile_seconds": 0.004628374939784408, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_47", + "seconds": 0.0060841660015285015, + "compile_seconds": 0.002733916975557804, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_48", + "seconds": 0.05329462490044534, + "compile_seconds": 0.008538000052794814, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_49", + "seconds": 0.05929183401167393, + "compile_seconds": 0.009858582983724773, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_50", + "seconds": 0.06462424993515015, + "compile_seconds": 0.013657290954142809, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_51", + "seconds": 0.020457540987990797, + "compile_seconds": 0.0035487499553710222, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_52", + "seconds": 0.059627957991324365, + "compile_seconds": 0.010420250007882714, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_53", + "seconds": 0.05455745803192258, + "compile_seconds": 0.010575624997727573, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_54", + "seconds": 0.05240212508942932, + "compile_seconds": 0.01301924989093095, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_55", + "seconds": 0.054520250065252185, + "compile_seconds": 0.013572791940532625, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_56", + "seconds": 0.0008353749290108681, + "compile_seconds": 0.002818542066961527, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_57", + "seconds": 0.053498500026762486, + "compile_seconds": 0.014118333929218352, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_58", + "seconds": 0.0008029160089790821, + "compile_seconds": 0.0026116659864783287, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_59", + "seconds": 0.0015218330081552267, + "compile_seconds": 0.004043249995447695, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_60", + "seconds": 0.005960374954156578, + "compile_seconds": 0.0025432079564779997, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_61", + "seconds": 0.05861554201692343, + "compile_seconds": 0.009700166992843151, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_62", + "seconds": 0.051567915943451226, + "compile_seconds": 0.01130879099946469, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_63_part_0", + "seconds": 0.06273645907640457, + "compile_seconds": 0.01316566695459187, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_63", + "seconds": 0.0006960000609979033, + "compile_seconds": 0.001643582945689559, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_64", + "seconds": 0.017440584022551775, + "compile_seconds": 0.0028496660524979234, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_65", + "seconds": 0.06134970800485462, + "compile_seconds": 0.011415999964810908, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_66", + "seconds": 0.05696291709318757, + "compile_seconds": 0.012642832938581705, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_67_part_0", + "seconds": 2.1999934688210487e-05, + "compile_seconds": 0.0015259579522535205, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_67", + "seconds": 0.08472491696011275, + "compile_seconds": 0.015915750060230494, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_68_part_0", + "seconds": 2.3499946109950542e-05, + "compile_seconds": 0.001605625031515956, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_68", + "seconds": 0.07972091704141349, + "compile_seconds": 0.015865791938267648, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_69", + "seconds": 0.0009352500783279538, + "compile_seconds": 0.002799249952659011, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_70_part_0", + "seconds": 1.954101026058197e-05, + "compile_seconds": 0.0014436669880524278, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_70", + "seconds": 0.07359195896424353, + "compile_seconds": 0.014270124956965446, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_71", + "seconds": 0.0007327500497922301, + "compile_seconds": 0.002424667007289827, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_72", + "seconds": 0.00169945799279958, + "compile_seconds": 0.0041219169506803155, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_73", + "seconds": 0.005134207895025611, + "compile_seconds": 0.0026177919935435057, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_74", + "seconds": 0.05470154108479619, + "compile_seconds": 0.012403458938933909, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_75", + "seconds": 0.05315379204694182, + "compile_seconds": 0.012195499963127077, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_76_part_0", + "seconds": 1.8874998204410076e-05, + "compile_seconds": 0.0014263750053942204, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_76", + "seconds": 0.11953041702508926, + "compile_seconds": 0.017187958932481706, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_77", + "seconds": 0.019182375050149858, + "compile_seconds": 0.0032188750337809324, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_78", + "seconds": 0.06053770799189806, + "compile_seconds": 0.014517750008963048, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_79_part_0", + "seconds": 0.05902054090984166, + "compile_seconds": 0.014215332921594381, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_79", + "seconds": 0.0004983330145478249, + "compile_seconds": 0.00175762502476573, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_80_part_0", + "seconds": 2.154102548956871e-05, + "compile_seconds": 0.001450291951186955, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_80", + "seconds": 0.07504270796198398, + "compile_seconds": 0.016939957975409925, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_81_part_0", + "seconds": 2.0625069737434387e-05, + "compile_seconds": 0.0015538750449195504, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_81", + "seconds": 0.07734462502412498, + "compile_seconds": 0.017020458006300032, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_82", + "seconds": 0.0009088750230148435, + "compile_seconds": 0.0030365410493686795, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_83_part_0", + "seconds": 2.6124995201826096e-05, + "compile_seconds": 0.0015622499631717801, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_83", + "seconds": 0.0809478749288246, + "compile_seconds": 0.01777970790863037, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_84", + "seconds": 0.0007779999868944287, + "compile_seconds": 0.0027065000031143427, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_85", + "seconds": 0.001550624961964786, + "compile_seconds": 0.004074541968293488, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_86", + "seconds": 0.00504458392970264, + "compile_seconds": 0.002626959001645446, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_87_part_0", + "seconds": 0.05858916603028774, + "compile_seconds": 0.013078500051051378, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_87", + "seconds": 0.0023463749093934894, + "compile_seconds": 0.00188045809045434, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_88_part_0", + "seconds": 1.9458006136119366e-05, + "compile_seconds": 0.0014139999402686954, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_88", + "seconds": 0.057106415973976254, + "compile_seconds": 0.014472250011749566, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_89_part_0", + "seconds": 2.0209001377224922e-05, + "compile_seconds": 0.0014980420237407088, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_89", + "seconds": 0.1176451250212267, + "compile_seconds": 0.01937649992760271, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_90", + "seconds": 0.0184882499743253, + "compile_seconds": 0.0033029590267688036, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_91_part_0", + "seconds": 2.1249987185001373e-05, + "compile_seconds": 0.001444583060219884, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_91", + "seconds": 0.06738783291075379, + "compile_seconds": 0.014661167049780488, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_92_part_0", + "seconds": 2.0749983377754688e-05, + "compile_seconds": 0.0015021669678390026, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_92", + "seconds": 0.060717875021509826, + "compile_seconds": 0.017558999941684306, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_93_part_0", + "seconds": 2.0582927390933037e-05, + "compile_seconds": 0.0014643750619143248, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_93", + "seconds": 0.08012904098723084, + "compile_seconds": 0.01925441692583263, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_94_part_0", + "seconds": 2.53340695053339e-05, + "compile_seconds": 0.0015880001010373235, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_94", + "seconds": 0.08035737508907914, + "compile_seconds": 0.0236563750077039, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_95", + "seconds": 0.0007972500752657652, + "compile_seconds": 0.0026308329543098807, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_96_part_0", + "seconds": 2.166605554521084e-05, + "compile_seconds": 0.0014783339574933052, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_96", + "seconds": 0.07718183402903378, + "compile_seconds": 0.01936279097571969, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_97", + "seconds": 0.0007686669705435634, + "compile_seconds": 0.0025952080031856894, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_98", + "seconds": 0.00141587492544204, + "compile_seconds": 0.003882999997586012, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_99", + "seconds": 0.005173791898414493, + "compile_seconds": 0.002454542089253664, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_100_part_0", + "seconds": 2.0875013433396816e-05, + "compile_seconds": 0.0014902499970048666, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_100", + "seconds": 0.05924170801881701, + "compile_seconds": 0.016229374916292727, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_101_part_0", + "seconds": 2.0249979570508003e-05, + "compile_seconds": 0.0014751249691471457, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_101", + "seconds": 0.05841391708236188, + "compile_seconds": 0.016520250006578863, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_102_part_0", + "seconds": 2.2582942619919777e-05, + "compile_seconds": 0.001438500010408461, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_102", + "seconds": 0.12608712504152209, + "compile_seconds": 0.02300650009419769, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_103", + "seconds": 0.018111290992237628, + "compile_seconds": 0.0031458749435842037, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_104_part_0", + "seconds": 2.4666893295943737e-05, + "compile_seconds": 0.0015657920157536864, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_104", + "seconds": 0.06421841692645103, + "compile_seconds": 0.018554375041276217, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_105_part_0", + "seconds": 2.037500962615013e-05, + "compile_seconds": 0.0015566250076517463, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_105", + "seconds": 0.0615038750693202, + "compile_seconds": 0.018063166993670166, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_106_part_0", + "seconds": 1.9666040316224098e-05, + "compile_seconds": 0.001452583004720509, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_106", + "seconds": 0.07896025001537055, + "compile_seconds": 0.021436374983750284, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_107_part_0", + "seconds": 1.9082915969192982e-05, + "compile_seconds": 0.0014262920012697577, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_107", + "seconds": 0.0745100419735536, + "compile_seconds": 0.02112545899581164, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_108", + "seconds": 0.0007736249826848507, + "compile_seconds": 0.002623375039547682, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_109_part_0", + "seconds": 1.7459038645029068e-05, + "compile_seconds": 0.0013612080365419388, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_109", + "seconds": 0.07880845793988556, + "compile_seconds": 0.02091479196678847, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_110", + "seconds": 0.0007120410446077585, + "compile_seconds": 0.0024127919459715486, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_111", + "seconds": 0.0014529579784721136, + "compile_seconds": 0.003952958039008081, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_112", + "seconds": 0.004962582956068218, + "compile_seconds": 0.0024466669419780374, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_113_part_0", + "seconds": 1.9999919459223747e-05, + "compile_seconds": 0.0014387089759111404, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_113", + "seconds": 0.05711387493647635, + "compile_seconds": 0.017955083050765097, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_114_part_0", + "seconds": 2.0082923583686352e-05, + "compile_seconds": 0.0014420829247683287, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_114", + "seconds": 0.05365895794238895, + "compile_seconds": 0.01870116696227342, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_115_part_0", + "seconds": 2.0749983377754688e-05, + "compile_seconds": 0.001509249908849597, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_115", + "seconds": 0.12159687501844019, + "compile_seconds": 0.02498570899479091, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_116", + "seconds": 0.01710716704837978, + "compile_seconds": 0.0034953750437125564, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_117_part_0", + "seconds": 2.145790494978428e-05, + "compile_seconds": 0.001499667065218091, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_117", + "seconds": 0.06690058298408985, + "compile_seconds": 0.02076904207933694, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_118_part_0", + "seconds": 2.0165927708148956e-05, + "compile_seconds": 0.0015129579696804285, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_118", + "seconds": 0.054969207965768874, + "compile_seconds": 0.02009350003208965, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_119_part_0", + "seconds": 2.0125065930187702e-05, + "compile_seconds": 0.0014895410276949406, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_119", + "seconds": 0.0781761669786647, + "compile_seconds": 0.02682908298447728, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_120_part_0", + "seconds": 2.0042061805725098e-05, + "compile_seconds": 0.0014812499284744263, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_120", + "seconds": 0.07781291601713747, + "compile_seconds": 0.023626541951671243, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_121", + "seconds": 0.000795625033788383, + "compile_seconds": 0.0027032920625060797, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_122_part_0", + "seconds": 1.825008075684309e-05, + "compile_seconds": 0.001378374989144504, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_122", + "seconds": 0.08042933396063745, + "compile_seconds": 0.023630290990695357, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_123", + "seconds": 0.0007453750586137176, + "compile_seconds": 0.002522749942727387, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_124", + "seconds": 0.0015254170866683125, + "compile_seconds": 0.0037977920146659017, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_125", + "seconds": 0.00478404201567173, + "compile_seconds": 0.002424375037662685, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_126_part_0", + "seconds": 1.9915984012186527e-05, + "compile_seconds": 0.0014588750200346112, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_126", + "seconds": 0.054853832931257784, + "compile_seconds": 0.019299666048027575, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_127_part_0", + "seconds": 1.9042054191231728e-05, + "compile_seconds": 0.0014787079999223351, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_127", + "seconds": 0.0594222079962492, + "compile_seconds": 0.02076245890930295, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_128_part_0", + "seconds": 1.8208054825663567e-05, + "compile_seconds": 0.0014424169203266501, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_128", + "seconds": 0.11392916704062372, + "compile_seconds": 0.024782249936833978, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_129", + "seconds": 0.017257834086194634, + "compile_seconds": 0.0031429160153493285, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_130_part_0", + "seconds": 2.025009598582983e-05, + "compile_seconds": 0.0014459999511018395, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_130", + "seconds": 0.05996866710484028, + "compile_seconds": 0.02237400005105883, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_131_part_0", + "seconds": 1.7874990589916706e-05, + "compile_seconds": 0.001482499996200204, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_131", + "seconds": 0.05600533296819776, + "compile_seconds": 0.021665708045475185, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_132_part_0", + "seconds": 1.9332976080477238e-05, + "compile_seconds": 0.0014520410913974047, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_132", + "seconds": 0.09680150006897748, + "compile_seconds": 0.029319250024855137, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_133_part_0", + "seconds": 1.9832979887723923e-05, + "compile_seconds": 0.0014745410298928618, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_133", + "seconds": 0.07702995906583965, + "compile_seconds": 0.026825082954019308, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_134", + "seconds": 0.0008926669834181666, + "compile_seconds": 0.0026455840561538935, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_135_part_0", + "seconds": 1.7457990907132626e-05, + "compile_seconds": 0.001384790986776352, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_135", + "seconds": 0.07823824998922646, + "compile_seconds": 0.025803458993323147, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_136", + "seconds": 0.0006571249105036259, + "compile_seconds": 0.0024697909830138087, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_137", + "seconds": 0.0015913749812170863, + "compile_seconds": 0.003896875074133277, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_138", + "seconds": 0.004804749973118305, + "compile_seconds": 0.0024987499928101897, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_139_part_0", + "seconds": 1.8666964024305344e-05, + "compile_seconds": 0.001446707989089191, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_139", + "seconds": 0.057325667003169656, + "compile_seconds": 0.022502582985907793, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_140_part_0", + "seconds": 1.9249971956014633e-05, + "compile_seconds": 0.0014639999717473984, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_140", + "seconds": 0.05669745791237801, + "compile_seconds": 0.022868208005093038, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_141_part_0", + "seconds": 2.0666979253292084e-05, + "compile_seconds": 0.00145800004247576, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_141", + "seconds": 0.12026929203420877, + "compile_seconds": 0.029101249994710088, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_142", + "seconds": 0.016605834010988474, + "compile_seconds": 0.0031705000437796116, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_143_part_0", + "seconds": 1.9875005818903446e-05, + "compile_seconds": 0.0014593329979106784, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_143", + "seconds": 0.06324304093141109, + "compile_seconds": 0.025186541955918074, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_144_part_0", + "seconds": 2.0749983377754688e-05, + "compile_seconds": 0.0014678339939564466, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_144", + "seconds": 0.058321750024333596, + "compile_seconds": 0.02417183294892311, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_145_part_0", + "seconds": 2.0332983694970608e-05, + "compile_seconds": 0.0014711669646203518, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_145", + "seconds": 0.07919845799915493, + "compile_seconds": 0.03544483392033726, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_146_part_0", + "seconds": 1.9583036191761494e-05, + "compile_seconds": 0.001467707916162908, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_146", + "seconds": 0.0800856250571087, + "compile_seconds": 0.028771250043064356, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_147", + "seconds": 0.00076299998909235, + "compile_seconds": 0.002632667077705264, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_saved_148_part_0", + "seconds": 1.979200169444084e-05, + "compile_seconds": 0.0013979169307276607, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_148", + "seconds": 0.07911212497856468, + "compile_seconds": 0.028708167024888098, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_149", + "seconds": 0.0007784159388393164, + "compile_seconds": 0.002472874941304326, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_saved_150", + "seconds": 0.0013545829569920897, + "compile_seconds": 0.0038694579852744937, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_saved_151", + "seconds": 0.005703542032279074, + "compile_seconds": 0.0024008749751374125, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_152_part_0", + "seconds": 1.9083963707089424e-05, + "compile_seconds": 0.0014087919844314456, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_152", + "seconds": 0.06190299999434501, + "compile_seconds": 0.024998834007419646, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_153_part_0", + "seconds": 1.895800232887268e-05, + "compile_seconds": 0.0014729580143466592, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_153", + "seconds": 0.05785020801704377, + "compile_seconds": 0.02721991704311222, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_154_part_0", + "seconds": 2.0125065930187702e-05, + "compile_seconds": 0.0014526669401675463, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_154", + "seconds": 0.11779062496498227, + "compile_seconds": 0.030219540931284428, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_saved_155", + "seconds": 0.020701292087323964, + "compile_seconds": 0.0031356250401586294, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_saved_156_part_0", + "seconds": 1.9915984012186527e-05, + "compile_seconds": 0.001437624916434288, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_156", + "seconds": 0.06392541597597301, + "compile_seconds": 0.028020833036862314, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_157_part_0", + "seconds": 2.004194539040327e-05, + "compile_seconds": 0.0014643330359831452, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_157", + "seconds": 0.06477587495464832, + "compile_seconds": 0.02722004195675254, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_saved_158_part_0", + "seconds": 1.979200169444084e-05, + "compile_seconds": 0.0014601250877603889, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_saved_158", + "seconds": 0.5678354169940576, + "compile_seconds": 0.030550166964530945, + "deferred": false, + "graph": false, + "rows": 804112, + "bytes": 25731584, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_saved_159", + "seconds": 0.03626533399801701, + "compile_seconds": 0.0024101249873638153, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 512 + }, + { + "name": "__ddx_77668c50_1_saved_160", + "seconds": 0.0016221250407397747, + "compile_seconds": 0.0029980839462950826, + "deferred": false, + "graph": false, + "rows": 1, + "bytes": 8 + }, + { + "name": "__ddx_77668c50_1_value", + "seconds": 0.00034991593565791845, + "compile_seconds": 0.0009661660296842456, + "deferred": false, + "graph": false, + "rows": 1, + "bytes": 8 + }, + { + "name": "__ddx_77668c50_1_cotangent_160", + "seconds": 0.00034370901994407177, + "compile_seconds": 0.0010902909561991692, + "deferred": false, + "graph": false, + "rows": 1, + "bytes": 8 + }, + { + "name": "__ddx_77668c50_1_cotangent_159", + "seconds": 0.002033500000834465, + "compile_seconds": 0.003444415982812643, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 512 + }, + { + "name": "__ddx_77668c50_1_cotangent_158", + "seconds": 0.13599233294371516, + "compile_seconds": 0.02805329207330942, + "deferred": false, + "graph": false, + "rows": 804112, + "bytes": 25731584 + }, + { + "name": "__ddx_77668c50_1_contraction_1", + "seconds": 0.22918816597666591, + "compile_seconds": 0.003110458957962692, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_region_2_part_0", + "seconds": 2.4957931600511074e-05, + "compile_seconds": 0.002741917036473751, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_2_part_1", + "seconds": 3.0082999728620052e-05, + "compile_seconds": 0.04550204204861075, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_2", + "seconds": 1.737498678267002e-05, + "compile_seconds": 0.008734249975532293, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_157", + "seconds": 0.057705708080902696, + "compile_seconds": 0.039392041973769665, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_3_part_0", + "seconds": 2.7542002499103546e-05, + "compile_seconds": 0.011706082965247333, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_3", + "seconds": 0.08386504207737744, + "compile_seconds": 0.12162016704678535, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 10321920 + }, + { + "name": "__ddx_77668c50_1_cotangent_156", + "seconds": 0.06506649998482317, + "compile_seconds": 0.040028166957199574, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_4_part_0", + "seconds": 3.0791969038546085e-05, + "compile_seconds": 0.011789124924689531, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_4", + "seconds": 0.07941408397164196, + "compile_seconds": 0.1038732910528779, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 10027008 + }, + { + "name": "__ddx_77668c50_1_cotangent_155", + "seconds": 0.07148295792285353, + "compile_seconds": 0.04229179199319333, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_154", + "seconds": 0.051860042032785714, + "compile_seconds": 0.0066067499574273825, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_contraction_5", + "seconds": 0.04317337495740503, + "compile_seconds": 0.003544667037203908, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_6_part_0", + "seconds": 3.162503708153963e-05, + "compile_seconds": 0.012305042007938027, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_6_part_1", + "seconds": 2.9999995604157448e-05, + "compile_seconds": 0.05828016600571573, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_6", + "seconds": 1.579197123646736e-05, + "compile_seconds": 0.004509374964982271, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_153", + "seconds": 0.06272958300542086, + "compile_seconds": 0.041380624985322356, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_7_part_0", + "seconds": 2.6499968953430653e-05, + "compile_seconds": 0.011626124964095652, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_7", + "seconds": 0.0973525830777362, + "compile_seconds": 0.10750408296007663, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 9928704 + }, + { + "name": "__ddx_77668c50_1_cotangent_152", + "seconds": 0.06466041703242809, + "compile_seconds": 0.04502995905932039, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_8_part_0", + "seconds": 2.8665992431342602e-05, + "compile_seconds": 0.011971750063821673, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_8", + "seconds": 0.07794387510512024, + "compile_seconds": 0.09400124999228865, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 9633792 + }, + { + "name": "__ddx_77668c50_1_cotangent_151", + "seconds": 0.13767904206179082, + "compile_seconds": 0.086554916924797, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_150", + "seconds": 0.02812570903915912, + "compile_seconds": 0.0052417919505387545, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_9", + "seconds": 0.010312624974176288, + "compile_seconds": 0.003735874895937741, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_149", + "seconds": 0.0009931669337674975, + "compile_seconds": 0.005610583000816405, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_148", + "seconds": 0.011217292048968375, + "compile_seconds": 0.007307792082428932, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_12", + "seconds": 0.025719125056639314, + "compile_seconds": 0.003098333952948451, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_13_part_0", + "seconds": 2.38749198615551e-05, + "compile_seconds": 0.011664208024740219, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_13", + "seconds": 2.7499976567924023e-05, + "compile_seconds": 0.052580458926968277, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_147", + "seconds": 0.008898917003534734, + "compile_seconds": 0.029386417008936405, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_146", + "seconds": 0.010533542023040354, + "compile_seconds": 0.005343124968931079, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_14", + "seconds": 0.026544541004113853, + "compile_seconds": 0.0031235830392688513, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_15_part_0", + "seconds": 2.6250025257468224e-05, + "compile_seconds": 0.01138891710434109, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_15", + "seconds": 2.9957969672977924e-05, + "compile_seconds": 0.051680166041478515, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_145", + "seconds": 0.01141758298035711, + "compile_seconds": 0.005365290911868215, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_16", + "seconds": 0.025995583971962333, + "compile_seconds": 0.003113916958682239, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_17_part_0", + "seconds": 2.3707980290055275e-05, + "compile_seconds": 0.011320333927869797, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_17", + "seconds": 2.7499976567924023e-05, + "compile_seconds": 0.05143624998163432, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_144", + "seconds": 0.19908858300186694, + "compile_seconds": 0.11194199998863041, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_18_part_0", + "seconds": 2.7207890525460243e-05, + "compile_seconds": 0.011791791068390012, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_18", + "seconds": 0.09256537503097206, + "compile_seconds": 0.09934608300682157, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 9535488 + }, + { + "name": "__ddx_77668c50_1_cotangent_143", + "seconds": 0.17266795900650322, + "compile_seconds": 0.12144662509672344, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_19_part_0", + "seconds": 2.74579506367445e-05, + "compile_seconds": 0.011669666040688753, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_19", + "seconds": 0.07691554189659655, + "compile_seconds": 0.08884816593490541, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 9240576 + }, + { + "name": "__ddx_77668c50_1_cotangent_142", + "seconds": 0.3228439169470221, + "compile_seconds": 0.206687374971807, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_cotangent_141", + "seconds": 0.05171054194215685, + "compile_seconds": 0.006638082908466458, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_contraction_20", + "seconds": 0.04569295793771744, + "compile_seconds": 0.0036437910748645663, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_21_part_0", + "seconds": 2.9333052225410938e-05, + "compile_seconds": 0.011750624980777502, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_21", + "seconds": 2.970895729959011e-05, + "compile_seconds": 0.04634245904162526, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_140", + "seconds": 0.0652399999089539, + "compile_seconds": 0.03665616700891405, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_22_part_0", + "seconds": 2.7250032871961594e-05, + "compile_seconds": 0.011776083032600582, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_22", + "seconds": 0.07569837500341237, + "compile_seconds": 0.09046620898880064, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 9142272 + }, + { + "name": "__ddx_77668c50_1_cotangent_139", + "seconds": 0.05956666695419699, + "compile_seconds": 0.04097700002603233, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_23_part_0", + "seconds": 3.133295103907585e-05, + "compile_seconds": 0.011853582924231887, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_23", + "seconds": 0.07251979201100767, + "compile_seconds": 0.07906654197722673, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 8847360 + }, + { + "name": "__ddx_77668c50_1_cotangent_138", + "seconds": 0.3728405419969931, + "compile_seconds": 0.2539528749184683, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_cotangent_137", + "seconds": 0.03157075005583465, + "compile_seconds": 0.0055565000511705875, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_24", + "seconds": 0.0093461669748649, + "compile_seconds": 0.0035052080638706684, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_136", + "seconds": 0.0009614999871701002, + "compile_seconds": 0.00556495797354728, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_135", + "seconds": 0.010936209000647068, + "compile_seconds": 0.00849070807453245, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_27", + "seconds": 0.02625533298123628, + "compile_seconds": 0.0032344170613214374, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_28_part_0", + "seconds": 2.5916961021721363e-05, + "compile_seconds": 0.011628375039435923, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_28", + "seconds": 3.14170029014349e-05, + "compile_seconds": 0.04145387501921505, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_134", + "seconds": 0.009425833006389439, + "compile_seconds": 0.028511333046481013, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_133", + "seconds": 0.01087650004774332, + "compile_seconds": 0.00521733402274549, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_29", + "seconds": 0.026591750094667077, + "compile_seconds": 0.003663124982267618, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_30_part_0", + "seconds": 2.6291003450751305e-05, + "compile_seconds": 0.011572540970519185, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_30", + "seconds": 2.995808608829975e-05, + "compile_seconds": 0.04069595807231963, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_132", + "seconds": 0.011079875053837895, + "compile_seconds": 0.005590750020928681, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_31", + "seconds": 0.025288415956310928, + "compile_seconds": 0.003247041953727603, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_32_part_0", + "seconds": 2.7749920263886452e-05, + "compile_seconds": 0.011588708031922579, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_32", + "seconds": 2.462498378008604e-05, + "compile_seconds": 0.04063608392607421, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_131", + "seconds": 0.18548504100181162, + "compile_seconds": 0.10335533297620714, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_33_part_0", + "seconds": 3.074994310736656e-05, + "compile_seconds": 0.012400500010699034, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_33", + "seconds": 0.07831391599029303, + "compile_seconds": 0.08260212501045316, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 8749056 + }, + { + "name": "__ddx_77668c50_1_cotangent_130", + "seconds": 0.17859341704752296, + "compile_seconds": 0.10871195804793388, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_34_part_0", + "seconds": 2.675002906471491e-05, + "compile_seconds": 0.011790041928179562, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_34", + "seconds": 0.07308904093224555, + "compile_seconds": 0.07053470809478313, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 8454144 + }, + { + "name": "__ddx_77668c50_1_cotangent_129", + "seconds": 0.6042519160546362, + "compile_seconds": 0.35767175001092255, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_cotangent_128", + "seconds": 0.053498624940402806, + "compile_seconds": 0.006516291992738843, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_contraction_35", + "seconds": 0.04020279203541577, + "compile_seconds": 0.0036590000381693244, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_36_part_0", + "seconds": 2.6834080927073956e-05, + "compile_seconds": 0.011683457996696234, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_36", + "seconds": 2.595805563032627e-05, + "compile_seconds": 0.035845624981448054, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_127", + "seconds": 0.06694491696543992, + "compile_seconds": 0.03464375005569309, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_37_part_0", + "seconds": 3.0374969355762005e-05, + "compile_seconds": 0.011699041933752596, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_37", + "seconds": 0.07562120794318616, + "compile_seconds": 0.07609829097054899, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 8355840 + }, + { + "name": "__ddx_77668c50_1_cotangent_126", + "seconds": 0.06294774997513741, + "compile_seconds": 0.03713737509679049, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_38_part_0", + "seconds": 2.6416033506393433e-05, + "compile_seconds": 0.011579374899156392, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_38", + "seconds": 0.07143750006798655, + "compile_seconds": 0.0653774170204997, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 8060928 + }, + { + "name": "__ddx_77668c50_1_cotangent_125", + "seconds": 0.6721989169018343, + "compile_seconds": 0.406039834022522, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_cotangent_124", + "seconds": 0.03276158298831433, + "compile_seconds": 0.005526917055249214, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_39", + "seconds": 0.010723292012698948, + "compile_seconds": 0.0036303329980000854, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_123", + "seconds": 0.000927999964915216, + "compile_seconds": 0.005670208018273115, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_122", + "seconds": 0.012553540989756584, + "compile_seconds": 0.007388082914985716, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_42", + "seconds": 0.026480833999812603, + "compile_seconds": 0.003238750039599836, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_43_part_0", + "seconds": 3.0290917493402958e-05, + "compile_seconds": 0.011769999982789159, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_43", + "seconds": 2.8250040486454964e-05, + "compile_seconds": 0.031140792067162693, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_121", + "seconds": 0.00952016597148031, + "compile_seconds": 0.02955070894677192, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_120", + "seconds": 0.01110208302270621, + "compile_seconds": 0.005223874934017658, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_44", + "seconds": 0.02635954099241644, + "compile_seconds": 0.00316062499769032, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_45_part_0", + "seconds": 2.4792039766907692e-05, + "compile_seconds": 0.01149324991274625, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_45", + "seconds": 2.5916029699146748e-05, + "compile_seconds": 0.03130391601007432, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_119", + "seconds": 0.011356958071701229, + "compile_seconds": 0.005437541054561734, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_46", + "seconds": 0.026319041964598, + "compile_seconds": 0.0031440419843420386, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_47_part_0", + "seconds": 2.350006252527237e-05, + "compile_seconds": 0.011348792002536356, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_47", + "seconds": 2.437504008412361e-05, + "compile_seconds": 0.030942499986849725, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_118", + "seconds": 0.20648729102686048, + "compile_seconds": 0.09259179094806314, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_region_48_part_0", + "seconds": 2.916704397648573e-05, + "compile_seconds": 0.011826374917291105, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_48", + "seconds": 0.076638541999273, + "compile_seconds": 0.06782650004606694, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 7962624 + }, + { + "name": "__ddx_77668c50_1_cotangent_117", + "seconds": 0.21248766605276614, + "compile_seconds": 0.09813037503045052, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_region_49_part_0", + "seconds": 2.862501423805952e-05, + "compile_seconds": 0.011744917021133006, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_49", + "seconds": 0.07313508295919746, + "compile_seconds": 0.05785570794250816, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 7667712 + }, + { + "name": "__ddx_77668c50_1_cotangent_116", + "seconds": 0.9251605829922482, + "compile_seconds": 0.5073600000469014, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_cotangent_115", + "seconds": 0.06705462501849979, + "compile_seconds": 0.006723625003360212, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_contraction_50", + "seconds": 0.04027845896780491, + "compile_seconds": 0.0036059579579159617, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_51_part_0", + "seconds": 2.8083915822207928e-05, + "compile_seconds": 0.011772749945521355, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_51", + "seconds": 2.5042099878191948e-05, + "compile_seconds": 0.027346292044967413, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_114", + "seconds": 0.06475766689982265, + "compile_seconds": 0.032758458983153105, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_52_part_0", + "seconds": 2.6499968953430653e-05, + "compile_seconds": 0.011563125066459179, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_52", + "seconds": 0.06857654196210206, + "compile_seconds": 0.06209483300335705, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 7569408 + }, + { + "name": "__ddx_77668c50_1_cotangent_113", + "seconds": 0.06009624991565943, + "compile_seconds": 0.033519208896905184, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_53_part_0", + "seconds": 2.9666000045835972e-05, + "compile_seconds": 0.01210299995727837, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_53", + "seconds": 0.06808991695288569, + "compile_seconds": 0.05267820798326284, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 7274496 + }, + { + "name": "__ddx_77668c50_1_cotangent_112", + "seconds": 0.9887757909018546, + "compile_seconds": 0.543890458997339, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_cotangent_111", + "seconds": 0.036072000046260655, + "compile_seconds": 0.005328958970494568, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_54", + "seconds": 0.009716957923956215, + "compile_seconds": 0.0035917910281568766, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_110", + "seconds": 0.000960333039984107, + "compile_seconds": 0.00548270798753947, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_109", + "seconds": 0.012180833029560745, + "compile_seconds": 0.007668665959499776, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_57", + "seconds": 0.027386458008550107, + "compile_seconds": 0.003232083050534129, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_58_part_0", + "seconds": 2.7791946195065975e-05, + "compile_seconds": 0.011548458016477525, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_58", + "seconds": 3.1458097510039806e-05, + "compile_seconds": 0.02353120897896588, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_108", + "seconds": 0.008764583035372198, + "compile_seconds": 0.02999112498946488, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_107", + "seconds": 0.01332791696768254, + "compile_seconds": 0.005306417006067932, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_59", + "seconds": 0.027412874973379076, + "compile_seconds": 0.003244832972995937, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_60_part_0", + "seconds": 2.5707995519042015e-05, + "compile_seconds": 0.011522958986461163, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_60", + "seconds": 2.6124995201826096e-05, + "compile_seconds": 0.02305033290758729, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_106", + "seconds": 0.012232374981977046, + "compile_seconds": 0.005339875002391636, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_61", + "seconds": 0.025628041941672564, + "compile_seconds": 0.0032557910308241844, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_62_part_0", + "seconds": 2.6250025257468224e-05, + "compile_seconds": 0.011404791963286698, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_62", + "seconds": 2.762500662356615e-05, + "compile_seconds": 0.022924666991457343, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_105", + "seconds": 0.17379587492905557, + "compile_seconds": 0.0856778749730438, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_63_part_0", + "seconds": 2.879102248698473e-05, + "compile_seconds": 0.011745000025257468, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_63", + "seconds": 0.07283958292100579, + "compile_seconds": 0.05498354195151478, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 7176192 + }, + { + "name": "__ddx_77668c50_1_cotangent_104", + "seconds": 0.16518420807551593, + "compile_seconds": 0.0893152920762077, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_64_part_0", + "seconds": 2.7709058485925198e-05, + "compile_seconds": 0.011845041066408157, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_64", + "seconds": 0.07042208395432681, + "compile_seconds": 0.04594945802818984, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 6881280 + }, + { + "name": "__ddx_77668c50_1_cotangent_103", + "seconds": 1.8376121249748394, + "compile_seconds": 0.03470079193357378, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_102", + "seconds": 0.06407316704280674, + "compile_seconds": 0.007088124984875321, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_contraction_65", + "seconds": 0.04242841701488942, + "compile_seconds": 0.00468758400529623, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_66_part_0", + "seconds": 2.9582995921373367e-05, + "compile_seconds": 0.011704874923452735, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_66", + "seconds": 3.0458904802799225e-05, + "compile_seconds": 0.01934754103422165, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_101", + "seconds": 0.06234683294314891, + "compile_seconds": 0.02772925002500415, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_67_part_0", + "seconds": 2.6332912966609e-05, + "compile_seconds": 0.011643540929071605, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_67", + "seconds": 0.0691028330475092, + "compile_seconds": 0.0486531670903787, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 6782976 + }, + { + "name": "__ddx_77668c50_1_cotangent_100", + "seconds": 0.06059570796787739, + "compile_seconds": 0.03069904202129692, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_68_part_0", + "seconds": 2.720800694078207e-05, + "compile_seconds": 0.011584958992898464, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_68", + "seconds": 0.06989633396733552, + "compile_seconds": 0.04107229202054441, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 6488064 + }, + { + "name": "__ddx_77668c50_1_cotangent_99", + "seconds": 1.9933694589417428, + "compile_seconds": 0.03675045899581164, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_98", + "seconds": 0.029232957982458174, + "compile_seconds": 0.0057004590053111315, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_69", + "seconds": 0.010601499932818115, + "compile_seconds": 0.003769458970054984, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_97", + "seconds": 0.0009832920040935278, + "compile_seconds": 0.005755042075179517, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_96", + "seconds": 0.010922832996584475, + "compile_seconds": 0.007701499969698489, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_72", + "seconds": 0.02666370803490281, + "compile_seconds": 0.0032403749646618962, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_73_part_0", + "seconds": 2.6875059120357037e-05, + "compile_seconds": 0.011619584052823484, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_73", + "seconds": 2.791709266602993e-05, + "compile_seconds": 0.016496207914315164, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_95", + "seconds": 0.00900787499267608, + "compile_seconds": 0.029762167017906904, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_94", + "seconds": 0.010672166012227535, + "compile_seconds": 0.005437292042188346, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_74", + "seconds": 0.02639195800293237, + "compile_seconds": 0.003215332981199026, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_75_part_0", + "seconds": 2.537504769861698e-05, + "compile_seconds": 0.011409249971620739, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_75", + "seconds": 2.437504008412361e-05, + "compile_seconds": 0.01625533401966095, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_93", + "seconds": 0.010961374966427684, + "compile_seconds": 0.00520704104565084, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_76", + "seconds": 0.02626679209060967, + "compile_seconds": 0.003199749975465238, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_77_part_0", + "seconds": 2.8709066100418568e-05, + "compile_seconds": 0.011888708919286728, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_77", + "seconds": 2.9209069907665253e-05, + "compile_seconds": 0.016578500042669475, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_92", + "seconds": 0.1962049170397222, + "compile_seconds": 0.07689724990632385, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_78_part_0", + "seconds": 2.90420139208436e-05, + "compile_seconds": 0.011646167025901377, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_78", + "seconds": 0.06969262508209795, + "compile_seconds": 0.04352429194841534, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 6389760 + }, + { + "name": "__ddx_77668c50_1_cotangent_91", + "seconds": 0.170298625016585, + "compile_seconds": 0.08065458305645734, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_79_part_0", + "seconds": 2.650008536875248e-05, + "compile_seconds": 0.011797291925176978, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_79", + "seconds": 0.06789029203355312, + "compile_seconds": 0.03733595903031528, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 6094848 + }, + { + "name": "__ddx_77668c50_1_cotangent_90", + "seconds": 2.2552877909038216, + "compile_seconds": 0.041345917037688196, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_89", + "seconds": 0.050599750014953315, + "compile_seconds": 0.006371792056597769, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_contraction_80", + "seconds": 0.04058629192877561, + "compile_seconds": 0.0036297079641371965, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_81_part_0", + "seconds": 2.74579506367445e-05, + "compile_seconds": 0.011559000005945563, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_81", + "seconds": 3.12920892611146e-05, + "compile_seconds": 0.013845834066160023, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_88", + "seconds": 0.058036250062286854, + "compile_seconds": 0.024539084057323635, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_82_part_0", + "seconds": 2.775003667920828e-05, + "compile_seconds": 0.011498917010612786, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_82", + "seconds": 0.06511633307673037, + "compile_seconds": 0.03824333392549306, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 5996544 + }, + { + "name": "__ddx_77668c50_1_cotangent_87", + "seconds": 0.05621399998199195, + "compile_seconds": 0.027035792008973658, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_83_part_0", + "seconds": 0.06669704196974635, + "compile_seconds": 0.03458187496289611, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4620288 + }, + { + "name": "__ddx_77668c50_1_region_83", + "seconds": 2.0832987502217293e-05, + "compile_seconds": 0.0019288749899715185, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_86", + "seconds": 2.363810332957655, + "compile_seconds": 0.04275016696192324, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_85", + "seconds": 0.027546625002287328, + "compile_seconds": 0.005023041972890496, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_84", + "seconds": 0.009953915956430137, + "compile_seconds": 0.003416249994188547, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_84", + "seconds": 0.002023500041104853, + "compile_seconds": 0.006091917050071061, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_83", + "seconds": 0.010290708974935114, + "compile_seconds": 0.007447166950441897, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_87", + "seconds": 0.026293499977327883, + "compile_seconds": 0.003218207973986864, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_88_part_0", + "seconds": 2.7875066734850407e-05, + "compile_seconds": 0.011983415926806629, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_88", + "seconds": 1.750001683831215e-05, + "compile_seconds": 0.010475084069184959, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_82", + "seconds": 0.008754208101890981, + "compile_seconds": 0.028206625021994114, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_81", + "seconds": 0.01026749995071441, + "compile_seconds": 0.00510745809879154, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_89", + "seconds": 0.025893959100358188, + "compile_seconds": 0.00315091700758785, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_90_part_0", + "seconds": 2.4374923668801785e-05, + "compile_seconds": 0.011387709062546492, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_90", + "seconds": 1.6666948795318604e-05, + "compile_seconds": 0.010333458078093827, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_80", + "seconds": 0.010728333028964698, + "compile_seconds": 0.00529170804657042, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_91", + "seconds": 0.025848207995295525, + "compile_seconds": 0.0030550839146599174, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_92_part_0", + "seconds": 2.3417058400809765e-05, + "compile_seconds": 0.011448750039562583, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_92", + "seconds": 2.1124957129359245e-05, + "compile_seconds": 0.010592291015200317, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_79", + "seconds": 0.17710591608192772, + "compile_seconds": 0.06626916700042784, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_93_part_0", + "seconds": 0.06697737495414913, + "compile_seconds": 0.034610125003382564, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4718592 + }, + { + "name": "__ddx_77668c50_1_region_93", + "seconds": 2.0416919142007828e-05, + "compile_seconds": 0.0024646249366924167, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_78", + "seconds": 0.16314199997577816, + "compile_seconds": 0.07332362502347678, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_94_part_0", + "seconds": 0.07889883301686496, + "compile_seconds": 0.033951333025470376, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4030464 + }, + { + "name": "__ddx_77668c50_1_region_94", + "seconds": 0.0018761249957606196, + "compile_seconds": 0.0030290830181911588, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 5308416 + }, + { + "name": "__ddx_77668c50_1_cotangent_77", + "seconds": 2.63158937508706, + "compile_seconds": 0.04782091605011374, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_76", + "seconds": 0.06362154195085168, + "compile_seconds": 0.014570208033546805, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_contraction_95", + "seconds": 0.04376295895781368, + "compile_seconds": 0.004724459024146199, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_96_part_0", + "seconds": 4.02920413762331e-05, + "compile_seconds": 0.013322958024218678, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_96", + "seconds": 2.679100725799799e-05, + "compile_seconds": 0.00901024998165667, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_75", + "seconds": 0.06775716692209244, + "compile_seconds": 0.025390333961695433, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_97_part_0", + "seconds": 0.07081891596317291, + "compile_seconds": 0.03714045905508101, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4128768 + }, + { + "name": "__ddx_77668c50_1_region_97", + "seconds": 0.008294790983200073, + "compile_seconds": 0.0045278751058503985, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 5210112 + }, + { + "name": "__ddx_77668c50_1_region_98", + "seconds": 0.1231116660637781, + "compile_seconds": 0.061827249941416085, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4915200 + }, + { + "name": "__ddx_77668c50_1_cotangent_73", + "seconds": 2.7597610409138724, + "compile_seconds": 0.05152237508445978, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_72", + "seconds": 0.027777082985267043, + "compile_seconds": 0.005162208923138678, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_99", + "seconds": 0.009504959103651345, + "compile_seconds": 0.0035467089619487524, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_71", + "seconds": 0.0009360420517623425, + "compile_seconds": 0.005569417029619217, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_70", + "seconds": 0.010849666083231568, + "compile_seconds": 0.007470250013284385, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_102", + "seconds": 0.026070208987221122, + "compile_seconds": 0.003215541015379131, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_103_part_0", + "seconds": 2.833292819559574e-05, + "compile_seconds": 0.011425624950788915, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_103", + "seconds": 1.579197123646736e-05, + "compile_seconds": 0.005781750078313053, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_69", + "seconds": 0.00907666701823473, + "compile_seconds": 0.028724542004056275, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_68", + "seconds": 0.01040354196447879, + "compile_seconds": 0.005442208028398454, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_104", + "seconds": 0.026295042014680803, + "compile_seconds": 0.0033749579451978207, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_105_part_0", + "seconds": 2.4250010028481483e-05, + "compile_seconds": 0.011232167016714811, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_105", + "seconds": 2.091703936457634e-05, + "compile_seconds": 0.0058536670403555036, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_67", + "seconds": 0.012566374964080751, + "compile_seconds": 0.005404957919381559, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_106", + "seconds": 0.026038791052997112, + "compile_seconds": 0.003160750027745962, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_107_part_0", + "seconds": 2.762500662356615e-05, + "compile_seconds": 0.011432458064518869, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_107", + "seconds": 1.3708020560443401e-05, + "compile_seconds": 0.005714417085982859, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_66", + "seconds": 0.17567620798945427, + "compile_seconds": 0.05783370905555785, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_region_108_part_0", + "seconds": 0.06950541702099144, + "compile_seconds": 0.034653250011615455, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4816896 + }, + { + "name": "__ddx_77668c50_1_region_108", + "seconds": 0.0024469579802826047, + "compile_seconds": 0.0025826250202953815, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4816896 + }, + { + "name": "__ddx_77668c50_1_region_109", + "seconds": 0.24583500006701797, + "compile_seconds": 0.10360699996817857, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4521984, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_cotangent_64", + "seconds": 2.99034879100509, + "compile_seconds": 0.05448579101357609, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_63", + "seconds": 0.06042691704351455, + "compile_seconds": 0.006627541966736317, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_contraction_110", + "seconds": 0.0385143329622224, + "compile_seconds": 0.003515958902426064, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_111_part_0", + "seconds": 2.8165988624095917e-05, + "compile_seconds": 0.011423000018112361, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_111", + "seconds": 1.4542019926011562e-05, + "compile_seconds": 0.003562791971489787, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_112", + "seconds": 0.1062448329757899, + "compile_seconds": 0.04903666605241597, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4423680 + }, + { + "name": "__ddx_77668c50_1_region_113", + "seconds": 0.10739079199265689, + "compile_seconds": 0.04750866605900228, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4128768 + }, + { + "name": "__ddx_77668c50_1_cotangent_60", + "seconds": 3.0548657500185072, + "compile_seconds": 0.054998665931634605, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_59", + "seconds": 0.029375583049841225, + "compile_seconds": 0.004981667036190629, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_114", + "seconds": 0.01032187498640269, + "compile_seconds": 0.0036512919468805194, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_58", + "seconds": 0.0009709169389680028, + "compile_seconds": 0.005776125006377697, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_57", + "seconds": 0.01194716605823487, + "compile_seconds": 0.007706957985647023, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_117", + "seconds": 0.026498667080886662, + "compile_seconds": 0.0032870410941541195, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_118_part_0", + "seconds": 2.6165973395109177e-05, + "compile_seconds": 0.011207666946575046, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_118", + "seconds": 1.108401920646429e-05, + "compile_seconds": 0.0017679999582469463, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_56", + "seconds": 0.009193707956001163, + "compile_seconds": 0.02843512501567602, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_55", + "seconds": 0.010250957915559411, + "compile_seconds": 0.005305167054757476, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_119", + "seconds": 0.026454624952748418, + "compile_seconds": 0.003257708973251283, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_120_part_0", + "seconds": 2.3667002096772194e-05, + "compile_seconds": 0.011236999998800457, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_120", + "seconds": 1.108401920646429e-05, + "compile_seconds": 0.0017364999512210488, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_cotangent_54", + "seconds": 0.012044292059727013, + "compile_seconds": 0.005392666906118393, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_121", + "seconds": 0.026352624990977347, + "compile_seconds": 0.003366874996572733, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_region_122_part_0", + "seconds": 3.0791969038546085e-05, + "compile_seconds": 0.011377666960470378, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_122", + "seconds": 1.1625001206994057e-05, + "compile_seconds": 0.0017608749913051724, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_region_123", + "seconds": 0.26778837491292506, + "compile_seconds": 0.08040733309462667, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4030464, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_region_124", + "seconds": 0.23227116593625396, + "compile_seconds": 0.0782474169973284, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3735552, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_cotangent_51", + "seconds": 3.34633904194925, + "compile_seconds": 0.05778675002511591, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_50", + "seconds": 0.0507029170403257, + "compile_seconds": 0.006459833006374538, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_region_126", + "seconds": 0.09893162501975894, + "compile_seconds": 0.03097762493416667, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 4227072 + }, + { + "name": "__ddx_77668c50_1_region_127", + "seconds": 0.061002292088232934, + "compile_seconds": 0.023876375053077936, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3637248 + }, + { + "name": "__ddx_77668c50_1_region_128", + "seconds": 0.058004042017273605, + "compile_seconds": 0.021266708965413272, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3342336 + }, + { + "name": "__ddx_77668c50_1_cotangent_47", + "seconds": 3.381214957917109, + "compile_seconds": 0.05891899997368455, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_46", + "seconds": 0.02793266694061458, + "compile_seconds": 0.005083916941657662, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_129", + "seconds": 0.009574875002726912, + "compile_seconds": 0.003459500032477081, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_45", + "seconds": 0.0009919999865815043, + "compile_seconds": 0.0055542909540236, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_44", + "seconds": 0.011616250034421682, + "compile_seconds": 0.007256333017721772, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_133", + "seconds": 0.07911391602829099, + "compile_seconds": 0.029719957965426147, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3833856 + }, + { + "name": "__ddx_77668c50_1_cotangent_43", + "seconds": 0.009020666941069067, + "compile_seconds": 0.028833790915086865, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_42", + "seconds": 0.010508999926969409, + "compile_seconds": 0.005333125009201467, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_135", + "seconds": 0.08656133397016674, + "compile_seconds": 0.027415249962359667, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3833856 + }, + { + "name": "__ddx_77668c50_1_cotangent_41", + "seconds": 0.011059666983783245, + "compile_seconds": 0.005785749992355704, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_137", + "seconds": 0.08138579095248133, + "compile_seconds": 0.02805308299139142, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3833856 + }, + { + "name": "__ddx_77668c50_1_region_138", + "seconds": 0.057322915992699564, + "compile_seconds": 0.022695333929732442, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3244032 + }, + { + "name": "__ddx_77668c50_1_region_139", + "seconds": 0.06327116605825722, + "compile_seconds": 0.020586917060427368, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2949120 + }, + { + "name": "__ddx_77668c50_1_cotangent_38", + "seconds": 3.414045708021149, + "compile_seconds": 0.05989945901092142, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_37", + "seconds": 0.06071445799898356, + "compile_seconds": 0.0064308339497074485, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_region_141", + "seconds": 0.10015566693618894, + "compile_seconds": 0.024487041984684765, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3440640 + }, + { + "name": "__ddx_77668c50_1_region_142", + "seconds": 0.058600957971066236, + "compile_seconds": 0.017650291905738413, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2850816 + }, + { + "name": "__ddx_77668c50_1_region_143", + "seconds": 0.05597191606648266, + "compile_seconds": 0.01613041607197374, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2555904 + }, + { + "name": "__ddx_77668c50_1_cotangent_34", + "seconds": 3.444257374969311, + "compile_seconds": 0.06056887505110353, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_33", + "seconds": 0.027718459023162723, + "compile_seconds": 0.0052541670156642795, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_144", + "seconds": 0.009659832925535738, + "compile_seconds": 0.003527207998558879, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_32", + "seconds": 0.0011042499681934714, + "compile_seconds": 0.005674332962371409, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_31", + "seconds": 0.011012959061190486, + "compile_seconds": 0.007521667052060366, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_148", + "seconds": 0.08023829106241465, + "compile_seconds": 0.02160674997139722, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3047424 + }, + { + "name": "__ddx_77668c50_1_cotangent_30", + "seconds": 0.009136374923400581, + "compile_seconds": 0.029007208067923784, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_29", + "seconds": 0.010522959055379033, + "compile_seconds": 0.005348958075046539, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_150", + "seconds": 0.07610620791092515, + "compile_seconds": 0.021843124995939434, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3047424 + }, + { + "name": "__ddx_77668c50_1_cotangent_28", + "seconds": 0.011425500037148595, + "compile_seconds": 0.0058180829510092735, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_152", + "seconds": 0.07607670791912824, + "compile_seconds": 0.021466000005602837, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 3047424 + }, + { + "name": "__ddx_77668c50_1_region_153", + "seconds": 0.062334165908396244, + "compile_seconds": 0.017567124916240573, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2457600 + }, + { + "name": "__ddx_77668c50_1_region_154", + "seconds": 0.05879200005438179, + "compile_seconds": 0.01708087499719113, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2162688 + }, + { + "name": "__ddx_77668c50_1_cotangent_25", + "seconds": 3.748901999904774, + "compile_seconds": 0.06500524992588907, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_24", + "seconds": 0.06139124999754131, + "compile_seconds": 0.00659308407921344, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_region_156", + "seconds": 0.10600324999541044, + "compile_seconds": 0.02058308303821832, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2654208 + }, + { + "name": "__ddx_77668c50_1_region_157", + "seconds": 0.058738791034556925, + "compile_seconds": 0.01392395794391632, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2064384 + }, + { + "name": "__ddx_77668c50_1_region_158", + "seconds": 0.05971570801921189, + "compile_seconds": 0.012444458086974919, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 1769472 + }, + { + "name": "__ddx_77668c50_1_cotangent_21", + "seconds": 3.78360225004144, + "compile_seconds": 0.06185545795597136, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_20", + "seconds": 0.033147166948765516, + "compile_seconds": 0.0058624999364838, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_159", + "seconds": 0.01250512502156198, + "compile_seconds": 0.0045458750100806355, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_19", + "seconds": 0.0014916249783709645, + "compile_seconds": 0.006031458033248782, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_18", + "seconds": 0.010886540985666215, + "compile_seconds": 0.008245042059570551, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_163", + "seconds": 0.08381516602821648, + "compile_seconds": 0.020811166032217443, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2260992 + }, + { + "name": "__ddx_77668c50_1_cotangent_17", + "seconds": 0.011634666938334703, + "compile_seconds": 0.03308783296961337, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_16", + "seconds": 0.011490958044305444, + "compile_seconds": 0.00705700006801635, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_165", + "seconds": 0.09473212494049221, + "compile_seconds": 0.021300625056028366, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2260992 + }, + { + "name": "__ddx_77668c50_1_cotangent_15", + "seconds": 0.012906749965623021, + "compile_seconds": 0.006430084002204239, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_167", + "seconds": 0.09221333300229162, + "compile_seconds": 0.022528458037413657, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 2260992 + }, + { + "name": "__ddx_77668c50_1_region_168", + "seconds": 0.06276095798239112, + "compile_seconds": 0.015548708033747971, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 1671168 + }, + { + "name": "__ddx_77668c50_1_region_169", + "seconds": 0.0598087910329923, + "compile_seconds": 0.012990999966859818, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 1376256 + }, + { + "name": "__ddx_77668c50_1_cotangent_12", + "seconds": 3.8010427500121295, + "compile_seconds": 0.07009562500752509, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_11", + "seconds": 0.06231862504500896, + "compile_seconds": 0.006643209024332464, + "deferred": false, + "graph": false, + "rows": 49152, + "bytes": 1572864 + }, + { + "name": "__ddx_77668c50_1_region_171", + "seconds": 0.09283883403986692, + "compile_seconds": 0.014934042003005743, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 1867776 + }, + { + "name": "__ddx_77668c50_1_region_172", + "seconds": 0.05955383402761072, + "compile_seconds": 0.009208125062286854, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 1277952 + }, + { + "name": "__ddx_77668c50_1_region_173", + "seconds": 0.049293708987534046, + "compile_seconds": 0.009170499979518354, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 983040 + }, + { + "name": "__ddx_77668c50_1_cotangent_8", + "seconds": 3.5893434589961544, + "compile_seconds": 0.06416683399584144, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 393216 + }, + { + "name": "__ddx_77668c50_1_cotangent_7", + "seconds": 0.0292200839612633, + "compile_seconds": 0.00515641609672457, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_contraction_174", + "seconds": 0.009429875062778592, + "compile_seconds": 0.003512584022246301, + "deferred": false, + "graph": false, + "rows": 3072, + "bytes": 122880 + }, + { + "name": "__ddx_77668c50_1_cotangent_6", + "seconds": 0.000930874957703054, + "compile_seconds": 0.005673832958564162, + "deferred": false, + "graph": false, + "rows": 192, + "bytes": 6144 + }, + { + "name": "__ddx_77668c50_1_cotangent_5", + "seconds": 0.010891583049669862, + "compile_seconds": 0.007465333095751703, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_178", + "seconds": 0.07440466701518744, + "compile_seconds": 0.011995582957752049, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 1474560 + }, + { + "name": "__ddx_77668c50_1_cotangent_4", + "seconds": 0.009021625039167702, + "compile_seconds": 0.029010333004407585, + "deferred": false, + "graph": false, + "rows": 1632, + "bytes": 65280 + }, + { + "name": "__ddx_77668c50_1_cotangent_3", + "seconds": 0.010404875036329031, + "compile_seconds": 0.005516084027476609, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_180", + "seconds": 0.07736595906317234, + "compile_seconds": 0.012461874983273447, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 1474560 + }, + { + "name": "__ddx_77668c50_1_cotangent_2", + "seconds": 0.01177833299152553, + "compile_seconds": 0.0060718749882653356, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 491520 + }, + { + "name": "__ddx_77668c50_1_region_182", + "seconds": 0.07210104097612202, + "compile_seconds": 0.012475040974095464, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 1474560 + }, + { + "name": "__ddx_77668c50_1_region_183", + "seconds": 0.057141083059832454, + "compile_seconds": 0.009619582910090685, + "deferred": false, + "graph": false, + "rows": 12288, + "bytes": 884736 + }, + { + "name": "__ddx_77668c50_1_cotangent_0", + "seconds": 0.008561374968849123, + "compile_seconds": 0.0049606249667704105, + "deferred": false, + "graph": false, + "rows": 16, + "bytes": 384 + }, + { + "name": "__ddx_77668c50_1_grad_0_wte_part_0", + "seconds": 2.1709012798964977e-05, + "compile_seconds": 0.003888209001161158, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_0_wte_part_1", + "seconds": 16.806742792017758, + "compile_seconds": 0.043255291995592415, + "deferred": false, + "graph": false, + "rows": 38597376, + "bytes": 926337024, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_0_wte", + "seconds": 18.579415083979256, + "compile_seconds": 0.0926483339862898, + "deferred": false, + "graph": true, + "rows": 38597376, + "bytes": 926337024 + }, + { + "name": "__ddx_77668c50_1_grad_1_wpe", + "seconds": 3.673719041980803, + "compile_seconds": 0.07160525000654161, + "deferred": false, + "graph": true, + "rows": 12288, + "bytes": 294912 + }, + { + "name": "__ddx_77668c50_1_grad_2_ln_1", + "seconds": 2.2079905829159543, + "compile_seconds": 0.045101875090040267, + "deferred": false, + "graph": true, + "rows": 9216, + "bytes": 221184 + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_0", + "seconds": 2.90420139208436e-05, + "compile_seconds": 0.0040955409640446305, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_1", + "seconds": 0.2470030830008909, + "compile_seconds": 0.04245795798487961, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 23592960, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_2", + "seconds": 2.629193477332592e-05, + "compile_seconds": 0.004128750064410269, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_3", + "seconds": 2.3083994165062904e-05, + "compile_seconds": 0.0038661659928038716, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_4", + "seconds": 2.0708073861896992e-05, + "compile_seconds": 0.0036420420510694385, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_5", + "seconds": 1.825008075684309e-05, + "compile_seconds": 0.0037308750906959176, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_6", + "seconds": 1.7417012713849545e-05, + "compile_seconds": 0.0036279159830883145, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_7", + "seconds": 1.3915938325226307e-05, + "compile_seconds": 0.0036509999772533774, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_8", + "seconds": 0.17627004103269428, + "compile_seconds": 0.02181895903777331, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 42467328 + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_9", + "seconds": 0.22835233400110155, + "compile_seconds": 0.019840375054627657, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 23592960, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_3_wq_part_10", + "seconds": 1.7885197499999776, + "compile_seconds": 0.224489290965721, + "deferred": false, + "graph": false, + "rows": 7077888, + "bytes": 566231040, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_3_wq", + "seconds": 0.32888204196933657, + "compile_seconds": 0.002308333059772849, + "deferred": false, + "graph": false, + "rows": 7077888, + "bytes": 283115520, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_0", + "seconds": 2.4458044208586216e-05, + "compile_seconds": 0.004090667003765702, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_1", + "seconds": 0.2516533749876544, + "compile_seconds": 0.048214040929451585, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 23592960, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_2", + "seconds": 2.279097680002451e-05, + "compile_seconds": 0.004125542007386684, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_3", + "seconds": 2.0041014067828655e-05, + "compile_seconds": 0.0038715829141438007, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_4", + "seconds": 1.750001683831215e-05, + "compile_seconds": 0.0037531660636886954, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_5", + "seconds": 1.5583005733788013e-05, + "compile_seconds": 0.0038021670188754797, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_6", + "seconds": 1.5458092093467712e-05, + "compile_seconds": 0.0037269999738782644, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_7", + "seconds": 2.1125073544681072e-05, + "compile_seconds": 0.003829624969512224, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_8", + "seconds": 0.16841829207260162, + "compile_seconds": 0.02176437503658235, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 42467328 + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_9", + "seconds": 0.23982133297249675, + "compile_seconds": 0.019535915926098824, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 23592960, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_4_wk_part_10", + "seconds": 1.844542457954958, + "compile_seconds": 0.22304433293174952, + "deferred": false, + "graph": false, + "rows": 7077888, + "bytes": 566231040, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_4_wk", + "seconds": 0.3327935839770362, + "compile_seconds": 0.00220512505620718, + "deferred": false, + "graph": false, + "rows": 7077888, + "bytes": 283115520, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_0", + "seconds": 2.4290988221764565e-05, + "compile_seconds": 0.004074417054653168, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_1", + "seconds": 0.2448859999421984, + "compile_seconds": 0.04944137507118285, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 23592960, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_2", + "seconds": 2.266606315970421e-05, + "compile_seconds": 0.004057083977386355, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_3", + "seconds": 2.7499976567924023e-05, + "compile_seconds": 0.003940667025744915, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_4", + "seconds": 1.7666956409811974e-05, + "compile_seconds": 0.0038107080617919564, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_5", + "seconds": 2.0832987502217293e-05, + "compile_seconds": 0.0038120420649647713, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_6", + "seconds": 1.6584061086177826e-05, + "compile_seconds": 0.0037594590103253722, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_7", + "seconds": 1.3541080988943577e-05, + "compile_seconds": 0.003660332993604243, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_8", + "seconds": 0.1656713749980554, + "compile_seconds": 0.02324887504801154, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 42467328 + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_9", + "seconds": 0.25231537502259016, + "compile_seconds": 0.019424165948294103, + "deferred": false, + "graph": false, + "rows": 589824, + "bytes": 23592960, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_5_wv_part_10", + "seconds": 1.7878661249997094, + "compile_seconds": 0.22105375002138317, + "deferred": false, + "graph": false, + "rows": 7077888, + "bytes": 566231040, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_5_wv", + "seconds": 0.3340878749731928, + "compile_seconds": 0.002158166957087815, + "deferred": false, + "graph": false, + "rows": 7077888, + "bytes": 283115520, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_6_wo", + "seconds": 1.5174185830401257, + "compile_seconds": 0.05680670798756182, + "deferred": false, + "graph": false, + "rows": 7077888, + "bytes": 283115520, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_7_ln_2", + "seconds": 0.4984346659621224, + "compile_seconds": 0.2459673749981448, + "deferred": false, + "graph": false, + "rows": 9216, + "bytes": 221184, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_0", + "seconds": 2.8000096790492535e-05, + "compile_seconds": 0.004188999999314547, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_1", + "seconds": 0.3908048749435693, + "compile_seconds": 0.04449691600166261, + "deferred": false, + "graph": false, + "rows": 2359296, + "bytes": 75497472, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_2", + "seconds": 2.4292035959661007e-05, + "compile_seconds": 0.004078416037373245, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_3", + "seconds": 2.4958979338407516e-05, + "compile_seconds": 0.004094708012416959, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_4", + "seconds": 1.9415980204939842e-05, + "compile_seconds": 0.0038844170048832893, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_5", + "seconds": 2.60830856859684e-05, + "compile_seconds": 0.00402324995957315, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_6", + "seconds": 1.6291975043714046e-05, + "compile_seconds": 0.003976207925006747, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_7", + "seconds": 1.8749968148767948e-05, + "compile_seconds": 0.003908250015228987, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_8", + "seconds": 1.9042054191231728e-05, + "compile_seconds": 0.0037055419525131583, + "deferred": true, + "graph": false, + "rows": null, + "bytes": null + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_9", + "seconds": 0.37979208305478096, + "compile_seconds": 0.019957208074629307, + "deferred": false, + "graph": false, + "rows": 2359296, + "bytes": 75497472, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc_part_10", + "seconds": 5.487648917012848, + "compile_seconds": 0.2593871250282973, + "deferred": false, + "graph": false, + "rows": 28311552, + "bytes": 905969664, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_8_w_fc", + "seconds": 3.759419917012565, + "compile_seconds": 0.0035656249383464456, + "deferred": false, + "graph": false, + "rows": 28311552, + "bytes": 905969664, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_9_w_proj", + "seconds": 6.316149958991446, + "compile_seconds": 0.05333429202437401, + "deferred": false, + "graph": false, + "rows": 28311552, + "bytes": 905969664, + "physical": "FFI_ExecutionPlan: einfold, number_of_children=0\n" + }, + { + "name": "__ddx_77668c50_1_grad_10_ln_f", + "seconds": 0.09902308299206197, + "compile_seconds": 0.03932916698977351, + "deferred": false, + "graph": false, + "rows": 768, + "bytes": 12288 + } + ], + "statistics": { + "table_registers": 12, + "query_preparations": 1, + "gradient_preparations": 1, + "cpu_matmul": 142, + "gpu_matmul": 0, + "gpu_attention": 0, + "declined": 12, + "physical_plans": 1804, + "plan_executions": 1804, + "stage_buffer_allocations": 1781, + "key_cache_hits": 7356, + "key_copy_bytes_avoided": 4976716160, + "forward_stages": 65, + "deferred_stages": 175, + "union_reduction_graphs": 19, + "reduction_stage_executions": 1291, + "planning_seconds": 29.620746106491424, + "execution_seconds": 98.48422984522767, + "stage_storage_seconds": 33.43930972169619, + "key_cache_bytes": 2890244224 + }, + "source_sha256": "fec8936a632b9e9cb9dffee237118e0bde0c30c75b0e557b819c03387c28d8a7", + "scope": "Original stage timings, shapes and statistics; generated SQL text omitted." +} diff --git a/benchmarks/apple/dense-before-stages.log b/benchmarks/apple/dense-before-stages.log new file mode 100644 index 00000000..649c1ac5 --- /dev/null +++ b/benchmarks/apple/dense-before-stages.log @@ -0,0 +1,1261 @@ +FORWARD {'l': [11.218448458220335]} +BEGIN __ddx_77668c50_1_saved_0 +END __ddx_77668c50_1_saved_0 0.0830829170299694 16 +BEGIN __ddx_77668c50_1_saved_1 +END __ddx_77668c50_1_saved_1 0.053183124982751906 16 +BEGIN __ddx_77668c50_1_saved_2 +END __ddx_77668c50_1_saved_2 0.04624150006566197 12288 +BEGIN __ddx_77668c50_1_saved_3 +END __ddx_77668c50_1_saved_3 0.046304875053465366 12288 +BEGIN __ddx_77668c50_1_saved_4 +END __ddx_77668c50_1_saved_4 0.0009725000709295273 1632 +BEGIN __ddx_77668c50_1_saved_5 +END __ddx_77668c50_1_saved_5 0.04911620810162276 12288 +BEGIN __ddx_77668c50_1_saved_6 +END __ddx_77668c50_1_saved_6 0.0007793749682605267 192 +BEGIN __ddx_77668c50_1_saved_7 +END __ddx_77668c50_1_saved_7 0.001510916044935584 12288 +BEGIN __ddx_77668c50_1_saved_8 +END __ddx_77668c50_1_saved_8 0.005846834043040872 12288 +BEGIN __ddx_77668c50_1_saved_9 +END __ddx_77668c50_1_saved_9 0.049393208930268884 16 +BEGIN __ddx_77668c50_1_saved_10 +END __ddx_77668c50_1_saved_10 0.04457054194062948 16 +BEGIN __ddx_77668c50_1_saved_11 +END __ddx_77668c50_1_saved_11 0.057055250043049455 49152 +BEGIN __ddx_77668c50_1_saved_12 +END __ddx_77668c50_1_saved_12 0.015947000007145107 12288 +BEGIN __ddx_77668c50_1_saved_13 +END __ddx_77668c50_1_saved_13 0.04988808301277459 16 +BEGIN __ddx_77668c50_1_saved_14 +END __ddx_77668c50_1_saved_14 0.049163166899234056 16 +BEGIN __ddx_77668c50_1_saved_15 +END __ddx_77668c50_1_saved_15 0.0474832080071792 12288 +BEGIN __ddx_77668c50_1_saved_16 +END __ddx_77668c50_1_saved_16 0.05072245793417096 12288 +BEGIN __ddx_77668c50_1_saved_17 +END __ddx_77668c50_1_saved_17 0.0008639999432489276 1632 +BEGIN __ddx_77668c50_1_saved_18 +END __ddx_77668c50_1_saved_18 0.04841679101809859 12288 +BEGIN __ddx_77668c50_1_saved_19 +END __ddx_77668c50_1_saved_19 0.0007794169941917062 192 +BEGIN __ddx_77668c50_1_saved_20 +END __ddx_77668c50_1_saved_20 0.0015972499968484044 12288 +BEGIN __ddx_77668c50_1_saved_21 +END __ddx_77668c50_1_saved_21 0.005822625011205673 12288 +BEGIN __ddx_77668c50_1_saved_22 +END __ddx_77668c50_1_saved_22 0.04901312501169741 16 +BEGIN __ddx_77668c50_1_saved_23 +END __ddx_77668c50_1_saved_23 0.05005024990532547 16 +BEGIN __ddx_77668c50_1_saved_24 +END __ddx_77668c50_1_saved_24 0.06101837498135865 49152 +BEGIN __ddx_77668c50_1_saved_25 +END __ddx_77668c50_1_saved_25 0.02461233304347843 12288 +BEGIN __ddx_77668c50_1_saved_26 +END __ddx_77668c50_1_saved_26 0.054812459042295814 16 +BEGIN __ddx_77668c50_1_saved_27 +END __ddx_77668c50_1_saved_27 0.04935095796827227 16 +BEGIN __ddx_77668c50_1_saved_28 +END __ddx_77668c50_1_saved_28 0.049880500067956746 12288 +BEGIN __ddx_77668c50_1_saved_29 +END __ddx_77668c50_1_saved_29 0.052384582930244505 12288 +BEGIN __ddx_77668c50_1_saved_30 +END __ddx_77668c50_1_saved_30 0.000992208020761609 1632 +BEGIN __ddx_77668c50_1_saved_31 +END __ddx_77668c50_1_saved_31 0.05212916596792638 12288 +BEGIN __ddx_77668c50_1_saved_32 +END __ddx_77668c50_1_saved_32 0.0008572089718654752 192 +BEGIN __ddx_77668c50_1_saved_33 +END __ddx_77668c50_1_saved_33 0.0017859999788925052 12288 +BEGIN __ddx_77668c50_1_saved_34 +END __ddx_77668c50_1_saved_34 0.006042041932232678 12288 +BEGIN __ddx_77668c50_1_saved_35 +END __ddx_77668c50_1_saved_35 0.05019187496509403 16 +BEGIN __ddx_77668c50_1_saved_36 +END __ddx_77668c50_1_saved_36 0.04953099996782839 16 +BEGIN __ddx_77668c50_1_saved_37 +END __ddx_77668c50_1_saved_37 0.06315358402207494 49152 +BEGIN __ddx_77668c50_1_saved_38 +END __ddx_77668c50_1_saved_38 0.01671562495175749 12288 +BEGIN __ddx_77668c50_1_saved_39 +END __ddx_77668c50_1_saved_39 0.060394749976694584 16 +BEGIN __ddx_77668c50_1_saved_40 +END __ddx_77668c50_1_saved_40 0.06937808298971504 16 +BEGIN __ddx_77668c50_1_saved_41 +END __ddx_77668c50_1_saved_41 0.05446574999950826 12288 +BEGIN __ddx_77668c50_1_saved_42 +END __ddx_77668c50_1_saved_42 0.05196700000669807 12288 +BEGIN __ddx_77668c50_1_saved_43 +END __ddx_77668c50_1_saved_43 0.001343625015579164 1632 +BEGIN __ddx_77668c50_1_saved_44 +END __ddx_77668c50_1_saved_44 0.05275612499099225 12288 +BEGIN __ddx_77668c50_1_saved_45 +END __ddx_77668c50_1_saved_45 0.0007840830367058516 192 +BEGIN __ddx_77668c50_1_saved_46 +END __ddx_77668c50_1_saved_46 0.0016745839966461062 12288 +BEGIN __ddx_77668c50_1_saved_47 +END __ddx_77668c50_1_saved_47 0.0060841660015285015 12288 +BEGIN __ddx_77668c50_1_saved_48 +END __ddx_77668c50_1_saved_48 0.05329462490044534 16 +BEGIN __ddx_77668c50_1_saved_49 +END __ddx_77668c50_1_saved_49 0.05929183401167393 16 +BEGIN __ddx_77668c50_1_saved_50 +END __ddx_77668c50_1_saved_50 0.06462424993515015 49152 +BEGIN __ddx_77668c50_1_saved_51 +END __ddx_77668c50_1_saved_51 0.020457540987990797 12288 +BEGIN __ddx_77668c50_1_saved_52 +END __ddx_77668c50_1_saved_52 0.059627957991324365 16 +BEGIN __ddx_77668c50_1_saved_53 +END __ddx_77668c50_1_saved_53 0.05455745803192258 16 +BEGIN __ddx_77668c50_1_saved_54 +END __ddx_77668c50_1_saved_54 0.05240212508942932 12288 +BEGIN __ddx_77668c50_1_saved_55 +END __ddx_77668c50_1_saved_55 0.054520250065252185 12288 +BEGIN __ddx_77668c50_1_saved_56 +END __ddx_77668c50_1_saved_56 0.0008353749290108681 1632 +BEGIN __ddx_77668c50_1_saved_57 +END __ddx_77668c50_1_saved_57 0.053498500026762486 12288 +BEGIN __ddx_77668c50_1_saved_58 +END __ddx_77668c50_1_saved_58 0.0008029160089790821 192 +BEGIN __ddx_77668c50_1_saved_59 +END __ddx_77668c50_1_saved_59 0.0015218330081552267 12288 +BEGIN __ddx_77668c50_1_saved_60 +END __ddx_77668c50_1_saved_60 0.005960374954156578 12288 +BEGIN __ddx_77668c50_1_saved_61 +END __ddx_77668c50_1_saved_61 0.05861554201692343 16 +BEGIN __ddx_77668c50_1_saved_62 +END __ddx_77668c50_1_saved_62 0.051567915943451226 16 +BEGIN __ddx_77668c50_1_saved_63_part_0 +END __ddx_77668c50_1_saved_63_part_0 0.06273645907640457 49152 +BEGIN __ddx_77668c50_1_saved_63 +END __ddx_77668c50_1_saved_63 0.0006960000609979033 49152 +BEGIN __ddx_77668c50_1_saved_64 +END __ddx_77668c50_1_saved_64 0.017440584022551775 12288 +BEGIN __ddx_77668c50_1_saved_65 +END __ddx_77668c50_1_saved_65 0.06134970800485462 16 +BEGIN __ddx_77668c50_1_saved_66 +END __ddx_77668c50_1_saved_66 0.05696291709318757 16 +BEGIN __ddx_77668c50_1_saved_67_part_0 +END __ddx_77668c50_1_saved_67_part_0 2.1999934688210487e-05 None +BEGIN __ddx_77668c50_1_saved_67 +END __ddx_77668c50_1_saved_67 0.08472491696011275 12288 +BEGIN __ddx_77668c50_1_saved_68_part_0 +END __ddx_77668c50_1_saved_68_part_0 2.3499946109950542e-05 None +BEGIN __ddx_77668c50_1_saved_68 +END __ddx_77668c50_1_saved_68 0.07972091704141349 12288 +BEGIN __ddx_77668c50_1_saved_69 +END __ddx_77668c50_1_saved_69 0.0009352500783279538 1632 +BEGIN __ddx_77668c50_1_saved_70_part_0 +END __ddx_77668c50_1_saved_70_part_0 1.954101026058197e-05 None +BEGIN __ddx_77668c50_1_saved_70 +END __ddx_77668c50_1_saved_70 0.07359195896424353 12288 +BEGIN __ddx_77668c50_1_saved_71 +END __ddx_77668c50_1_saved_71 0.0007327500497922301 192 +BEGIN __ddx_77668c50_1_saved_72 +END __ddx_77668c50_1_saved_72 0.00169945799279958 12288 +BEGIN __ddx_77668c50_1_saved_73 +END __ddx_77668c50_1_saved_73 0.005134207895025611 12288 +BEGIN __ddx_77668c50_1_saved_74 +END __ddx_77668c50_1_saved_74 0.05470154108479619 16 +BEGIN __ddx_77668c50_1_saved_75 +END __ddx_77668c50_1_saved_75 0.05315379204694182 16 +BEGIN __ddx_77668c50_1_saved_76_part_0 +END __ddx_77668c50_1_saved_76_part_0 1.8874998204410076e-05 None +BEGIN __ddx_77668c50_1_saved_76 +END __ddx_77668c50_1_saved_76 0.11953041702508926 49152 +BEGIN __ddx_77668c50_1_saved_77 +END __ddx_77668c50_1_saved_77 0.019182375050149858 12288 +BEGIN __ddx_77668c50_1_saved_78 +END __ddx_77668c50_1_saved_78 0.06053770799189806 16 +BEGIN __ddx_77668c50_1_saved_79_part_0 +END __ddx_77668c50_1_saved_79_part_0 0.05902054090984166 16 +BEGIN __ddx_77668c50_1_saved_79 +END __ddx_77668c50_1_saved_79 0.0004983330145478249 16 +BEGIN __ddx_77668c50_1_saved_80_part_0 +END __ddx_77668c50_1_saved_80_part_0 2.154102548956871e-05 None +BEGIN __ddx_77668c50_1_saved_80 +END __ddx_77668c50_1_saved_80 0.07504270796198398 12288 +BEGIN __ddx_77668c50_1_saved_81_part_0 +END __ddx_77668c50_1_saved_81_part_0 2.0625069737434387e-05 None +BEGIN __ddx_77668c50_1_saved_81 +END __ddx_77668c50_1_saved_81 0.07734462502412498 12288 +BEGIN __ddx_77668c50_1_saved_82 +END __ddx_77668c50_1_saved_82 0.0009088750230148435 1632 +BEGIN __ddx_77668c50_1_saved_83_part_0 +END __ddx_77668c50_1_saved_83_part_0 2.6124995201826096e-05 None +BEGIN __ddx_77668c50_1_saved_83 +END __ddx_77668c50_1_saved_83 0.0809478749288246 12288 +BEGIN __ddx_77668c50_1_saved_84 +END __ddx_77668c50_1_saved_84 0.0007779999868944287 192 +BEGIN __ddx_77668c50_1_saved_85 +END __ddx_77668c50_1_saved_85 0.001550624961964786 12288 +BEGIN __ddx_77668c50_1_saved_86 +END __ddx_77668c50_1_saved_86 0.00504458392970264 12288 +BEGIN __ddx_77668c50_1_saved_87_part_0 +END __ddx_77668c50_1_saved_87_part_0 0.05858916603028774 16 +BEGIN __ddx_77668c50_1_saved_87 +END __ddx_77668c50_1_saved_87 0.0023463749093934894 16 +BEGIN __ddx_77668c50_1_saved_88_part_0 +END __ddx_77668c50_1_saved_88_part_0 1.9458006136119366e-05 None +BEGIN __ddx_77668c50_1_saved_88 +END __ddx_77668c50_1_saved_88 0.057106415973976254 16 +BEGIN __ddx_77668c50_1_saved_89_part_0 +END __ddx_77668c50_1_saved_89_part_0 2.0209001377224922e-05 None +BEGIN __ddx_77668c50_1_saved_89 +END __ddx_77668c50_1_saved_89 0.1176451250212267 49152 +BEGIN __ddx_77668c50_1_saved_90 +END __ddx_77668c50_1_saved_90 0.0184882499743253 12288 +BEGIN __ddx_77668c50_1_saved_91_part_0 +END __ddx_77668c50_1_saved_91_part_0 2.1249987185001373e-05 None +BEGIN __ddx_77668c50_1_saved_91 +END __ddx_77668c50_1_saved_91 0.06738783291075379 16 +BEGIN __ddx_77668c50_1_saved_92_part_0 +END __ddx_77668c50_1_saved_92_part_0 2.0749983377754688e-05 None +BEGIN __ddx_77668c50_1_saved_92 +END __ddx_77668c50_1_saved_92 0.060717875021509826 16 +BEGIN __ddx_77668c50_1_saved_93_part_0 +END __ddx_77668c50_1_saved_93_part_0 2.0582927390933037e-05 None +BEGIN __ddx_77668c50_1_saved_93 +END __ddx_77668c50_1_saved_93 0.08012904098723084 12288 +BEGIN __ddx_77668c50_1_saved_94_part_0 +END __ddx_77668c50_1_saved_94_part_0 2.53340695053339e-05 None +BEGIN __ddx_77668c50_1_saved_94 +END __ddx_77668c50_1_saved_94 0.08035737508907914 12288 +BEGIN __ddx_77668c50_1_saved_95 +END __ddx_77668c50_1_saved_95 0.0007972500752657652 1632 +BEGIN __ddx_77668c50_1_saved_96_part_0 +END __ddx_77668c50_1_saved_96_part_0 2.166605554521084e-05 None +BEGIN __ddx_77668c50_1_saved_96 +END __ddx_77668c50_1_saved_96 0.07718183402903378 12288 +BEGIN __ddx_77668c50_1_saved_97 +END __ddx_77668c50_1_saved_97 0.0007686669705435634 192 +BEGIN __ddx_77668c50_1_saved_98 +END __ddx_77668c50_1_saved_98 0.00141587492544204 12288 +BEGIN __ddx_77668c50_1_saved_99 +END __ddx_77668c50_1_saved_99 0.005173791898414493 12288 +BEGIN __ddx_77668c50_1_saved_100_part_0 +END __ddx_77668c50_1_saved_100_part_0 2.0875013433396816e-05 None +BEGIN __ddx_77668c50_1_saved_100 +END __ddx_77668c50_1_saved_100 0.05924170801881701 16 +BEGIN __ddx_77668c50_1_saved_101_part_0 +END __ddx_77668c50_1_saved_101_part_0 2.0249979570508003e-05 None +BEGIN __ddx_77668c50_1_saved_101 +END __ddx_77668c50_1_saved_101 0.05841391708236188 16 +BEGIN __ddx_77668c50_1_saved_102_part_0 +END __ddx_77668c50_1_saved_102_part_0 2.2582942619919777e-05 None +BEGIN __ddx_77668c50_1_saved_102 +END __ddx_77668c50_1_saved_102 0.12608712504152209 49152 +BEGIN __ddx_77668c50_1_saved_103 +END __ddx_77668c50_1_saved_103 0.018111290992237628 12288 +BEGIN __ddx_77668c50_1_saved_104_part_0 +END __ddx_77668c50_1_saved_104_part_0 2.4666893295943737e-05 None +BEGIN __ddx_77668c50_1_saved_104 +END __ddx_77668c50_1_saved_104 0.06421841692645103 16 +BEGIN __ddx_77668c50_1_saved_105_part_0 +END __ddx_77668c50_1_saved_105_part_0 2.037500962615013e-05 None +BEGIN __ddx_77668c50_1_saved_105 +END __ddx_77668c50_1_saved_105 0.0615038750693202 16 +BEGIN __ddx_77668c50_1_saved_106_part_0 +END __ddx_77668c50_1_saved_106_part_0 1.9666040316224098e-05 None +BEGIN __ddx_77668c50_1_saved_106 +END __ddx_77668c50_1_saved_106 0.07896025001537055 12288 +BEGIN __ddx_77668c50_1_saved_107_part_0 +END __ddx_77668c50_1_saved_107_part_0 1.9082915969192982e-05 None +BEGIN __ddx_77668c50_1_saved_107 +END __ddx_77668c50_1_saved_107 0.0745100419735536 12288 +BEGIN __ddx_77668c50_1_saved_108 +END __ddx_77668c50_1_saved_108 0.0007736249826848507 1632 +BEGIN __ddx_77668c50_1_saved_109_part_0 +END __ddx_77668c50_1_saved_109_part_0 1.7459038645029068e-05 None +BEGIN __ddx_77668c50_1_saved_109 +END __ddx_77668c50_1_saved_109 0.07880845793988556 12288 +BEGIN __ddx_77668c50_1_saved_110 +END __ddx_77668c50_1_saved_110 0.0007120410446077585 192 +BEGIN __ddx_77668c50_1_saved_111 +END __ddx_77668c50_1_saved_111 0.0014529579784721136 12288 +BEGIN __ddx_77668c50_1_saved_112 +END __ddx_77668c50_1_saved_112 0.004962582956068218 12288 +BEGIN __ddx_77668c50_1_saved_113_part_0 +END __ddx_77668c50_1_saved_113_part_0 1.9999919459223747e-05 None +BEGIN __ddx_77668c50_1_saved_113 +END __ddx_77668c50_1_saved_113 0.05711387493647635 16 +BEGIN __ddx_77668c50_1_saved_114_part_0 +END __ddx_77668c50_1_saved_114_part_0 2.0082923583686352e-05 None +BEGIN __ddx_77668c50_1_saved_114 +END __ddx_77668c50_1_saved_114 0.05365895794238895 16 +BEGIN __ddx_77668c50_1_saved_115_part_0 +END __ddx_77668c50_1_saved_115_part_0 2.0749983377754688e-05 None +BEGIN __ddx_77668c50_1_saved_115 +END __ddx_77668c50_1_saved_115 0.12159687501844019 49152 +BEGIN __ddx_77668c50_1_saved_116 +END __ddx_77668c50_1_saved_116 0.01710716704837978 12288 +BEGIN __ddx_77668c50_1_saved_117_part_0 +END __ddx_77668c50_1_saved_117_part_0 2.145790494978428e-05 None +BEGIN __ddx_77668c50_1_saved_117 +END __ddx_77668c50_1_saved_117 0.06690058298408985 16 +BEGIN __ddx_77668c50_1_saved_118_part_0 +END __ddx_77668c50_1_saved_118_part_0 2.0165927708148956e-05 None +BEGIN __ddx_77668c50_1_saved_118 +END __ddx_77668c50_1_saved_118 0.054969207965768874 16 +BEGIN __ddx_77668c50_1_saved_119_part_0 +END __ddx_77668c50_1_saved_119_part_0 2.0125065930187702e-05 None +BEGIN __ddx_77668c50_1_saved_119 +END __ddx_77668c50_1_saved_119 0.0781761669786647 12288 +BEGIN __ddx_77668c50_1_saved_120_part_0 +END __ddx_77668c50_1_saved_120_part_0 2.0042061805725098e-05 None +BEGIN __ddx_77668c50_1_saved_120 +END __ddx_77668c50_1_saved_120 0.07781291601713747 12288 +BEGIN __ddx_77668c50_1_saved_121 +END __ddx_77668c50_1_saved_121 0.000795625033788383 1632 +BEGIN __ddx_77668c50_1_saved_122_part_0 +END __ddx_77668c50_1_saved_122_part_0 1.825008075684309e-05 None +BEGIN __ddx_77668c50_1_saved_122 +END __ddx_77668c50_1_saved_122 0.08042933396063745 12288 +BEGIN __ddx_77668c50_1_saved_123 +END __ddx_77668c50_1_saved_123 0.0007453750586137176 192 +BEGIN __ddx_77668c50_1_saved_124 +END __ddx_77668c50_1_saved_124 0.0015254170866683125 12288 +BEGIN __ddx_77668c50_1_saved_125 +END __ddx_77668c50_1_saved_125 0.00478404201567173 12288 +BEGIN __ddx_77668c50_1_saved_126_part_0 +END __ddx_77668c50_1_saved_126_part_0 1.9915984012186527e-05 None +BEGIN __ddx_77668c50_1_saved_126 +END __ddx_77668c50_1_saved_126 0.054853832931257784 16 +BEGIN __ddx_77668c50_1_saved_127_part_0 +END __ddx_77668c50_1_saved_127_part_0 1.9042054191231728e-05 None +BEGIN __ddx_77668c50_1_saved_127 +END __ddx_77668c50_1_saved_127 0.0594222079962492 16 +BEGIN __ddx_77668c50_1_saved_128_part_0 +END __ddx_77668c50_1_saved_128_part_0 1.8208054825663567e-05 None +BEGIN __ddx_77668c50_1_saved_128 +END __ddx_77668c50_1_saved_128 0.11392916704062372 49152 +BEGIN __ddx_77668c50_1_saved_129 +END __ddx_77668c50_1_saved_129 0.017257834086194634 12288 +BEGIN __ddx_77668c50_1_saved_130_part_0 +END __ddx_77668c50_1_saved_130_part_0 2.025009598582983e-05 None +BEGIN __ddx_77668c50_1_saved_130 +END __ddx_77668c50_1_saved_130 0.05996866710484028 16 +BEGIN __ddx_77668c50_1_saved_131_part_0 +END __ddx_77668c50_1_saved_131_part_0 1.7874990589916706e-05 None +BEGIN __ddx_77668c50_1_saved_131 +END __ddx_77668c50_1_saved_131 0.05600533296819776 16 +BEGIN __ddx_77668c50_1_saved_132_part_0 +END __ddx_77668c50_1_saved_132_part_0 1.9332976080477238e-05 None +BEGIN __ddx_77668c50_1_saved_132 +END __ddx_77668c50_1_saved_132 0.09680150006897748 12288 +BEGIN __ddx_77668c50_1_saved_133_part_0 +END __ddx_77668c50_1_saved_133_part_0 1.9832979887723923e-05 None +BEGIN __ddx_77668c50_1_saved_133 +END __ddx_77668c50_1_saved_133 0.07702995906583965 12288 +BEGIN __ddx_77668c50_1_saved_134 +END __ddx_77668c50_1_saved_134 0.0008926669834181666 1632 +BEGIN __ddx_77668c50_1_saved_135_part_0 +END __ddx_77668c50_1_saved_135_part_0 1.7457990907132626e-05 None +BEGIN __ddx_77668c50_1_saved_135 +END __ddx_77668c50_1_saved_135 0.07823824998922646 12288 +BEGIN __ddx_77668c50_1_saved_136 +END __ddx_77668c50_1_saved_136 0.0006571249105036259 192 +BEGIN __ddx_77668c50_1_saved_137 +END __ddx_77668c50_1_saved_137 0.0015913749812170863 12288 +BEGIN __ddx_77668c50_1_saved_138 +END __ddx_77668c50_1_saved_138 0.004804749973118305 12288 +BEGIN __ddx_77668c50_1_saved_139_part_0 +END __ddx_77668c50_1_saved_139_part_0 1.8666964024305344e-05 None +BEGIN __ddx_77668c50_1_saved_139 +END __ddx_77668c50_1_saved_139 0.057325667003169656 16 +BEGIN __ddx_77668c50_1_saved_140_part_0 +END __ddx_77668c50_1_saved_140_part_0 1.9249971956014633e-05 None +BEGIN __ddx_77668c50_1_saved_140 +END __ddx_77668c50_1_saved_140 0.05669745791237801 16 +BEGIN __ddx_77668c50_1_saved_141_part_0 +END __ddx_77668c50_1_saved_141_part_0 2.0666979253292084e-05 None +BEGIN __ddx_77668c50_1_saved_141 +END __ddx_77668c50_1_saved_141 0.12026929203420877 49152 +BEGIN __ddx_77668c50_1_saved_142 +END __ddx_77668c50_1_saved_142 0.016605834010988474 12288 +BEGIN __ddx_77668c50_1_saved_143_part_0 +END __ddx_77668c50_1_saved_143_part_0 1.9875005818903446e-05 None +BEGIN __ddx_77668c50_1_saved_143 +END __ddx_77668c50_1_saved_143 0.06324304093141109 16 +BEGIN __ddx_77668c50_1_saved_144_part_0 +END __ddx_77668c50_1_saved_144_part_0 2.0749983377754688e-05 None +BEGIN __ddx_77668c50_1_saved_144 +END __ddx_77668c50_1_saved_144 0.058321750024333596 16 +BEGIN __ddx_77668c50_1_saved_145_part_0 +END __ddx_77668c50_1_saved_145_part_0 2.0332983694970608e-05 None +BEGIN __ddx_77668c50_1_saved_145 +END __ddx_77668c50_1_saved_145 0.07919845799915493 12288 +BEGIN __ddx_77668c50_1_saved_146_part_0 +END __ddx_77668c50_1_saved_146_part_0 1.9583036191761494e-05 None +BEGIN __ddx_77668c50_1_saved_146 +END __ddx_77668c50_1_saved_146 0.0800856250571087 12288 +BEGIN __ddx_77668c50_1_saved_147 +END __ddx_77668c50_1_saved_147 0.00076299998909235 1632 +BEGIN __ddx_77668c50_1_saved_148_part_0 +END __ddx_77668c50_1_saved_148_part_0 1.979200169444084e-05 None +BEGIN __ddx_77668c50_1_saved_148 +END __ddx_77668c50_1_saved_148 0.07911212497856468 12288 +BEGIN __ddx_77668c50_1_saved_149 +END __ddx_77668c50_1_saved_149 0.0007784159388393164 192 +BEGIN __ddx_77668c50_1_saved_150 +END __ddx_77668c50_1_saved_150 0.0013545829569920897 12288 +BEGIN __ddx_77668c50_1_saved_151 +END __ddx_77668c50_1_saved_151 0.005703542032279074 12288 +BEGIN __ddx_77668c50_1_saved_152_part_0 +END __ddx_77668c50_1_saved_152_part_0 1.9083963707089424e-05 None +BEGIN __ddx_77668c50_1_saved_152 +END __ddx_77668c50_1_saved_152 0.06190299999434501 16 +BEGIN __ddx_77668c50_1_saved_153_part_0 +END __ddx_77668c50_1_saved_153_part_0 1.895800232887268e-05 None +BEGIN __ddx_77668c50_1_saved_153 +END __ddx_77668c50_1_saved_153 0.05785020801704377 16 +BEGIN __ddx_77668c50_1_saved_154_part_0 +END __ddx_77668c50_1_saved_154_part_0 2.0125065930187702e-05 None +BEGIN __ddx_77668c50_1_saved_154 +END __ddx_77668c50_1_saved_154 0.11779062496498227 49152 +BEGIN __ddx_77668c50_1_saved_155 +END __ddx_77668c50_1_saved_155 0.020701292087323964 12288 +BEGIN __ddx_77668c50_1_saved_156_part_0 +END __ddx_77668c50_1_saved_156_part_0 1.9915984012186527e-05 None +BEGIN __ddx_77668c50_1_saved_156 +END __ddx_77668c50_1_saved_156 0.06392541597597301 16 +BEGIN __ddx_77668c50_1_saved_157_part_0 +END __ddx_77668c50_1_saved_157_part_0 2.004194539040327e-05 None +BEGIN __ddx_77668c50_1_saved_157 +END __ddx_77668c50_1_saved_157 0.06477587495464832 16 +BEGIN __ddx_77668c50_1_saved_158_part_0 +END __ddx_77668c50_1_saved_158_part_0 1.979200169444084e-05 None +BEGIN __ddx_77668c50_1_saved_158 +END __ddx_77668c50_1_saved_158 0.5678354169940576 804112 +BEGIN __ddx_77668c50_1_saved_159 +END __ddx_77668c50_1_saved_159 0.03626533399801701 16 +BEGIN __ddx_77668c50_1_saved_160 +END __ddx_77668c50_1_saved_160 0.0016221250407397747 1 +BEGIN __ddx_77668c50_1_value +END __ddx_77668c50_1_value 0.00034991593565791845 1 +BEGIN __ddx_77668c50_1_cotangent_160 +END __ddx_77668c50_1_cotangent_160 0.00034370901994407177 1 +BEGIN __ddx_77668c50_1_cotangent_159 +END __ddx_77668c50_1_cotangent_159 0.002033500000834465 16 +BEGIN __ddx_77668c50_1_cotangent_158 +END __ddx_77668c50_1_cotangent_158 0.13599233294371516 804112 +BEGIN __ddx_77668c50_1_contraction_1 +END __ddx_77668c50_1_contraction_1 0.22918816597666591 12288 +BEGIN __ddx_77668c50_1_region_2_part_0 +END __ddx_77668c50_1_region_2_part_0 2.4957931600511074e-05 None +BEGIN __ddx_77668c50_1_region_2_part_1 +END __ddx_77668c50_1_region_2_part_1 3.0082999728620052e-05 None +BEGIN __ddx_77668c50_1_region_2 +END __ddx_77668c50_1_region_2 1.737498678267002e-05 None +BEGIN __ddx_77668c50_1_cotangent_157 +END __ddx_77668c50_1_cotangent_157 0.057705708080902696 16 +BEGIN __ddx_77668c50_1_region_3_part_0 +END __ddx_77668c50_1_region_3_part_0 2.7542002499103546e-05 None +BEGIN __ddx_77668c50_1_region_3 +END __ddx_77668c50_1_region_3 0.08386504207737744 12288 +BEGIN __ddx_77668c50_1_cotangent_156 +END __ddx_77668c50_1_cotangent_156 0.06506649998482317 16 +BEGIN __ddx_77668c50_1_region_4_part_0 +END __ddx_77668c50_1_region_4_part_0 3.0791969038546085e-05 None +BEGIN __ddx_77668c50_1_region_4 +END __ddx_77668c50_1_region_4 0.07941408397164196 12288 +BEGIN __ddx_77668c50_1_cotangent_155 +END __ddx_77668c50_1_cotangent_155 0.07148295792285353 12288 +BEGIN __ddx_77668c50_1_cotangent_154 +END __ddx_77668c50_1_cotangent_154 0.051860042032785714 49152 +BEGIN __ddx_77668c50_1_contraction_5 +END __ddx_77668c50_1_contraction_5 0.04317337495740503 12288 +BEGIN __ddx_77668c50_1_region_6_part_0 +END __ddx_77668c50_1_region_6_part_0 3.162503708153963e-05 None +BEGIN __ddx_77668c50_1_region_6_part_1 +END __ddx_77668c50_1_region_6_part_1 2.9999995604157448e-05 None +BEGIN __ddx_77668c50_1_region_6 +END __ddx_77668c50_1_region_6 1.579197123646736e-05 None +BEGIN __ddx_77668c50_1_cotangent_153 +END __ddx_77668c50_1_cotangent_153 0.06272958300542086 16 +BEGIN __ddx_77668c50_1_region_7_part_0 +END __ddx_77668c50_1_region_7_part_0 2.6499968953430653e-05 None +BEGIN __ddx_77668c50_1_region_7 +END __ddx_77668c50_1_region_7 0.0973525830777362 12288 +BEGIN __ddx_77668c50_1_cotangent_152 +END __ddx_77668c50_1_cotangent_152 0.06466041703242809 16 +BEGIN __ddx_77668c50_1_region_8_part_0 +END __ddx_77668c50_1_region_8_part_0 2.8665992431342602e-05 None +BEGIN __ddx_77668c50_1_region_8 +END __ddx_77668c50_1_region_8 0.07794387510512024 12288 +BEGIN __ddx_77668c50_1_cotangent_151 +END __ddx_77668c50_1_cotangent_151 0.13767904206179082 12288 +BEGIN __ddx_77668c50_1_cotangent_150 +END __ddx_77668c50_1_cotangent_150 0.02812570903915912 12288 +BEGIN __ddx_77668c50_1_contraction_9 +END __ddx_77668c50_1_contraction_9 0.010312624974176288 3072 +BEGIN __ddx_77668c50_1_cotangent_149 +END __ddx_77668c50_1_cotangent_149 0.0009931669337674975 192 +BEGIN __ddx_77668c50_1_cotangent_148 +END __ddx_77668c50_1_cotangent_148 0.011217292048968375 12288 +BEGIN __ddx_77668c50_1_contraction_12 +END __ddx_77668c50_1_contraction_12 0.025719125056639314 12288 +BEGIN __ddx_77668c50_1_region_13_part_0 +END __ddx_77668c50_1_region_13_part_0 2.38749198615551e-05 None +BEGIN __ddx_77668c50_1_region_13 +END __ddx_77668c50_1_region_13 2.7499976567924023e-05 None +BEGIN __ddx_77668c50_1_cotangent_147 +END __ddx_77668c50_1_cotangent_147 0.008898917003534734 1632 +BEGIN __ddx_77668c50_1_cotangent_146 +END __ddx_77668c50_1_cotangent_146 0.010533542023040354 12288 +BEGIN __ddx_77668c50_1_contraction_14 +END __ddx_77668c50_1_contraction_14 0.026544541004113853 12288 +BEGIN __ddx_77668c50_1_region_15_part_0 +END __ddx_77668c50_1_region_15_part_0 2.6250025257468224e-05 None +BEGIN __ddx_77668c50_1_region_15 +END __ddx_77668c50_1_region_15 2.9957969672977924e-05 None +BEGIN __ddx_77668c50_1_cotangent_145 +END __ddx_77668c50_1_cotangent_145 0.01141758298035711 12288 +BEGIN __ddx_77668c50_1_contraction_16 +END __ddx_77668c50_1_contraction_16 0.025995583971962333 12288 +BEGIN __ddx_77668c50_1_region_17_part_0 +END __ddx_77668c50_1_region_17_part_0 2.3707980290055275e-05 None +BEGIN __ddx_77668c50_1_region_17 +END __ddx_77668c50_1_region_17 2.7499976567924023e-05 None +BEGIN __ddx_77668c50_1_cotangent_144 +END __ddx_77668c50_1_cotangent_144 0.19908858300186694 16 +BEGIN __ddx_77668c50_1_region_18_part_0 +END __ddx_77668c50_1_region_18_part_0 2.7207890525460243e-05 None +BEGIN __ddx_77668c50_1_region_18 +END __ddx_77668c50_1_region_18 0.09256537503097206 12288 +BEGIN __ddx_77668c50_1_cotangent_143 +END __ddx_77668c50_1_cotangent_143 0.17266795900650322 16 +BEGIN __ddx_77668c50_1_region_19_part_0 +END __ddx_77668c50_1_region_19_part_0 2.74579506367445e-05 None +BEGIN __ddx_77668c50_1_region_19 +END __ddx_77668c50_1_region_19 0.07691554189659655 12288 +BEGIN __ddx_77668c50_1_cotangent_142 +END __ddx_77668c50_1_cotangent_142 0.3228439169470221 12288 +BEGIN __ddx_77668c50_1_cotangent_141 +END __ddx_77668c50_1_cotangent_141 0.05171054194215685 49152 +BEGIN __ddx_77668c50_1_contraction_20 +END __ddx_77668c50_1_contraction_20 0.04569295793771744 12288 +BEGIN __ddx_77668c50_1_region_21_part_0 +END __ddx_77668c50_1_region_21_part_0 2.9333052225410938e-05 None +BEGIN __ddx_77668c50_1_region_21 +END __ddx_77668c50_1_region_21 2.970895729959011e-05 None +BEGIN __ddx_77668c50_1_cotangent_140 +END __ddx_77668c50_1_cotangent_140 0.0652399999089539 16 +BEGIN __ddx_77668c50_1_region_22_part_0 +END __ddx_77668c50_1_region_22_part_0 2.7250032871961594e-05 None +BEGIN __ddx_77668c50_1_region_22 +END __ddx_77668c50_1_region_22 0.07569837500341237 12288 +BEGIN __ddx_77668c50_1_cotangent_139 +END __ddx_77668c50_1_cotangent_139 0.05956666695419699 16 +BEGIN __ddx_77668c50_1_region_23_part_0 +END __ddx_77668c50_1_region_23_part_0 3.133295103907585e-05 None +BEGIN __ddx_77668c50_1_region_23 +END __ddx_77668c50_1_region_23 0.07251979201100767 12288 +BEGIN __ddx_77668c50_1_cotangent_138 +END __ddx_77668c50_1_cotangent_138 0.3728405419969931 12288 +BEGIN __ddx_77668c50_1_cotangent_137 +END __ddx_77668c50_1_cotangent_137 0.03157075005583465 12288 +BEGIN __ddx_77668c50_1_contraction_24 +END __ddx_77668c50_1_contraction_24 0.0093461669748649 3072 +BEGIN __ddx_77668c50_1_cotangent_136 +END __ddx_77668c50_1_cotangent_136 0.0009614999871701002 192 +BEGIN __ddx_77668c50_1_cotangent_135 +END __ddx_77668c50_1_cotangent_135 0.010936209000647068 12288 +BEGIN __ddx_77668c50_1_contraction_27 +END __ddx_77668c50_1_contraction_27 0.02625533298123628 12288 +BEGIN __ddx_77668c50_1_region_28_part_0 +END __ddx_77668c50_1_region_28_part_0 2.5916961021721363e-05 None +BEGIN __ddx_77668c50_1_region_28 +END __ddx_77668c50_1_region_28 3.14170029014349e-05 None +BEGIN __ddx_77668c50_1_cotangent_134 +END __ddx_77668c50_1_cotangent_134 0.009425833006389439 1632 +BEGIN __ddx_77668c50_1_cotangent_133 +END __ddx_77668c50_1_cotangent_133 0.01087650004774332 12288 +BEGIN __ddx_77668c50_1_contraction_29 +END __ddx_77668c50_1_contraction_29 0.026591750094667077 12288 +BEGIN __ddx_77668c50_1_region_30_part_0 +END __ddx_77668c50_1_region_30_part_0 2.6291003450751305e-05 None +BEGIN __ddx_77668c50_1_region_30 +END __ddx_77668c50_1_region_30 2.995808608829975e-05 None +BEGIN __ddx_77668c50_1_cotangent_132 +END __ddx_77668c50_1_cotangent_132 0.011079875053837895 12288 +BEGIN __ddx_77668c50_1_contraction_31 +END __ddx_77668c50_1_contraction_31 0.025288415956310928 12288 +BEGIN __ddx_77668c50_1_region_32_part_0 +END __ddx_77668c50_1_region_32_part_0 2.7749920263886452e-05 None +BEGIN __ddx_77668c50_1_region_32 +END __ddx_77668c50_1_region_32 2.462498378008604e-05 None +BEGIN __ddx_77668c50_1_cotangent_131 +END __ddx_77668c50_1_cotangent_131 0.18548504100181162 16 +BEGIN __ddx_77668c50_1_region_33_part_0 +END __ddx_77668c50_1_region_33_part_0 3.074994310736656e-05 None +BEGIN __ddx_77668c50_1_region_33 +END __ddx_77668c50_1_region_33 0.07831391599029303 12288 +BEGIN __ddx_77668c50_1_cotangent_130 +END __ddx_77668c50_1_cotangent_130 0.17859341704752296 16 +BEGIN __ddx_77668c50_1_region_34_part_0 +END __ddx_77668c50_1_region_34_part_0 2.675002906471491e-05 None +BEGIN __ddx_77668c50_1_region_34 +END __ddx_77668c50_1_region_34 0.07308904093224555 12288 +BEGIN __ddx_77668c50_1_cotangent_129 +END __ddx_77668c50_1_cotangent_129 0.6042519160546362 12288 +BEGIN __ddx_77668c50_1_cotangent_128 +END __ddx_77668c50_1_cotangent_128 0.053498624940402806 49152 +BEGIN __ddx_77668c50_1_contraction_35 +END __ddx_77668c50_1_contraction_35 0.04020279203541577 12288 +BEGIN __ddx_77668c50_1_region_36_part_0 +END __ddx_77668c50_1_region_36_part_0 2.6834080927073956e-05 None +BEGIN __ddx_77668c50_1_region_36 +END __ddx_77668c50_1_region_36 2.595805563032627e-05 None +BEGIN __ddx_77668c50_1_cotangent_127 +END __ddx_77668c50_1_cotangent_127 0.06694491696543992 16 +BEGIN __ddx_77668c50_1_region_37_part_0 +END __ddx_77668c50_1_region_37_part_0 3.0374969355762005e-05 None +BEGIN __ddx_77668c50_1_region_37 +END __ddx_77668c50_1_region_37 0.07562120794318616 12288 +BEGIN __ddx_77668c50_1_cotangent_126 +END __ddx_77668c50_1_cotangent_126 0.06294774997513741 16 +BEGIN __ddx_77668c50_1_region_38_part_0 +END __ddx_77668c50_1_region_38_part_0 2.6416033506393433e-05 None +BEGIN __ddx_77668c50_1_region_38 +END __ddx_77668c50_1_region_38 0.07143750006798655 12288 +BEGIN __ddx_77668c50_1_cotangent_125 +END __ddx_77668c50_1_cotangent_125 0.6721989169018343 12288 +BEGIN __ddx_77668c50_1_cotangent_124 +END __ddx_77668c50_1_cotangent_124 0.03276158298831433 12288 +BEGIN __ddx_77668c50_1_contraction_39 +END __ddx_77668c50_1_contraction_39 0.010723292012698948 3072 +BEGIN __ddx_77668c50_1_cotangent_123 +END __ddx_77668c50_1_cotangent_123 0.000927999964915216 192 +BEGIN __ddx_77668c50_1_cotangent_122 +END __ddx_77668c50_1_cotangent_122 0.012553540989756584 12288 +BEGIN __ddx_77668c50_1_contraction_42 +END __ddx_77668c50_1_contraction_42 0.026480833999812603 12288 +BEGIN __ddx_77668c50_1_region_43_part_0 +END __ddx_77668c50_1_region_43_part_0 3.0290917493402958e-05 None +BEGIN __ddx_77668c50_1_region_43 +END __ddx_77668c50_1_region_43 2.8250040486454964e-05 None +BEGIN __ddx_77668c50_1_cotangent_121 +END __ddx_77668c50_1_cotangent_121 0.00952016597148031 1632 +BEGIN __ddx_77668c50_1_cotangent_120 +END __ddx_77668c50_1_cotangent_120 0.01110208302270621 12288 +BEGIN __ddx_77668c50_1_contraction_44 +END __ddx_77668c50_1_contraction_44 0.02635954099241644 12288 +BEGIN __ddx_77668c50_1_region_45_part_0 +END __ddx_77668c50_1_region_45_part_0 2.4792039766907692e-05 None +BEGIN __ddx_77668c50_1_region_45 +END __ddx_77668c50_1_region_45 2.5916029699146748e-05 None +BEGIN __ddx_77668c50_1_cotangent_119 +END __ddx_77668c50_1_cotangent_119 0.011356958071701229 12288 +BEGIN __ddx_77668c50_1_contraction_46 +END __ddx_77668c50_1_contraction_46 0.026319041964598 12288 +BEGIN __ddx_77668c50_1_region_47_part_0 +END __ddx_77668c50_1_region_47_part_0 2.350006252527237e-05 None +BEGIN __ddx_77668c50_1_region_47 +END __ddx_77668c50_1_region_47 2.437504008412361e-05 None +BEGIN __ddx_77668c50_1_cotangent_118 +END __ddx_77668c50_1_cotangent_118 0.20648729102686048 16 +BEGIN __ddx_77668c50_1_region_48_part_0 +END __ddx_77668c50_1_region_48_part_0 2.916704397648573e-05 None +BEGIN __ddx_77668c50_1_region_48 +END __ddx_77668c50_1_region_48 0.076638541999273 12288 +BEGIN __ddx_77668c50_1_cotangent_117 +END __ddx_77668c50_1_cotangent_117 0.21248766605276614 16 +BEGIN __ddx_77668c50_1_region_49_part_0 +END __ddx_77668c50_1_region_49_part_0 2.862501423805952e-05 None +BEGIN __ddx_77668c50_1_region_49 +END __ddx_77668c50_1_region_49 0.07313508295919746 12288 +BEGIN __ddx_77668c50_1_cotangent_116 +END __ddx_77668c50_1_cotangent_116 0.9251605829922482 12288 +BEGIN __ddx_77668c50_1_cotangent_115 +END __ddx_77668c50_1_cotangent_115 0.06705462501849979 49152 +BEGIN __ddx_77668c50_1_contraction_50 +END __ddx_77668c50_1_contraction_50 0.04027845896780491 12288 +BEGIN __ddx_77668c50_1_region_51_part_0 +END __ddx_77668c50_1_region_51_part_0 2.8083915822207928e-05 None +BEGIN __ddx_77668c50_1_region_51 +END __ddx_77668c50_1_region_51 2.5042099878191948e-05 None +BEGIN __ddx_77668c50_1_cotangent_114 +END __ddx_77668c50_1_cotangent_114 0.06475766689982265 16 +BEGIN __ddx_77668c50_1_region_52_part_0 +END __ddx_77668c50_1_region_52_part_0 2.6499968953430653e-05 None +BEGIN __ddx_77668c50_1_region_52 +END __ddx_77668c50_1_region_52 0.06857654196210206 12288 +BEGIN __ddx_77668c50_1_cotangent_113 +END __ddx_77668c50_1_cotangent_113 0.06009624991565943 16 +BEGIN __ddx_77668c50_1_region_53_part_0 +END __ddx_77668c50_1_region_53_part_0 2.9666000045835972e-05 None +BEGIN __ddx_77668c50_1_region_53 +END __ddx_77668c50_1_region_53 0.06808991695288569 12288 +BEGIN __ddx_77668c50_1_cotangent_112 +END __ddx_77668c50_1_cotangent_112 0.9887757909018546 12288 +BEGIN __ddx_77668c50_1_cotangent_111 +END __ddx_77668c50_1_cotangent_111 0.036072000046260655 12288 +BEGIN __ddx_77668c50_1_contraction_54 +END __ddx_77668c50_1_contraction_54 0.009716957923956215 3072 +BEGIN __ddx_77668c50_1_cotangent_110 +END __ddx_77668c50_1_cotangent_110 0.000960333039984107 192 +BEGIN __ddx_77668c50_1_cotangent_109 +END __ddx_77668c50_1_cotangent_109 0.012180833029560745 12288 +BEGIN __ddx_77668c50_1_contraction_57 +END __ddx_77668c50_1_contraction_57 0.027386458008550107 12288 +BEGIN __ddx_77668c50_1_region_58_part_0 +END __ddx_77668c50_1_region_58_part_0 2.7791946195065975e-05 None +BEGIN __ddx_77668c50_1_region_58 +END __ddx_77668c50_1_region_58 3.1458097510039806e-05 None +BEGIN __ddx_77668c50_1_cotangent_108 +END __ddx_77668c50_1_cotangent_108 0.008764583035372198 1632 +BEGIN __ddx_77668c50_1_cotangent_107 +END __ddx_77668c50_1_cotangent_107 0.01332791696768254 12288 +BEGIN __ddx_77668c50_1_contraction_59 +END __ddx_77668c50_1_contraction_59 0.027412874973379076 12288 +BEGIN __ddx_77668c50_1_region_60_part_0 +END __ddx_77668c50_1_region_60_part_0 2.5707995519042015e-05 None +BEGIN __ddx_77668c50_1_region_60 +END __ddx_77668c50_1_region_60 2.6124995201826096e-05 None +BEGIN __ddx_77668c50_1_cotangent_106 +END __ddx_77668c50_1_cotangent_106 0.012232374981977046 12288 +BEGIN __ddx_77668c50_1_contraction_61 +END __ddx_77668c50_1_contraction_61 0.025628041941672564 12288 +BEGIN __ddx_77668c50_1_region_62_part_0 +END __ddx_77668c50_1_region_62_part_0 2.6250025257468224e-05 None +BEGIN __ddx_77668c50_1_region_62 +END __ddx_77668c50_1_region_62 2.762500662356615e-05 None +BEGIN __ddx_77668c50_1_cotangent_105 +END __ddx_77668c50_1_cotangent_105 0.17379587492905557 16 +BEGIN __ddx_77668c50_1_region_63_part_0 +END __ddx_77668c50_1_region_63_part_0 2.879102248698473e-05 None +BEGIN __ddx_77668c50_1_region_63 +END __ddx_77668c50_1_region_63 0.07283958292100579 12288 +BEGIN __ddx_77668c50_1_cotangent_104 +END __ddx_77668c50_1_cotangent_104 0.16518420807551593 16 +BEGIN __ddx_77668c50_1_region_64_part_0 +END __ddx_77668c50_1_region_64_part_0 2.7709058485925198e-05 None +BEGIN __ddx_77668c50_1_region_64 +END __ddx_77668c50_1_region_64 0.07042208395432681 12288 +BEGIN __ddx_77668c50_1_cotangent_103 +END __ddx_77668c50_1_cotangent_103 1.8376121249748394 12288 +BEGIN __ddx_77668c50_1_cotangent_102 +END __ddx_77668c50_1_cotangent_102 0.06407316704280674 49152 +BEGIN __ddx_77668c50_1_contraction_65 +END __ddx_77668c50_1_contraction_65 0.04242841701488942 12288 +BEGIN __ddx_77668c50_1_region_66_part_0 +END __ddx_77668c50_1_region_66_part_0 2.9582995921373367e-05 None +BEGIN __ddx_77668c50_1_region_66 +END __ddx_77668c50_1_region_66 3.0458904802799225e-05 None +BEGIN __ddx_77668c50_1_cotangent_101 +END __ddx_77668c50_1_cotangent_101 0.06234683294314891 16 +BEGIN __ddx_77668c50_1_region_67_part_0 +END __ddx_77668c50_1_region_67_part_0 2.6332912966609e-05 None +BEGIN __ddx_77668c50_1_region_67 +END __ddx_77668c50_1_region_67 0.0691028330475092 12288 +BEGIN __ddx_77668c50_1_cotangent_100 +END __ddx_77668c50_1_cotangent_100 0.06059570796787739 16 +BEGIN __ddx_77668c50_1_region_68_part_0 +END __ddx_77668c50_1_region_68_part_0 2.720800694078207e-05 None +BEGIN __ddx_77668c50_1_region_68 +END __ddx_77668c50_1_region_68 0.06989633396733552 12288 +BEGIN __ddx_77668c50_1_cotangent_99 +END __ddx_77668c50_1_cotangent_99 1.9933694589417428 12288 +BEGIN __ddx_77668c50_1_cotangent_98 +END __ddx_77668c50_1_cotangent_98 0.029232957982458174 12288 +BEGIN __ddx_77668c50_1_contraction_69 +END __ddx_77668c50_1_contraction_69 0.010601499932818115 3072 +BEGIN __ddx_77668c50_1_cotangent_97 +END __ddx_77668c50_1_cotangent_97 0.0009832920040935278 192 +BEGIN __ddx_77668c50_1_cotangent_96 +END __ddx_77668c50_1_cotangent_96 0.010922832996584475 12288 +BEGIN __ddx_77668c50_1_contraction_72 +END __ddx_77668c50_1_contraction_72 0.02666370803490281 12288 +BEGIN __ddx_77668c50_1_region_73_part_0 +END __ddx_77668c50_1_region_73_part_0 2.6875059120357037e-05 None +BEGIN __ddx_77668c50_1_region_73 +END __ddx_77668c50_1_region_73 2.791709266602993e-05 None +BEGIN __ddx_77668c50_1_cotangent_95 +END __ddx_77668c50_1_cotangent_95 0.00900787499267608 1632 +BEGIN __ddx_77668c50_1_cotangent_94 +END __ddx_77668c50_1_cotangent_94 0.010672166012227535 12288 +BEGIN __ddx_77668c50_1_contraction_74 +END __ddx_77668c50_1_contraction_74 0.02639195800293237 12288 +BEGIN __ddx_77668c50_1_region_75_part_0 +END __ddx_77668c50_1_region_75_part_0 2.537504769861698e-05 None +BEGIN __ddx_77668c50_1_region_75 +END __ddx_77668c50_1_region_75 2.437504008412361e-05 None +BEGIN __ddx_77668c50_1_cotangent_93 +END __ddx_77668c50_1_cotangent_93 0.010961374966427684 12288 +BEGIN __ddx_77668c50_1_contraction_76 +END __ddx_77668c50_1_contraction_76 0.02626679209060967 12288 +BEGIN __ddx_77668c50_1_region_77_part_0 +END __ddx_77668c50_1_region_77_part_0 2.8709066100418568e-05 None +BEGIN __ddx_77668c50_1_region_77 +END __ddx_77668c50_1_region_77 2.9209069907665253e-05 None +BEGIN __ddx_77668c50_1_cotangent_92 +END __ddx_77668c50_1_cotangent_92 0.1962049170397222 16 +BEGIN __ddx_77668c50_1_region_78_part_0 +END __ddx_77668c50_1_region_78_part_0 2.90420139208436e-05 None +BEGIN __ddx_77668c50_1_region_78 +END __ddx_77668c50_1_region_78 0.06969262508209795 12288 +BEGIN __ddx_77668c50_1_cotangent_91 +END __ddx_77668c50_1_cotangent_91 0.170298625016585 16 +BEGIN __ddx_77668c50_1_region_79_part_0 +END __ddx_77668c50_1_region_79_part_0 2.650008536875248e-05 None +BEGIN __ddx_77668c50_1_region_79 +END __ddx_77668c50_1_region_79 0.06789029203355312 12288 +BEGIN __ddx_77668c50_1_cotangent_90 +END __ddx_77668c50_1_cotangent_90 2.2552877909038216 12288 +BEGIN __ddx_77668c50_1_cotangent_89 +END __ddx_77668c50_1_cotangent_89 0.050599750014953315 49152 +BEGIN __ddx_77668c50_1_contraction_80 +END __ddx_77668c50_1_contraction_80 0.04058629192877561 12288 +BEGIN __ddx_77668c50_1_region_81_part_0 +END __ddx_77668c50_1_region_81_part_0 2.74579506367445e-05 None +BEGIN __ddx_77668c50_1_region_81 +END __ddx_77668c50_1_region_81 3.12920892611146e-05 None +BEGIN __ddx_77668c50_1_cotangent_88 +END __ddx_77668c50_1_cotangent_88 0.058036250062286854 16 +BEGIN __ddx_77668c50_1_region_82_part_0 +END __ddx_77668c50_1_region_82_part_0 2.775003667920828e-05 None +BEGIN __ddx_77668c50_1_region_82 +END __ddx_77668c50_1_region_82 0.06511633307673037 12288 +BEGIN __ddx_77668c50_1_cotangent_87 +END __ddx_77668c50_1_cotangent_87 0.05621399998199195 16 +BEGIN __ddx_77668c50_1_region_83_part_0 +END __ddx_77668c50_1_region_83_part_0 0.06669704196974635 12288 +BEGIN __ddx_77668c50_1_region_83 +END __ddx_77668c50_1_region_83 2.0832987502217293e-05 None +BEGIN __ddx_77668c50_1_cotangent_86 +END __ddx_77668c50_1_cotangent_86 2.363810332957655 12288 +BEGIN __ddx_77668c50_1_cotangent_85 +END __ddx_77668c50_1_cotangent_85 0.027546625002287328 12288 +BEGIN __ddx_77668c50_1_contraction_84 +END __ddx_77668c50_1_contraction_84 0.009953915956430137 3072 +BEGIN __ddx_77668c50_1_cotangent_84 +END __ddx_77668c50_1_cotangent_84 0.002023500041104853 192 +BEGIN __ddx_77668c50_1_cotangent_83 +END __ddx_77668c50_1_cotangent_83 0.010290708974935114 12288 +BEGIN __ddx_77668c50_1_contraction_87 +END __ddx_77668c50_1_contraction_87 0.026293499977327883 12288 +BEGIN __ddx_77668c50_1_region_88_part_0 +END __ddx_77668c50_1_region_88_part_0 2.7875066734850407e-05 None +BEGIN __ddx_77668c50_1_region_88 +END __ddx_77668c50_1_region_88 1.750001683831215e-05 None +BEGIN __ddx_77668c50_1_cotangent_82 +END __ddx_77668c50_1_cotangent_82 0.008754208101890981 1632 +BEGIN __ddx_77668c50_1_cotangent_81 +END __ddx_77668c50_1_cotangent_81 0.01026749995071441 12288 +BEGIN __ddx_77668c50_1_contraction_89 +END __ddx_77668c50_1_contraction_89 0.025893959100358188 12288 +BEGIN __ddx_77668c50_1_region_90_part_0 +END __ddx_77668c50_1_region_90_part_0 2.4374923668801785e-05 None +BEGIN __ddx_77668c50_1_region_90 +END __ddx_77668c50_1_region_90 1.6666948795318604e-05 None +BEGIN __ddx_77668c50_1_cotangent_80 +END __ddx_77668c50_1_cotangent_80 0.010728333028964698 12288 +BEGIN __ddx_77668c50_1_contraction_91 +END __ddx_77668c50_1_contraction_91 0.025848207995295525 12288 +BEGIN __ddx_77668c50_1_region_92_part_0 +END __ddx_77668c50_1_region_92_part_0 2.3417058400809765e-05 None +BEGIN __ddx_77668c50_1_region_92 +END __ddx_77668c50_1_region_92 2.1124957129359245e-05 None +BEGIN __ddx_77668c50_1_cotangent_79 +END __ddx_77668c50_1_cotangent_79 0.17710591608192772 16 +BEGIN __ddx_77668c50_1_region_93_part_0 +END __ddx_77668c50_1_region_93_part_0 0.06697737495414913 12288 +BEGIN __ddx_77668c50_1_region_93 +END __ddx_77668c50_1_region_93 2.0416919142007828e-05 None +BEGIN __ddx_77668c50_1_cotangent_78 +END __ddx_77668c50_1_cotangent_78 0.16314199997577816 16 +BEGIN __ddx_77668c50_1_region_94_part_0 +END __ddx_77668c50_1_region_94_part_0 0.07889883301686496 12288 +BEGIN __ddx_77668c50_1_region_94 +END __ddx_77668c50_1_region_94 0.0018761249957606196 12288 +BEGIN __ddx_77668c50_1_cotangent_77 +END __ddx_77668c50_1_cotangent_77 2.63158937508706 12288 +BEGIN __ddx_77668c50_1_cotangent_76 +END __ddx_77668c50_1_cotangent_76 0.06362154195085168 49152 +BEGIN __ddx_77668c50_1_contraction_95 +END __ddx_77668c50_1_contraction_95 0.04376295895781368 12288 +BEGIN __ddx_77668c50_1_region_96_part_0 +END __ddx_77668c50_1_region_96_part_0 4.02920413762331e-05 None +BEGIN __ddx_77668c50_1_region_96 +END __ddx_77668c50_1_region_96 2.679100725799799e-05 None +BEGIN __ddx_77668c50_1_cotangent_75 +END __ddx_77668c50_1_cotangent_75 0.06775716692209244 16 +BEGIN __ddx_77668c50_1_region_97_part_0 +END __ddx_77668c50_1_region_97_part_0 0.07081891596317291 12288 +BEGIN __ddx_77668c50_1_region_97 +END __ddx_77668c50_1_region_97 0.008294790983200073 12288 +BEGIN __ddx_77668c50_1_region_98 +END __ddx_77668c50_1_region_98 0.1231116660637781 12288 +BEGIN __ddx_77668c50_1_cotangent_73 +END __ddx_77668c50_1_cotangent_73 2.7597610409138724 12288 +BEGIN __ddx_77668c50_1_cotangent_72 +END __ddx_77668c50_1_cotangent_72 0.027777082985267043 12288 +BEGIN __ddx_77668c50_1_contraction_99 +END __ddx_77668c50_1_contraction_99 0.009504959103651345 3072 +BEGIN __ddx_77668c50_1_cotangent_71 +END __ddx_77668c50_1_cotangent_71 0.0009360420517623425 192 +BEGIN __ddx_77668c50_1_cotangent_70 +END __ddx_77668c50_1_cotangent_70 0.010849666083231568 12288 +BEGIN __ddx_77668c50_1_contraction_102 +END __ddx_77668c50_1_contraction_102 0.026070208987221122 12288 +BEGIN __ddx_77668c50_1_region_103_part_0 +END __ddx_77668c50_1_region_103_part_0 2.833292819559574e-05 None +BEGIN __ddx_77668c50_1_region_103 +END __ddx_77668c50_1_region_103 1.579197123646736e-05 None +BEGIN __ddx_77668c50_1_cotangent_69 +END __ddx_77668c50_1_cotangent_69 0.00907666701823473 1632 +BEGIN __ddx_77668c50_1_cotangent_68 +END __ddx_77668c50_1_cotangent_68 0.01040354196447879 12288 +BEGIN __ddx_77668c50_1_contraction_104 +END __ddx_77668c50_1_contraction_104 0.026295042014680803 12288 +BEGIN __ddx_77668c50_1_region_105_part_0 +END __ddx_77668c50_1_region_105_part_0 2.4250010028481483e-05 None +BEGIN __ddx_77668c50_1_region_105 +END __ddx_77668c50_1_region_105 2.091703936457634e-05 None +BEGIN __ddx_77668c50_1_cotangent_67 +END __ddx_77668c50_1_cotangent_67 0.012566374964080751 12288 +BEGIN __ddx_77668c50_1_contraction_106 +END __ddx_77668c50_1_contraction_106 0.026038791052997112 12288 +BEGIN __ddx_77668c50_1_region_107_part_0 +END __ddx_77668c50_1_region_107_part_0 2.762500662356615e-05 None +BEGIN __ddx_77668c50_1_region_107 +END __ddx_77668c50_1_region_107 1.3708020560443401e-05 None +BEGIN __ddx_77668c50_1_cotangent_66 +END __ddx_77668c50_1_cotangent_66 0.17567620798945427 16 +BEGIN __ddx_77668c50_1_region_108_part_0 +END __ddx_77668c50_1_region_108_part_0 0.06950541702099144 12288 +BEGIN __ddx_77668c50_1_region_108 +END __ddx_77668c50_1_region_108 0.0024469579802826047 12288 +BEGIN __ddx_77668c50_1_region_109 +END __ddx_77668c50_1_region_109 0.24583500006701797 12288 +BEGIN __ddx_77668c50_1_cotangent_64 +END __ddx_77668c50_1_cotangent_64 2.99034879100509 12288 +BEGIN __ddx_77668c50_1_cotangent_63 +END __ddx_77668c50_1_cotangent_63 0.06042691704351455 49152 +BEGIN __ddx_77668c50_1_contraction_110 +END __ddx_77668c50_1_contraction_110 0.0385143329622224 12288 +BEGIN __ddx_77668c50_1_region_111_part_0 +END __ddx_77668c50_1_region_111_part_0 2.8165988624095917e-05 None +BEGIN __ddx_77668c50_1_region_111 +END __ddx_77668c50_1_region_111 1.4542019926011562e-05 None +BEGIN __ddx_77668c50_1_region_112 +END __ddx_77668c50_1_region_112 0.1062448329757899 12288 +BEGIN __ddx_77668c50_1_region_113 +END __ddx_77668c50_1_region_113 0.10739079199265689 12288 +BEGIN __ddx_77668c50_1_cotangent_60 +END __ddx_77668c50_1_cotangent_60 3.0548657500185072 12288 +BEGIN __ddx_77668c50_1_cotangent_59 +END __ddx_77668c50_1_cotangent_59 0.029375583049841225 12288 +BEGIN __ddx_77668c50_1_contraction_114 +END __ddx_77668c50_1_contraction_114 0.01032187498640269 3072 +BEGIN __ddx_77668c50_1_cotangent_58 +END __ddx_77668c50_1_cotangent_58 0.0009709169389680028 192 +BEGIN __ddx_77668c50_1_cotangent_57 +END __ddx_77668c50_1_cotangent_57 0.01194716605823487 12288 +BEGIN __ddx_77668c50_1_contraction_117 +END __ddx_77668c50_1_contraction_117 0.026498667080886662 12288 +BEGIN __ddx_77668c50_1_region_118_part_0 +END __ddx_77668c50_1_region_118_part_0 2.6165973395109177e-05 None +BEGIN __ddx_77668c50_1_region_118 +END __ddx_77668c50_1_region_118 1.108401920646429e-05 None +BEGIN __ddx_77668c50_1_cotangent_56 +END __ddx_77668c50_1_cotangent_56 0.009193707956001163 1632 +BEGIN __ddx_77668c50_1_cotangent_55 +END __ddx_77668c50_1_cotangent_55 0.010250957915559411 12288 +BEGIN __ddx_77668c50_1_contraction_119 +END __ddx_77668c50_1_contraction_119 0.026454624952748418 12288 +BEGIN __ddx_77668c50_1_region_120_part_0 +END __ddx_77668c50_1_region_120_part_0 2.3667002096772194e-05 None +BEGIN __ddx_77668c50_1_region_120 +END __ddx_77668c50_1_region_120 1.108401920646429e-05 None +BEGIN __ddx_77668c50_1_cotangent_54 +END __ddx_77668c50_1_cotangent_54 0.012044292059727013 12288 +BEGIN __ddx_77668c50_1_contraction_121 +END __ddx_77668c50_1_contraction_121 0.026352624990977347 12288 +BEGIN __ddx_77668c50_1_region_122_part_0 +END __ddx_77668c50_1_region_122_part_0 3.0791969038546085e-05 None +BEGIN __ddx_77668c50_1_region_122 +END __ddx_77668c50_1_region_122 1.1625001206994057e-05 None +BEGIN __ddx_77668c50_1_region_123 +END __ddx_77668c50_1_region_123 0.26778837491292506 12288 +BEGIN __ddx_77668c50_1_region_124 +END __ddx_77668c50_1_region_124 0.23227116593625396 12288 +BEGIN __ddx_77668c50_1_cotangent_51 +END __ddx_77668c50_1_cotangent_51 3.34633904194925 12288 +BEGIN __ddx_77668c50_1_cotangent_50 +END __ddx_77668c50_1_cotangent_50 0.0507029170403257 49152 +BEGIN __ddx_77668c50_1_region_126 +END __ddx_77668c50_1_region_126 0.09893162501975894 12288 +BEGIN __ddx_77668c50_1_region_127 +END __ddx_77668c50_1_region_127 0.061002292088232934 12288 +BEGIN __ddx_77668c50_1_region_128 +END __ddx_77668c50_1_region_128 0.058004042017273605 12288 +BEGIN __ddx_77668c50_1_cotangent_47 +END __ddx_77668c50_1_cotangent_47 3.381214957917109 12288 +BEGIN __ddx_77668c50_1_cotangent_46 +END __ddx_77668c50_1_cotangent_46 0.02793266694061458 12288 +BEGIN __ddx_77668c50_1_contraction_129 +END __ddx_77668c50_1_contraction_129 0.009574875002726912 3072 +BEGIN __ddx_77668c50_1_cotangent_45 +END __ddx_77668c50_1_cotangent_45 0.0009919999865815043 192 +BEGIN __ddx_77668c50_1_cotangent_44 +END __ddx_77668c50_1_cotangent_44 0.011616250034421682 12288 +BEGIN __ddx_77668c50_1_region_133 +END __ddx_77668c50_1_region_133 0.07911391602829099 12288 +BEGIN __ddx_77668c50_1_cotangent_43 +END __ddx_77668c50_1_cotangent_43 0.009020666941069067 1632 +BEGIN __ddx_77668c50_1_cotangent_42 +END __ddx_77668c50_1_cotangent_42 0.010508999926969409 12288 +BEGIN __ddx_77668c50_1_region_135 +END __ddx_77668c50_1_region_135 0.08656133397016674 12288 +BEGIN __ddx_77668c50_1_cotangent_41 +END __ddx_77668c50_1_cotangent_41 0.011059666983783245 12288 +BEGIN __ddx_77668c50_1_region_137 +END __ddx_77668c50_1_region_137 0.08138579095248133 12288 +BEGIN __ddx_77668c50_1_region_138 +END __ddx_77668c50_1_region_138 0.057322915992699564 12288 +BEGIN __ddx_77668c50_1_region_139 +END __ddx_77668c50_1_region_139 0.06327116605825722 12288 +BEGIN __ddx_77668c50_1_cotangent_38 +END __ddx_77668c50_1_cotangent_38 3.414045708021149 12288 +BEGIN __ddx_77668c50_1_cotangent_37 +END __ddx_77668c50_1_cotangent_37 0.06071445799898356 49152 +BEGIN __ddx_77668c50_1_region_141 +END __ddx_77668c50_1_region_141 0.10015566693618894 12288 +BEGIN __ddx_77668c50_1_region_142 +END __ddx_77668c50_1_region_142 0.058600957971066236 12288 +BEGIN __ddx_77668c50_1_region_143 +END __ddx_77668c50_1_region_143 0.05597191606648266 12288 +BEGIN __ddx_77668c50_1_cotangent_34 +END __ddx_77668c50_1_cotangent_34 3.444257374969311 12288 +BEGIN __ddx_77668c50_1_cotangent_33 +END __ddx_77668c50_1_cotangent_33 0.027718459023162723 12288 +BEGIN __ddx_77668c50_1_contraction_144 +END __ddx_77668c50_1_contraction_144 0.009659832925535738 3072 +BEGIN __ddx_77668c50_1_cotangent_32 +END __ddx_77668c50_1_cotangent_32 0.0011042499681934714 192 +BEGIN __ddx_77668c50_1_cotangent_31 +END __ddx_77668c50_1_cotangent_31 0.011012959061190486 12288 +BEGIN __ddx_77668c50_1_region_148 +END __ddx_77668c50_1_region_148 0.08023829106241465 12288 +BEGIN __ddx_77668c50_1_cotangent_30 +END __ddx_77668c50_1_cotangent_30 0.009136374923400581 1632 +BEGIN __ddx_77668c50_1_cotangent_29 +END __ddx_77668c50_1_cotangent_29 0.010522959055379033 12288 +BEGIN __ddx_77668c50_1_region_150 +END __ddx_77668c50_1_region_150 0.07610620791092515 12288 +BEGIN __ddx_77668c50_1_cotangent_28 +END __ddx_77668c50_1_cotangent_28 0.011425500037148595 12288 +BEGIN __ddx_77668c50_1_region_152 +END __ddx_77668c50_1_region_152 0.07607670791912824 12288 +BEGIN __ddx_77668c50_1_region_153 +END __ddx_77668c50_1_region_153 0.062334165908396244 12288 +BEGIN __ddx_77668c50_1_region_154 +END __ddx_77668c50_1_region_154 0.05879200005438179 12288 +BEGIN __ddx_77668c50_1_cotangent_25 +END __ddx_77668c50_1_cotangent_25 3.748901999904774 12288 +BEGIN __ddx_77668c50_1_cotangent_24 +END __ddx_77668c50_1_cotangent_24 0.06139124999754131 49152 +BEGIN __ddx_77668c50_1_region_156 +END __ddx_77668c50_1_region_156 0.10600324999541044 12288 +BEGIN __ddx_77668c50_1_region_157 +END __ddx_77668c50_1_region_157 0.058738791034556925 12288 +BEGIN __ddx_77668c50_1_region_158 +END __ddx_77668c50_1_region_158 0.05971570801921189 12288 +BEGIN __ddx_77668c50_1_cotangent_21 +END __ddx_77668c50_1_cotangent_21 3.78360225004144 12288 +BEGIN __ddx_77668c50_1_cotangent_20 +END __ddx_77668c50_1_cotangent_20 0.033147166948765516 12288 +BEGIN __ddx_77668c50_1_contraction_159 +END __ddx_77668c50_1_contraction_159 0.01250512502156198 3072 +BEGIN __ddx_77668c50_1_cotangent_19 +END __ddx_77668c50_1_cotangent_19 0.0014916249783709645 192 +BEGIN __ddx_77668c50_1_cotangent_18 +END __ddx_77668c50_1_cotangent_18 0.010886540985666215 12288 +BEGIN __ddx_77668c50_1_region_163 +END __ddx_77668c50_1_region_163 0.08381516602821648 12288 +BEGIN __ddx_77668c50_1_cotangent_17 +END __ddx_77668c50_1_cotangent_17 0.011634666938334703 1632 +BEGIN __ddx_77668c50_1_cotangent_16 +END __ddx_77668c50_1_cotangent_16 0.011490958044305444 12288 +BEGIN __ddx_77668c50_1_region_165 +END __ddx_77668c50_1_region_165 0.09473212494049221 12288 +BEGIN __ddx_77668c50_1_cotangent_15 +END __ddx_77668c50_1_cotangent_15 0.012906749965623021 12288 +BEGIN __ddx_77668c50_1_region_167 +END __ddx_77668c50_1_region_167 0.09221333300229162 12288 +BEGIN __ddx_77668c50_1_region_168 +END __ddx_77668c50_1_region_168 0.06276095798239112 12288 +BEGIN __ddx_77668c50_1_region_169 +END __ddx_77668c50_1_region_169 0.0598087910329923 12288 +BEGIN __ddx_77668c50_1_cotangent_12 +END __ddx_77668c50_1_cotangent_12 3.8010427500121295 12288 +BEGIN __ddx_77668c50_1_cotangent_11 +END __ddx_77668c50_1_cotangent_11 0.06231862504500896 49152 +BEGIN __ddx_77668c50_1_region_171 +END __ddx_77668c50_1_region_171 0.09283883403986692 12288 +BEGIN __ddx_77668c50_1_region_172 +END __ddx_77668c50_1_region_172 0.05955383402761072 12288 +BEGIN __ddx_77668c50_1_region_173 +END __ddx_77668c50_1_region_173 0.049293708987534046 12288 +BEGIN __ddx_77668c50_1_cotangent_8 +END __ddx_77668c50_1_cotangent_8 3.5893434589961544 12288 +BEGIN __ddx_77668c50_1_cotangent_7 +END __ddx_77668c50_1_cotangent_7 0.0292200839612633 12288 +BEGIN __ddx_77668c50_1_contraction_174 +END __ddx_77668c50_1_contraction_174 0.009429875062778592 3072 +BEGIN __ddx_77668c50_1_cotangent_6 +END __ddx_77668c50_1_cotangent_6 0.000930874957703054 192 +BEGIN __ddx_77668c50_1_cotangent_5 +END __ddx_77668c50_1_cotangent_5 0.010891583049669862 12288 +BEGIN __ddx_77668c50_1_region_178 +END __ddx_77668c50_1_region_178 0.07440466701518744 12288 +BEGIN __ddx_77668c50_1_cotangent_4 +END __ddx_77668c50_1_cotangent_4 0.009021625039167702 1632 +BEGIN __ddx_77668c50_1_cotangent_3 +END __ddx_77668c50_1_cotangent_3 0.010404875036329031 12288 +BEGIN __ddx_77668c50_1_region_180 +END __ddx_77668c50_1_region_180 0.07736595906317234 12288 +BEGIN __ddx_77668c50_1_cotangent_2 +END __ddx_77668c50_1_cotangent_2 0.01177833299152553 12288 +BEGIN __ddx_77668c50_1_region_182 +END __ddx_77668c50_1_region_182 0.07210104097612202 12288 +BEGIN __ddx_77668c50_1_region_183 +END __ddx_77668c50_1_region_183 0.057141083059832454 12288 +BEGIN __ddx_77668c50_1_cotangent_0 +END __ddx_77668c50_1_cotangent_0 0.008561374968849123 16 +BEGIN __ddx_77668c50_1_grad_0_wte_part_0 +END __ddx_77668c50_1_grad_0_wte_part_0 2.1709012798964977e-05 None +BEGIN __ddx_77668c50_1_grad_0_wte_part_1 +END __ddx_77668c50_1_grad_0_wte_part_1 16.806742792017758 38597376 +BEGIN __ddx_77668c50_1_grad_0_wte +END __ddx_77668c50_1_grad_0_wte 18.579415083979256 38597376 +BEGIN __ddx_77668c50_1_grad_1_wpe +END __ddx_77668c50_1_grad_1_wpe 3.673719041980803 12288 +BEGIN __ddx_77668c50_1_grad_2_ln_1 +END __ddx_77668c50_1_grad_2_ln_1 2.2079905829159543 9216 +BEGIN __ddx_77668c50_1_grad_3_wq_part_0 +END __ddx_77668c50_1_grad_3_wq_part_0 2.90420139208436e-05 None +BEGIN __ddx_77668c50_1_grad_3_wq_part_1 +END __ddx_77668c50_1_grad_3_wq_part_1 0.2470030830008909 589824 +BEGIN __ddx_77668c50_1_grad_3_wq_part_2 +END __ddx_77668c50_1_grad_3_wq_part_2 2.629193477332592e-05 None +BEGIN __ddx_77668c50_1_grad_3_wq_part_3 +END __ddx_77668c50_1_grad_3_wq_part_3 2.3083994165062904e-05 None +BEGIN __ddx_77668c50_1_grad_3_wq_part_4 +END __ddx_77668c50_1_grad_3_wq_part_4 2.0708073861896992e-05 None +BEGIN __ddx_77668c50_1_grad_3_wq_part_5 +END __ddx_77668c50_1_grad_3_wq_part_5 1.825008075684309e-05 None +BEGIN __ddx_77668c50_1_grad_3_wq_part_6 +END __ddx_77668c50_1_grad_3_wq_part_6 1.7417012713849545e-05 None +BEGIN __ddx_77668c50_1_grad_3_wq_part_7 +END __ddx_77668c50_1_grad_3_wq_part_7 1.3915938325226307e-05 None +BEGIN __ddx_77668c50_1_grad_3_wq_part_8 +END __ddx_77668c50_1_grad_3_wq_part_8 0.17627004103269428 589824 +BEGIN __ddx_77668c50_1_grad_3_wq_part_9 +END __ddx_77668c50_1_grad_3_wq_part_9 0.22835233400110155 589824 +BEGIN __ddx_77668c50_1_grad_3_wq_part_10 +END __ddx_77668c50_1_grad_3_wq_part_10 1.7885197499999776 7077888 +BEGIN __ddx_77668c50_1_grad_3_wq +END __ddx_77668c50_1_grad_3_wq 0.32888204196933657 7077888 +BEGIN __ddx_77668c50_1_grad_4_wk_part_0 +END __ddx_77668c50_1_grad_4_wk_part_0 2.4458044208586216e-05 None +BEGIN __ddx_77668c50_1_grad_4_wk_part_1 +END __ddx_77668c50_1_grad_4_wk_part_1 0.2516533749876544 589824 +BEGIN __ddx_77668c50_1_grad_4_wk_part_2 +END __ddx_77668c50_1_grad_4_wk_part_2 2.279097680002451e-05 None +BEGIN __ddx_77668c50_1_grad_4_wk_part_3 +END __ddx_77668c50_1_grad_4_wk_part_3 2.0041014067828655e-05 None +BEGIN __ddx_77668c50_1_grad_4_wk_part_4 +END __ddx_77668c50_1_grad_4_wk_part_4 1.750001683831215e-05 None +BEGIN __ddx_77668c50_1_grad_4_wk_part_5 +END __ddx_77668c50_1_grad_4_wk_part_5 1.5583005733788013e-05 None +BEGIN __ddx_77668c50_1_grad_4_wk_part_6 +END __ddx_77668c50_1_grad_4_wk_part_6 1.5458092093467712e-05 None +BEGIN __ddx_77668c50_1_grad_4_wk_part_7 +END __ddx_77668c50_1_grad_4_wk_part_7 2.1125073544681072e-05 None +BEGIN __ddx_77668c50_1_grad_4_wk_part_8 +END __ddx_77668c50_1_grad_4_wk_part_8 0.16841829207260162 589824 +BEGIN __ddx_77668c50_1_grad_4_wk_part_9 +END __ddx_77668c50_1_grad_4_wk_part_9 0.23982133297249675 589824 +BEGIN __ddx_77668c50_1_grad_4_wk_part_10 +END __ddx_77668c50_1_grad_4_wk_part_10 1.844542457954958 7077888 +BEGIN __ddx_77668c50_1_grad_4_wk +END __ddx_77668c50_1_grad_4_wk 0.3327935839770362 7077888 +BEGIN __ddx_77668c50_1_grad_5_wv_part_0 +END __ddx_77668c50_1_grad_5_wv_part_0 2.4290988221764565e-05 None +BEGIN __ddx_77668c50_1_grad_5_wv_part_1 +END __ddx_77668c50_1_grad_5_wv_part_1 0.2448859999421984 589824 +BEGIN __ddx_77668c50_1_grad_5_wv_part_2 +END __ddx_77668c50_1_grad_5_wv_part_2 2.266606315970421e-05 None +BEGIN __ddx_77668c50_1_grad_5_wv_part_3 +END __ddx_77668c50_1_grad_5_wv_part_3 2.7499976567924023e-05 None +BEGIN __ddx_77668c50_1_grad_5_wv_part_4 +END __ddx_77668c50_1_grad_5_wv_part_4 1.7666956409811974e-05 None +BEGIN __ddx_77668c50_1_grad_5_wv_part_5 +END __ddx_77668c50_1_grad_5_wv_part_5 2.0832987502217293e-05 None +BEGIN __ddx_77668c50_1_grad_5_wv_part_6 +END __ddx_77668c50_1_grad_5_wv_part_6 1.6584061086177826e-05 None +BEGIN __ddx_77668c50_1_grad_5_wv_part_7 +END __ddx_77668c50_1_grad_5_wv_part_7 1.3541080988943577e-05 None +BEGIN __ddx_77668c50_1_grad_5_wv_part_8 +END __ddx_77668c50_1_grad_5_wv_part_8 0.1656713749980554 589824 +BEGIN __ddx_77668c50_1_grad_5_wv_part_9 +END __ddx_77668c50_1_grad_5_wv_part_9 0.25231537502259016 589824 +BEGIN __ddx_77668c50_1_grad_5_wv_part_10 +END __ddx_77668c50_1_grad_5_wv_part_10 1.7878661249997094 7077888 +BEGIN __ddx_77668c50_1_grad_5_wv +END __ddx_77668c50_1_grad_5_wv 0.3340878749731928 7077888 +BEGIN __ddx_77668c50_1_grad_6_wo +END __ddx_77668c50_1_grad_6_wo 1.5174185830401257 7077888 +BEGIN __ddx_77668c50_1_grad_7_ln_2 +END __ddx_77668c50_1_grad_7_ln_2 0.4984346659621224 9216 +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_0 +END __ddx_77668c50_1_grad_8_w_fc_part_0 2.8000096790492535e-05 None +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_1 +END __ddx_77668c50_1_grad_8_w_fc_part_1 0.3908048749435693 2359296 +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_2 +END __ddx_77668c50_1_grad_8_w_fc_part_2 2.4292035959661007e-05 None +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_3 +END __ddx_77668c50_1_grad_8_w_fc_part_3 2.4958979338407516e-05 None +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_4 +END __ddx_77668c50_1_grad_8_w_fc_part_4 1.9415980204939842e-05 None +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_5 +END __ddx_77668c50_1_grad_8_w_fc_part_5 2.60830856859684e-05 None +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_6 +END __ddx_77668c50_1_grad_8_w_fc_part_6 1.6291975043714046e-05 None +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_7 +END __ddx_77668c50_1_grad_8_w_fc_part_7 1.8749968148767948e-05 None +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_8 +END __ddx_77668c50_1_grad_8_w_fc_part_8 1.9042054191231728e-05 None +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_9 +END __ddx_77668c50_1_grad_8_w_fc_part_9 0.37979208305478096 2359296 +BEGIN __ddx_77668c50_1_grad_8_w_fc_part_10 +END __ddx_77668c50_1_grad_8_w_fc_part_10 5.487648917012848 28311552 +BEGIN __ddx_77668c50_1_grad_8_w_fc +END __ddx_77668c50_1_grad_8_w_fc 3.759419917012565 28311552 +BEGIN __ddx_77668c50_1_grad_9_w_proj +END __ddx_77668c50_1_grad_9_w_proj 6.316149958991446 28311552 +BEGIN __ddx_77668c50_1_grad_10_ln_f +END __ddx_77668c50_1_grad_10_ln_f 0.09902308299206197 768 +__ddx_77668c50_1_grad_0_wte 18.579415083979256 38597376 +__ddx_77668c50_1_grad_0_wte_part_1 16.806742792017758 38597376 +__ddx_77668c50_1_grad_9_w_proj 6.316149958991446 28311552 +__ddx_77668c50_1_grad_8_w_fc_part_10 5.487648917012848 28311552 +__ddx_77668c50_1_cotangent_12 3.8010427500121295 12288 +__ddx_77668c50_1_cotangent_21 3.78360225004144 12288 +__ddx_77668c50_1_grad_8_w_fc 3.759419917012565 28311552 +__ddx_77668c50_1_cotangent_25 3.748901999904774 12288 +__ddx_77668c50_1_grad_1_wpe 3.673719041980803 12288 +__ddx_77668c50_1_cotangent_8 3.5893434589961544 12288 +__ddx_77668c50_1_cotangent_34 3.444257374969311 12288 +__ddx_77668c50_1_cotangent_38 3.414045708021149 12288 +__ddx_77668c50_1_cotangent_47 3.381214957917109 12288 +__ddx_77668c50_1_cotangent_51 3.34633904194925 12288 +__ddx_77668c50_1_cotangent_60 3.0548657500185072 12288 +__ddx_77668c50_1_cotangent_64 2.99034879100509 12288 +__ddx_77668c50_1_cotangent_73 2.7597610409138724 12288 +__ddx_77668c50_1_cotangent_77 2.63158937508706 12288 +__ddx_77668c50_1_cotangent_86 2.363810332957655 12288 +__ddx_77668c50_1_cotangent_90 2.2552877909038216 12288 diff --git a/benchmarks/apple/dense-build.log b/benchmarks/apple/dense-build.log new file mode 100644 index 00000000..39c2b394 --- /dev/null +++ b/benchmarks/apple/dense-build.log @@ -0,0 +1,16 @@ +🍹 Building a mixed python/rust project +🐍 Found CPython 3.12 at /Users/cromwellian/xql/xarray-sql/.venv/bin/python +🔗 Found pyo3 bindings with abi3-py3.10 support +📡 Using build options features from pyproject.toml +💻 Using `MACOSX_DEPLOYMENT_TARGET=11.0` for aarch64-apple-darwin by default +warning: constant `MAIN_STACK` is never used + --> src/stack.rs:25:18 + | +25 | pub(crate) const MAIN_STACK: u64 = 64 << 20; + | ^^^^^^^^^^ + | + = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default + +warning: `einfold-python` (lib) generated 1 warning + Finished `release` profile [optimized] target(s) in 0.09s +📦 Built wheel for abi3 Python ≥ 3.10 to /private/tmp/xql-mlx-wheels/einfold-0.2.0+xqlmlx.4-cp310-abi3-macosx_11_0_arm64.whl diff --git a/benchmarks/apple/dense-cache32-gpt2-c1024-dense_cpu.csv b/benchmarks/apple/dense-cache32-gpt2-c1024-dense_cpu.csv new file mode 100644 index 00000000..55bf9cbf --- /dev/null +++ b/benchmarks/apple/dense-cache32-gpt2-c1024-dense_cpu.csv @@ -0,0 +1,4 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",dense_cpu,1,4.113954583066516,ok (0e+00),10.783063,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,65,65,0,17246218899,0.19002637185622007,3.955314708990045,0,,,,, +nanogpt,"train (sgd), step 0",dense_cpu,1,171.2869603750296,ok (3e-16),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,707,150733,34359738299,2.290518462541513,174.99057333392557,0,,0.14480854175053537,0.005089541897177696,1,11 +nanogpt,"loss, step 1",dense_cpu,1,3.746646665968001,ok (0e+00),8.782008,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,772,150733,34359738299,2.4281566716963425,178.546197833959,0,,,,1,12 diff --git a/benchmarks/apple/dense-cache32-gpt2-c1024-dense_cpu.log b/benchmarks/apple/dense-cache32-gpt2-c1024-dense_cpu.log new file mode 100644 index 00000000..e8ba2996 --- /dev/null +++ b/benchmarks/apple/dense-cache32-gpt2-c1024-dense_cpu.log @@ -0,0 +1,6 @@ + 12-layer GPT, vocab 50257, 124,337,664 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=1024, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_cpu batch=1 4.114s + train (sgd), step 0 dense_cpu batch=1 171.287s + loss, step 1 dense_cpu batch=1 3.747s diff --git a/benchmarks/apple/dense-cache32-gpt2-c1024-dense_gpu.csv b/benchmarks/apple/dense-cache32-gpt2-c1024-dense_gpu.csv new file mode 100644 index 00000000..d587bb83 --- /dev/null +++ b/benchmarks/apple/dense-cache32-gpt2-c1024-dense_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates,gpu_union_reductions,gpu_scatter_reductions +nanogpt,"loss, step 0",dense_gpu,1,0.5839858750114217,ok (2e-08),10.783063,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,65,65,0,5806422667,0.19667258020490408,0.583741124952212,0,96,994701992,10,188,197,1,4,,,,,,, +nanogpt,"train (sgd), step 0",dense_gpu,1,7.217241375008598,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,707,24579,34359738259,2.311262867413461,7.800123083987273,0,1589,1989415160,148,9603,6814,34,535949432,,0.12084812729153782,0.005107792094349861,1,11,254,227 +nanogpt,"loss, step 1",dense_gpu,1,0.35345016699284315,ok (3e-08),8.782008,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,772,24579,34359738259,2.4369680357631296,8.153323167003691,0,1685,2984117152,148,9791,7011,35,535949436,,,,1,12,254,227 +nanogpt,"train (sgd), step 1",dense_gpu,1,5.712546291993931,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,1414,49158,34359738259,4.692176157492213,13.864777376060374,0,3178,3978830320,148,19206,13628,68,1071898864,,0.1672550031216815,0.005184834124520421,1,23,508,454 +nanogpt,"loss, step 2",dense_gpu,1,0.3430515000363812,ok (4e-08),7.639927,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,1479,49158,34359738259,4.816358660697006,14.207540168077685,0,3274,4973532312,148,19394,13825,69,1071898868,,,,1,24,508,454 +nanogpt,"train (sgd), step 2",dense_gpu,1,5.414057458052412,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2121,73737,34359738259,6.938842817442492,19.620573126128875,0,4767,5968245480,148,28809,20442,102,1607848296,,0.21186996053438634,0.005261417129077017,1,35,762,681 +nanogpt,"loss, step 3",dense_gpu,1,0.3541462919674814,ok (1e-08),6.953781,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2186,73737,34359738259,7.064828693168238,19.974493376212195,0,4863,6962947472,148,28997,20639,103,1607848300,,,,1,36,762,681 +nanogpt,"train (sgd), step 3",dense_gpu,1,5.349663708009757,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2828,98316,34359738316,9.110842370428145,25.32349170919042,0,6356,7957660640,148,38412,27256,136,2143797728,,0.25804446171969175,0.005334625137038529,1,47,1016,908 +nanogpt,"loss, step 4",dense_gpu,1,0.35993366700131446,ok (7e-09),6.847609,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2893,98316,34359738316,9.23799170262646,25.683073543244973,0,6452,8952362632,148,38600,27453,137,2143797732,,,,1,48,1016,908 +nanogpt,"train (sgd), step 4",dense_gpu,1,5.454650625004433,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,3535,122895,34359738316,11.413957489887252,31.136904293322004,0,7945,9947075800,148,48015,34070,170,2679747160,,0.3064046708168462,0.005411083111539483,1,59,1270,1135 diff --git a/benchmarks/apple/dense-cache32-gpt2-c1024-dense_gpu.log b/benchmarks/apple/dense-cache32-gpt2-c1024-dense_gpu.log new file mode 100644 index 00000000..7f6e2512 --- /dev/null +++ b/benchmarks/apple/dense-cache32-gpt2-c1024-dense_gpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 124,337,664 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=1024, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_gpu batch=1 0.584s + train (sgd), step 0 dense_gpu batch=1 7.217s + loss, step 1 dense_gpu batch=1 0.353s + train (sgd), step 1 dense_gpu batch=1 5.713s + loss, step 2 dense_gpu batch=1 0.343s + train (sgd), step 2 dense_gpu batch=1 5.414s + loss, step 3 dense_gpu batch=1 0.354s + train (sgd), step 3 dense_gpu batch=1 5.350s + loss, step 4 dense_gpu batch=1 0.360s + train (sgd), step 4 dense_gpu batch=1 5.455s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | gpu_input_arrays | gpu_input_bytes | gpu_fused_programs | gpu_fused_executions | gpu_contractions | gpu_readbacks | gpu_readback_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | gpu_union_reductions | gpu_scatter_reductions | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_gpu | 1 | 0.584 | ok (2e-08) | 10.783 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 65 | 65 | 0 | 5806422667 | 0.197 | 0.584 | 0 | 96 | 994701992 | 10 | 188 | 197 | 1 | 4 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_gpu | 1 | 7.217 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 707 | 24579 | 34359738259 | 2.311 | 7.800 | 0 | 1589 | 1989415160 | 148 | 9603 | 6814 | 34 | 535949432 | n/a | 0.121 | 0.005 | 1 | 11 | 254 | 227 | +| nanogpt | loss, step 1 | dense_gpu | 1 | 0.353 | ok (3e-08) | 8.782 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 772 | 24579 | 34359738259 | 2.437 | 8.153 | 0 | 1685 | 2984117152 | 148 | 9791 | 7011 | 35 | 535949436 | n/a | n/a | n/a | 1 | 12 | 254 | 227 | +| nanogpt | train (sgd), step 1 | dense_gpu | 1 | 5.713 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 1414 | 49158 | 34359738259 | 4.692 | 13.865 | 0 | 3178 | 3978830320 | 148 | 19206 | 13628 | 68 | 1071898864 | n/a | 0.167 | 0.005 | 1 | 23 | 508 | 454 | +| nanogpt | loss, step 2 | dense_gpu | 1 | 0.343 | ok (4e-08) | 7.640 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 1479 | 49158 | 34359738259 | 4.816 | 14.208 | 0 | 3274 | 4973532312 | 148 | 19394 | 13825 | 69 | 1071898868 | n/a | n/a | n/a | 1 | 24 | 508 | 454 | +| nanogpt | train (sgd), step 2 | dense_gpu | 1 | 5.414 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2121 | 73737 | 34359738259 | 6.939 | 19.621 | 0 | 4767 | 5968245480 | 148 | 28809 | 20442 | 102 | 1607848296 | n/a | 0.212 | 0.005 | 1 | 35 | 762 | 681 | +| nanogpt | loss, step 3 | dense_gpu | 1 | 0.354 | ok (1e-08) | 6.954 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2186 | 73737 | 34359738259 | 7.065 | 19.974 | 0 | 4863 | 6962947472 | 148 | 28997 | 20639 | 103 | 1607848300 | n/a | n/a | n/a | 1 | 36 | 762 | 681 | +| nanogpt | train (sgd), step 3 | dense_gpu | 1 | 5.350 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2828 | 98316 | 34359738316 | 9.111 | 25.323 | 0 | 6356 | 7957660640 | 148 | 38412 | 27256 | 136 | 2143797728 | n/a | 0.258 | 0.005 | 1 | 47 | 1016 | 908 | +| nanogpt | loss, step 4 | dense_gpu | 1 | 0.360 | ok (7e-09) | 6.848 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2893 | 98316 | 34359738316 | 9.238 | 25.683 | 0 | 6452 | 8952362632 | 148 | 38600 | 27453 | 137 | 2143797732 | n/a | n/a | n/a | 1 | 48 | 1016 | 908 | +| nanogpt | train (sgd), step 4 | dense_gpu | 1 | 5.455 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 3535 | 122895 | 34359738316 | 11.414 | 31.137 | 0 | 7945 | 9947075800 | 148 | 48015 | 34070 | 170 | 2679747160 | n/a | 0.306 | 0.005 | 1 | 59 | 1270 | 1135 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-cache32-gpt2-c1024-dense_gpu.csv diff --git a/benchmarks/apple/dense-cache32-gpt2-c1024-runs.json b/benchmarks/apple/dense-cache32-gpt2-c1024-runs.json new file mode 100644 index 00000000..9fa2b550 --- /dev/null +++ b/benchmarks/apple/dense-cache32-gpt2-c1024-runs.json @@ -0,0 +1,74 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-cache32-gpt2-c1024-dense_cpu.csv", + "--context", + "1024", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-cache32-gpt2-c1024-dense_cpu.log", + "csv": "dense-cache32-gpt2-c1024-dense_cpu.csv", + "peak_rss_gib": 53.74195861816406, + "status": "timeout", + "started_utc": "2026-11-01T22:27:38.352408+00:00", + "exit_code": -15, + "wall_seconds": 301.30939437495545, + "checked_rows": 3 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-cache32-gpt2-c1024-dense_gpu.csv", + "--context", + "1024", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-cache32-gpt2-c1024-dense_gpu.log", + "csv": "dense-cache32-gpt2-c1024-dense_gpu.csv", + "peak_rss_gib": 28.424163818359375, + "status": "passed", + "started_utc": "2026-11-01T22:32:39.663819+00:00", + "exit_code": 0, + "wall_seconds": 59.11476125009358, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/dense-controls-runs.json b/benchmarks/apple/dense-controls-runs.json new file mode 100644 index 00000000..d5b62db5 --- /dev/null +++ b/benchmarks/apple/dense-controls-runs.json @@ -0,0 +1,98 @@ +[ + { + "host": "duckdb", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "duckdb", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-duckdb.csv" + ], + "contended": false, + "log": "dense-final-tiny-duckdb.log", + "csv": "dense-final-tiny-duckdb.csv", + "peak_rss_gib": 0.6759490966796875, + "status": "passed", + "started_utc": "2026-11-01T21:59:53.598085+00:00", + "exit_code": 0, + "wall_seconds": 3.886874582967721, + "checked_rows": 10, + "source_manifest": "dense-final-tiny-runs.json" + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-shared_cpu.csv" + ], + "contended": false, + "log": "dense-final-tiny-shared_cpu.log", + "csv": "dense-final-tiny-shared_cpu.csv", + "peak_rss_gib": 0.6473541259765625, + "status": "passed", + "started_utc": "2026-11-01T21:59:57.485449+00:00", + "exit_code": 0, + "wall_seconds": 3.6641790000721812, + "checked_rows": 10, + "source_manifest": "dense-final-tiny-runs.json" + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-shared_gpu.csv" + ], + "contended": false, + "log": "dense-final-tiny-shared_gpu.log", + "csv": "dense-final-tiny-shared_gpu.csv", + "peak_rss_gib": 0.64837646484375, + "status": "passed", + "started_utc": "2026-11-01T22:00:01.150111+00:00", + "exit_code": 0, + "wall_seconds": 3.8351630830438808, + "checked_rows": 10, + "source_manifest": "dense-final-tiny-runs.json" + } +] diff --git a/benchmarks/apple/dense-cpu-profile.log b/benchmarks/apple/dense-cpu-profile.log new file mode 100644 index 00000000..faa196c2 --- /dev/null +++ b/benchmarks/apple/dense-cpu-profile.log @@ -0,0 +1,53 @@ +/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/pyarrow/compute.py:231: FutureWarning: Specifying null_placement in SortOptions is deprecated as of 25.0.0. Specify null_placement per sort_key instead. + return options_class(*args, **kwargs) + 8994777 function calls (8623284 primitive calls) in 10.319 seconds + + Ordered by: cumulative time + List reduced from 390 to 45 due to restriction <45> + + ncalls tottime percall cumtime percall filename:lineno(function) + 1 0.000 0.000 10.321 10.321 _hosts.py:341(query_all) + 1 0.000 0.000 10.321 10.321 dense.py:217() + 1 0.000 0.000 10.321 10.321 dense.py:154(_invoke) + 1 0.000 0.000 10.321 10.321 dense.py:203(execute) + 11 0.000 0.000 8.742 0.795 dense_program.py:930(arrow) + 22 0.670 0.030 5.282 0.240 dense_program.py:299(arrow) +143079/1095 0.627 0.000 4.502 0.004 dense_program.py:56(get) + 11 3.302 0.300 3.302 0.300 compute.py:257(wrapper) + 168/94 1.685 0.010 2.710 0.029 dense_program.py:848(compute) +11570/982 0.192 0.000 1.801 0.002 dense_program.py:385(compute) + 84/10 0.747 0.009 1.513 0.151 dense_program.py:898(valid_count) + 300/297 0.001 0.000 1.340 0.005 dense_program.py:770(compute) + 1 0.000 0.000 1.314 1.314 __init__.py:270(run) + 1 0.030 0.030 1.314 1.314 {method 'run' of 'ddxdb._Prepared' objects} + 620 0.000 0.000 1.187 0.002 dense.py:65(materialize_compiled) + 642 0.001 0.000 1.177 0.002 dense.py:49(run) + 642 0.001 0.000 1.176 0.002 dense_program.py:963(run) +27458/642 0.062 0.000 1.175 0.002 dense_program.py:964(visit) + 11570 0.023 0.000 0.662 0.000 einsumfunc.py:1242(einsum) + 13204 0.288 0.000 0.427 0.000 einsumfunc.py:1144(bmm_einsum) + 21991 0.008 0.000 0.405 0.000 dense_program.py:692(_map) + 4580 0.023 0.000 0.389 0.000 dense_program.py:590(_join) + 11424 0.027 0.000 0.296 0.000 dense_program.py:999() +186165/151211 0.084 0.000 0.268 0.000 dense_program.py:471(_expr) + 16387 0.020 0.000 0.236 0.000 dense_program.py:254(rename) + 11 0.227 0.021 0.227 0.021 compute.py:468(take) + 3767 0.015 0.000 0.217 0.000 dense_program.py:700(_aggregate) + 77363 0.101 0.000 0.196 0.000 dense_program.py:40(_align) + 11570 0.057 0.000 0.194 0.000 einsumfunc.py:634(einsum_path) + 42206 0.010 0.000 0.185 0.000 fromnumeric.py:48(_wrapfunc) + 125/101 0.003 0.000 0.180 0.002 dense_program.py:432(compute) + 63788 0.020 0.000 0.180 0.000 dense_program.py:217(rename) + 143963 0.068 0.000 0.178 0.000 dense_program.py:109(rename) + 283529 0.081 0.000 0.160 0.000 dense_program.py:463(_key) + 620 0.156 0.000 0.159 0.000 dense.py:74(drop_table) + 1 0.000 0.000 0.155 0.155 __init__.py:198(__exit__) + 1 0.000 0.000 0.155 0.155 __init__.py:187(release) + 1 0.000 0.000 0.155 0.155 {method 'release' of 'ddxdb._Release' objects} + 4086 0.132 0.000 0.132 0.000 {built-in method numpy._core._multiarray_umath.c_einsum} + 84 0.001 0.000 0.129 0.002 dense_program.py:826(_aggregate_bag) +44833/44591 0.040 0.000 0.127 0.000 dense_program.py:137(_op) + 3898 0.004 0.000 0.119 0.000 dense_program.py:418(_reduce) + 11 0.106 0.010 0.106 0.010 shared.py:83(_snapshot) + 744 0.106 0.000 0.106 0.000 {method 'ravel' of 'numpy.ndarray' objects} + 186400 0.029 0.000 0.102 0.000 {built-in method builtins.any} diff --git a/benchmarks/apple/dense-environment.json b/benchmarks/apple/dense-environment.json new file mode 100644 index 00000000..4933cfa8 --- /dev/null +++ b/benchmarks/apple/dense-environment.json @@ -0,0 +1,32 @@ +{ + "python": "3.12.14", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "cpu_count": 30, + "machine_cpu": "Apple M5 Ultra", + "memory_bytes": 274877906944, + "git_head": "b05eac34d1293e1d43ab9a83ef8b4c1a444fdf72", + "uncommitted_work": true, + "versions": { + "numpy": "2.5.4", + "pyarrow": "26.0.0", + "xarray": "2026.9.0", + "datafusion": "55.0.0", + "einfold": "0.2.0+xqlmlx.4", + "ddxdb": "0.4.6", + "mlx": "0.32.4", + "jax": "0.12.0.1", + "duckdb": "1.5.6" + }, + "source_sha256": { + "accelerators/einfold_mlx/dense_program.py": "cbc024270fcb2021509dab7598bc285649ea41a786e20e5b1c760b0e12f16dfe", + "accelerators/einfold_mlx/dense-backend.patch": "a8836032a0fe06554145f00510f3d0a54ec0229f3ed371db397576b3950abfb2", + "accelerators/einfold_mlx/build.py": "457de5434290cc0ad7574babee15e98d2a44e3fb7b9c0568e5a1f40927b511b7", + "xarray_sql/dense.py": "7fea6bd1f83872ff96cb4e0ee2979de2d2d25d7722fde64f9009d7a7111be14e", + "xarray_sql/_dense_mlx.py": "d17677d824219264bf97e44e11b88ea9dc925ef8dc9bb0db2024d96619ed243f", + "xarray_sql/shared.py": "3f8ad3f0366d5a5677a487d169ed17818e70993141ae1be6ff4d5bda039bac8c", + "xarray_sql/sql.py": "a3981e362badfc41300ad5730360be016ad7875924710e6acb9c4c1cf89245b8", + "benchmarks/ml/02_nanogpt.py": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "benchmarks/ml/_hosts.py": "e6132bd0e65330ae4413f86aa7fa1d958863aa331a291f8ea70bb38b36069f78" + }, + "installed_dense_module_sha256": "cbc024270fcb2021509dab7598bc285649ea41a786e20e5b1c760b0e12f16dfe" +} diff --git a/benchmarks/apple/dense-execution.md b/benchmarks/apple/dense-execution.md new file mode 100644 index 00000000..3c98087a --- /dev/null +++ b/benchmarks/apple/dense-execution.md @@ -0,0 +1,301 @@ +# Compact SQL execution on Apple M5 Ultra + +November 1, 2026. [EinFold](https://github.com/xqlsystems/einfold) lowers +SQL numerical operations to arrays. [ddx](https://github.com/xqlsystems/ddx) +still differentiates the original SQL. The new optional executor retains +compact arrays across shared forward and backward stages, using the +[NumPy](https://numpy.org/) array library with float64 on CPU or +[MLX](https://github.com/ml-explore/mlx) float32 on GPU. +It does not substitute framework model code or MLX automatic differentiation. + +See the [research and execution plan](../../accelerators/einfold_mlx/dense-execution-plan.md) +and [installation/API instructions](../../accelerators/einfold_mlx/README.md). +The implementation is local and experimental, packaged in +`einfold==0.2.0+xqlmlx.4`; it is not in the ordinary upstream wheel. + +## Method + +Machine: Apple M5 Ultra, 30 CPU cores, 64 GPU cores, 256 GiB RAM, +macOS 27.0.1. GPU game contention was off. Hosts ran serially. Every +reported training run passed **five steps**, with each loss and every +parameter update checked against the same float64 +[JAX](https://docs.jax.dev/) array-framework reference outside +the API timer. CPU tolerances remain `rtol=atol=1e-9`; GPU tolerances +remain `3e-5`. GPU runs launched with `MLX_ENABLE_TF32=0`. + +Warm time is the mean loss-plus-training API time at steps 1–4, excluding +step 0. The SQL interface makes separate loss and update calls, including +full [Apache Arrow](https://arrow.apache.org/) columnar-table output +materialization and synchronization. Dataset loading, +model definition, cold first-use time, and total API time are retained in +the summary. Reference computation, parameter validation and conversion +of returned rows into Datasets are outside these API timers. These are +four warm observations, not a confidence interval. + +All SGD runs use the same model SQL, dimensions, tied embeddings, tanh +GELU, inputs and seeds as their controls. The tiny run uses learning rate +0.5; larger runs use 0.01. GPT-2-size runs use vocabulary 50257 and batch 1. +Inputs remain Shakespeare character IDs with a padded vocabulary, rather +than GPT-2 byte-pair tokenization. Context 16 has 123,563,520 parameters; +context 128 has 123,649,536; context 1024 has 124,337,664. All vocabulary +classes and declared parameters participate. + +Sampled RSS is process memory measured once per second. It is not a hard +allocator limit or a complete accounting of Metal driver/unified memory. +The runner kills only its own benchmark process group when a configured +RSS/time bound is exceeded. Incomplete runs contribute no speedup. + +## Measurements + + + +All rows below are completed five-step training runs. Times are seconds. + +| Parameters / context / batch | CPU before | CPU now | CPU speedup | GPU before | GPU now | GPU speedup | +|---|---:|---:|---:|---:|---:|---:| +| 4,416 / 16 / 4 | 0.193 | 0.052 | 3.7× | 0.219 | 0.078 | 2.8× | +| 213,504 / 64 / 1 | 0.431 | 0.061 | 7.1× | 0.482 | 0.083 | 5.8× | +| 10,745,088 / 256 / 1 | 29.257 | 1.406 | 20.8× | 29.937 | 0.589 | 50.8× | +| 123,563,520 / 16 / 1 | 164.034 | 4.037 | 40.6× | 163.646 | 2.434 | 67.2× | + +The 123.6M/context-16 comparison is **40.6× faster on CPU and 67.2× on GPU** than the earlier shared SQL implementation. The new GPU path is 1.66× faster than the new CPU path on that case. CPU remains faster for the smallest models. + +| Parameters / context | Host | Cache GiB | Cold first pair | Warm pair | Five API + loading/model | Peak RSS GiB | +|---|---|---:|---:|---:|---:|---:| +| 4,416 / 16 | [dense_cpu](dense-measured-tiny-runs.json) | 8 | 0.128 | 0.052 | 0.348 | 0.53 | +| 4,416 / 16 | [dense_gpu](dense-measured-tiny-runs.json) | 8 | 0.356 | 0.078 | 0.679 | 0.57 | +| 213,504 / 64 | [dense_cpu](dense-measured-demo-runs.json) | 8 | 0.139 | 0.061 | 0.396 | 0.60 | +| 213,504 / 64 | [dense_gpu](dense-measured-demo-runs.json) | 8 | 0.188 | 0.083 | 0.534 | 0.58 | +| 10,745,088 / 256 | [dense_cpu](dense-measured-shakespeare-runs.json) | 8 | 1.870 | 1.406 | 7.548 | 7.97 | +| 10,745,088 / 256 | [dense_gpu](dense-measured-shakespeare-runs.json) | 8 | 0.993 | 0.589 | 3.401 | 2.78 | +| 123,563,520 / 16 | [dense_cpu](dense-measured-gpt2-c16-runs.json) | 8 | 5.193 | 4.037 | 21.627 | 22.90 | +| 123,563,520 / 16 | [dense_gpu](dense-measured-gpt2-c16-runs.json) | 8 | 3.750 | 2.434 | 13.774 | 17.91 | +| 123,649,536 / 128 | [dense_cpu](dense-measured-gpt2-c128-runs.json) | 32 | 7.194 | 5.787 | 30.638 | 35.78 | +| 123,649,536 / 128 | [dense_gpu](dense-measured-gpt2-c128-runs.json) | 32 | 3.882 | 2.540 | 14.330 | 18.46 | +| 124,337,664 / 1024 | [dense_gpu](dense-measured-gpt2-c1024-runs.json) | 32 | 7.496 | 5.575 | 30.085 | 28.72 | + +Fresh tiny-model DuckDB time is 0.274 s. Compact CPU is 5.29× faster, and compact GPU is 3.53× faster. This is a bounded tiny-model DuckDB comparison, not a claim about full-size DuckDB training. + +The compatible API includes cold ddx preparation and initial MLX compilation in the first pair. Loading/model time includes cumulative input/state writes over the run. Full-context CPU trials and strict dense AdamW refusals are excluded from these tables. The earlier compact cache study improved context-1024 GPU time from 14.242 s at 8 GiB to 5.835 s at 32 GiB; the final 32 GiB run measures 5.575 s. Cache-study records are in [the 8 GiB trial](dense-masks-gpt2-c1024-runs.json) and [32 GiB trial](dense-cache32-gpt2-c1024-runs.json). + +[All final aggregate records](dense-summary.json). The fourteen completed runs contain 140 checked loss/update checkpoints. + + +The [previous shared execution report](nanogpt-shared.md) supplies the +large-model baselines. Tiny DuckDB/shared CPU/shared GPU controls were +rerun during this work, with their commands in +[the control manifest](dense-final-tiny-runs.json). Development runs with +older intermediate implementations are retained separately and are not +used as final measurements. + +## What changed + +- Coordinate axes are represented once. Numerical values remain compact + arrays between SQL stages. Scalar key columns expand at public output + boundaries. +- Actual [Apache DataFusion](https://datafusion.apache.org/) SQL-engine + logical joins, projections, CASE expressions, + contractions, full-partition windows, gathers and grouped reductions + lower to an EinFold array program. Shared CTEs and ddx named reads bind + to current array slots. Numerical stages do not build hash joins or + reconstruct intermediate Arrow coordinates. +- Embedding gradients use segmented/scatter reductions. UNION remains a + bag union until a following grouped SUM justifies accumulating its + contributions. Domain left joins still fill missing gradients correctly. +- Exact domains keep row-presence and NULL masks compact. Constant + coordinate predicates slice axes before numerical work, including + predicates within conjunctions. Coordinate ordering is established + before row expansion when its exact domains prove the requested order. +- Structural logical programs and compiled pointwise templates are reused + across steps. Every compiled MLX function receives its numerical leaves + explicitly; it does not capture previous parameter values. Input buffers + are imported afresh per call, including external in-place writes. +- Computed arrays have a bounded LRU cache; sources remain pinned for the + call. Values may be recomputed after eviction. Graphs are released at + call completion. Default cache capacity is 8 GiB; the full-context GPU + experiment also measures 32 GiB. This bounds retained computed values, + not all allocations or process memory. + +Private schema-only providers let ddx serialize named read schemas. They +are not executed as numerical scans. Runtime guards and existence checks +evaluate actual data. Unsupported execution restarts the call through the +original shared SQL executor before any parameter update is committed. +Benchmarks set `fallback=False`, so a refusal cannot silently become an +accelerated result. + +## Profiling and device crossings + +The original instrumented backward profile identified these large stages: + +| Stage | Time | Rows | Arrow bytes | +|---|---:|---:|---:| +| Combined token embedding gradient | 18.58 s | 38,597,376 | 926,337,024 | +| One token embedding gradient contribution | 16.81 s | 38,597,376 | 926,337,024 | +| One stacked projection gradient | 6.32 s | 28,311,552 | 905,969,664 | + +One cotangent stage took 3.8 seconds to produce just 12,288 values. Its SQL +contained many independently expanded gradient branches. DataFusion's +outer FFI node count hid those internal operators; it was not a useful +cross-backend operator count. The full diagnostic is in +[the stage JSON](dense-before-stages-summary.json) and +[log](dense-before-stages.log). + +An intermediate compact CPU profile then exposed about 3.3 seconds of +sorting fully expanded output coordinates. Ordering compact axes removed +that work. It also exposed repeated full-domain gradient validity arrays; +domain-aware masks now avoid those allocations where exact facts permit. +[That diagnostic profile](dense-cpu-profile.log) is not a final benchmark. + + +The final diagnostic warm pair takes 5.19 seconds under CPU profiling and +3.62 seconds under GPU profiling. These include profiler overhead and are +not substituted for the uninstrumented benchmark times. + +The GPU profile records 0.278 seconds in input conversion, 0.257 seconds +waiting for pending GPU computation, and 0.00044 seconds creating NumPy +views after that wait. The rest includes Python graph construction, +domain/contribution processing, coordinate expansion, Arrow conversion and +profiler overhead. Every training call returns **3,871,488,000 Arrow bytes** +at context 16, even though its compact GPU parameter values occupy roughly +494 MB. The final CPU profile attributes 1.80 seconds cumulatively to dense +stage execution and 3.08 seconds to output collection, including deferred +numerical work; these nested timings should not be added to other entries. + +[CPU profile](dense-final-cpu-profile.txt), +[CPU phase record](dense-final-cpu-profile.json), +[GPU profile](dense-final-gpu-profile.txt), +[GPU phase record](dense-final-gpu-profile.json). + + +The raw counters distinguish source imports, numerical operations and +readback requests. Readback byte counts are logical array bytes made +available to NumPy; they are not measurements of PCIe traffic. Apple +silicon uses unified memory, and float64-to-float32 imports still copy and +convert. The returned Arrow values keep SQL dtypes, so GPU float32 values +also convert back to Arrow float64. + +Across the five context-16 steps, counters record 140 reusable fused MLX +templates, 20,020 contractions/reductions and 170 readback requests. Source +imports total 9,885,144,280 bytes: approximately one import of each parameter +buffer per loss/update call, plus scalar constants. Requested readback +arrays total 2,664,259,240 bytes, including validity/presence data. The +context-1024 run also uses 170 readback requests, while larger arrays and +cache eviction increase its numerical work. These counts verify actual +GPU dispatch, not just an explained GPU node. + +The 12-layer runs retain 707 logical programs after the first training +step. Their count stays fixed over five steps. The earlier shared backend +built 7,148 physical plans over five steps. Those are different categories +of plan, not equivalent operator counts: the new numerical path uses +logical programs without per-stage physical planning. + +## Native framework comparison + + +The native GPU control uses +[PyTorch](https://pytorch.org/), a tensor framework, through its Metal +Performance Shaders (MPS) backend and the +[upstream nanoGPT model](https://github.com/karpathy/nanoGPT). +For **124,337,664 parameters, context 1024, batch 1 and five training steps**: + +| Implementation | Warm step | Precision | +|---|---:|---| +| Compact SQL + MLX | 5.575 s | float32 arithmetic, Arrow outputs | +| JAX CPU, compiled | 2.445 s | float64, resident arrays | +| Upstream nanoGPT / PyTorch MPS | 0.099 s | float32, resident arrays | + +The SQL API is approximately **2.28× slower than compiled JAX CPU** and +**56.6× slower than PyTorch MPS** in these measurements. Its substantial +improvement over the earlier SQL backend does not eliminate the remaining +gap to native array execution. + +[JAX result](dense-native-gpt2-c1024-jax_jit.json), +[JAX log](dense-native-gpt2-c1024-jax_jit.log), +[nanoGPT/MPS result](dense-native-gpt2-c1024-torch_mps.json), +[nanoGPT/MPS log](dense-native-gpt2-c1024-torch_mps.log). + + +These are API comparisons with important scope differences. SQL performs +a loss call and an update call and returns complete Arrow parameter tables. +The native frameworks keep array state resident and synchronize numerical +work inside their timer, with validation/readback outside. CPU reference +arithmetic is float64; GPU arithmetic is explicitly float32. The framework +harness uses the actual upstream nanoGPT `GPT` class, matching the SQL +math with tanh GELU and SGD; it is not the repository's untouched CUDA +AdamW speed benchmark. Source revisions, actual allocated parameter counts +and all per-step checks are saved in the JSON records. See also the +[original comparison and harness documentation](nanogpt-frameworks.md). + +## Correctness and limitations + + +The frozen implementation passes **695 tests**, with 277 skipped and 102 +network integration tests deselected. The 83 existing warnings are retained +in [the full test log](dense-full-correctness.log). Focused CPU/Metal SQL +and backward tests pass 73 cases. Ruff and focused mypy checks pass; +`git diff --check` and clean application of the pinned native patch pass. +No Rust numerical kernel changed in this phase; the prior native kernel +validation remains in the shared-backend report. + +[Environment and source hashes](dense-environment.json) identify the exact +checkout, dependencies and hardware. The installed EinFold dense module is +byte-identical to the adjacent source included by the wheel builder. +[Build log](dense-build.log), [focused test log](dense-final-contract.log), +[mypy log](dense-mypy.log). + + +Public contract checks cover live values and snapshots, changed coordinate +domains, non-positional/permuted labels, matrix gradients through normalized +operands, nullable and null-safe joins, bag unions, all-NULL/empty groups, +integer accumulators and averages, filtered-out rows, function overrides, +grad/VJP/JVP, stop-gradient and typed refusals. The existing suite continues +to check xarray dimensions, coordinates and attributes. Source arrays are +not mutated by numerical execution. No tolerance was relaxed. + +Full-context CPU execution remains sensitive to recomputation and cache +capacity. An early unbounded trial crossed the 64 GiB RSS guard; bounded +trials stayed within the process bound but did not complete five steps +within their chosen time limits. The completed GPU context-1024 run is +training; the earlier context-1024 shared-backend comparison was only +forward execution. There is no successful old full-context training run +from which to calculate a training speedup. +The final CPU trial used a 32 GiB computed cache and a 90-second process +time bound; it did not finish its first update. An earlier five-minute +trial completed one checked update in 171 seconds but not five steps. +Both are retained as timeouts, not averaged into a training result. + +The strict dense AdamW demo currently declines an aggregate with multiple +dependent or nullable keys. It is excluded from performance claims. The +public executor's default SQL fallback remains available and is tested +through a partial backward refusal and subsequent input updates. Existing +float32 AdamW sensitivity is not concealed by weakening its tolerance or +changing its update. For exact arithmetic, use the default float64 CPU +mode or the original SQL executor. + +Remaining work is an explicit API that owns GPU training state across +calls, compilation of larger portions of the whole ddx program, and better +liveness/sparse handling of lifted backward contributions. They address +the remaining native-framework gap. The compatible API still imports and +exports full state, constructs numerical graphs in Python, and may +recompute after cache eviction. It is not native array-training parity. + +## Reproduce + +```sh +.venv/bin/python accelerators/einfold_mlx/build.py +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python \ + benchmarks/apple/run_nanogpt_sizes.py --preset gpt2 --context 16 \ + --steps 5 --batch 1 --hosts dense_cpu dense_gpu --prefix local-gpt2-16 \ + --rss-limit-gib 64 --max-minutes 5 --keep-going +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python \ + benchmarks/apple/run_nanogpt_sizes.py --preset gpt2 --context 1024 \ + --steps 5 --batch 1 --hosts dense_gpu --dense-cache-gib 32 \ + --prefix local-gpt2-1024 --rss-limit-gib 64 --max-minutes 5 +``` + +Exact commands, exits, checks and sampled memory are in each linked run +manifest. `summarize_dense.py` regenerates the aggregate JSON from those +manifests and CSVs and excludes failed runs. `profile_dense_execution.py` +records a separate diagnostic warm pair, including import conversion, +device wait and NumPy view creation on GPU. Profiles are not used as +performance measurements. diff --git a/benchmarks/apple/dense-final-contract.log b/benchmarks/apple/dense-final-contract.log new file mode 100644 index 00000000..bf6132ce --- /dev/null +++ b/benchmarks/apple/dense-final-contract.log @@ -0,0 +1,3 @@ +........................................................................ [ 98%] +. [100%] +73 passed in 21.27s diff --git a/benchmarks/apple/dense-final-cpu-profile.json b/benchmarks/apple/dense-final-cpu-profile.json new file mode 100644 index 00000000..be4b6697 --- /dev/null +++ b/benchmarks/apple/dense-final-cpu-profile.json @@ -0,0 +1,27 @@ +{ + "diagnostic": true, + "host": "dense_cpu", + "context": 16, + "parameters": 123563520, + "cache_gib": 8, + "paired_seconds": 5.193765166099183, + "public_output_bytes": 3871488000, + "phases": { + "input_conversion_seconds": 0.0, + "device_wait_seconds": 0.0, + "numpy_view_seconds": 0.0 + }, + "statistics": { + "table_registers": 12, + "query_preparations": 1, + "gradient_preparations": 1, + "dense_programs": 707, + "dense_executions": 1414, + "dense_cache_evictions": 52510, + "dense_peak_cached_bytes": 8589934589, + "dense_lowering_seconds": 2.8820676773320884, + "dense_call_seconds": 10.176030291011557, + "key_cache_bytes": 0, + "dense_last_decline": null + } +} diff --git a/benchmarks/apple/dense-final-cpu-profile.txt b/benchmarks/apple/dense-final-cpu-profile.txt new file mode 100644 index 00000000..b9d9443d --- /dev/null +++ b/benchmarks/apple/dense-final-cpu-profile.txt @@ -0,0 +1,46 @@ + 9655625 function calls (9327488 primitive calls) in 5.192 seconds + + Ordered by: cumulative time + List reduced from 434 to 40 due to restriction <40> + + ncalls tottime percall cumtime percall filename:lineno(function) + 2 0.000 0.000 5.101 2.550 dense.py:180(_invoke) + 1 0.000 0.000 4.965 4.965 _hosts.py:342(query_all) + 1 0.074 0.074 4.965 4.965 dense.py:268() + 1 0.001 0.001 4.891 4.891 dense.py:254(execute) + 23 0.614 0.027 3.079 0.134 dense_program.py:442(arrow) + 11 0.000 0.000 3.077 0.280 dense_program.py:1265(arrow) +102608/6127 0.325 0.000 2.847 0.000 dense_program.py:100(get) + 707 0.001 0.000 1.797 0.003 dense.py:54(run) + 707 0.001 0.000 1.796 0.003 dense_program.py:1324(run) +29824/707 0.071 0.000 1.795 0.003 dense_program.py:1325(visit) + 1 0.000 0.000 1.661 1.661 __init__.py:270(run) + 1 0.029 0.029 1.661 1.661 {method 'run' of 'ddxdb._Prepared' objects} + 620 0.000 0.000 1.604 0.003 dense.py:70(materialize_compiled) + 84/10 0.634 0.008 1.553 0.155 dense_program.py:1192(compute) +12194/5257 0.206 0.000 1.514 0.000 dense_program.py:526(compute) + 300/299 0.001 0.000 0.725 0.002 dense_program.py:1042(compute) + 12194 0.022 0.000 0.692 0.000 einsumfunc.py:1242(einsum) + 3964 0.018 0.000 0.627 0.000 dense_program.py:948(_aggregate) + 23862 0.012 0.000 0.471 0.000 dense_program.py:938(_map) + 180/94 0.226 0.001 0.455 0.005 dense_program.py:1128(compute) + 9907 0.274 0.000 0.455 0.000 einsumfunc.py:1144(bmm_einsum) + 4853 0.031 0.000 0.411 0.000 dense_program.py:824(_join) + 4121 0.006 0.000 0.395 0.000 dense_program.py:587(_reduce) +191804/155509 0.092 0.000 0.318 0.000 dense_program.py:648(_expr) + 11922 0.029 0.000 0.312 0.000 dense_program.py:1363() + 1 0.000 0.000 0.229 0.229 _hosts.py:334(query) + 1 0.019 0.019 0.229 0.229 dense.py:243() + 1 0.000 0.000 0.210 0.210 dense.py:239(execute) + 2/1 0.000 0.000 0.209 0.209 dense.py:86(run) + 8566 0.002 0.000 0.199 0.000 dense_program.py:192(_truth) + 12194 0.065 0.000 0.198 0.000 einsumfunc.py:634(einsum_path) + 17166 0.022 0.000 0.192 0.000 dense_program.py:397(rename) + 84 0.001 0.000 0.185 0.002 dense_program.py:1165(_aggregate_bag) + 8871 0.179 0.000 0.179 0.000 {built-in method numpy._core._multiarray_umath.c_einsum} + 31142 0.007 0.000 0.179 0.000 fromnumeric.py:48(_wrapfunc) + 69274 0.040 0.000 0.170 0.000 dense_program.py:210(_op) + 4678 0.169 0.000 0.169 0.000 dense_program.py:264(_arithmetic) + 290883 0.085 0.000 0.165 0.000 dense_program.py:640(_key) + 66921 0.021 0.000 0.150 0.000 dense_program.py:360(rename) + 12 0.148 0.012 0.148 0.012 shared.py:83(_snapshot) diff --git a/benchmarks/apple/dense-final-gpu-profile.json b/benchmarks/apple/dense-final-gpu-profile.json new file mode 100644 index 00000000..261232d9 --- /dev/null +++ b/benchmarks/apple/dense-final-gpu-profile.json @@ -0,0 +1,36 @@ +{ + "diagnostic": true, + "host": "dense_gpu", + "context": 16, + "parameters": 123563520, + "cache_gib": 8, + "paired_seconds": 3.6194783749524504, + "public_output_bytes": 3871488000, + "phases": { + "input_conversion_seconds": 0.2776629664003849, + "device_wait_seconds": 0.2570341653190553, + "numpy_view_seconds": 0.00044416775926947594 + }, + "statistics": { + "table_registers": 12, + "query_preparations": 1, + "gradient_preparations": 1, + "dense_programs": 707, + "dense_executions": 1414, + "dense_cache_evictions": 0, + "dense_peak_cached_bytes": 5152709312, + "dense_lowering_seconds": 3.1360446005128324, + "dense_call_seconds": 7.324241542024538, + "key_cache_bytes": 0, + "gpu_input_arrays": 3178, + "gpu_input_bytes": 3954057712, + "gpu_fused_programs": 140, + "gpu_fused_executions": 10998, + "gpu_contractions": 8008, + "gpu_readbacks": 68, + "gpu_readback_bytes": 1065703696, + "gpu_union_reductions": 336, + "gpu_scatter_reductions": 398, + "dense_last_decline": null + } +} diff --git a/benchmarks/apple/dense-final-gpu-profile.txt b/benchmarks/apple/dense-final-gpu-profile.txt new file mode 100644 index 00000000..92676b64 --- /dev/null +++ b/benchmarks/apple/dense-final-gpu-profile.txt @@ -0,0 +1,46 @@ + 8851514 function calls (8455532 primitive calls) in 3.618 seconds + + Ordered by: cumulative time + List reduced from 450 to 40 due to restriction <40> + + ncalls tottime percall cumtime percall filename:lineno(function) + 2 0.000 0.000 3.614 1.807 dense.py:180(_invoke) + 1 0.000 0.000 3.369 3.369 _hosts.py:342(query_all) + 1 0.005 0.005 3.369 3.369 dense.py:268() + 1 0.000 0.000 3.364 3.364 dense.py:254(execute) + 707 0.001 0.000 1.966 0.003 dense.py:54(run) + 707 0.001 0.000 1.966 0.003 dense_program.py:1324(run) +29824/707 0.072 0.000 1.965 0.003 dense_program.py:1325(visit) + 1 0.000 0.000 1.814 1.814 __init__.py:270(run) + 1 0.030 0.030 1.814 1.814 {method 'run' of 'ddxdb._Prepared' objects} + 620 0.000 0.000 1.752 0.003 dense.py:70(materialize_compiled) + 23 0.610 0.027 1.342 0.058 dense_program.py:442(arrow) + 11 0.000 0.000 1.322 0.120 dense_program.py:1265(arrow) +109505/6127 0.060 0.000 0.960 0.000 dense_program.py:100(get) +11074/4639 0.047 0.000 0.895 0.000 _dense_mlx.py:105(pointwise) +53197/4639 0.037 0.000 0.857 0.000 _dense_mlx.py:108(describe) +63904/13928 0.010 0.000 0.853 0.000 _dense_mlx.py:112() + 3964 0.018 0.000 0.712 0.000 dense_program.py:948(_aggregate) +12186/5257 0.041 0.000 0.690 0.000 dense_program.py:526(compute) + 23862 0.012 0.000 0.495 0.000 dense_program.py:938(_map) + 4853 0.032 0.000 0.494 0.000 dense_program.py:824(_join) + 4121 0.006 0.000 0.394 0.000 dense_program.py:587(_reduce) + 168/22 0.033 0.000 0.367 0.017 _dense_mlx.py:233(merge) + 300/299 0.001 0.000 0.313 0.001 dense_program.py:1042(compute) + 11922 0.028 0.000 0.311 0.000 dense_program.py:1363() + 8566 0.003 0.000 0.303 0.000 dense_program.py:192(_truth) +191804/155509 0.090 0.000 0.288 0.000 dense_program.py:648(_expr) + 22416 0.008 0.000 0.282 0.000 profile_dense_execution.py:70(import_array) + 17166 0.022 0.000 0.275 0.000 dense_program.py:397(rename) + 22416 0.260 0.000 0.272 0.000 _dense_mlx.py:82(array) + 109 0.000 0.000 0.258 0.002 dense_program.py:58(_numpy) + 109 0.257 0.002 0.258 0.002 profile_dense_execution.py:76(export_array) + 1 0.000 0.000 0.250 0.250 _hosts.py:334(query) + 1 0.000 0.000 0.250 0.250 dense.py:243() + 1 0.000 0.000 0.250 0.250 dense.py:239(execute) + 2/1 0.000 0.000 0.236 0.236 dense.py:86(run) + 66921 0.021 0.000 0.232 0.000 dense_program.py:360(rename) + 12 0.224 0.019 0.224 0.019 shared.py:83(_snapshot) + 84 0.001 0.000 0.214 0.003 dense_program.py:1165(_aggregate_bag) + 151008 0.056 0.000 0.213 0.000 dense_program.py:168(rename) + 168/94 0.010 0.000 0.208 0.002 dense_program.py:1128(compute) diff --git a/benchmarks/apple/dense-final-tiny-dense_cpu.csv b/benchmarks/apple/dense-final-tiny-dense_cpu.csv new file mode 100644 index 00000000..baa53153 --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-dense_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",dense_cpu,4,0.007884833961725235,ok (2e-16),4.179251,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,10,10,0.0026251659728586674,0.007798084057867527,0,,,,, +nanogpt,"train (sgd), step 0",dense_cpu,4,0.12847254099324346,ok (2e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,86,0.022136001498438418,0.1362576250685379,0,,0.004613624885678291,0.0008226670324802399,1,11 +nanogpt,"loss, step 1",dense_cpu,4,0.005080000031739473,ok (0e+00),4.104661,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,96,0.024650501552969217,0.14132458402309567,0,,,,1,12 +nanogpt,"train (sgd), step 1",dense_cpu,4,0.0635539999930188,ok (3e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,172,0.06324329227209091,0.20487216708716005,0,,0.006329747615382075,0.0008589581120759249,1,23 +nanogpt,"loss, step 2",dense_cpu,4,0.004218916990794241,ok (1e-16),3.883857,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,182,0.06565121025778353,0.20908100111410022,0,,,,1,24 +nanogpt,"train (sgd), step 2",dense_cpu,4,0.03972570889163762,ok (5e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,258,0.08231550129130483,0.24880183418281376,0,,0.008083163993433118,0.000881499145179987,1,35 +nanogpt,"loss, step 3",dense_cpu,4,0.004158874973654747,ok (0e+00),3.641763,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,268,0.08465558243915439,0.25294562615454197,0,,,,1,36 +nanogpt,"train (sgd), step 3",dense_cpu,4,0.03930358309298754,ok (5e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,344,0.10160387447103858,0.29224416811484843,0,,0.0097903311252594,0.0009022901067510247,1,47 +nanogpt,"loss, step 4",dense_cpu,4,0.004242292023263872,ok (3e-16),3.842234,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,354,0.10400824761018157,0.2964766271179542,0,,,,1,48 +nanogpt,"train (sgd), step 4",dense_cpu,4,0.03997116710525006,ok (7e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,430,0.12095858063548803,0.33644287707284093,0,,0.011496665072627366,0.000923123094253242,1,59 diff --git a/benchmarks/apple/dense-final-tiny-dense_cpu.log b/benchmarks/apple/dense-final-tiny-dense_cpu.log new file mode 100644 index 00000000..9b0f2543 --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-dense_cpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 dense_cpu batch=4 0.008s + train (sgd), step 0 dense_cpu batch=4 0.128s + loss, step 1 dense_cpu batch=4 0.005s + train (sgd), step 1 dense_cpu batch=4 0.064s + loss, step 2 dense_cpu batch=4 0.004s + train (sgd), step 2 dense_cpu batch=4 0.040s + loss, step 3 dense_cpu batch=4 0.004s + train (sgd), step 3 dense_cpu batch=4 0.039s + loss, step 4 dense_cpu batch=4 0.004s + train (sgd), step 4 dense_cpu batch=4 0.040s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_cpu | 4 | 0.008 | ok (2e-16) | 4.179 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 10 | 10 | 0.003 | 0.008 | 0 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_cpu | 4 | 0.128 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 86 | 0.022 | 0.136 | 0 | n/a | 0.005 | 0.001 | 1 | 11 | +| nanogpt | loss, step 1 | dense_cpu | 4 | 0.005 | ok (0e+00) | 4.105 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 96 | 0.025 | 0.141 | 0 | n/a | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | dense_cpu | 4 | 0.064 | ok (3e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 172 | 0.063 | 0.205 | 0 | n/a | 0.006 | 0.001 | 1 | 23 | +| nanogpt | loss, step 2 | dense_cpu | 4 | 0.004 | ok (1e-16) | 3.884 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 182 | 0.066 | 0.209 | 0 | n/a | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | dense_cpu | 4 | 0.040 | ok (5e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 258 | 0.082 | 0.249 | 0 | n/a | 0.008 | 0.001 | 1 | 35 | +| nanogpt | loss, step 3 | dense_cpu | 4 | 0.004 | ok (0e+00) | 3.642 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 268 | 0.085 | 0.253 | 0 | n/a | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | dense_cpu | 4 | 0.039 | ok (5e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 344 | 0.102 | 0.292 | 0 | n/a | 0.010 | 0.001 | 1 | 47 | +| nanogpt | loss, step 4 | dense_cpu | 4 | 0.004 | ok (3e-16) | 3.842 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 354 | 0.104 | 0.296 | 0 | n/a | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | dense_cpu | 4 | 0.040 | ok (7e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 430 | 0.121 | 0.336 | 0 | n/a | 0.011 | 0.001 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-dense_cpu.csv diff --git a/benchmarks/apple/dense-final-tiny-dense_gpu.csv b/benchmarks/apple/dense-final-tiny-dense_gpu.csv new file mode 100644 index 00000000..488a4aca --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-dense_gpu.csv @@ -0,0 +1,2 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline +nanogpt,"loss, step 0",dense_gpu,4,0.018822374986484647,ok (5e-08),4.179251,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,10,10,0.0026654581306502223,0.018717166036367416,0,19,35392,10,23,21,1,4, diff --git a/benchmarks/apple/dense-final-tiny-dense_gpu.log b/benchmarks/apple/dense-final-tiny-dense_gpu.log new file mode 100644 index 00000000..2cfc2091 --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-dense_gpu.log @@ -0,0 +1,188 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 dense_gpu batch=4 0.019s +Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 792, in + main() + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 723, in main + updated = report.attempt( + ^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_bench.py", line 127, in attempt + return self.time(step, host, size, run, *args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_bench.py", line 91, in time + result = run(*args) + ^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_hosts.py", line 346, in query_all + return self._programs[key]() + ^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 221, in + return lambda **tables: self._invoke(execute, fallback, tables) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 169, in _invoke + return execute() + ^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 217, in execute + _snapshot(compiled.run(self).arrow(compiled.schema)) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1035, in arrow + ).arrow(schema) + ^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 331, in arrow + _align(_numpy(col.tensor.get()), col.tensor.axes, self.axes), + ^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 105, in get + result = runtime.pointwise(self) + ^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 116, in pointwise + describe(tensor) + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 109, in describe + leaves.append(_align(node.get(), node.axes, tensor.axes)) + ^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 71, in get + self.cached = self.args[0].get() + ^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 71, in get + self.cached = self.args[0].get() + ^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 71, in get + self.cached = self.args[0].get() + ^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 105, in get + result = runtime.pointwise(self) + ^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 116, in pointwise + describe(tensor) + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 109, in describe + leaves.append(_align(node.get(), node.axes, tensor.axes)) + ^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 71, in get + self.cached = self.args[0].get() + ^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 71, in get + self.cached = self.args[0].get() + ^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 71, in get + self.cached = self.args[0].get() + ^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 105, in get + result = runtime.pointwise(self) + ^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 116, in pointwise + describe(tensor) + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 109, in describe + leaves.append(_align(node.get(), node.axes, tensor.axes)) + ^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 101, in get + self.cached = self.value() + ^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 973, in valid_count + if runtime is not None and any( + ^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 974, in + runtime.owns(_op("and", p.present, p.columns[key].valid).get()) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 105, in get + result = runtime.pointwise(self) + ^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 116, in pointwise + describe(tensor) + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 109, in describe + leaves.append(_align(node.get(), node.axes, tensor.axes)) + ^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 101, in get + self.cached = self.value() + ^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 404, in compute + arrays = [factor.get() for factor in factors] + ^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 105, in get + result = runtime.pointwise(self) + ^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 116, in pointwise + describe(tensor) + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in describe + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 106, in + children = tuple(describe(a) for a in node.args) + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 109, in describe + leaves.append(_align(node.get(), node.axes, tensor.axes)) + ^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 101, in get + self.cached = self.value() + ^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 831, in compute + return runtime.scatter(data, indexes, len(unique)).reshape( + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/_dense_mlx.py", line 209, in scatter + ].add(data) + ^^^^^^^^^ +ValueError: [scatter] GPU scatter does not yet support int64 for the input or updates. diff --git a/benchmarks/apple/dense-final-tiny-duckdb.csv b/benchmarks/apple/dense-final-tiny-duckdb.csv new file mode 100644 index 00000000..f17ee005 --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-duckdb.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,load_seconds,model_seconds +nanogpt,"loss, step 0",duckdb,4,0.02979412500280887,ok (0e+00),4.179251,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 0",duckdb,4,0.2989180830772966,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.010787540581077337,0.0011049999156966805 +nanogpt,"loss, step 1",duckdb,4,0.024936834000982344,ok (0e+00),4.104661,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 1",duckdb,4,0.25098350003827363,ok (4e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.0161727488739416,0.0020935829961672425 +nanogpt,"loss, step 2",duckdb,4,0.02436333301011473,ok (1e-16),3.883857,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 2",duckdb,4,0.2476455409778282,ok (5e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.021583832683973014,0.0030175409046933055 +nanogpt,"loss, step 3",duckdb,4,0.02453645900823176,ok (0e+00),3.641763,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 3",duckdb,4,0.2526984999421984,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.02712412376422435,0.003926208009943366 +nanogpt,"loss, step 4",duckdb,4,0.024509625043720007,ok (2e-16),3.842234,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 4",duckdb,4,0.246333249961026,ok (1e-15),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.032413956709206104,0.004844874027185142 diff --git a/benchmarks/apple/dense-final-tiny-duckdb.log b/benchmarks/apple/dense-final-tiny-duckdb.log new file mode 100644 index 00000000..8eea29da --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-duckdb.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 duckdb batch=4 0.030s + train (sgd), step 0 duckdb batch=4 0.299s + loss, step 1 duckdb batch=4 0.025s + train (sgd), step 1 duckdb batch=4 0.251s + loss, step 2 duckdb batch=4 0.024s + train (sgd), step 2 duckdb batch=4 0.248s + loss, step 3 duckdb batch=4 0.025s + train (sgd), step 3 duckdb batch=4 0.253s + loss, step 4 duckdb batch=4 0.025s + train (sgd), step 4 duckdb batch=4 0.246s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | load_seconds | model_seconds | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | duckdb | 4 | 0.030 | ok (0e+00) | 4.179 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 0 | duckdb | 4 | 0.299 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.011 | 0.001 | +| nanogpt | loss, step 1 | duckdb | 4 | 0.025 | ok (0e+00) | 4.105 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 1 | duckdb | 4 | 0.251 | ok (4e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.016 | 0.002 | +| nanogpt | loss, step 2 | duckdb | 4 | 0.024 | ok (1e-16) | 3.884 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 2 | duckdb | 4 | 0.248 | ok (5e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.022 | 0.003 | +| nanogpt | loss, step 3 | duckdb | 4 | 0.025 | ok (0e+00) | 3.642 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 3 | duckdb | 4 | 0.253 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.027 | 0.004 | +| nanogpt | loss, step 4 | duckdb | 4 | 0.025 | ok (2e-16) | 3.842 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 4 | duckdb | 4 | 0.246 | ok (1e-15) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.032 | 0.005 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-duckdb.csv diff --git a/benchmarks/apple/dense-final-tiny-runs.json b/benchmarks/apple/dense-final-tiny-runs.json new file mode 100644 index 00000000..44a52ed0 --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-runs.json @@ -0,0 +1,157 @@ +[ + { + "host": "duckdb", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "duckdb", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-duckdb.csv" + ], + "contended": false, + "log": "dense-final-tiny-duckdb.log", + "csv": "dense-final-tiny-duckdb.csv", + "peak_rss_gib": 0.6759490966796875, + "status": "passed", + "started_utc": "2026-11-01T21:59:53.598085+00:00", + "exit_code": 0, + "wall_seconds": 3.886874582967721, + "checked_rows": 10 + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-shared_cpu.csv" + ], + "contended": false, + "log": "dense-final-tiny-shared_cpu.log", + "csv": "dense-final-tiny-shared_cpu.csv", + "peak_rss_gib": 0.6473541259765625, + "status": "passed", + "started_utc": "2026-11-01T21:59:57.485449+00:00", + "exit_code": 0, + "wall_seconds": 3.6641790000721812, + "checked_rows": 10 + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-shared_gpu.csv" + ], + "contended": false, + "log": "dense-final-tiny-shared_gpu.log", + "csv": "dense-final-tiny-shared_gpu.csv", + "peak_rss_gib": 0.64837646484375, + "status": "passed", + "started_utc": "2026-11-01T22:00:01.150111+00:00", + "exit_code": 0, + "wall_seconds": 3.8351630830438808, + "checked_rows": 10 + }, + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-dense_cpu.csv" + ], + "contended": false, + "log": "dense-final-tiny-dense_cpu.log", + "csv": "dense-final-tiny-dense_cpu.csv", + "peak_rss_gib": 0.532806396484375, + "status": "passed", + "started_utc": "2026-11-01T22:00:04.985866+00:00", + "exit_code": 0, + "wall_seconds": 2.826964083011262, + "checked_rows": 10 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-dense_gpu.csv" + ], + "contended": false, + "log": "dense-final-tiny-dense_gpu.log", + "csv": "dense-final-tiny-dense_gpu.csv", + "peak_rss_gib": 0.4050140380859375, + "status": "failed", + "started_utc": "2026-11-01T22:00:07.813307+00:00", + "exit_code": 1, + "wall_seconds": 1.4619372499873862, + "checked_rows": 1 + } +] diff --git a/benchmarks/apple/dense-final-tiny-shared_cpu.csv b/benchmarks/apple/dense-final-tiny-shared_cpu.csv new file mode 100644 index 00000000..bb6fffbd --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-shared_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,stage_buffer_writes,stage_replacements_keys_or_nulls +nanogpt,"loss, step 0",shared_cpu,4,0.04386008298024535,ok (2e-16),4.179251,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,7,0,0,1,10,10,9,16,175616,10,0.03309287293814123,0.0060567090986296535,0.0009612920694053173,125440,,,,,, +nanogpt,"train (sgd), step 0",shared_cpu,4,0.4670621659606695,ok (3e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,32,0,0,1,86,86,63,206,1706880,10,0.30510158836841583,0.1077326238155365,0.007829788140952587,440320,0.004450415959581733,0.0008739999029785395,1,11,, +nanogpt,"loss, step 1",shared_cpu,4,0.008216958027333021,ok (0e+00),4.104661,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,39,0,0,2,86,96,63,206,1706880,10,0.30510158836841583,0.11501595785375684,0.008579872315749526,440320,,,1,12,9, +nanogpt,"train (sgd), step 1",shared_cpu,4,0.1937360829906538,ok (4e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,64,0,0,2,99,172,73,304,2509696,10,0.3775937953032553,0.21949787461198866,0.014731829171068966,473088,0.006173956673592329,0.0009051669621840119,1,23,53,10 +nanogpt,"loss, step 2",shared_cpu,4,0.00547970796469599,ok (1e-16),3.883857,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,71,0,0,3,99,182,73,304,2509696,10,0.3775937953032553,0.22439899947494268,0.015191870159469545,473088,,,1,24,62,10 +nanogpt,"train (sgd), step 2",shared_cpu,4,0.18465083301998675,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,96,0,0,3,112,258,83,402,3312512,10,0.44696238136384636,0.3248937438474968,0.020634743152186275,505856,0.007919747615233064,0.0009321249090135098,1,35,106,20 +nanogpt,"loss, step 3",shared_cpu,4,0.005424499977380037,ok (1e-16),3.641763,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,103,0,0,4,112,268,83,402,3312512,10,0.44696238136384636,0.3297681597759947,0.021070284070447087,505856,,,1,36,115,20 +nanogpt,"train (sgd), step 3",shared_cpu,4,0.18376395793166012,ok (4e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,128,0,0,4,125,344,93,500,4115328,10,0.5161318393656984,0.4297826630063355,0.026502496446482837,538624,0.009638619725592434,0.0009549588430672884,1,47,159,30 +nanogpt,"loss, step 4",shared_cpu,4,0.005554000032134354,ok (3e-16),3.842234,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,135,0,0,5,125,354,93,500,4115328,10,0.5161318393656984,0.43475328874774277,0.026971536339260638,538624,,,1,48,168,30 +nanogpt,"train (sgd), step 4",shared_cpu,4,0.18708095897454768,ok (1e-15),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,160,0,0,5,138,430,103,598,4918144,10,0.5864103413186967,0.5365529175614938,0.03259224316570908,571392,0.011414202628657222,0.000977833871729672,1,59,212,40 diff --git a/benchmarks/apple/dense-final-tiny-shared_cpu.log b/benchmarks/apple/dense-final-tiny-shared_cpu.log new file mode 100644 index 00000000..a90cf15d --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-shared_cpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 shared_cpu batch=4 0.044s + train (sgd), step 0 shared_cpu batch=4 0.467s + loss, step 1 shared_cpu batch=4 0.008s + train (sgd), step 1 shared_cpu batch=4 0.194s + loss, step 2 shared_cpu batch=4 0.005s + train (sgd), step 2 shared_cpu batch=4 0.185s + loss, step 3 shared_cpu batch=4 0.005s + train (sgd), step 3 shared_cpu batch=4 0.184s + loss, step 4 shared_cpu batch=4 0.006s + train (sgd), step 4 shared_cpu batch=4 0.187s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | stage_buffer_writes | stage_replacements_keys_or_nulls | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_cpu | 4 | 0.044 | ok (2e-16) | 4.179 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 7 | 0 | 0 | 1 | 10 | 10 | 9 | 16 | 175616 | 10 | 0.033 | 0.006 | 0.001 | 125440 | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_cpu | 4 | 0.467 | ok (3e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 32 | 0 | 0 | 1 | 86 | 86 | 63 | 206 | 1706880 | 10 | 0.305 | 0.108 | 0.008 | 440320 | 0.004 | 0.001 | 1 | 11 | n/a | n/a | +| nanogpt | loss, step 1 | shared_cpu | 4 | 0.008 | ok (0e+00) | 4.105 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 39 | 0 | 0 | 2 | 86 | 96 | 63 | 206 | 1706880 | 10 | 0.305 | 0.115 | 0.009 | 440320 | n/a | n/a | 1 | 12 | 9 | n/a | +| nanogpt | train (sgd), step 1 | shared_cpu | 4 | 0.194 | ok (4e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 64 | 0 | 0 | 2 | 99 | 172 | 73 | 304 | 2509696 | 10 | 0.378 | 0.219 | 0.015 | 473088 | 0.006 | 0.001 | 1 | 23 | 53 | 10 | +| nanogpt | loss, step 2 | shared_cpu | 4 | 0.005 | ok (1e-16) | 3.884 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 71 | 0 | 0 | 3 | 99 | 182 | 73 | 304 | 2509696 | 10 | 0.378 | 0.224 | 0.015 | 473088 | n/a | n/a | 1 | 24 | 62 | 10 | +| nanogpt | train (sgd), step 2 | shared_cpu | 4 | 0.185 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 96 | 0 | 0 | 3 | 112 | 258 | 83 | 402 | 3312512 | 10 | 0.447 | 0.325 | 0.021 | 505856 | 0.008 | 0.001 | 1 | 35 | 106 | 20 | +| nanogpt | loss, step 3 | shared_cpu | 4 | 0.005 | ok (1e-16) | 3.642 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 103 | 0 | 0 | 4 | 112 | 268 | 83 | 402 | 3312512 | 10 | 0.447 | 0.330 | 0.021 | 505856 | n/a | n/a | 1 | 36 | 115 | 20 | +| nanogpt | train (sgd), step 3 | shared_cpu | 4 | 0.184 | ok (4e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 128 | 0 | 0 | 4 | 125 | 344 | 93 | 500 | 4115328 | 10 | 0.516 | 0.430 | 0.027 | 538624 | 0.010 | 0.001 | 1 | 47 | 159 | 30 | +| nanogpt | loss, step 4 | shared_cpu | 4 | 0.006 | ok (3e-16) | 3.842 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 135 | 0 | 0 | 5 | 125 | 354 | 93 | 500 | 4115328 | 10 | 0.516 | 0.435 | 0.027 | 538624 | n/a | n/a | 1 | 48 | 168 | 30 | +| nanogpt | train (sgd), step 4 | shared_cpu | 4 | 0.187 | ok (1e-15) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 160 | 0 | 0 | 5 | 138 | 430 | 103 | 598 | 4918144 | 10 | 0.586 | 0.537 | 0.033 | 571392 | 0.011 | 0.001 | 1 | 59 | 212 | 40 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-shared_cpu.csv diff --git a/benchmarks/apple/dense-final-tiny-shared_gpu.csv b/benchmarks/apple/dense-final-tiny-shared_gpu.csv new file mode 100644 index 00000000..5f503fad --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-shared_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,stage_buffer_writes,stage_replacements_keys_or_nulls +nanogpt,"loss, step 0",shared_gpu,4,0.05680545896757394,ok (5e-10),4.179251,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,7,1,0,10,10,9,16,175616,10,0.03394321002997458,0.017971210181713104,0.0010203311685472727,125440,,,,,, +nanogpt,"train (sgd), step 0",shared_gpu,4,0.4983854169258848,ok (1e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,32,1,0,86,86,63,206,1706880,10,0.306403880007565,0.14996395958587527,0.007952913059853017,440320,0.004543794319033623,0.0008421660168096423,1,11,, +nanogpt,"loss, step 1",shared_gpu,4,0.01118729100562632,ok (1e-09),4.104661,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,39,2,0,86,96,63,206,1706880,10,0.306403880007565,0.16023787471931428,0.00868811912368983,440320,,,1,12,9, +nanogpt,"train (sgd), step 1",shared_gpu,4,0.2053503750357777,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,64,2,0,99,172,73,304,2509696,10,0.376053381129168,0.27962975401896983,0.014452201314270496,473088,0.006287793396040797,0.0008724579820409417,1,23,53,10 +nanogpt,"loss, step 2",shared_gpu,4,0.011190332937985659,ok (7e-10),3.883857,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,71,3,0,99,182,73,304,2509696,10,0.376053381129168,0.2901977129513398,0.014952657395042479,473088,,,1,24,62,10 +nanogpt,"train (sgd), step 2",shared_gpu,4,0.21012354199774563,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,96,3,0,112,258,83,402,3312512,10,0.4458500910550356,0.4153064179699868,0.020680740009993315,505856,0.007964127231389284,0.0008972079958766699,1,35,106,20 +nanogpt,"loss, step 3",shared_gpu,4,0.010834375047124922,ok (4e-09),3.641763,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,103,4,0,112,268,83,402,3312512,10,0.4458500910550356,0.42551683296915144,0.02118182205595076,505856,,,1,36,115,20 +nanogpt,"train (sgd), step 3",shared_gpu,4,0.20861883298493922,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,128,4,0,125,344,93,500,4115328,10,0.5144741751719266,0.5508878785185516,0.026629691012203693,538624,0.009663502336479723,0.0009191239951178432,1,47,159,30 +nanogpt,"loss, step 4",shared_gpu,4,0.01025291602127254,ok (2e-09),3.842234,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,135,5,0,125,354,93,500,4115328,10,0.5144741751719266,0.5605452506570145,0.02710252406541258,538624,,,1,48,168,30 +nanogpt,"train (sgd), step 4",shared_gpu,4,0.20838645799085498,ok (3e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,160,5,0,138,430,103,598,4918144,10,0.5840442591579631,0.6845704186707735,0.032620440353639424,571392,0.011468461365438998,0.0009424160234630108,1,59,212,40 diff --git a/benchmarks/apple/dense-final-tiny-shared_gpu.log b/benchmarks/apple/dense-final-tiny-shared_gpu.log new file mode 100644 index 00000000..e8a24f30 --- /dev/null +++ b/benchmarks/apple/dense-final-tiny-shared_gpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 shared_gpu batch=4 0.057s + train (sgd), step 0 shared_gpu batch=4 0.498s + loss, step 1 shared_gpu batch=4 0.011s + train (sgd), step 1 shared_gpu batch=4 0.205s + loss, step 2 shared_gpu batch=4 0.011s + train (sgd), step 2 shared_gpu batch=4 0.210s + loss, step 3 shared_gpu batch=4 0.011s + train (sgd), step 3 shared_gpu batch=4 0.209s + loss, step 4 shared_gpu batch=4 0.010s + train (sgd), step 4 shared_gpu batch=4 0.208s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | stage_buffer_writes | stage_replacements_keys_or_nulls | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_gpu | 4 | 0.057 | ok (5e-10) | 4.179 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 7 | 1 | 0 | 10 | 10 | 9 | 16 | 175616 | 10 | 0.034 | 0.018 | 0.001 | 125440 | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_gpu | 4 | 0.498 | ok (1e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 32 | 1 | 0 | 86 | 86 | 63 | 206 | 1706880 | 10 | 0.306 | 0.150 | 0.008 | 440320 | 0.005 | 0.001 | 1 | 11 | n/a | n/a | +| nanogpt | loss, step 1 | shared_gpu | 4 | 0.011 | ok (1e-09) | 4.105 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 39 | 2 | 0 | 86 | 96 | 63 | 206 | 1706880 | 10 | 0.306 | 0.160 | 0.009 | 440320 | n/a | n/a | 1 | 12 | 9 | n/a | +| nanogpt | train (sgd), step 1 | shared_gpu | 4 | 0.205 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 64 | 2 | 0 | 99 | 172 | 73 | 304 | 2509696 | 10 | 0.376 | 0.280 | 0.014 | 473088 | 0.006 | 0.001 | 1 | 23 | 53 | 10 | +| nanogpt | loss, step 2 | shared_gpu | 4 | 0.011 | ok (7e-10) | 3.884 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 71 | 3 | 0 | 99 | 182 | 73 | 304 | 2509696 | 10 | 0.376 | 0.290 | 0.015 | 473088 | n/a | n/a | 1 | 24 | 62 | 10 | +| nanogpt | train (sgd), step 2 | shared_gpu | 4 | 0.210 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 96 | 3 | 0 | 112 | 258 | 83 | 402 | 3312512 | 10 | 0.446 | 0.415 | 0.021 | 505856 | 0.008 | 0.001 | 1 | 35 | 106 | 20 | +| nanogpt | loss, step 3 | shared_gpu | 4 | 0.011 | ok (4e-09) | 3.642 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 103 | 4 | 0 | 112 | 268 | 83 | 402 | 3312512 | 10 | 0.446 | 0.426 | 0.021 | 505856 | n/a | n/a | 1 | 36 | 115 | 20 | +| nanogpt | train (sgd), step 3 | shared_gpu | 4 | 0.209 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 128 | 4 | 0 | 125 | 344 | 93 | 500 | 4115328 | 10 | 0.514 | 0.551 | 0.027 | 538624 | 0.010 | 0.001 | 1 | 47 | 159 | 30 | +| nanogpt | loss, step 4 | shared_gpu | 4 | 0.010 | ok (2e-09) | 3.842 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 135 | 5 | 0 | 125 | 354 | 93 | 500 | 4115328 | 10 | 0.514 | 0.561 | 0.027 | 538624 | n/a | n/a | 1 | 48 | 168 | 30 | +| nanogpt | train (sgd), step 4 | shared_gpu | 4 | 0.208 | ok (3e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 160 | 5 | 0 | 138 | 430 | 103 | 598 | 4918144 | 10 | 0.584 | 0.685 | 0.033 | 571392 | 0.011 | 0.001 | 1 | 59 | 212 | 40 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-shared_gpu.csv diff --git a/benchmarks/apple/dense-full-correctness.log b/benchmarks/apple/dense-full-correctness.log new file mode 100644 index 00000000..4a7a07e5 --- /dev/null +++ b/benchmarks/apple/dense-full-correctness.log @@ -0,0 +1,88 @@ +.............sssssssss...sssssssss...ssssssssss..ssssssssss..ssssssssss. [ 7%] +.ssssssssss..sssssssss.....ssssssssssssssssss....sssssssss...sssssssss.. [ 14%] +...ssssssssssssssssss....sssssssss...sssssssss...sssssssss...sssssssss.. [ 22%] +.sssssssss...sssssssss...sssssssss...sssssssss...sssssssss..ssssssssssss [ 29%] +sssssssssssssssssssssssssssssssssssssssssssssssssssssssssss............. [ 37%] +........................................................................ [ 44%] +........................................................................ [ 51%] +........................................................................ [ 59%] +...............................................sss.....s................ [ 66%] +........................................................................ [ 74%] +........................................................................ [ 81%] +........................................................................ [ 88%] +........................................................................ [ 96%] +.................................... [100%] +=============================== warnings summary =============================== +tests/test_df.py:688 + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:688: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT"), np.datetime64("2000-01-01")], + +tests/test_adbc_backend.py: 28 warnings +tests/test_flight_sql.py: 10 warnings + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py:371: Warning: Cannot disable autocommit; conn will not be DB-API 2.0 compliant + warnings.warn( + +tests/test_arrow_dataset.py::test_unchunked_zarr_scans_by_storage_chunks + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:260: ZarrUserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future. + warnings.warn( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/tests/test_coord_lookup.py:111: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + template = xr.Dataset( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/xarray_sql/ds.py:707: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + return xr.Dataset(data_vars=data_vars, coords=coords_arg) + +tests/test_df.py::test_partition_metadata_preserves_coordinate_dtype[float16-values9-expected9] + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:639: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"v": (["x"], np.arange(len(coord)))}, coords={"x": coord}) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'z' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'u' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'v' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_flight_sql.py::test_shutdown_does_not_wait_forever_on_an_unread_result + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:63: PytestUnraisableExceptionWarning: Exception ignored in: + + Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 1053, in __del__ + self.close() + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 851, in close + self._stmt.close() + File "adbc_driver_manager/_lib.pyx", line 1555, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 1561, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 262, in adbc_driver_manager._lib.check_error + adbc_driver_manager.OperationalError: IO: [FlightSQL] connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:57215: connect: connection refused" (Unavailable; ClosePreparedStatement). Vendor code: 14 + + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) + +tests/test_lazy_roundtrip.py: 32 warnings + /Users/cromwellian/xql/xarray-sql/xarray_sql/lazyscan.py:63: DeprecationWarning: `is_in` with a collection of the same datatype is ambiguous and deprecated. + Please use `implode` to return to previous behavior. + + See https://github.com/pola-rs/polars/issues/22149 for more information. + return lf.collect(engine="streaming") + +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place + /Users/cromwellian/xql/xarray-sql/tests/test_prepare.py:169: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT") if d is None else np.datetime64(d) for d in days], + +tests/test_reader.py::TestFilterPushdown::test_float_coordinate_types_prune_with_natural_literal[float16] + /Users/cromwellian/xql/xarray-sql/tests/test_reader.py:1067: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"value": (["x"], np.arange(8))}, coords={"x": coord}) + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +695 passed, 277 skipped, 102 deselected, 83 warnings in 48.21s diff --git a/benchmarks/apple/dense-masks-gpt2-c1024-dense_cpu.csv b/benchmarks/apple/dense-masks-gpt2-c1024-dense_cpu.csv new file mode 100644 index 00000000..9da9b461 --- /dev/null +++ b/benchmarks/apple/dense-masks-gpt2-c1024-dense_cpu.csv @@ -0,0 +1,2 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline +nanogpt,"loss, step 0",dense_cpu,1,3.5377670830348507,ok (0e+00),10.783063,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,65,65,1294,8589572304,0.17863825091626495,3.4335250839358196,0, diff --git a/benchmarks/apple/dense-masks-gpt2-c1024-dense_cpu.log b/benchmarks/apple/dense-masks-gpt2-c1024-dense_cpu.log new file mode 100644 index 00000000..d367450a --- /dev/null +++ b/benchmarks/apple/dense-masks-gpt2-c1024-dense_cpu.log @@ -0,0 +1,4 @@ + 12-layer GPT, vocab 50257, 124,337,664 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=1024, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_cpu batch=1 3.538s diff --git a/benchmarks/apple/dense-masks-gpt2-c1024-dense_gpu.csv b/benchmarks/apple/dense-masks-gpt2-c1024-dense_gpu.csv new file mode 100644 index 00000000..7e05eba9 --- /dev/null +++ b/benchmarks/apple/dense-masks-gpt2-c1024-dense_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates,gpu_union_reductions,gpu_scatter_reductions +nanogpt,"loss, step 0",dense_gpu,1,0.5693485420197248,ok (2e-08),10.783063,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,65,65,0,5806422667,0.16695500095374882,0.5690589998848736,0,96,994701992,10,188,197,1,4,,,,,,, +nanogpt,"train (sgd), step 0",dense_gpu,1,18.07497258298099,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,707,119385,8589934552,2.3658397102262825,18.64375324989669,0,1589,1989415160,147,45775,19707,34,535949432,,0.12966945697553456,0.004977041040547192,1,11,1165,401 +nanogpt,"loss, step 1",dense_gpu,1,0.4707324169576168,ok (3e-08),8.782008,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,772,119385,8589934552,2.5769948760280386,19.114228749880567,0,1685,2984117152,147,45963,19904,35,535949436,,,,1,12,1165,401 +nanogpt,"train (sgd), step 1",dense_gpu,1,13.666450457996689,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,1414,238770,8589934552,4.6814343796577305,32.78041866689455,0,3178,3978830320,147,91550,39414,68,1071898864,,0.17528983193915337,0.005054833134636283,1,23,2330,802 +nanogpt,"loss, step 2",dense_gpu,1,0.36269783310126513,ok (4e-08),7.639927,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,1479,238770,8589934552,4.8086735852994025,33.14277658390347,0,3274,4973532312,147,91738,39611,69,1071898868,,,,1,24,2330,802 +nanogpt,"train (sgd), step 2",dense_gpu,1,14.152979207923636,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2121,358155,8589934552,7.068855741294101,47.29550837597344,0,4767,5968245480,147,137325,59121,102,1607848296,,0.22014666500035673,0.0051302912179380655,1,35,3495,1203 +nanogpt,"loss, step 3",dense_gpu,1,0.3625659580575302,ok (1e-08),6.953781,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2186,358155,8589934552,7.194439584971406,47.657654833979905,0,4863,6962947472,147,137513,59318,103,1607848300,,,,1,36,3495,1203 +nanogpt,"train (sgd), step 3",dense_gpu,1,13.68710524996277,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2828,477540,8589934552,9.402995257056318,61.344518000027165,0,6356,7957660640,147,183100,78828,136,2143797728,,0.2660044969525188,0.005241166218183935,1,47,4660,1604 +nanogpt,"loss, step 4",dense_gpu,1,0.36802916694432497,ok (7e-09),6.847609,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2893,477540,8589934552,9.53312700917013,61.712194374995306,0,6452,8952362632,147,183288,79025,137,2143797732,,,,1,48,4660,1604 +nanogpt,"train (sgd), step 4",dense_gpu,1,13.895598458009772,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,3535,596925,8589934552,11.84630450536497,75.60747250006534,0,7945,9947075800,147,228875,98535,170,2679747160,,0.3113979957997799,0.005314874229952693,1,59,5825,2005 diff --git a/benchmarks/apple/dense-masks-gpt2-c1024-dense_gpu.log b/benchmarks/apple/dense-masks-gpt2-c1024-dense_gpu.log new file mode 100644 index 00000000..65ff8d7f --- /dev/null +++ b/benchmarks/apple/dense-masks-gpt2-c1024-dense_gpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 124,337,664 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=1024, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_gpu batch=1 0.569s + train (sgd), step 0 dense_gpu batch=1 18.075s + loss, step 1 dense_gpu batch=1 0.471s + train (sgd), step 1 dense_gpu batch=1 13.666s + loss, step 2 dense_gpu batch=1 0.363s + train (sgd), step 2 dense_gpu batch=1 14.153s + loss, step 3 dense_gpu batch=1 0.363s + train (sgd), step 3 dense_gpu batch=1 13.687s + loss, step 4 dense_gpu batch=1 0.368s + train (sgd), step 4 dense_gpu batch=1 13.896s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | gpu_input_arrays | gpu_input_bytes | gpu_fused_programs | gpu_fused_executions | gpu_contractions | gpu_readbacks | gpu_readback_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | gpu_union_reductions | gpu_scatter_reductions | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_gpu | 1 | 0.569 | ok (2e-08) | 10.783 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 65 | 65 | 0 | 5806422667 | 0.167 | 0.569 | 0 | 96 | 994701992 | 10 | 188 | 197 | 1 | 4 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_gpu | 1 | 18.075 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 707 | 119385 | 8589934552 | 2.366 | 18.644 | 0 | 1589 | 1989415160 | 147 | 45775 | 19707 | 34 | 535949432 | n/a | 0.130 | 0.005 | 1 | 11 | 1165 | 401 | +| nanogpt | loss, step 1 | dense_gpu | 1 | 0.471 | ok (3e-08) | 8.782 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 772 | 119385 | 8589934552 | 2.577 | 19.114 | 0 | 1685 | 2984117152 | 147 | 45963 | 19904 | 35 | 535949436 | n/a | n/a | n/a | 1 | 12 | 1165 | 401 | +| nanogpt | train (sgd), step 1 | dense_gpu | 1 | 13.666 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 1414 | 238770 | 8589934552 | 4.681 | 32.780 | 0 | 3178 | 3978830320 | 147 | 91550 | 39414 | 68 | 1071898864 | n/a | 0.175 | 0.005 | 1 | 23 | 2330 | 802 | +| nanogpt | loss, step 2 | dense_gpu | 1 | 0.363 | ok (4e-08) | 7.640 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 1479 | 238770 | 8589934552 | 4.809 | 33.143 | 0 | 3274 | 4973532312 | 147 | 91738 | 39611 | 69 | 1071898868 | n/a | n/a | n/a | 1 | 24 | 2330 | 802 | +| nanogpt | train (sgd), step 2 | dense_gpu | 1 | 14.153 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2121 | 358155 | 8589934552 | 7.069 | 47.296 | 0 | 4767 | 5968245480 | 147 | 137325 | 59121 | 102 | 1607848296 | n/a | 0.220 | 0.005 | 1 | 35 | 3495 | 1203 | +| nanogpt | loss, step 3 | dense_gpu | 1 | 0.363 | ok (1e-08) | 6.954 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2186 | 358155 | 8589934552 | 7.194 | 47.658 | 0 | 4863 | 6962947472 | 147 | 137513 | 59318 | 103 | 1607848300 | n/a | n/a | n/a | 1 | 36 | 3495 | 1203 | +| nanogpt | train (sgd), step 3 | dense_gpu | 1 | 13.687 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2828 | 477540 | 8589934552 | 9.403 | 61.345 | 0 | 6356 | 7957660640 | 147 | 183100 | 78828 | 136 | 2143797728 | n/a | 0.266 | 0.005 | 1 | 47 | 4660 | 1604 | +| nanogpt | loss, step 4 | dense_gpu | 1 | 0.368 | ok (7e-09) | 6.848 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2893 | 477540 | 8589934552 | 9.533 | 61.712 | 0 | 6452 | 8952362632 | 147 | 183288 | 79025 | 137 | 2143797732 | n/a | n/a | n/a | 1 | 48 | 4660 | 1604 | +| nanogpt | train (sgd), step 4 | dense_gpu | 1 | 13.896 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 3535 | 596925 | 8589934552 | 11.846 | 75.607 | 0 | 7945 | 9947075800 | 147 | 228875 | 98535 | 170 | 2679747160 | n/a | 0.311 | 0.005 | 1 | 59 | 5825 | 2005 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-masks-gpt2-c1024-dense_gpu.csv diff --git a/benchmarks/apple/dense-masks-gpt2-c1024-runs.json b/benchmarks/apple/dense-masks-gpt2-c1024-runs.json new file mode 100644 index 00000000..3b682dcf --- /dev/null +++ b/benchmarks/apple/dense-masks-gpt2-c1024-runs.json @@ -0,0 +1,72 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-masks-gpt2-c1024-dense_cpu.csv", + "--context", + "1024", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "dense-masks-gpt2-c1024-dense_cpu.log", + "csv": "dense-masks-gpt2-c1024-dense_cpu.csv", + "peak_rss_gib": 21.509750366210938, + "status": "failed", + "started_utc": "2026-11-01T22:21:17.776817+00:00", + "exit_code": -15, + "wall_seconds": 138.8612410420319, + "checked_rows": 1 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-masks-gpt2-c1024-dense_gpu.csv", + "--context", + "1024", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "dense-masks-gpt2-c1024-dense_gpu.log", + "csv": "dense-masks-gpt2-c1024-dense_gpu.csv", + "peak_rss_gib": 28.481826782226562, + "status": "passed", + "started_utc": "2026-11-01T22:23:36.639416+00:00", + "exit_code": 0, + "wall_seconds": 103.18275220796932, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/dense-measured-adamw-dense_cpu.csv b/benchmarks/apple/dense-measured-adamw-dense_cpu.csv new file mode 100644 index 00000000..a4247042 --- /dev/null +++ b/benchmarks/apple/dense-measured-adamw-dense_cpu.csv @@ -0,0 +1,2 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline +nanogpt,"loss, step 0",dense_cpu,12,0.018254542024806142,ok (6e-16),4.178527,,1e-09,1e-09,1,128,4,64,65,213504,adamw,,False,13,1,10,10,0,61634251,0.014421955915167928,0.018121959059499204,0, diff --git a/benchmarks/apple/dense-measured-adamw-dense_cpu.log b/benchmarks/apple/dense-measured-adamw-dense_cpu.log new file mode 100644 index 00000000..150390ea --- /dev/null +++ b/benchmarks/apple/dense-measured-adamw-dense_cpu.log @@ -0,0 +1,54 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=adamw, context=64, heads=4, embed=128, forward_only=False + loss, step 0 dense_cpu batch=12 0.018s +Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 792, in + main() + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 719, in main + updated = report.attempt( + ^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_bench.py", line 127, in attempt + return self.time(step, host, size, run, *args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_bench.py", line 91, in time + result = run(*args) + ^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 524, in adamw_step + gradients = host.query_all(gradient_statements()) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_hosts.py", line 347, in query_all + return self._programs[key]() + ^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 268, in + return lambda **tables: self._invoke(execute, fallback, tables) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 207, in _invoke + return execute() + ^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 255, in execute + with prepared.run() as rewritten: + ^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/ddxdb/__init__.py", line 279, in run + prepared, out, tables, release = _prepare(self.backend, self.statements, self.dialect) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 74, in returns_rows_compiled + table = compiled.run(self.owner) + ^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 56, in run + result = self.program.run(owner._read) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1454, in run + return visit(self.plan) + ^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1348, in visit + inputs = [visit(p) for p in plan.inputs()] + ^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1348, in visit + inputs = [visit(p) for p in plan.inputs()] + ^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1396, in visit + return _aggregate( + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 953, in _aggregate + raise _Decline("aggregation by multiple dependent or nullable keys") +einfold.dense_program._Decline: aggregation by multiple dependent or nullable keys diff --git a/benchmarks/apple/dense-measured-adamw-dense_gpu.csv b/benchmarks/apple/dense-measured-adamw-dense_gpu.csv new file mode 100644 index 00000000..f4969f52 --- /dev/null +++ b/benchmarks/apple/dense-measured-adamw-dense_gpu.csv @@ -0,0 +1,2 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline +nanogpt,"loss, step 0",dense_gpu,12,0.021150832995772362,ok (5e-09),4.178527,,3e-05,3e-05,1,128,4,64,65,213504,adamw,,False,13,1,10,10,0,16524995,0.006783918011933565,0.02104495803359896,0,19,1708096,9,23,21,1,4, diff --git a/benchmarks/apple/dense-measured-adamw-dense_gpu.log b/benchmarks/apple/dense-measured-adamw-dense_gpu.log new file mode 100644 index 00000000..e073c0f8 --- /dev/null +++ b/benchmarks/apple/dense-measured-adamw-dense_gpu.log @@ -0,0 +1,54 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=adamw, context=64, heads=4, embed=128, forward_only=False + loss, step 0 dense_gpu batch=12 0.021s +Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 792, in + main() + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 719, in main + updated = report.attempt( + ^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_bench.py", line 127, in attempt + return self.time(step, host, size, run, *args) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_bench.py", line 91, in time + result = run(*args) + ^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 524, in adamw_step + gradients = host.query_all(gradient_statements()) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_hosts.py", line 347, in query_all + return self._programs[key]() + ^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 268, in + return lambda **tables: self._invoke(execute, fallback, tables) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 207, in _invoke + return execute() + ^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 255, in execute + with prepared.run() as rewritten: + ^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/ddxdb/__init__.py", line 279, in run + prepared, out, tables, release = _prepare(self.backend, self.statements, self.dialect) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 74, in returns_rows_compiled + table = compiled.run(self.owner) + ^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/xarray_sql/dense.py", line 56, in run + result = self.program.run(owner._read) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1454, in run + return visit(self.plan) + ^^^^^^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1348, in visit + inputs = [visit(p) for p in plan.inputs()] + ^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1348, in visit + inputs = [visit(p) for p in plan.inputs()] + ^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 1396, in visit + return _aggregate( + ^^^^^^^^^^^ + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/einfold/dense_program.py", line 953, in _aggregate + raise _Decline("aggregation by multiple dependent or nullable keys") +einfold.dense_program._Decline: aggregation by multiple dependent or nullable keys diff --git a/benchmarks/apple/dense-measured-adamw-runs.json b/benchmarks/apple/dense-measured-adamw-runs.json new file mode 100644 index 00000000..bc0c6c2a --- /dev/null +++ b/benchmarks/apple/dense-measured-adamw-runs.json @@ -0,0 +1,66 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "12", + "--optimizer", + "adamw", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-adamw-dense_cpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-adamw-dense_cpu.log", + "csv": "dense-measured-adamw-dense_cpu.csv", + "peak_rss_gib": 0.4573822021484375, + "status": "failed", + "started_utc": "2026-11-01T23:33:40.960780+00:00", + "exit_code": 1, + "wall_seconds": 1.4579214999685064, + "checked_rows": 1 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "12", + "--optimizer", + "adamw", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-adamw-dense_gpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-adamw-dense_gpu.log", + "csv": "dense-measured-adamw-dense_gpu.csv", + "peak_rss_gib": 0.427886962890625, + "status": "failed", + "started_utc": "2026-11-01T23:33:42.419377+00:00", + "exit_code": 1, + "wall_seconds": 1.464974624919705, + "checked_rows": 1 + } +] diff --git a/benchmarks/apple/dense-measured-demo-dense_cpu.csv b/benchmarks/apple/dense-measured-demo-dense_cpu.csv new file mode 100644 index 00000000..ae8d9646 --- /dev/null +++ b/benchmarks/apple/dense-measured-demo-dense_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",dense_cpu,1,0.009688499965704978,ok (0e+00),4.185923,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,10,10,0,6981323,0.0062256649835035205,0.00956412497907877,0,,,,, +nanogpt,"train (sgd), step 0",dense_cpu,1,0.12959120806772262,ok (1e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,86,0,59292293,0.03397195669822395,0.13859062502160668,0,,0.0054640809539705515,0.0008809160208329558,1,11 +nanogpt,"loss, step 1",dense_cpu,1,0.005519666941836476,ok (2e-16),4.111482,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,96,0,59292293,0.03931195673067123,0.14405125007033348,0,,,,1,12 +nanogpt,"train (sgd), step 1",dense_cpu,1,0.07193350000306964,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,172,0,59292293,0.08684895373880863,0.2154025830095634,0,,0.00728737423196435,0.0009191660210490227,1,23 +nanogpt,"loss, step 2",dense_cpu,1,0.005148417083546519,ok (4e-16),4.147195,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,182,0,59292293,0.09185508079826832,0.2204982079565525,0,,,,1,24 +nanogpt,"train (sgd), step 2",dense_cpu,1,0.04956449999008328,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,258,0,59292293,0.11803237244021147,0.26950433291494846,0,,0.009122581104747951,0.0009479159489274025,1,35 +nanogpt,"loss, step 3",dense_cpu,1,0.0053911660797894,ok (0e+00),4.065422,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,268,0,59292293,0.12327703938353807,0.2748351239133626,0,,,,1,36 +nanogpt,"train (sgd), step 3",dense_cpu,1,0.04916995903477073,ok (3e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,344,0,59292293,0.14992341527249664,0.3234464159468189,0,,0.011121872114017606,0.0009722079848870635,1,47 +nanogpt,"loss, step 4",dense_cpu,1,0.005612666951492429,ok (0e+00),4.03082,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,354,0,59292293,0.15536679152864963,0.3289989159675315,0,,,,1,48 +nanogpt,"train (sgd), step 4",dense_cpu,1,0.05015191691927612,ok (3e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,430,0,59292293,0.182141374447383,0.37847587396390736,0,,0.012952539953403175,0.0010030410485342145,1,59 diff --git a/benchmarks/apple/dense-measured-demo-dense_cpu.log b/benchmarks/apple/dense-measured-demo-dense_cpu.log new file mode 100644 index 00000000..86e252ba --- /dev/null +++ b/benchmarks/apple/dense-measured-demo-dense_cpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=64, heads=4, embed=128, forward_only=False + loss, step 0 dense_cpu batch=1 0.010s + train (sgd), step 0 dense_cpu batch=1 0.130s + loss, step 1 dense_cpu batch=1 0.006s + train (sgd), step 1 dense_cpu batch=1 0.072s + loss, step 2 dense_cpu batch=1 0.005s + train (sgd), step 2 dense_cpu batch=1 0.050s + loss, step 3 dense_cpu batch=1 0.005s + train (sgd), step 3 dense_cpu batch=1 0.049s + loss, step 4 dense_cpu batch=1 0.006s + train (sgd), step 4 dense_cpu batch=1 0.050s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_cpu | 1 | 0.010 | ok (0e+00) | 4.186 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 10 | 10 | 0 | 6981323 | 0.006 | 0.010 | 0 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_cpu | 1 | 0.130 | ok (1e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 86 | 0 | 59292293 | 0.034 | 0.139 | 0 | n/a | 0.005 | 0.001 | 1 | 11 | +| nanogpt | loss, step 1 | dense_cpu | 1 | 0.006 | ok (2e-16) | 4.111 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 96 | 0 | 59292293 | 0.039 | 0.144 | 0 | n/a | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | dense_cpu | 1 | 0.072 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 172 | 0 | 59292293 | 0.087 | 0.215 | 0 | n/a | 0.007 | 0.001 | 1 | 23 | +| nanogpt | loss, step 2 | dense_cpu | 1 | 0.005 | ok (4e-16) | 4.147 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 182 | 0 | 59292293 | 0.092 | 0.220 | 0 | n/a | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | dense_cpu | 1 | 0.050 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 258 | 0 | 59292293 | 0.118 | 0.270 | 0 | n/a | 0.009 | 0.001 | 1 | 35 | +| nanogpt | loss, step 3 | dense_cpu | 1 | 0.005 | ok (0e+00) | 4.065 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 268 | 0 | 59292293 | 0.123 | 0.275 | 0 | n/a | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | dense_cpu | 1 | 0.049 | ok (3e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 344 | 0 | 59292293 | 0.150 | 0.323 | 0 | n/a | 0.011 | 0.001 | 1 | 47 | +| nanogpt | loss, step 4 | dense_cpu | 1 | 0.006 | ok (0e+00) | 4.031 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 354 | 0 | 59292293 | 0.155 | 0.329 | 0 | n/a | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | dense_cpu | 1 | 0.050 | ok (3e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 430 | 0 | 59292293 | 0.182 | 0.378 | 0 | n/a | 0.013 | 0.001 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-demo-dense_cpu.csv diff --git a/benchmarks/apple/dense-measured-demo-dense_gpu.csv b/benchmarks/apple/dense-measured-demo-dense_gpu.csv new file mode 100644 index 00000000..0da4f0f9 --- /dev/null +++ b/benchmarks/apple/dense-measured-demo-dense_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates,gpu_union_reductions,gpu_scatter_reductions +nanogpt,"loss, step 0",dense_gpu,1,0.02100791595876217,ok (6e-08),4.185923,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,10,10,0,2380227,0.006869332864880562,0.020904292003251612,0,19,1708096,9,23,21,1,4,,,,,,, +nanogpt,"train (sgd), step 0",dense_gpu,1,0.16685320797841996,ok (9e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,86,0,16902347,0.036571041913703084,0.18755995901301503,0,214,3417600,51,377,191,34,862430,,0.005300377495586872,0.0008726660162210464,1,11,22,23 +nanogpt,"loss, step 1",dense_gpu,1,0.007050457992590964,ok (3e-08),4.111482,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,96,0,16902347,0.04157958319410682,0.19458270899485797,0,233,5125696,51,400,212,35,862434,,,,1,12,22,23 +nanogpt,"train (sgd), step 1",dense_gpu,1,0.08541129191871732,ok (1e-07),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,172,0,16902347,0.09119475656189024,0.27970325003843755,0,428,6835200,51,754,382,68,1724860,,0.007235003518871963,0.0009054159745573997,1,23,44,46 +nanogpt,"loss, step 2",dense_gpu,1,0.006592749967239797,ok (6e-08),4.147195,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,182,0,16902347,0.09607317228801548,0.28627241612412035,0,447,8543296,51,777,403,69,1724864,,,,1,24,44,46 +nanogpt,"train (sgd), step 2",dense_gpu,1,0.06153591605834663,ok (1e-07),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,258,0,16902347,0.1244273844640702,0.3473610411165282,0,642,10252800,51,1131,573,102,2587290,,0.009026793297380209,0.000931665999814868,1,35,66,69 +nanogpt,"loss, step 3",dense_gpu,1,0.006558167049661279,ok (4e-08),4.065422,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,268,0,16902347,0.12919084227178246,0.35388899920508265,0,661,11960896,51,1154,594,103,2587294,,,,1,36,66,69 +nanogpt,"train (sgd), step 3",dense_gpu,1,0.05877591692842543,ok (2e-07),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,344,0,16902347,0.15663634415250272,0.41245641524437815,0,856,13670400,51,1508,764,136,3449720,,0.010819749324582517,0.0009562500054016709,1,47,88,92 +nanogpt,"loss, step 4",dense_gpu,1,0.030258083948865533,ok (7e-08),4.03082,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,354,0,16902347,0.18498422158882022,0.44268212420865893,0,875,15378496,51,1531,785,137,3449724,,,,1,48,88,92 +nanogpt,"train (sgd), step 4",dense_gpu,1,0.07594120898284018,ok (2e-07),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,86,430,0,16902347,0.2127329718787223,0.5184563321527094,0,1070,17088000,51,1885,955,170,4312150,,0.01261241442989558,0.000981333083473146,1,59,110,115 diff --git a/benchmarks/apple/dense-measured-demo-dense_gpu.log b/benchmarks/apple/dense-measured-demo-dense_gpu.log new file mode 100644 index 00000000..e9acc1cb --- /dev/null +++ b/benchmarks/apple/dense-measured-demo-dense_gpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=64, heads=4, embed=128, forward_only=False + loss, step 0 dense_gpu batch=1 0.021s + train (sgd), step 0 dense_gpu batch=1 0.167s + loss, step 1 dense_gpu batch=1 0.007s + train (sgd), step 1 dense_gpu batch=1 0.085s + loss, step 2 dense_gpu batch=1 0.007s + train (sgd), step 2 dense_gpu batch=1 0.062s + loss, step 3 dense_gpu batch=1 0.007s + train (sgd), step 3 dense_gpu batch=1 0.059s + loss, step 4 dense_gpu batch=1 0.030s + train (sgd), step 4 dense_gpu batch=1 0.076s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | gpu_input_arrays | gpu_input_bytes | gpu_fused_programs | gpu_fused_executions | gpu_contractions | gpu_readbacks | gpu_readback_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | gpu_union_reductions | gpu_scatter_reductions | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_gpu | 1 | 0.021 | ok (6e-08) | 4.186 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 10 | 10 | 0 | 2380227 | 0.007 | 0.021 | 0 | 19 | 1708096 | 9 | 23 | 21 | 1 | 4 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_gpu | 1 | 0.167 | ok (9e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 86 | 0 | 16902347 | 0.037 | 0.188 | 0 | 214 | 3417600 | 51 | 377 | 191 | 34 | 862430 | n/a | 0.005 | 0.001 | 1 | 11 | 22 | 23 | +| nanogpt | loss, step 1 | dense_gpu | 1 | 0.007 | ok (3e-08) | 4.111 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 96 | 0 | 16902347 | 0.042 | 0.195 | 0 | 233 | 5125696 | 51 | 400 | 212 | 35 | 862434 | n/a | n/a | n/a | 1 | 12 | 22 | 23 | +| nanogpt | train (sgd), step 1 | dense_gpu | 1 | 0.085 | ok (1e-07) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 172 | 0 | 16902347 | 0.091 | 0.280 | 0 | 428 | 6835200 | 51 | 754 | 382 | 68 | 1724860 | n/a | 0.007 | 0.001 | 1 | 23 | 44 | 46 | +| nanogpt | loss, step 2 | dense_gpu | 1 | 0.007 | ok (6e-08) | 4.147 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 182 | 0 | 16902347 | 0.096 | 0.286 | 0 | 447 | 8543296 | 51 | 777 | 403 | 69 | 1724864 | n/a | n/a | n/a | 1 | 24 | 44 | 46 | +| nanogpt | train (sgd), step 2 | dense_gpu | 1 | 0.062 | ok (1e-07) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 258 | 0 | 16902347 | 0.124 | 0.347 | 0 | 642 | 10252800 | 51 | 1131 | 573 | 102 | 2587290 | n/a | 0.009 | 0.001 | 1 | 35 | 66 | 69 | +| nanogpt | loss, step 3 | dense_gpu | 1 | 0.007 | ok (4e-08) | 4.065 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 268 | 0 | 16902347 | 0.129 | 0.354 | 0 | 661 | 11960896 | 51 | 1154 | 594 | 103 | 2587294 | n/a | n/a | n/a | 1 | 36 | 66 | 69 | +| nanogpt | train (sgd), step 3 | dense_gpu | 1 | 0.059 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 344 | 0 | 16902347 | 0.157 | 0.412 | 0 | 856 | 13670400 | 51 | 1508 | 764 | 136 | 3449720 | n/a | 0.011 | 0.001 | 1 | 47 | 88 | 92 | +| nanogpt | loss, step 4 | dense_gpu | 1 | 0.030 | ok (7e-08) | 4.031 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 354 | 0 | 16902347 | 0.185 | 0.443 | 0 | 875 | 15378496 | 51 | 1531 | 785 | 137 | 3449724 | n/a | n/a | n/a | 1 | 48 | 88 | 92 | +| nanogpt | train (sgd), step 4 | dense_gpu | 1 | 0.076 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 86 | 430 | 0 | 16902347 | 0.213 | 0.518 | 0 | 1070 | 17088000 | 51 | 1885 | 955 | 170 | 4312150 | n/a | 0.013 | 0.001 | 1 | 59 | 110 | 115 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-demo-dense_gpu.csv diff --git a/benchmarks/apple/dense-measured-demo-runs.json b/benchmarks/apple/dense-measured-demo-runs.json new file mode 100644 index 00000000..b540e79d --- /dev/null +++ b/benchmarks/apple/dense-measured-demo-runs.json @@ -0,0 +1,66 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-demo-dense_cpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-demo-dense_cpu.log", + "csv": "dense-measured-demo-dense_cpu.csv", + "peak_rss_gib": 0.5987548828125, + "status": "passed", + "started_utc": "2026-11-01T23:31:58.067931+00:00", + "exit_code": 0, + "wall_seconds": 2.7708311669994146, + "checked_rows": 10 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-demo-dense_gpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-demo-dense_gpu.log", + "csv": "dense-measured-demo-dense_gpu.csv", + "peak_rss_gib": 0.5789947509765625, + "status": "passed", + "started_utc": "2026-11-01T23:32:00.839351+00:00", + "exit_code": 0, + "wall_seconds": 2.8991056670201942, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/dense-measured-gpt2-c1024-dense_cpu.csv b/benchmarks/apple/dense-measured-gpt2-c1024-dense_cpu.csv new file mode 100644 index 00000000..27000c54 --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c1024-dense_cpu.csv @@ -0,0 +1,2 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline +nanogpt,"loss, step 0",dense_cpu,1,3.9096898330608383,ok (0e+00),10.783063,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,65,65,0,17246218899,3.63800504617393,3.671857916051522,0, diff --git a/benchmarks/apple/dense-measured-gpt2-c1024-dense_cpu.log b/benchmarks/apple/dense-measured-gpt2-c1024-dense_cpu.log new file mode 100644 index 00000000..62cce20e --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c1024-dense_cpu.log @@ -0,0 +1,4 @@ + 12-layer GPT, vocab 50257, 124,337,664 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=1024, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_cpu batch=1 3.910s diff --git a/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.csv b/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.csv new file mode 100644 index 00000000..01ca0ed5 --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates,gpu_union_reductions,gpu_scatter_reductions +nanogpt,"loss, step 0",dense_gpu,1,0.514757625060156,ok (2e-08),10.783063,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,65,65,0,5806422667,0.35029807942919433,0.5144586659735069,0,96,994701992,9,188,197,1,4,,,,,,, +nanogpt,"train (sgd), step 0",dense_gpu,1,6.980883750016801,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,707,24585,34359737980,2.590446741087362,7.494380624964833,0,1589,1989415160,149,9607,6817,34,535949432,,0.10694933612830937,0.005029166000895202,1,11,254,227 +nanogpt,"loss, step 1",dense_gpu,1,0.3492700409842655,ok (3e-08),8.782008,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,772,24585,34359737980,2.861444994341582,7.843389915884472,0,1685,2984117152,149,9795,7014,35,535949436,,,,1,12,254,227 +nanogpt,"train (sgd), step 1",dense_gpu,1,5.1961897920118645,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,1414,49170,34359737980,5.061550997314043,13.038670915877447,0,3178,3978830320,149,19214,13634,68,1071898864,,0.1509792524157092,0.0051039159297943115,1,23,508,454 +nanogpt,"loss, step 2",dense_gpu,1,0.3389572079759091,ok (2e-08),7.639927,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,1479,49170,34359737980,5.323658370645717,13.377335665863939,0,3274,4973532312,149,19402,13831,69,1071898868,,,,1,24,508,454 +nanogpt,"train (sgd), step 2",dense_gpu,1,5.259192208060995,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2121,73755,34359737980,7.6417727797525,18.635580457863398,0,4767,5968245480,149,28821,20451,102,1607848296,,0.1954242930514738,0.00518529093824327,1,35,762,681 +nanogpt,"loss, step 3",dense_gpu,1,0.34998762502800673,ok (1e-08),6.953781,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2186,73755,34359737980,7.9122199434787035,18.98526666685939,0,4863,6962947472,149,29009,20648,103,1607848300,,,,1,36,762,681 +nanogpt,"train (sgd), step 3",dense_gpu,1,5.214400500059128,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2828,98340,34359737980,10.214102042838931,24.198648083838634,0,6356,7957660640,149,38428,27268,136,2143797728,,0.23918933363165706,0.0052569989347830415,1,47,1016,908 +nanogpt,"loss, step 4",dense_gpu,1,0.35393929202109575,ok (7e-09),6.847609,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,2893,98340,34359737980,10.492515878984705,24.55229570879601,0,6452,8952362632,149,38616,27465,137,2143797732,,,,1,48,1016,908 +nanogpt,"train (sgd), step 4",dense_gpu,1,5.238639291957952,ok (2e-07),,,,,12,768,12,1024,50257,124337664,sgd,0.01,False,13,1,707,3535,122925,34359737980,12.795586302410811,29.79013208381366,0,7945,9947075800,149,48035,34085,170,2679747160,,0.28358350263442844,0.005331457010470331,1,59,1270,1135 diff --git a/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.log b/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.log new file mode 100644 index 00000000..7a1bfd20 --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 124,337,664 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=1024, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_gpu batch=1 0.515s + train (sgd), step 0 dense_gpu batch=1 6.981s + loss, step 1 dense_gpu batch=1 0.349s + train (sgd), step 1 dense_gpu batch=1 5.196s + loss, step 2 dense_gpu batch=1 0.339s + train (sgd), step 2 dense_gpu batch=1 5.259s + loss, step 3 dense_gpu batch=1 0.350s + train (sgd), step 3 dense_gpu batch=1 5.214s + loss, step 4 dense_gpu batch=1 0.354s + train (sgd), step 4 dense_gpu batch=1 5.239s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | gpu_input_arrays | gpu_input_bytes | gpu_fused_programs | gpu_fused_executions | gpu_contractions | gpu_readbacks | gpu_readback_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | gpu_union_reductions | gpu_scatter_reductions | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_gpu | 1 | 0.515 | ok (2e-08) | 10.783 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 65 | 65 | 0 | 5806422667 | 0.350 | 0.514 | 0 | 96 | 994701992 | 9 | 188 | 197 | 1 | 4 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_gpu | 1 | 6.981 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 707 | 24585 | 34359737980 | 2.590 | 7.494 | 0 | 1589 | 1989415160 | 149 | 9607 | 6817 | 34 | 535949432 | n/a | 0.107 | 0.005 | 1 | 11 | 254 | 227 | +| nanogpt | loss, step 1 | dense_gpu | 1 | 0.349 | ok (3e-08) | 8.782 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 772 | 24585 | 34359737980 | 2.861 | 7.843 | 0 | 1685 | 2984117152 | 149 | 9795 | 7014 | 35 | 535949436 | n/a | n/a | n/a | 1 | 12 | 254 | 227 | +| nanogpt | train (sgd), step 1 | dense_gpu | 1 | 5.196 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 1414 | 49170 | 34359737980 | 5.062 | 13.039 | 0 | 3178 | 3978830320 | 149 | 19214 | 13634 | 68 | 1071898864 | n/a | 0.151 | 0.005 | 1 | 23 | 508 | 454 | +| nanogpt | loss, step 2 | dense_gpu | 1 | 0.339 | ok (2e-08) | 7.640 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 1479 | 49170 | 34359737980 | 5.324 | 13.377 | 0 | 3274 | 4973532312 | 149 | 19402 | 13831 | 69 | 1071898868 | n/a | n/a | n/a | 1 | 24 | 508 | 454 | +| nanogpt | train (sgd), step 2 | dense_gpu | 1 | 5.259 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2121 | 73755 | 34359737980 | 7.642 | 18.636 | 0 | 4767 | 5968245480 | 149 | 28821 | 20451 | 102 | 1607848296 | n/a | 0.195 | 0.005 | 1 | 35 | 762 | 681 | +| nanogpt | loss, step 3 | dense_gpu | 1 | 0.350 | ok (1e-08) | 6.954 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2186 | 73755 | 34359737980 | 7.912 | 18.985 | 0 | 4863 | 6962947472 | 149 | 29009 | 20648 | 103 | 1607848300 | n/a | n/a | n/a | 1 | 36 | 762 | 681 | +| nanogpt | train (sgd), step 3 | dense_gpu | 1 | 5.214 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2828 | 98340 | 34359737980 | 10.214 | 24.199 | 0 | 6356 | 7957660640 | 149 | 38428 | 27268 | 136 | 2143797728 | n/a | 0.239 | 0.005 | 1 | 47 | 1016 | 908 | +| nanogpt | loss, step 4 | dense_gpu | 1 | 0.354 | ok (7e-09) | 6.848 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 2893 | 98340 | 34359737980 | 10.493 | 24.552 | 0 | 6452 | 8952362632 | 149 | 38616 | 27465 | 137 | 2143797732 | n/a | n/a | n/a | 1 | 48 | 1016 | 908 | +| nanogpt | train (sgd), step 4 | dense_gpu | 1 | 5.239 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | False | 13 | 1 | 707 | 3535 | 122925 | 34359737980 | 12.796 | 29.790 | 0 | 7945 | 9947075800 | 149 | 48035 | 34085 | 170 | 2679747160 | n/a | 0.284 | 0.005 | 1 | 59 | 1270 | 1135 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.csv diff --git a/benchmarks/apple/dense-measured-gpt2-c1024-runs.json b/benchmarks/apple/dense-measured-gpt2-c1024-runs.json new file mode 100644 index 00000000..633283db --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c1024-runs.json @@ -0,0 +1,74 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c1024-dense_cpu.csv", + "--context", + "1024", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-measured-gpt2-c1024-dense_cpu.log", + "csv": "dense-measured-gpt2-c1024-dense_cpu.csv", + "peak_rss_gib": 47.749176025390625, + "status": "timeout", + "started_utc": "2026-11-01T23:26:27.747100+00:00", + "exit_code": -15, + "wall_seconds": 90.51262095803395, + "checked_rows": 1 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.csv", + "--context", + "1024", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-measured-gpt2-c1024-dense_gpu.log", + "csv": "dense-measured-gpt2-c1024-dense_gpu.csv", + "peak_rss_gib": 28.716506958007812, + "status": "passed", + "started_utc": "2026-11-01T23:27:58.261043+00:00", + "exit_code": 0, + "wall_seconds": 56.74237979191821, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.csv b/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.csv new file mode 100644 index 00000000..1cc8f34a --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",dense_cpu,1,0.4022469168994576,ok (2e-16),10.937082,,1e-09,1e-09,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,65,65,0,1602653843,0.35478441417217255,0.38202233298216015,0,,,,, +nanogpt,"train (sgd), step 0",dense_cpu,1,6.791802417021245,ok (6e-16),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,707,0,22743236662,1.6311142548220232,7.000483499956317,0,,0.11224525386933237,0.00507900002412498,1,11 +nanogpt,"loss, step 1",dense_cpu,1,0.3098156669875607,ok (2e-16),8.742184,,1e-09,1e-09,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,772,0,22743236662,1.9137374586425722,7.283495874959044,0,,,,1,12 +nanogpt,"train (sgd), step 1",dense_cpu,1,5.512956790975295,ok (6e-16),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,1414,0,22743236662,3.0957732666283846,12.643677333951928,0,,0.15629637974780053,0.005189709016121924,1,23 +nanogpt,"loss, step 2",dense_cpu,1,0.30115254200063646,ok (0e+00),7.755612,,1e-09,1e-09,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,1479,0,22743236662,3.369680884759873,12.917963708983734,0,,,,1,24 +nanogpt,"train (sgd), step 2",dense_cpu,1,5.358630583039485,ok (6e-16),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,2121,0,22743236662,4.522470458643511,18.108521917951293,0,,0.19990583788603544,0.005268251057714224,1,35 +nanogpt,"loss, step 3",dense_cpu,1,0.305430082953535,ok (3e-16),6.895476,,1e-09,1e-09,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,2186,0,22743236662,4.797578918747604,18.384011250920594,0,,,,1,36 +nanogpt,"train (sgd), step 3",dense_cpu,1,5.375662000034936,ok (6e-16),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,2828,0,22743236662,5.915788219426759,23.56538595992606,0,,0.2473457132000476,0.005348418140783906,1,47 +nanogpt,"loss, step 4",dense_cpu,1,0.3124701250344515,ok (0e+00),6.484016,,1e-09,1e-09,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,2893,0,22743236662,6.200249511515722,23.850227417889982,0,,,,1,48 +nanogpt,"train (sgd), step 4",dense_cpu,1,5.671188749955036,ok (6e-16),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,3535,0,22743236662,7.350553346215747,29.36832120991312,0,,0.2909224620088935,0.0054317101603373885,1,59 diff --git a/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.log b/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.log new file mode 100644 index 00000000..b5bbbc6d --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 123,649,536 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=128, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_cpu batch=1 0.402s + train (sgd), step 0 dense_cpu batch=1 6.792s + loss, step 1 dense_cpu batch=1 0.310s + train (sgd), step 1 dense_cpu batch=1 5.513s + loss, step 2 dense_cpu batch=1 0.301s + train (sgd), step 2 dense_cpu batch=1 5.359s + loss, step 3 dense_cpu batch=1 0.305s + train (sgd), step 3 dense_cpu batch=1 5.376s + loss, step 4 dense_cpu batch=1 0.312s + train (sgd), step 4 dense_cpu batch=1 5.671s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_cpu | 1 | 0.402 | ok (2e-16) | 10.937 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 65 | 65 | 0 | 1602653843 | 0.355 | 0.382 | 0 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_cpu | 1 | 6.792 | ok (6e-16) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 707 | 0 | 22743236662 | 1.631 | 7.000 | 0 | n/a | 0.112 | 0.005 | 1 | 11 | +| nanogpt | loss, step 1 | dense_cpu | 1 | 0.310 | ok (2e-16) | 8.742 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 772 | 0 | 22743236662 | 1.914 | 7.283 | 0 | n/a | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | dense_cpu | 1 | 5.513 | ok (6e-16) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 1414 | 0 | 22743236662 | 3.096 | 12.644 | 0 | n/a | 0.156 | 0.005 | 1 | 23 | +| nanogpt | loss, step 2 | dense_cpu | 1 | 0.301 | ok (0e+00) | 7.756 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 1479 | 0 | 22743236662 | 3.370 | 12.918 | 0 | n/a | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | dense_cpu | 1 | 5.359 | ok (6e-16) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 2121 | 0 | 22743236662 | 4.522 | 18.109 | 0 | n/a | 0.200 | 0.005 | 1 | 35 | +| nanogpt | loss, step 3 | dense_cpu | 1 | 0.305 | ok (3e-16) | 6.895 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 2186 | 0 | 22743236662 | 4.798 | 18.384 | 0 | n/a | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | dense_cpu | 1 | 5.376 | ok (6e-16) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 2828 | 0 | 22743236662 | 5.916 | 23.565 | 0 | n/a | 0.247 | 0.005 | 1 | 47 | +| nanogpt | loss, step 4 | dense_cpu | 1 | 0.312 | ok (0e+00) | 6.484 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 2893 | 0 | 22743236662 | 6.200 | 23.850 | 0 | n/a | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | dense_cpu | 1 | 5.671 | ok (6e-16) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 3535 | 0 | 22743236662 | 7.351 | 29.368 | 0 | n/a | 0.291 | 0.005 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.csv diff --git a/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.csv b/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.csv new file mode 100644 index 00000000..a6007ba2 --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates,gpu_union_reductions,gpu_scatter_reductions +nanogpt,"loss, step 0",dense_gpu,1,0.32601212500594556,ok (1e-09),10.937082,,3e-05,3e-05,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,65,65,0,618521739,0.26744179415982217,0.3257542080245912,0,96,989196968,9,188,197,1,4,,,,,,, +nanogpt,"train (sgd), step 0",dense_gpu,1,3.555519917048514,ok (1e-07),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,707,0,7219030384,1.2814124781871215,3.877060208003968,0,1589,1978405112,140,5499,4004,34,533196024,,0.10567708313465118,0.005067208083346486,1,11,168,199 +nanogpt,"loss, step 1",dense_gpu,1,0.20585112494882196,ok (1e-08),8.742184,,3e-05,3e-05,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,772,0,7219030384,1.465973440092057,4.082688416936435,0,1685,2967602080,140,5687,4201,35,533196028,,,,1,12,168,199 +nanogpt,"train (sgd), step 1",dense_gpu,1,2.3626125409500673,ok (1e-07),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,1414,0,7219030384,2.441833976772614,6.440963374916464,0,3178,3956810224,140,10998,8008,68,1066392048,,0.15066116699017584,0.00514070806093514,1,23,336,398 +nanogpt,"loss, step 2",dense_gpu,1,0.21013649995438755,ok (2e-10),7.755612,,3e-05,3e-05,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,1479,0,7219030384,2.626783176790923,6.650833832914941,0,3274,4946007192,140,11186,8205,69,1066392052,,,,1,24,336,398 +nanogpt,"train (sgd), step 2",dense_gpu,1,2.330647209077142,ok (2e-07),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,2121,0,7219030384,3.653418047935702,8.976494582835585,0,4767,5935215336,140,16497,12012,102,1599588072,,0.19475058291573077,0.0052194580202922225,1,35,504,597 +nanogpt,"loss, step 3",dense_gpu,1,0.1992492499994114,ok (9e-08),6.895476,,3e-05,3e-05,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,2186,0,7219030384,3.8269422934390604,9.17550745792687,0,4863,6924412304,140,16685,12209,103,1599588076,,,,1,36,504,597 +nanogpt,"train (sgd), step 3",dense_gpu,1,2.3210190830286592,ok (2e-07),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,2828,0,7219030384,4.808491181116551,11.491605957970023,0,6356,7913620448,140,21996,16016,136,2132784096,,0.23844341619405895,0.00529620808083564,1,47,672,796 +nanogpt,"loss, step 4",dense_gpu,1,0.19663012505043298,ok (8e-08),6.484016,,3e-05,3e-05,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,2893,0,7219030384,4.981497059343383,11.687991000013426,0,6452,8902817416,140,22184,16213,137,2132784100,,,,1,48,672,796 +nanogpt,"train (sgd), step 4",dense_gpu,1,2.334231665940024,ok (3e-07),,,,,12,768,12,128,50257,123649536,sgd,0.01,False,13,1,707,3535,0,7219030384,6.000592845142819,14.017265541944653,0,7945,9892025560,140,27495,20020,170,2665980120,,0.28235145821236074,0.005367833073250949,1,59,840,995 diff --git a/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.log b/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.log new file mode 100644 index 00000000..b6e30fb4 --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 123,649,536 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=128, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_gpu batch=1 0.326s + train (sgd), step 0 dense_gpu batch=1 3.556s + loss, step 1 dense_gpu batch=1 0.206s + train (sgd), step 1 dense_gpu batch=1 2.363s + loss, step 2 dense_gpu batch=1 0.210s + train (sgd), step 2 dense_gpu batch=1 2.331s + loss, step 3 dense_gpu batch=1 0.199s + train (sgd), step 3 dense_gpu batch=1 2.321s + loss, step 4 dense_gpu batch=1 0.197s + train (sgd), step 4 dense_gpu batch=1 2.334s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | gpu_input_arrays | gpu_input_bytes | gpu_fused_programs | gpu_fused_executions | gpu_contractions | gpu_readbacks | gpu_readback_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | gpu_union_reductions | gpu_scatter_reductions | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_gpu | 1 | 0.326 | ok (1e-09) | 10.937 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 65 | 65 | 0 | 618521739 | 0.267 | 0.326 | 0 | 96 | 989196968 | 9 | 188 | 197 | 1 | 4 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_gpu | 1 | 3.556 | ok (1e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 707 | 0 | 7219030384 | 1.281 | 3.877 | 0 | 1589 | 1978405112 | 140 | 5499 | 4004 | 34 | 533196024 | n/a | 0.106 | 0.005 | 1 | 11 | 168 | 199 | +| nanogpt | loss, step 1 | dense_gpu | 1 | 0.206 | ok (1e-08) | 8.742 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 772 | 0 | 7219030384 | 1.466 | 4.083 | 0 | 1685 | 2967602080 | 140 | 5687 | 4201 | 35 | 533196028 | n/a | n/a | n/a | 1 | 12 | 168 | 199 | +| nanogpt | train (sgd), step 1 | dense_gpu | 1 | 2.363 | ok (1e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 1414 | 0 | 7219030384 | 2.442 | 6.441 | 0 | 3178 | 3956810224 | 140 | 10998 | 8008 | 68 | 1066392048 | n/a | 0.151 | 0.005 | 1 | 23 | 336 | 398 | +| nanogpt | loss, step 2 | dense_gpu | 1 | 0.210 | ok (2e-10) | 7.756 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 1479 | 0 | 7219030384 | 2.627 | 6.651 | 0 | 3274 | 4946007192 | 140 | 11186 | 8205 | 69 | 1066392052 | n/a | n/a | n/a | 1 | 24 | 336 | 398 | +| nanogpt | train (sgd), step 2 | dense_gpu | 1 | 2.331 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 2121 | 0 | 7219030384 | 3.653 | 8.976 | 0 | 4767 | 5935215336 | 140 | 16497 | 12012 | 102 | 1599588072 | n/a | 0.195 | 0.005 | 1 | 35 | 504 | 597 | +| nanogpt | loss, step 3 | dense_gpu | 1 | 0.199 | ok (9e-08) | 6.895 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 2186 | 0 | 7219030384 | 3.827 | 9.176 | 0 | 4863 | 6924412304 | 140 | 16685 | 12209 | 103 | 1599588076 | n/a | n/a | n/a | 1 | 36 | 504 | 597 | +| nanogpt | train (sgd), step 3 | dense_gpu | 1 | 2.321 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 2828 | 0 | 7219030384 | 4.808 | 11.492 | 0 | 6356 | 7913620448 | 140 | 21996 | 16016 | 136 | 2132784096 | n/a | 0.238 | 0.005 | 1 | 47 | 672 | 796 | +| nanogpt | loss, step 4 | dense_gpu | 1 | 0.197 | ok (8e-08) | 6.484 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 2893 | 0 | 7219030384 | 4.981 | 11.688 | 0 | 6452 | 8902817416 | 140 | 22184 | 16213 | 137 | 2132784100 | n/a | n/a | n/a | 1 | 48 | 672 | 796 | +| nanogpt | train (sgd), step 4 | dense_gpu | 1 | 2.334 | ok (3e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 128 | 50257 | 123649536 | sgd | 0.010 | False | 13 | 1 | 707 | 3535 | 0 | 7219030384 | 6.001 | 14.017 | 0 | 7945 | 9892025560 | 140 | 27495 | 20020 | 170 | 2665980120 | n/a | 0.282 | 0.005 | 1 | 59 | 840 | 995 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.csv diff --git a/benchmarks/apple/dense-measured-gpt2-c128-runs.json b/benchmarks/apple/dense-measured-gpt2-c128-runs.json new file mode 100644 index 00000000..6a043bc9 --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c128-runs.json @@ -0,0 +1,74 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.csv", + "--context", + "128", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-measured-gpt2-c128-dense_cpu.log", + "csv": "dense-measured-gpt2-c128-dense_cpu.csv", + "peak_rss_gib": 35.78117370605469, + "status": "passed", + "started_utc": "2026-11-01T23:32:23.618737+00:00", + "exit_code": 0, + "wall_seconds": 46.86709191696718, + "checked_rows": 10 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.csv", + "--context", + "128", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-measured-gpt2-c128-dense_gpu.log", + "csv": "dense-measured-gpt2-c128-dense_gpu.csv", + "peak_rss_gib": 18.463027954101562, + "status": "passed", + "started_utc": "2026-11-01T23:33:10.486683+00:00", + "exit_code": 0, + "wall_seconds": 30.451320540974848, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.csv b/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.csv new file mode 100644 index 00000000..723c4e85 --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",dense_cpu,1,0.2411250410368666,ok (3e-16),11.218448,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,65,65,0,874613523,0.20421678957063705,0.23056858393829316,0,,,,, +nanogpt,"train (sgd), step 0",dense_cpu,1,4.951549417106435,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,707,26255,8589934589,1.1015663080615923,5.099310416961089,0,,0.10574604477733374,0.005019458010792732,1,11 +nanogpt,"loss, step 1",dense_cpu,1,0.17704154190141708,ok (4e-16),9.443567,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,772,26255,8589934589,1.2560201855376363,5.254137625917792,0,,,,1,12 +nanogpt,"train (sgd), step 1",dense_cpu,1,3.937083958997391,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,1414,52510,8589934589,2.1307576920371503,9.119261625921354,0,,0.14972742181271315,0.005096082924865186,1,23 +nanogpt,"loss, step 2",dense_cpu,1,0.17266641603782773,ok (3e-16),7.05059,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,1479,52510,8589934589,2.278495654463768,9.267355459975079,0,,,,1,24 +nanogpt,"train (sgd), step 2",dense_cpu,1,3.84204950009007,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,2121,78765,8589934589,3.108662732760422,13.035509417997673,0,,0.19344762770924717,0.005169791984371841,1,35 +nanogpt,"loss, step 3",dense_cpu,1,0.16423116705846041,ok (0e+00),8.578056,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,2186,78765,8589934589,3.252407315187156,13.179606584948488,0,,,,1,36 +nanogpt,"train (sgd), step 3",dense_cpu,1,3.857871084008366,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,2828,105020,8589934589,4.114749561529607,16.965993417892605,0,,0.23701816971879452,0.005251958966255188,1,47 +nanogpt,"loss, step 4",dense_cpu,1,0.16875254199840128,ok (3e-16),6.74504,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,2893,105020,8589934589,4.261292810901068,17.112894292804413,0,,,,1,48 +nanogpt,"train (sgd), step 4",dense_cpu,1,3.828551125014201,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,3535,131275,8589934589,5.09097968484275,20.868966875830665,0,,0.28081154357641935,0.005326375016011298,1,59 diff --git a/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.log b/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.log new file mode 100644 index 00000000..70f93b4a --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 123,563,520 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=16, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_cpu batch=1 0.241s + train (sgd), step 0 dense_cpu batch=1 4.952s + loss, step 1 dense_cpu batch=1 0.177s + train (sgd), step 1 dense_cpu batch=1 3.937s + loss, step 2 dense_cpu batch=1 0.173s + train (sgd), step 2 dense_cpu batch=1 3.842s + loss, step 3 dense_cpu batch=1 0.164s + train (sgd), step 3 dense_cpu batch=1 3.858s + loss, step 4 dense_cpu batch=1 0.169s + train (sgd), step 4 dense_cpu batch=1 3.829s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_cpu | 1 | 0.241 | ok (3e-16) | 11.218 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 65 | 65 | 0 | 874613523 | 0.204 | 0.231 | 0 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_cpu | 1 | 4.952 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 707 | 26255 | 8589934589 | 1.102 | 5.099 | 0 | n/a | 0.106 | 0.005 | 1 | 11 | +| nanogpt | loss, step 1 | dense_cpu | 1 | 0.177 | ok (4e-16) | 9.444 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 772 | 26255 | 8589934589 | 1.256 | 5.254 | 0 | n/a | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | dense_cpu | 1 | 3.937 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 1414 | 52510 | 8589934589 | 2.131 | 9.119 | 0 | n/a | 0.150 | 0.005 | 1 | 23 | +| nanogpt | loss, step 2 | dense_cpu | 1 | 0.173 | ok (3e-16) | 7.051 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 1479 | 52510 | 8589934589 | 2.278 | 9.267 | 0 | n/a | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | dense_cpu | 1 | 3.842 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 2121 | 78765 | 8589934589 | 3.109 | 13.036 | 0 | n/a | 0.193 | 0.005 | 1 | 35 | +| nanogpt | loss, step 3 | dense_cpu | 1 | 0.164 | ok (0e+00) | 8.578 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 2186 | 78765 | 8589934589 | 3.252 | 13.180 | 0 | n/a | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | dense_cpu | 1 | 3.858 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 2828 | 105020 | 8589934589 | 4.115 | 16.966 | 0 | n/a | 0.237 | 0.005 | 1 | 47 | +| nanogpt | loss, step 4 | dense_cpu | 1 | 0.169 | ok (3e-16) | 6.745 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 2893 | 105020 | 8589934589 | 4.261 | 17.113 | 0 | n/a | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | dense_cpu | 1 | 3.829 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 3535 | 131275 | 8589934589 | 5.091 | 20.869 | 0 | n/a | 0.281 | 0.005 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.csv diff --git a/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.csv b/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.csv new file mode 100644 index 00000000..030f0d95 --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates,gpu_union_reductions,gpu_scatter_reductions +nanogpt,"loss, step 0",dense_gpu,1,0.30283583304844797,ok (2e-08),11.218448,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,65,65,0,417102283,0.24769529106561095,0.30261050001718104,0,96,988508840,9,188,197,1,4,,,,,,, +nanogpt,"train (sgd), step 0",dense_gpu,1,3.4466827500145882,ok (3e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,707,0,5152709312,1.1685504958732054,3.74532695801463,0,1589,1977028856,140,5499,4004,34,532851848,,0.10397278983145952,0.004886207985691726,1,11,168,199 +nanogpt,"loss, step 1",dense_gpu,1,0.2056449999799952,ok (1e-07),9.443567,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,772,0,5152709312,1.3560123713687062,3.9507552080322057,0,1685,2965537696,140,5687,4201,35,532851852,,,,1,12,168,199 +nanogpt,"train (sgd), step 1",dense_gpu,1,2.2588730410207063,ok (3e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,1414,0,5152709312,2.2531927733216435,6.205487665953115,0,3178,3954057712,140,10998,8008,68,1065703696,,0.14989395369775593,0.0049617920303717256,1,23,336,398 +nanogpt,"loss, step 2",dense_gpu,1,0.20018020796123892,ok (1e-08),7.05059,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,1479,0,5152709312,2.4321851078420877,6.4054046659730375,0,3274,4942566552,140,11186,8205,69,1065703700,,,,1,24,336,398 +nanogpt,"train (sgd), step 2",dense_gpu,1,2.242123374948278,ok (3e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,2121,0,5152709312,3.3513271970441565,8.642735500005074,0,4767,5931086568,140,16497,12012,102,1598555544,,0.19350328878499568,0.005034249974414706,1,35,504,597 +nanogpt,"loss, step 3",dense_gpu,1,0.19902858301065862,ok (8e-08),8.578056,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,2186,0,5152709312,3.5282166552497074,8.84154616703745,0,4863,6919595408,140,16685,12209,103,1598555548,,,,1,36,504,597 +nanogpt,"train (sgd), step 3",dense_gpu,1,2.197064582956955,ok (3e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,2828,0,5152709312,4.422117635025643,11.033719958039,0,6356,7908115424,140,21996,16016,136,2131407392,,0.23741058073937893,0.005113207967951894,1,47,672,796 +nanogpt,"loss, step 4",dense_gpu,1,0.20738541590981185,ok (9e-08),6.74504,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,2893,0,5152709312,4.604739512666129,11.240865333005786,0,6452,8896624264,140,22184,16213,137,2131407396,,,,1,48,672,796 +nanogpt,"train (sgd), step 4",dense_gpu,1,2.2273182090139017,ok (3e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,707,3535,0,5152709312,5.535126848728396,13.463256041985005,0,7945,9885144280,140,27495,20020,170,2664259240,,0.2814445396652445,0.005190290976315737,1,59,840,995 diff --git a/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.log b/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.log new file mode 100644 index 00000000..54eb3bef --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 123,563,520 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=16, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 dense_gpu batch=1 0.303s + train (sgd), step 0 dense_gpu batch=1 3.447s + loss, step 1 dense_gpu batch=1 0.206s + train (sgd), step 1 dense_gpu batch=1 2.259s + loss, step 2 dense_gpu batch=1 0.200s + train (sgd), step 2 dense_gpu batch=1 2.242s + loss, step 3 dense_gpu batch=1 0.199s + train (sgd), step 3 dense_gpu batch=1 2.197s + loss, step 4 dense_gpu batch=1 0.207s + train (sgd), step 4 dense_gpu batch=1 2.227s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | gpu_input_arrays | gpu_input_bytes | gpu_fused_programs | gpu_fused_executions | gpu_contractions | gpu_readbacks | gpu_readback_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | gpu_union_reductions | gpu_scatter_reductions | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_gpu | 1 | 0.303 | ok (2e-08) | 11.218 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 65 | 65 | 0 | 417102283 | 0.248 | 0.303 | 0 | 96 | 988508840 | 9 | 188 | 197 | 1 | 4 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_gpu | 1 | 3.447 | ok (3e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 707 | 0 | 5152709312 | 1.169 | 3.745 | 0 | 1589 | 1977028856 | 140 | 5499 | 4004 | 34 | 532851848 | n/a | 0.104 | 0.005 | 1 | 11 | 168 | 199 | +| nanogpt | loss, step 1 | dense_gpu | 1 | 0.206 | ok (1e-07) | 9.444 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 772 | 0 | 5152709312 | 1.356 | 3.951 | 0 | 1685 | 2965537696 | 140 | 5687 | 4201 | 35 | 532851852 | n/a | n/a | n/a | 1 | 12 | 168 | 199 | +| nanogpt | train (sgd), step 1 | dense_gpu | 1 | 2.259 | ok (3e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 1414 | 0 | 5152709312 | 2.253 | 6.205 | 0 | 3178 | 3954057712 | 140 | 10998 | 8008 | 68 | 1065703696 | n/a | 0.150 | 0.005 | 1 | 23 | 336 | 398 | +| nanogpt | loss, step 2 | dense_gpu | 1 | 0.200 | ok (1e-08) | 7.051 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 1479 | 0 | 5152709312 | 2.432 | 6.405 | 0 | 3274 | 4942566552 | 140 | 11186 | 8205 | 69 | 1065703700 | n/a | n/a | n/a | 1 | 24 | 336 | 398 | +| nanogpt | train (sgd), step 2 | dense_gpu | 1 | 2.242 | ok (3e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 2121 | 0 | 5152709312 | 3.351 | 8.643 | 0 | 4767 | 5931086568 | 140 | 16497 | 12012 | 102 | 1598555544 | n/a | 0.194 | 0.005 | 1 | 35 | 504 | 597 | +| nanogpt | loss, step 3 | dense_gpu | 1 | 0.199 | ok (8e-08) | 8.578 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 2186 | 0 | 5152709312 | 3.528 | 8.842 | 0 | 4863 | 6919595408 | 140 | 16685 | 12209 | 103 | 1598555548 | n/a | n/a | n/a | 1 | 36 | 504 | 597 | +| nanogpt | train (sgd), step 3 | dense_gpu | 1 | 2.197 | ok (3e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 2828 | 0 | 5152709312 | 4.422 | 11.034 | 0 | 6356 | 7908115424 | 140 | 21996 | 16016 | 136 | 2131407392 | n/a | 0.237 | 0.005 | 1 | 47 | 672 | 796 | +| nanogpt | loss, step 4 | dense_gpu | 1 | 0.207 | ok (9e-08) | 6.745 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 2893 | 0 | 5152709312 | 4.605 | 11.241 | 0 | 6452 | 8896624264 | 140 | 22184 | 16213 | 137 | 2131407396 | n/a | n/a | n/a | 1 | 48 | 672 | 796 | +| nanogpt | train (sgd), step 4 | dense_gpu | 1 | 2.227 | ok (3e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 707 | 3535 | 0 | 5152709312 | 5.535 | 13.463 | 0 | 7945 | 9885144280 | 140 | 27495 | 20020 | 170 | 2664259240 | n/a | 0.281 | 0.005 | 1 | 59 | 840 | 995 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.csv diff --git a/benchmarks/apple/dense-measured-gpt2-c16-runs.json b/benchmarks/apple/dense-measured-gpt2-c16-runs.json new file mode 100644 index 00000000..07984b7a --- /dev/null +++ b/benchmarks/apple/dense-measured-gpt2-c16-runs.json @@ -0,0 +1,74 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-gpt2-c16-dense_cpu.log", + "csv": "dense-measured-gpt2-c16-dense_cpu.csv", + "peak_rss_gib": 22.903076171875, + "status": "passed", + "started_utc": "2026-11-01T23:25:22.753974+00:00", + "exit_code": 0, + "wall_seconds": 36.28782012499869, + "checked_rows": 10 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-gpt2-c16-dense_gpu.log", + "csv": "dense-measured-gpt2-c16-dense_gpu.csv", + "peak_rss_gib": 17.911361694335938, + "status": "passed", + "started_utc": "2026-11-01T23:25:59.042533+00:00", + "exit_code": 0, + "wall_seconds": 28.680158666917123, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/dense-measured-shakespeare-dense_cpu.csv b/benchmarks/apple/dense-measured-shakespeare-dense_cpu.csv new file mode 100644 index 00000000..eabde523 --- /dev/null +++ b/benchmarks/apple/dense-measured-shakespeare-dense_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",dense_cpu,1,0.12969304190482944,ok (0e+00),4.243064,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,35,35,0,527038851,0.11209549952764064,0.12614800001028925,0,,,,, +nanogpt,"train (sgd), step 0",dense_cpu,1,1.7400712919188663,ok (1e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,316,0,5340715780,0.46617837890516967,1.8110525000374764,0,,0.0269849969772622,0.0026606250321492553,1,11 +nanogpt,"loss, step 1",dense_cpu,1,0.09850491711404175,ok (0e+00),3.670806,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,351,0,5340715780,0.5607024627970532,1.9057971250731498,0,,,,1,12 +nanogpt,"train (sgd), step 1",dense_cpu,1,1.3464477499946952,ok (3e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,632,0,5340715780,0.898749205400236,3.197879083105363,0,,0.032657457049936056,0.0027331250021234155,1,23 +nanogpt,"loss, step 2",dense_cpu,1,0.09525075007695705,ok (2e-16),3.681532,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,667,0,5340715780,0.9900317442370579,3.2893898740876466,0,,,,1,24 +nanogpt,"train (sgd), step 2",dense_cpu,1,1.255308499909006,ok (3e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,948,0,5340715780,1.315386119298637,4.494990166043863,0,,0.03844083205331117,0.002784542040899396,1,35 +nanogpt,"loss, step 3",dense_cpu,1,0.09543283306993544,ok (1e-16),3.366265,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,983,0,5340715780,1.4068186182994395,4.586644332041033,0,,,,1,36 +nanogpt,"train (sgd), step 3",dense_cpu,1,1.3276504999957979,ok (3e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,1264,0,5340715780,1.753027451224625,5.846702957060188,0,,0.044066331116482615,0.0028370420914143324,1,47 +nanogpt,"loss, step 4",dense_cpu,1,0.09688924998044968,ok (0e+00),3.36678,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,1299,0,5340715780,1.845867117983289,5.9397677490487695,0,,,,1,48 +nanogpt,"train (sgd), step 4",dense_cpu,1,1.309704374987632,ok (3e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,1580,0,5340715780,2.171821787604131,7.200985666015185,0,,0.04971128888428211,0.002891167066991329,1,59 diff --git a/benchmarks/apple/dense-measured-shakespeare-dense_cpu.log b/benchmarks/apple/dense-measured-shakespeare-dense_cpu.log new file mode 100644 index 00000000..c9888e59 --- /dev/null +++ b/benchmarks/apple/dense-measured-shakespeare-dense_cpu.log @@ -0,0 +1,27 @@ + 6-layer GPT, vocab 65, 10,745,088 parameters: the loss is 543 lines of SQL + steps=5, optimizer=sgd, context=256, heads=6, embed=384, forward_only=False + loss, step 0 dense_cpu batch=1 0.130s + train (sgd), step 0 dense_cpu batch=1 1.740s + loss, step 1 dense_cpu batch=1 0.099s + train (sgd), step 1 dense_cpu batch=1 1.346s + loss, step 2 dense_cpu batch=1 0.095s + train (sgd), step 2 dense_cpu batch=1 1.255s + loss, step 3 dense_cpu batch=1 0.095s + train (sgd), step 3 dense_cpu batch=1 1.328s + loss, step 4 dense_cpu batch=1 0.097s + train (sgd), step 4 dense_cpu batch=1 1.310s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_cpu | 1 | 0.130 | ok (0e+00) | 4.243 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 35 | 35 | 0 | 527038851 | 0.112 | 0.126 | 0 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_cpu | 1 | 1.740 | ok (1e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 316 | 0 | 5340715780 | 0.466 | 1.811 | 0 | n/a | 0.027 | 0.003 | 1 | 11 | +| nanogpt | loss, step 1 | dense_cpu | 1 | 0.099 | ok (0e+00) | 3.671 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 351 | 0 | 5340715780 | 0.561 | 1.906 | 0 | n/a | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | dense_cpu | 1 | 1.346 | ok (3e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 632 | 0 | 5340715780 | 0.899 | 3.198 | 0 | n/a | 0.033 | 0.003 | 1 | 23 | +| nanogpt | loss, step 2 | dense_cpu | 1 | 0.095 | ok (2e-16) | 3.682 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 667 | 0 | 5340715780 | 0.990 | 3.289 | 0 | n/a | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | dense_cpu | 1 | 1.255 | ok (3e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 948 | 0 | 5340715780 | 1.315 | 4.495 | 0 | n/a | 0.038 | 0.003 | 1 | 35 | +| nanogpt | loss, step 3 | dense_cpu | 1 | 0.095 | ok (1e-16) | 3.366 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 983 | 0 | 5340715780 | 1.407 | 4.587 | 0 | n/a | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | dense_cpu | 1 | 1.328 | ok (3e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 1264 | 0 | 5340715780 | 1.753 | 5.847 | 0 | n/a | 0.044 | 0.003 | 1 | 47 | +| nanogpt | loss, step 4 | dense_cpu | 1 | 0.097 | ok (0e+00) | 3.367 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 1299 | 0 | 5340715780 | 1.846 | 5.940 | 0 | n/a | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | dense_cpu | 1 | 1.310 | ok (3e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 1580 | 0 | 5340715780 | 2.172 | 7.201 | 0 | n/a | 0.050 | 0.003 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-shakespeare-dense_cpu.csv diff --git a/benchmarks/apple/dense-measured-shakespeare-dense_gpu.csv b/benchmarks/apple/dense-measured-shakespeare-dense_gpu.csv new file mode 100644 index 00000000..1a103150 --- /dev/null +++ b/benchmarks/apple/dense-measured-shakespeare-dense_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates,gpu_union_reductions,gpu_scatter_reductions +nanogpt,"loss, step 0",dense_gpu,1,0.07125879207160324,ok (6e-08),4.243064,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,35,35,0,176565627,0.04190937418024987,0.07108991593122482,0,54,85961048,9,98,101,1,4,,,,,,, +nanogpt,"train (sgd), step 0",dense_gpu,1,0.9215724170207977,ok (8e-08),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,316,0,1483857024,0.32746717310510576,0.9908825408201665,0,839,171927944,91,2229,1448,34,43005692,,0.026845209999009967,0.00260245893150568,1,11,96,103 +nanogpt,"loss, step 1",dense_gpu,1,0.04705770802684128,ok (1e-09),3.670806,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,351,0,1483857024,0.3629802556242794,1.037770123919472,0,893,257888992,91,2327,1549,35,43005696,,,,1,12,96,103 +nanogpt,"train (sgd), step 1",dense_gpu,1,0.5563703749794513,ok (1e-07),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,632,0,1483857024,0.66398379963357,1.5927157069090754,0,1678,343855888,91,4458,2896,68,86011384,,0.032448251149617136,0.0026652509113773704,1,23,192,206 +nanogpt,"loss, step 2",dense_gpu,1,0.04081658401992172,ok (2e-08),3.681532,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,667,0,1483857024,0.6983936328906566,1.6333495408762246,0,1732,429816936,91,4556,2997,69,86011388,,,,1,24,192,206 +nanogpt,"train (sgd), step 2",dense_gpu,1,0.565831792075187,ok (2e-07),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,948,0,1483857024,0.9849822508404031,2.1975372908636928,0,2517,515783832,91,6687,4344,102,129017076,,0.038075793301686645,0.0027181259356439114,1,35,288,309 +nanogpt,"loss, step 3",dense_gpu,1,0.040763458935543895,ok (1e-09),3.366265,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,983,0,1483857024,1.0193667089333758,2.238110124832019,0,2571,601744880,91,6785,4445,103,129017080,,,,1,36,288,309 +nanogpt,"train (sgd), step 3",dense_gpu,1,0.5136696669505909,ok (2e-07),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,1264,0,1483857024,1.303268792340532,2.7495695418911055,0,3356,687711776,91,8916,5792,136,172022768,,0.04368383309338242,0.0027690428541973233,1,47,384,412 +nanogpt,"loss, step 4",dense_gpu,1,0.08205695904325694,ok (4e-08),3.36678,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,1299,0,1483857024,1.3787565431557596,2.8314313328592107,0,3410,773672824,91,9014,5893,137,172022772,,,,1,48,384,412 +nanogpt,"train (sgd), step 4",dense_gpu,1,0.509722915943712,ok (2e-07),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,316,1580,0,1483857024,1.6390505810268223,3.3393739578314126,0,4195,859639720,91,11145,7240,170,215028460,,0.04925570881459862,0.0028301269048824906,1,59,480,515 diff --git a/benchmarks/apple/dense-measured-shakespeare-dense_gpu.log b/benchmarks/apple/dense-measured-shakespeare-dense_gpu.log new file mode 100644 index 00000000..8b5d895b --- /dev/null +++ b/benchmarks/apple/dense-measured-shakespeare-dense_gpu.log @@ -0,0 +1,27 @@ + 6-layer GPT, vocab 65, 10,745,088 parameters: the loss is 543 lines of SQL + steps=5, optimizer=sgd, context=256, heads=6, embed=384, forward_only=False + loss, step 0 dense_gpu batch=1 0.071s + train (sgd), step 0 dense_gpu batch=1 0.922s + loss, step 1 dense_gpu batch=1 0.047s + train (sgd), step 1 dense_gpu batch=1 0.556s + loss, step 2 dense_gpu batch=1 0.041s + train (sgd), step 2 dense_gpu batch=1 0.566s + loss, step 3 dense_gpu batch=1 0.041s + train (sgd), step 3 dense_gpu batch=1 0.514s + loss, step 4 dense_gpu batch=1 0.082s + train (sgd), step 4 dense_gpu batch=1 0.510s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | gpu_input_arrays | gpu_input_bytes | gpu_fused_programs | gpu_fused_executions | gpu_contractions | gpu_readbacks | gpu_readback_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | gpu_union_reductions | gpu_scatter_reductions | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_gpu | 1 | 0.071 | ok (6e-08) | 4.243 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 35 | 35 | 0 | 176565627 | 0.042 | 0.071 | 0 | 54 | 85961048 | 9 | 98 | 101 | 1 | 4 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_gpu | 1 | 0.922 | ok (8e-08) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 316 | 0 | 1483857024 | 0.327 | 0.991 | 0 | 839 | 171927944 | 91 | 2229 | 1448 | 34 | 43005692 | n/a | 0.027 | 0.003 | 1 | 11 | 96 | 103 | +| nanogpt | loss, step 1 | dense_gpu | 1 | 0.047 | ok (1e-09) | 3.671 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 351 | 0 | 1483857024 | 0.363 | 1.038 | 0 | 893 | 257888992 | 91 | 2327 | 1549 | 35 | 43005696 | n/a | n/a | n/a | 1 | 12 | 96 | 103 | +| nanogpt | train (sgd), step 1 | dense_gpu | 1 | 0.556 | ok (1e-07) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 632 | 0 | 1483857024 | 0.664 | 1.593 | 0 | 1678 | 343855888 | 91 | 4458 | 2896 | 68 | 86011384 | n/a | 0.032 | 0.003 | 1 | 23 | 192 | 206 | +| nanogpt | loss, step 2 | dense_gpu | 1 | 0.041 | ok (2e-08) | 3.682 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 667 | 0 | 1483857024 | 0.698 | 1.633 | 0 | 1732 | 429816936 | 91 | 4556 | 2997 | 69 | 86011388 | n/a | n/a | n/a | 1 | 24 | 192 | 206 | +| nanogpt | train (sgd), step 2 | dense_gpu | 1 | 0.566 | ok (2e-07) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 948 | 0 | 1483857024 | 0.985 | 2.198 | 0 | 2517 | 515783832 | 91 | 6687 | 4344 | 102 | 129017076 | n/a | 0.038 | 0.003 | 1 | 35 | 288 | 309 | +| nanogpt | loss, step 3 | dense_gpu | 1 | 0.041 | ok (1e-09) | 3.366 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 983 | 0 | 1483857024 | 1.019 | 2.238 | 0 | 2571 | 601744880 | 91 | 6785 | 4445 | 103 | 129017080 | n/a | n/a | n/a | 1 | 36 | 288 | 309 | +| nanogpt | train (sgd), step 3 | dense_gpu | 1 | 0.514 | ok (2e-07) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 1264 | 0 | 1483857024 | 1.303 | 2.750 | 0 | 3356 | 687711776 | 91 | 8916 | 5792 | 136 | 172022768 | n/a | 0.044 | 0.003 | 1 | 47 | 384 | 412 | +| nanogpt | loss, step 4 | dense_gpu | 1 | 0.082 | ok (4e-08) | 3.367 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 1299 | 0 | 1483857024 | 1.379 | 2.831 | 0 | 3410 | 773672824 | 91 | 9014 | 5893 | 137 | 172022772 | n/a | n/a | n/a | 1 | 48 | 384 | 412 | +| nanogpt | train (sgd), step 4 | dense_gpu | 1 | 0.510 | ok (2e-07) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 316 | 1580 | 0 | 1483857024 | 1.639 | 3.339 | 0 | 4195 | 859639720 | 91 | 11145 | 7240 | 170 | 215028460 | n/a | 0.049 | 0.003 | 1 | 59 | 480 | 515 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-shakespeare-dense_gpu.csv diff --git a/benchmarks/apple/dense-measured-shakespeare-runs.json b/benchmarks/apple/dense-measured-shakespeare-runs.json new file mode 100644 index 00000000..19b1b431 --- /dev/null +++ b/benchmarks/apple/dense-measured-shakespeare-runs.json @@ -0,0 +1,66 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "shakespeare", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-shakespeare-dense_cpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-shakespeare-dense_cpu.log", + "csv": "dense-measured-shakespeare-dense_cpu.csv", + "peak_rss_gib": 7.9724884033203125, + "status": "passed", + "started_utc": "2026-11-01T23:32:03.760863+00:00", + "exit_code": 0, + "wall_seconds": 12.016731290961616, + "checked_rows": 10 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "shakespeare", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-shakespeare-dense_gpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-shakespeare-dense_gpu.log", + "csv": "dense-measured-shakespeare-dense_gpu.csv", + "peak_rss_gib": 2.7833099365234375, + "status": "passed", + "started_utc": "2026-11-01T23:32:15.778182+00:00", + "exit_code": 0, + "wall_seconds": 7.818204250070266, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/dense-measured-tiny-dense_cpu.csv b/benchmarks/apple/dense-measured-tiny-dense_cpu.csv new file mode 100644 index 00000000..311a964d --- /dev/null +++ b/benchmarks/apple/dense-measured-tiny-dense_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",dense_cpu,4,0.008622791967354715,ok (4e-16),4.179251,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,10,10,0,762795,0.005173332872800529,0.00850154203362763,0,,,,, +nanogpt,"train (sgd), step 0",dense_cpu,4,0.11979020806029439,ok (2e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,86,0,6620829,0.03189966210629791,0.12786275008693337,0,,0.0046209164429455996,0.0008603750029578805,1,11 +nanogpt,"loss, step 1",dense_cpu,4,0.005101999966427684,ok (0e+00),4.104661,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,96,0,6620829,0.03681991109624505,0.1329101660521701,0,,,,1,12 +nanogpt,"train (sgd), step 1",dense_cpu,4,0.06502658396493644,ok (4e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,172,0,6690877,0.08389065554365516,0.19744008197449148,0,,0.006402165396139026,0.0008917080704122782,1,23 +nanogpt,"loss, step 2",dense_cpu,4,0.004534624982625246,ok (1e-16),3.883857,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,182,0,6690877,0.08829236275050789,0.201925539993681,0,,,,1,24 +nanogpt,"train (sgd), step 2",dense_cpu,4,0.042114209034480155,ok (5e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,258,0,6690877,0.11350756301544607,0.24360029003582895,0,,0.008142665377818048,0.0009154161671176553,1,35 +nanogpt,"loss, step 3",dense_cpu,4,0.004402416059747338,ok (1e-16),3.641763,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,268,0,6690877,0.11778052197769284,0.24795316508971155,0,,,,1,36 +nanogpt,"train (sgd), step 3",dense_cpu,4,0.04103962506633252,ok (5e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,344,0,6690877,0.14231293846387416,0.28855874913278967,0,,0.009968249243684113,0.0009386661695316434,1,47 +nanogpt,"loss, step 4",dense_cpu,4,0.004441625089384615,ok (3e-16),3.842234,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,354,0,6690877,0.14662235660944134,0.2929502901388332,0,,,,1,48 +nanogpt,"train (sgd), step 4",dense_cpu,4,0.0406262920005247,ok (8e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,430,0,6713053,0.17087464581709355,0.3331449571996927,0,,0.011711789178662002,0.0009609580738469958,1,59 diff --git a/benchmarks/apple/dense-measured-tiny-dense_cpu.log b/benchmarks/apple/dense-measured-tiny-dense_cpu.log new file mode 100644 index 00000000..ea165a63 --- /dev/null +++ b/benchmarks/apple/dense-measured-tiny-dense_cpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 dense_cpu batch=4 0.009s + train (sgd), step 0 dense_cpu batch=4 0.120s + loss, step 1 dense_cpu batch=4 0.005s + train (sgd), step 1 dense_cpu batch=4 0.065s + loss, step 2 dense_cpu batch=4 0.005s + train (sgd), step 2 dense_cpu batch=4 0.042s + loss, step 3 dense_cpu batch=4 0.004s + train (sgd), step 3 dense_cpu batch=4 0.041s + loss, step 4 dense_cpu batch=4 0.004s + train (sgd), step 4 dense_cpu batch=4 0.041s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_cpu | 4 | 0.009 | ok (4e-16) | 4.179 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 10 | 10 | 0 | 762795 | 0.005 | 0.009 | 0 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_cpu | 4 | 0.120 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 86 | 0 | 6620829 | 0.032 | 0.128 | 0 | n/a | 0.005 | 0.001 | 1 | 11 | +| nanogpt | loss, step 1 | dense_cpu | 4 | 0.005 | ok (0e+00) | 4.105 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 96 | 0 | 6620829 | 0.037 | 0.133 | 0 | n/a | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | dense_cpu | 4 | 0.065 | ok (4e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 172 | 0 | 6690877 | 0.084 | 0.197 | 0 | n/a | 0.006 | 0.001 | 1 | 23 | +| nanogpt | loss, step 2 | dense_cpu | 4 | 0.005 | ok (1e-16) | 3.884 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 182 | 0 | 6690877 | 0.088 | 0.202 | 0 | n/a | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | dense_cpu | 4 | 0.042 | ok (5e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 258 | 0 | 6690877 | 0.114 | 0.244 | 0 | n/a | 0.008 | 0.001 | 1 | 35 | +| nanogpt | loss, step 3 | dense_cpu | 4 | 0.004 | ok (1e-16) | 3.642 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 268 | 0 | 6690877 | 0.118 | 0.248 | 0 | n/a | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | dense_cpu | 4 | 0.041 | ok (5e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 344 | 0 | 6690877 | 0.142 | 0.289 | 0 | n/a | 0.010 | 0.001 | 1 | 47 | +| nanogpt | loss, step 4 | dense_cpu | 4 | 0.004 | ok (3e-16) | 3.842 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 354 | 0 | 6690877 | 0.147 | 0.293 | 0 | n/a | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | dense_cpu | 4 | 0.041 | ok (8e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 430 | 0 | 6713053 | 0.171 | 0.333 | 0 | n/a | 0.012 | 0.001 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-tiny-dense_cpu.csv diff --git a/benchmarks/apple/dense-measured-tiny-dense_gpu.csv b/benchmarks/apple/dense-measured-tiny-dense_gpu.csv new file mode 100644 index 00000000..924c3c42 --- /dev/null +++ b/benchmarks/apple/dense-measured-tiny-dense_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,dense_programs,dense_executions,dense_cache_evictions,dense_peak_cached_bytes,dense_lowering_seconds,dense_call_seconds,key_cache_bytes,gpu_input_arrays,gpu_input_bytes,gpu_fused_programs,gpu_fused_executions,gpu_contractions,gpu_readbacks,gpu_readback_bytes,dense_last_decline,load_seconds,model_seconds,gradient_preparations,inplace_updates,gpu_union_reductions,gpu_scatter_reductions +nanogpt,"loss, step 0",dense_gpu,4,0.19149316602852196,ok (5e-08),4.179251,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,10,10,0,221219,0.005893289111554623,0.19139137491583824,0,19,35392,9,23,21,1,4,,,,,,, +nanogpt,"train (sgd), step 0",dense_gpu,4,0.16422079200856388,ok (4e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,86,0,1827051,0.03457403834909201,0.35542787495069206,0,214,72192,51,377,191,34,18746,,0.004631083109416068,0.0008682499174028635,1,11,22,23 +nanogpt,"loss, step 1",dense_gpu,4,0.007177999941632152,ok (2e-08),4.104661,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,96,0,1827051,0.03964228625409305,0.3625815419945866,0,233,107584,51,400,212,35,18750,,,,1,12,22,23 +nanogpt,"train (sgd), step 1",dense_gpu,4,0.0816356249852106,ok (3e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,172,0,1885515,0.08886882627848536,0.4440025419462472,0,428,144384,51,754,382,68,37492,,0.006363542168401182,0.0009006250184029341,1,23,44,46 +nanogpt,"loss, step 2",dense_gpu,4,0.006223625037819147,ok (5e-08),3.883857,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,182,0,1885515,0.09352836827747524,0.4502004999667406,0,447,179776,51,777,403,69,37496,,,,1,24,44,46 +nanogpt,"train (sgd), step 2",dense_gpu,4,0.0589954589959234,ok (3e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,258,0,1885515,0.12126461451407522,0.5087507090065628,0,642,216576,51,1131,573,102,56238,,0.008092333097010851,0.0009255839977413416,1,35,66,69 +nanogpt,"loss, step 3",dense_gpu,4,0.0062376250280067325,ok (1e-08),3.641763,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,268,0,1885515,0.1257922404911369,0.5149658749578521,0,661,251968,51,1154,594,103,56242,,,,1,36,66,69 +nanogpt,"train (sgd), step 3",dense_gpu,4,0.057437957962974906,ok (4e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,344,0,1885515,0.15403815894387662,0.5721922919619828,0,856,288768,51,1508,764,136,74984,,0.00980150094255805,0.0009495420381426811,1,47,88,92 +nanogpt,"loss, step 4",dense_gpu,4,0.02939050004351884,ok (3e-09),3.842234,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,354,0,1885515,0.1818226168397814,0.6015549999428913,0,875,324160,51,1531,785,137,74988,,,,1,48,88,92 +nanogpt,"train (sgd), step 4",dense_gpu,4,0.06349054200109094,ok (8e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,86,430,0,1903979,0.20953710924368352,0.6648724579717964,0,1070,360960,51,1885,955,170,93730,,0.011544793145731091,0.0009757500374689698,1,59,110,115 diff --git a/benchmarks/apple/dense-measured-tiny-dense_gpu.log b/benchmarks/apple/dense-measured-tiny-dense_gpu.log new file mode 100644 index 00000000..0da51fee --- /dev/null +++ b/benchmarks/apple/dense-measured-tiny-dense_gpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 dense_gpu batch=4 0.191s + train (sgd), step 0 dense_gpu batch=4 0.164s + loss, step 1 dense_gpu batch=4 0.007s + train (sgd), step 1 dense_gpu batch=4 0.082s + loss, step 2 dense_gpu batch=4 0.006s + train (sgd), step 2 dense_gpu batch=4 0.059s + loss, step 3 dense_gpu batch=4 0.006s + train (sgd), step 3 dense_gpu batch=4 0.057s + loss, step 4 dense_gpu batch=4 0.029s + train (sgd), step 4 dense_gpu batch=4 0.063s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | dense_programs | dense_executions | dense_cache_evictions | dense_peak_cached_bytes | dense_lowering_seconds | dense_call_seconds | key_cache_bytes | gpu_input_arrays | gpu_input_bytes | gpu_fused_programs | gpu_fused_executions | gpu_contractions | gpu_readbacks | gpu_readback_bytes | dense_last_decline | load_seconds | model_seconds | gradient_preparations | inplace_updates | gpu_union_reductions | gpu_scatter_reductions | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | dense_gpu | 4 | 0.191 | ok (5e-08) | 4.179 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 10 | 10 | 0 | 221219 | 0.006 | 0.191 | 0 | 19 | 35392 | 9 | 23 | 21 | 1 | 4 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | dense_gpu | 4 | 0.164 | ok (4e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 86 | 0 | 1827051 | 0.035 | 0.355 | 0 | 214 | 72192 | 51 | 377 | 191 | 34 | 18746 | n/a | 0.005 | 0.001 | 1 | 11 | 22 | 23 | +| nanogpt | loss, step 1 | dense_gpu | 4 | 0.007 | ok (2e-08) | 4.105 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 96 | 0 | 1827051 | 0.040 | 0.363 | 0 | 233 | 107584 | 51 | 400 | 212 | 35 | 18750 | n/a | n/a | n/a | 1 | 12 | 22 | 23 | +| nanogpt | train (sgd), step 1 | dense_gpu | 4 | 0.082 | ok (3e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 172 | 0 | 1885515 | 0.089 | 0.444 | 0 | 428 | 144384 | 51 | 754 | 382 | 68 | 37492 | n/a | 0.006 | 0.001 | 1 | 23 | 44 | 46 | +| nanogpt | loss, step 2 | dense_gpu | 4 | 0.006 | ok (5e-08) | 3.884 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 182 | 0 | 1885515 | 0.094 | 0.450 | 0 | 447 | 179776 | 51 | 777 | 403 | 69 | 37496 | n/a | n/a | n/a | 1 | 24 | 44 | 46 | +| nanogpt | train (sgd), step 2 | dense_gpu | 4 | 0.059 | ok (3e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 258 | 0 | 1885515 | 0.121 | 0.509 | 0 | 642 | 216576 | 51 | 1131 | 573 | 102 | 56238 | n/a | 0.008 | 0.001 | 1 | 35 | 66 | 69 | +| nanogpt | loss, step 3 | dense_gpu | 4 | 0.006 | ok (1e-08) | 3.642 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 268 | 0 | 1885515 | 0.126 | 0.515 | 0 | 661 | 251968 | 51 | 1154 | 594 | 103 | 56242 | n/a | n/a | n/a | 1 | 36 | 66 | 69 | +| nanogpt | train (sgd), step 3 | dense_gpu | 4 | 0.057 | ok (4e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 344 | 0 | 1885515 | 0.154 | 0.572 | 0 | 856 | 288768 | 51 | 1508 | 764 | 136 | 74984 | n/a | 0.010 | 0.001 | 1 | 47 | 88 | 92 | +| nanogpt | loss, step 4 | dense_gpu | 4 | 0.029 | ok (3e-09) | 3.842 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 354 | 0 | 1885515 | 0.182 | 0.602 | 0 | 875 | 324160 | 51 | 1531 | 785 | 137 | 74988 | n/a | n/a | n/a | 1 | 48 | 88 | 92 | +| nanogpt | train (sgd), step 4 | dense_gpu | 4 | 0.063 | ok (8e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 86 | 430 | 0 | 1903979 | 0.210 | 0.665 | 0 | 1070 | 360960 | 51 | 1885 | 955 | 170 | 93730 | n/a | 0.012 | 0.001 | 1 | 59 | 110 | 115 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-tiny-dense_gpu.csv diff --git a/benchmarks/apple/dense-measured-tiny-runs.json b/benchmarks/apple/dense-measured-tiny-runs.json new file mode 100644 index 00000000..89b85b18 --- /dev/null +++ b/benchmarks/apple/dense-measured-tiny-runs.json @@ -0,0 +1,66 @@ +[ + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-tiny-dense_cpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-tiny-dense_cpu.log", + "csv": "dense-measured-tiny-dense_cpu.csv", + "peak_rss_gib": 0.5282745361328125, + "status": "passed", + "started_utc": "2026-11-01T23:25:16.750396+00:00", + "exit_code": 0, + "wall_seconds": 2.846897584036924, + "checked_rows": 10 + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-tiny-dense_gpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-tiny-dense_gpu.log", + "csv": "dense-measured-tiny-dense_gpu.csv", + "peak_rss_gib": 0.570648193359375, + "status": "passed", + "started_utc": "2026-11-01T23:25:19.601839+00:00", + "exit_code": 0, + "wall_seconds": 3.1278892919654027, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/dense-mypy.log b/benchmarks/apple/dense-mypy.log new file mode 100644 index 00000000..bca2ac65 --- /dev/null +++ b/benchmarks/apple/dense-mypy.log @@ -0,0 +1,2 @@ +pyproject.toml: note: unused section(s): module = ['cftime.*', 'pandas.*', 'xarray.*', 'xarray_sql._native'] +Success: no issues found in 2 source files diff --git a/benchmarks/apple/dense-native-gpt2-c1024-jax_jit.json b/benchmarks/apple/dense-native-gpt2-c1024-jax_jit.json new file mode 100644 index 00000000..f7f59165 --- /dev/null +++ b/benchmarks/apple/dense-native-gpt2-c1024-jax_jit.json @@ -0,0 +1,130 @@ +{ + "backend": "jax_jit", + "context": 1024, + "parameters": 124337664, + "parameter_shapes": { + "wte": [ + 50257, + 768 + ], + "wpe": [ + 1024, + 768 + ], + "ln_1": [ + 12, + 768 + ], + "wq": [ + 12, + 768, + 12, + 64 + ], + "wk": [ + 12, + 768, + 12, + 64 + ], + "wv": [ + 12, + 768, + 12, + 64 + ], + "wo": [ + 12, + 12, + 64, + 768 + ], + "ln_2": [ + 12, + 768 + ], + "w_fc": [ + 12, + 768, + 3072 + ], + "w_proj": [ + 12, + 3072, + 768 + ], + "ln_f": [ + 768 + ] + }, + "parameter_counts": { + "wte": 38597376, + "wpe": 786432, + "ln_1": 9216, + "wq": 7077888, + "wk": 7077888, + "wv": 7077888, + "wo": 7077888, + "ln_2": 9216, + "w_fc": 28311552, + "w_proj": 28311552, + "ln_f": 768 + }, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 2.7095948749920353, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 1.5821289497231385e-16 + }, + { + "step": 1, + "seconds": 2.4120942499721423, + "loss": 8.782007900424375, + "check": "ok", + "max_scaled_error": 2.2200777105558492e-16 + }, + { + "step": 2, + "seconds": 2.4132071669446304, + "loss": 7.639927209727933, + "check": "ok", + "max_scaled_error": 2.2200913907570052e-16 + }, + { + "step": 3, + "seconds": 2.5045561250299215, + "loss": 6.953781219773996, + "check": "ok", + "max_scaled_error": 2.220100087616973e-16 + }, + { + "step": 4, + "seconds": 2.4481768329860643, + "loss": 6.847608614444603, + "check": "ok", + "max_scaled_error": 2.2588187400865964e-16 + } + ], + "compile_seconds": 2.769668708089739, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 2.8493559170747176, + "warm_seconds": 2.4445085937331896, + "five_step_seconds": 12.487629249924794, + "five_steps_with_setup_compile_seconds": 15.336985166999511 +} diff --git a/benchmarks/apple/dense-native-gpt2-c1024-jax_jit.log b/benchmarks/apple/dense-native-gpt2-c1024-jax_jit.log new file mode 100644 index 00000000..56aa7e1e --- /dev/null +++ b/benchmarks/apple/dense-native-gpt2-c1024-jax_jit.log @@ -0,0 +1,5 @@ +jax_jit {'step': 0, 'seconds': 2.7095948749920353, 'loss': 10.783063169305695, 'check': 'ok', 'max_scaled_error': 1.5821289497231385e-16} +jax_jit {'step': 1, 'seconds': 2.4120942499721423, 'loss': 8.782007900424375, 'check': 'ok', 'max_scaled_error': 2.2200777105558492e-16} +jax_jit {'step': 2, 'seconds': 2.4132071669446304, 'loss': 7.639927209727933, 'check': 'ok', 'max_scaled_error': 2.2200913907570052e-16} +jax_jit {'step': 3, 'seconds': 2.5045561250299215, 'loss': 6.953781219773996, 'check': 'ok', 'max_scaled_error': 2.220100087616973e-16} +jax_jit {'step': 4, 'seconds': 2.4481768329860643, 'loss': 6.847608614444603, 'check': 'ok', 'max_scaled_error': 2.2588187400865964e-16} diff --git a/benchmarks/apple/dense-native-gpt2-c1024-torch_mps.json b/benchmarks/apple/dense-native-gpt2-c1024-torch_mps.json new file mode 100644 index 00000000..b5f2caa5 --- /dev/null +++ b/benchmarks/apple/dense-native-gpt2-c1024-torch_mps.json @@ -0,0 +1,134 @@ +{ + "backend": "torch_mps", + "context": 1024, + "parameters": 124337664, + "parameter_shapes": { + "wte": [ + 50257, + 768 + ], + "wpe": [ + 1024, + 768 + ], + "ln_1": [ + 12, + 768 + ], + "wq": [ + 12, + 768, + 12, + 64 + ], + "wk": [ + 12, + 768, + 12, + 64 + ], + "wv": [ + 12, + 768, + 12, + 64 + ], + "wo": [ + 12, + 12, + 64, + 768 + ], + "ln_2": [ + 12, + 768 + ], + "w_fc": [ + 12, + 768, + 3072 + ], + "w_proj": [ + 12, + 3072, + 768 + ], + "ln_f": [ + 768 + ] + }, + "parameter_counts": { + "wte": 38597376, + "wpe": 786432, + "ln_1": 9216, + "wq": 7077888, + "wk": 7077888, + "wv": 7077888, + "wo": 7077888, + "ln_2": 9216, + "w_fc": 28311552, + "w_proj": 28311552, + "ln_f": 768 + }, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.4004810000769794, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 1.521540289056585e-07 + }, + { + "step": 1, + "seconds": 0.12742812500800937, + "loss": 8.782007900424375, + "check": "ok", + "max_scaled_error": 1.5396272019241336e-07 + }, + { + "step": 2, + "seconds": 0.08607137494254857, + "loss": 7.639927209727933, + "check": "ok", + "max_scaled_error": 1.7109354215391967e-07 + }, + { + "step": 3, + "seconds": 0.08894479100126773, + "loss": 6.953781219773996, + "check": "ok", + "max_scaled_error": 2.2278449722128608e-07 + }, + { + "step": 4, + "seconds": 0.09160724992398173, + "loss": 6.847608614444603, + "check": "ok", + "max_scaled_error": 2.4998383965869387e-07 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "mps" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 2.4589105000486597, + "warm_seconds": 0.09851288521895185, + "five_step_seconds": 0.7945325409527868, + "five_steps_with_setup_compile_seconds": 3.2534430410014465 +} diff --git a/benchmarks/apple/dense-native-gpt2-c1024-torch_mps.log b/benchmarks/apple/dense-native-gpt2-c1024-torch_mps.log new file mode 100644 index 00000000..ec30d9fb --- /dev/null +++ b/benchmarks/apple/dense-native-gpt2-c1024-torch_mps.log @@ -0,0 +1,6 @@ +number of parameters: 123.55M +torch_mps {'step': 0, 'seconds': 0.4004810000769794, 'loss': 10.783063169305695, 'check': 'ok', 'max_scaled_error': 1.521540289056585e-07} +torch_mps {'step': 1, 'seconds': 0.12742812500800937, 'loss': 8.782007900424375, 'check': 'ok', 'max_scaled_error': 1.5396272019241336e-07} +torch_mps {'step': 2, 'seconds': 0.08607137494254857, 'loss': 7.639927209727933, 'check': 'ok', 'max_scaled_error': 1.7109354215391967e-07} +torch_mps {'step': 3, 'seconds': 0.08894479100126773, 'loss': 6.953781219773996, 'check': 'ok', 'max_scaled_error': 2.2278449722128608e-07} +torch_mps {'step': 4, 'seconds': 0.09160724992398173, 'loss': 6.847608614444603, 'check': 'ok', 'max_scaled_error': 2.4998383965869387e-07} diff --git a/benchmarks/apple/dense-summary.json b/benchmarks/apple/dense-summary.json new file mode 100644 index 00000000..4758015b --- /dev/null +++ b/benchmarks/apple/dense-summary.json @@ -0,0 +1,1075 @@ +{ + "method": "Summarize checked serial SQL benchmarks without including failed runs.\n\nWarm time is the mean paired loss/update API time at steps 1\u20134. Setup and\nfirst-use costs are reported separately. Per-parameter validation and\nconversion of returned Arrow rows to Datasets are outside the API timers.\n", + "runs": [ + { + "host": "duckdb", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "duckdb", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-duckdb.csv" + ], + "contended": false, + "log": "dense-final-tiny-duckdb.log", + "csv": "dense-final-tiny-duckdb.csv", + "peak_rss_gib": 0.6759490966796875, + "status": "passed", + "started_utc": "2026-11-01T21:59:53.598085+00:00", + "exit_code": 0, + "wall_seconds": 3.886874582967721, + "checked_rows": 10, + "source_manifest": "dense-final-tiny-runs.json", + "manifest": "dense-controls-runs.json", + "steps": 5, + "batch": 4, + "parameters": 4416, + "context": 16, + "optimizer": "sgd", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "paired_api_seconds": [ + 0.3287122080801055, + 0.275920334039256, + 0.27200887398794293, + 0.27723495895043015, + 0.270842875004746 + ], + "warm_loss_seconds": 0.02458656276576221, + "warm_training_seconds": 0.24941519772983156, + "warm_combined_seconds": 0.27400176049559377, + "five_api_seconds": 1.4247192500624806, + "loading_and_model_seconds": 0.037258830736391246, + "five_api_with_loading_model_seconds": 1.4619780807988718, + "final_statistics": {} + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-shared_cpu.csv" + ], + "contended": false, + "log": "dense-final-tiny-shared_cpu.log", + "csv": "dense-final-tiny-shared_cpu.csv", + "peak_rss_gib": 0.6473541259765625, + "status": "passed", + "started_utc": "2026-11-01T21:59:57.485449+00:00", + "exit_code": 0, + "wall_seconds": 3.6641790000721812, + "checked_rows": 10, + "source_manifest": "dense-final-tiny-runs.json", + "manifest": "dense-controls-runs.json", + "steps": 5, + "batch": 4, + "parameters": 4416, + "context": 16, + "optimizer": "sgd", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "paired_api_seconds": [ + 0.5109222489409149, + 0.20195304101798683, + 0.19013054098468274, + 0.18918845790904015, + 0.19263495900668204 + ], + "warm_loss_seconds": 0.006168791500385851, + "warm_training_seconds": 0.1873079582292121, + "warm_combined_seconds": 0.19347674972959794, + "five_api_seconds": 1.2848292478593066, + "loading_and_model_seconds": 0.012392036500386894, + "five_api_with_loading_model_seconds": 1.2972212843596935, + "final_statistics": { + "gpu_matmul": "0", + "gpu_attention": "0", + "physical_plans": "138", + "stage_buffer_allocations": "103", + "key_cache_hits": "598", + "key_copy_bytes_avoided": "4918144", + "planning_seconds": "0.5864103413186967", + "stage_storage_seconds": "0.03259224316570908", + "key_cache_bytes": "571392", + "stage_buffer_writes": "212", + "stage_replacements_keys_or_nulls": "40" + } + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-final-tiny-shared_gpu.csv" + ], + "contended": false, + "log": "dense-final-tiny-shared_gpu.log", + "csv": "dense-final-tiny-shared_gpu.csv", + "peak_rss_gib": 0.64837646484375, + "status": "passed", + "started_utc": "2026-11-01T22:00:01.150111+00:00", + "exit_code": 0, + "wall_seconds": 3.8351630830438808, + "checked_rows": 10, + "source_manifest": "dense-final-tiny-runs.json", + "manifest": "dense-controls-runs.json", + "steps": 5, + "batch": 4, + "parameters": 4416, + "context": 16, + "optimizer": "sgd", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "paired_api_seconds": [ + 0.5551908758934587, + 0.216537666041404, + 0.2213138749357313, + 0.21945320803206414, + 0.21863937401212752 + ], + "warm_loss_seconds": 0.01086622875300236, + "warm_training_seconds": 0.20811980200232938, + "warm_combined_seconds": 0.21898603075533174, + "five_api_seconds": 1.4311349989147857, + "loading_and_model_seconds": 0.012410877388902009, + "five_api_with_loading_model_seconds": 1.4435458763036877, + "final_statistics": { + "gpu_matmul": "160", + "gpu_attention": "5", + "physical_plans": "138", + "stage_buffer_allocations": "103", + "key_cache_hits": "598", + "key_copy_bytes_avoided": "4918144", + "planning_seconds": "0.5840442591579631", + "stage_storage_seconds": "0.032620440353639424", + "key_cache_bytes": "571392", + "stage_buffer_writes": "212", + "stage_replacements_keys_or_nulls": "40" + } + }, + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-tiny-dense_cpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-tiny-dense_cpu.log", + "csv": "dense-measured-tiny-dense_cpu.csv", + "peak_rss_gib": 0.5282745361328125, + "status": "passed", + "started_utc": "2026-11-01T23:25:16.750396+00:00", + "exit_code": 0, + "wall_seconds": 2.846897584036924, + "checked_rows": 10, + "manifest": "dense-measured-tiny-runs.json", + "steps": 5, + "batch": 4, + "parameters": 4416, + "context": 16, + "optimizer": "sgd", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "paired_api_seconds": [ + 0.1284130000276491, + 0.07012858393136412, + 0.0466488340171054, + 0.04544204112607986, + 0.045067917089909315 + ], + "warm_loss_seconds": 0.004620166524546221, + "warm_training_seconds": 0.04720167751656845, + "warm_combined_seconds": 0.05182184404111467, + "five_api_seconds": 0.3357003761921078, + "loading_and_model_seconds": 0.012672747252508998, + "five_api_with_loading_model_seconds": 0.3483731234446168, + "final_statistics": { + "dense_programs": "86", + "dense_executions": "430", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "6713053", + "dense_lowering_seconds": "0.17087464581709355", + "dense_call_seconds": "0.3331449571996927", + "key_cache_bytes": "0", + "dense_last_decline": "" + } + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-tiny-dense_gpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-tiny-dense_gpu.log", + "csv": "dense-measured-tiny-dense_gpu.csv", + "peak_rss_gib": 0.570648193359375, + "status": "passed", + "started_utc": "2026-11-01T23:25:19.601839+00:00", + "exit_code": 0, + "wall_seconds": 3.1278892919654027, + "checked_rows": 10, + "manifest": "dense-measured-tiny-runs.json", + "steps": 5, + "batch": 4, + "parameters": 4416, + "context": 16, + "optimizer": "sgd", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "paired_api_seconds": [ + 0.35571395803708583, + 0.08881362492684275, + 0.06521908403374255, + 0.06367558299098164, + 0.09288104204460979 + ], + "warm_loss_seconds": 0.012257437512744218, + "warm_training_seconds": 0.06538989598629996, + "warm_combined_seconds": 0.07764733349904418, + "five_api_seconds": 0.6663032920332626, + "loading_and_model_seconds": 0.012520543183200061, + "five_api_with_loading_model_seconds": 0.6788238352164626, + "final_statistics": { + "dense_programs": "86", + "dense_executions": "430", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "1903979", + "dense_lowering_seconds": "0.20953710924368352", + "dense_call_seconds": "0.6648724579717964", + "key_cache_bytes": "0", + "gpu_input_arrays": "1070", + "gpu_input_bytes": "360960", + "gpu_fused_programs": "51", + "gpu_fused_executions": "1885", + "gpu_contractions": "955", + "gpu_readbacks": "170", + "gpu_readback_bytes": "93730", + "dense_last_decline": "", + "gpu_union_reductions": "110", + "gpu_scatter_reductions": "115" + } + }, + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-demo-dense_cpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-demo-dense_cpu.log", + "csv": "dense-measured-demo-dense_cpu.csv", + "peak_rss_gib": 0.5987548828125, + "status": "passed", + "started_utc": "2026-11-01T23:31:58.067931+00:00", + "exit_code": 0, + "wall_seconds": 2.7708311669994146, + "checked_rows": 10, + "manifest": "dense-measured-demo-runs.json", + "steps": 5, + "batch": 1, + "parameters": 213504, + "context": 64, + "optimizer": "sgd", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "paired_api_seconds": [ + 0.1392797080334276, + 0.07745316694490612, + 0.054712917073629797, + 0.05456112511456013, + 0.05576458387076855 + ], + "warm_loss_seconds": 0.005417979264166206, + "warm_training_seconds": 0.05520496898679994, + "warm_combined_seconds": 0.06062294825096615, + "five_api_seconds": 0.3817715010372922, + "loading_and_model_seconds": 0.01395558100193739, + "five_api_with_loading_model_seconds": 0.39572708203922957, + "final_statistics": { + "dense_programs": "86", + "dense_executions": "430", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "59292293", + "dense_lowering_seconds": "0.182141374447383", + "dense_call_seconds": "0.37847587396390736", + "key_cache_bytes": "0", + "dense_last_decline": "" + } + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-demo-dense_gpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-demo-dense_gpu.log", + "csv": "dense-measured-demo-dense_gpu.csv", + "peak_rss_gib": 0.5789947509765625, + "status": "passed", + "started_utc": "2026-11-01T23:32:00.839351+00:00", + "exit_code": 0, + "wall_seconds": 2.8991056670201942, + "checked_rows": 10, + "manifest": "dense-measured-demo-runs.json", + "steps": 5, + "batch": 1, + "parameters": 213504, + "context": 64, + "optimizer": "sgd", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "paired_api_seconds": [ + 0.18786112393718213, + 0.09246174991130829, + 0.06812866602558643, + 0.06533408397808671, + 0.10619929293170571 + ], + "warm_loss_seconds": 0.012614864739589393, + "warm_training_seconds": 0.07041608347208239, + "warm_combined_seconds": 0.08303094821167178, + "five_api_seconds": 0.5199849167838693, + "loading_and_model_seconds": 0.013593747513368726, + "five_api_with_loading_model_seconds": 0.533578664297238, + "final_statistics": { + "dense_programs": "86", + "dense_executions": "430", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "16902347", + "dense_lowering_seconds": "0.2127329718787223", + "dense_call_seconds": "0.5184563321527094", + "key_cache_bytes": "0", + "gpu_input_arrays": "1070", + "gpu_input_bytes": "17088000", + "gpu_fused_programs": "51", + "gpu_fused_executions": "1885", + "gpu_contractions": "955", + "gpu_readbacks": "170", + "gpu_readback_bytes": "4312150", + "dense_last_decline": "", + "gpu_union_reductions": "110", + "gpu_scatter_reductions": "115" + } + }, + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "shakespeare", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-shakespeare-dense_cpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-shakespeare-dense_cpu.log", + "csv": "dense-measured-shakespeare-dense_cpu.csv", + "peak_rss_gib": 7.9724884033203125, + "status": "passed", + "started_utc": "2026-11-01T23:32:03.760863+00:00", + "exit_code": 0, + "wall_seconds": 12.016731290961616, + "checked_rows": 10, + "manifest": "dense-measured-shakespeare-runs.json", + "steps": 5, + "batch": 1, + "parameters": 10745088, + "context": 256, + "optimizer": "sgd", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "paired_api_seconds": [ + 1.8697643338236958, + 1.444952667108737, + 1.350559249985963, + 1.4230833330657333, + 1.4065936249680817 + ], + "warm_loss_seconds": 0.09651943756034598, + "warm_training_seconds": 1.3097777812217828, + "warm_combined_seconds": 1.4062972187821288, + "five_api_seconds": 7.494953208952211, + "loading_and_model_seconds": 0.05260245595127344, + "five_api_with_loading_model_seconds": 7.547555664903484, + "final_statistics": { + "dense_programs": "316", + "dense_executions": "1580", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "5340715780", + "dense_lowering_seconds": "2.171821787604131", + "dense_call_seconds": "7.200985666015185", + "key_cache_bytes": "0", + "dense_last_decline": "" + } + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "shakespeare", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-shakespeare-dense_gpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-shakespeare-dense_gpu.log", + "csv": "dense-measured-shakespeare-dense_gpu.csv", + "peak_rss_gib": 2.7833099365234375, + "status": "passed", + "started_utc": "2026-11-01T23:32:15.778182+00:00", + "exit_code": 0, + "wall_seconds": 7.818204250070266, + "checked_rows": 10, + "manifest": "dense-measured-shakespeare-runs.json", + "steps": 5, + "batch": 1, + "parameters": 10745088, + "context": 256, + "optimizer": "sgd", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "paired_api_seconds": [ + 0.992831209092401, + 0.6034280830062926, + 0.6066483760951087, + 0.5544331258861348, + 0.5917798749869689 + ], + "warm_loss_seconds": 0.05267367750639096, + "warm_training_seconds": 0.5363986874872353, + "warm_combined_seconds": 0.5890723649936263, + "five_api_seconds": 3.349120669066906, + "loading_and_model_seconds": 0.05208583571948111, + "five_api_with_loading_model_seconds": 3.401206504786387, + "final_statistics": { + "dense_programs": "316", + "dense_executions": "1580", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "1483857024", + "dense_lowering_seconds": "1.6390505810268223", + "dense_call_seconds": "3.3393739578314126", + "key_cache_bytes": "0", + "gpu_input_arrays": "4195", + "gpu_input_bytes": "859639720", + "gpu_fused_programs": "91", + "gpu_fused_executions": "11145", + "gpu_contractions": "7240", + "gpu_readbacks": "170", + "gpu_readback_bytes": "215028460", + "dense_last_decline": "", + "gpu_union_reductions": "480", + "gpu_scatter_reductions": "515" + } + }, + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c16-dense_cpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-gpt2-c16-dense_cpu.log", + "csv": "dense-measured-gpt2-c16-dense_cpu.csv", + "peak_rss_gib": 22.903076171875, + "status": "passed", + "started_utc": "2026-11-01T23:25:22.753974+00:00", + "exit_code": 0, + "wall_seconds": 36.28782012499869, + "checked_rows": 10, + "manifest": "dense-measured-gpt2-c16-runs.json", + "steps": 5, + "batch": 1, + "parameters": 123563520, + "context": 16, + "optimizer": "sgd", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "paired_api_seconds": [ + 5.192674458143301, + 4.114125500898808, + 4.014715916127898, + 4.022102251066826, + 3.997303667012602 + ], + "warm_loss_seconds": 0.17067291674902663, + "warm_training_seconds": 3.866388917027507, + "warm_combined_seconds": 4.037061833776534, + "five_api_seconds": 21.340921793249436, + "loading_and_model_seconds": 0.28613791859243065, + "five_api_with_loading_model_seconds": 21.627059711841866, + "final_statistics": { + "dense_programs": "707", + "dense_executions": "3535", + "dense_cache_evictions": "131275", + "dense_peak_cached_bytes": "8589934589", + "dense_lowering_seconds": "5.09097968484275", + "dense_call_seconds": "20.868966875830665", + "key_cache_bytes": "0", + "dense_last_decline": "" + } + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c16-dense_gpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-gpt2-c16-dense_gpu.log", + "csv": "dense-measured-gpt2-c16-dense_gpu.csv", + "peak_rss_gib": 17.911361694335938, + "status": "passed", + "started_utc": "2026-11-01T23:25:59.042533+00:00", + "exit_code": 0, + "wall_seconds": 28.680158666917123, + "checked_rows": 10, + "manifest": "dense-measured-gpt2-c16-runs.json", + "steps": 5, + "batch": 1, + "parameters": 123563520, + "context": 16, + "optimizer": "sgd", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "paired_api_seconds": [ + 3.749518583063036, + 2.4645180410007015, + 2.442303582909517, + 2.3960931659676135, + 2.4347036249237135 + ], + "warm_loss_seconds": 0.20305980171542615, + "warm_training_seconds": 2.23134480198496, + "warm_combined_seconds": 2.4344046037003864, + "five_api_seconds": 13.487136997864582, + "loading_and_model_seconds": 0.28663483064156026, + "five_api_with_loading_model_seconds": 13.773771828506142, + "final_statistics": { + "dense_programs": "707", + "dense_executions": "3535", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "5152709312", + "dense_lowering_seconds": "5.535126848728396", + "dense_call_seconds": "13.463256041985005", + "key_cache_bytes": "0", + "gpu_input_arrays": "7945", + "gpu_input_bytes": "9885144280", + "gpu_fused_programs": "140", + "gpu_fused_executions": "27495", + "gpu_contractions": "20020", + "gpu_readbacks": "170", + "gpu_readback_bytes": "2664259240", + "dense_last_decline": "", + "gpu_union_reductions": "840", + "gpu_scatter_reductions": "995" + } + }, + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c128-dense_cpu.csv", + "--context", + "128", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-measured-gpt2-c128-dense_cpu.log", + "csv": "dense-measured-gpt2-c128-dense_cpu.csv", + "peak_rss_gib": 35.78117370605469, + "status": "passed", + "started_utc": "2026-11-01T23:32:23.618737+00:00", + "exit_code": 0, + "wall_seconds": 46.86709191696718, + "checked_rows": 10, + "manifest": "dense-measured-gpt2-c128-runs.json", + "steps": 5, + "batch": 1, + "parameters": 123649536, + "context": 128, + "optimizer": "sgd", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "paired_api_seconds": [ + 7.194049333920702, + 5.822772457962856, + 5.659783125040121, + 5.681092082988471, + 5.983658874989487 + ], + "warm_loss_seconds": 0.3072171042440459, + "warm_training_seconds": 5.479609531001188, + "warm_combined_seconds": 5.786826635245234, + "five_api_seconds": 30.341355874901637, + "loading_and_model_seconds": 0.2963541721692309, + "five_api_with_loading_model_seconds": 30.63771004707087, + "final_statistics": { + "dense_programs": "707", + "dense_executions": "3535", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "22743236662", + "dense_lowering_seconds": "7.350553346215747", + "dense_call_seconds": "29.36832120991312", + "key_cache_bytes": "0", + "dense_last_decline": "" + } + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c128-dense_gpu.csv", + "--context", + "128", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-measured-gpt2-c128-dense_gpu.log", + "csv": "dense-measured-gpt2-c128-dense_gpu.csv", + "peak_rss_gib": 18.463027954101562, + "status": "passed", + "started_utc": "2026-11-01T23:33:10.486683+00:00", + "exit_code": 0, + "wall_seconds": 30.451320540974848, + "checked_rows": 10, + "manifest": "dense-measured-gpt2-c128-runs.json", + "steps": 5, + "batch": 1, + "parameters": 123649536, + "context": 128, + "optimizer": "sgd", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "paired_api_seconds": [ + 3.8815320420544595, + 2.5684636658988893, + 2.5407837090315297, + 2.5202683330280706, + 2.530861790990457 + ], + "warm_loss_seconds": 0.20296674998826347, + "warm_training_seconds": 2.337127624748973, + "warm_combined_seconds": 2.5400943747372366, + "five_api_seconds": 14.041909541003406, + "loading_and_model_seconds": 0.2877192912856117, + "five_api_with_loading_model_seconds": 14.329628832289018, + "final_statistics": { + "dense_programs": "707", + "dense_executions": "3535", + "dense_cache_evictions": "0", + "dense_peak_cached_bytes": "7219030384", + "dense_lowering_seconds": "6.000592845142819", + "dense_call_seconds": "14.017265541944653", + "key_cache_bytes": "0", + "gpu_input_arrays": "7945", + "gpu_input_bytes": "9892025560", + "gpu_fused_programs": "140", + "gpu_fused_executions": "27495", + "gpu_contractions": "20020", + "gpu_readbacks": "170", + "gpu_readback_bytes": "2665980120", + "dense_last_decline": "", + "gpu_union_reductions": "840", + "gpu_scatter_reductions": "995" + } + }, + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c1024-dense_cpu.csv", + "--context", + "1024", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-measured-gpt2-c1024-dense_cpu.log", + "csv": "dense-measured-gpt2-c1024-dense_cpu.csv", + "peak_rss_gib": 47.749176025390625, + "status": "timeout", + "started_utc": "2026-11-01T23:26:27.747100+00:00", + "exit_code": -15, + "wall_seconds": 90.51262095803395, + "checked_rows": 1, + "manifest": "dense-measured-gpt2-c1024-runs.json" + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-gpt2-c1024-dense_gpu.csv", + "--context", + "1024", + "--vocab-size", + "50257" + ], + "contended": false, + "dense_cache_gib": 32.0, + "log": "dense-measured-gpt2-c1024-dense_gpu.log", + "csv": "dense-measured-gpt2-c1024-dense_gpu.csv", + "peak_rss_gib": 28.716506958007812, + "status": "passed", + "started_utc": "2026-11-01T23:27:58.261043+00:00", + "exit_code": 0, + "wall_seconds": 56.74237979191821, + "checked_rows": 10, + "manifest": "dense-measured-gpt2-c1024-runs.json", + "steps": 5, + "batch": 1, + "parameters": 124337664, + "context": 1024, + "optimizer": "sgd", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "paired_api_seconds": [ + 7.495641375076957, + 5.54545983299613, + 5.598149416036904, + 5.5643881250871345, + 5.592578583979048 + ], + "warm_loss_seconds": 0.3480385415023193, + "warm_training_seconds": 5.227105448022485, + "warm_combined_seconds": 5.575143989524804, + "five_api_seconds": 29.796217333176173, + "loading_and_model_seconds": 0.28891495964489877, + "five_api_with_loading_model_seconds": 30.085132292821072, + "final_statistics": { + "dense_programs": "707", + "dense_executions": "3535", + "dense_cache_evictions": "122925", + "dense_peak_cached_bytes": "34359737980", + "dense_lowering_seconds": "12.795586302410811", + "dense_call_seconds": "29.79013208381366", + "key_cache_bytes": "0", + "gpu_input_arrays": "7945", + "gpu_input_bytes": "9947075800", + "gpu_fused_programs": "149", + "gpu_fused_executions": "48035", + "gpu_contractions": "34085", + "gpu_readbacks": "170", + "gpu_readback_bytes": "2679747160", + "dense_last_decline": "", + "gpu_union_reductions": "1270", + "gpu_scatter_reductions": "1135" + } + }, + { + "host": "dense_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "12", + "--optimizer", + "adamw", + "--learning-rate", + "0.01", + "--hosts", + "dense_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-adamw-dense_cpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-adamw-dense_cpu.log", + "csv": "dense-measured-adamw-dense_cpu.csv", + "peak_rss_gib": 0.4573822021484375, + "status": "failed", + "started_utc": "2026-11-01T23:33:40.960780+00:00", + "exit_code": 1, + "wall_seconds": 1.4579214999685064, + "checked_rows": 1, + "manifest": "dense-measured-adamw-runs.json" + }, + { + "host": "dense_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "12", + "--optimizer", + "adamw", + "--learning-rate", + "0.01", + "--hosts", + "dense_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/dense-measured-adamw-dense_gpu.csv" + ], + "contended": false, + "dense_cache_gib": 8, + "log": "dense-measured-adamw-dense_gpu.log", + "csv": "dense-measured-adamw-dense_gpu.csv", + "peak_rss_gib": 0.427886962890625, + "status": "failed", + "started_utc": "2026-11-01T23:33:42.419377+00:00", + "exit_code": 1, + "wall_seconds": 1.464974624919705, + "checked_rows": 1, + "manifest": "dense-measured-adamw-runs.json" + } + ] +} diff --git a/benchmarks/apple/fine-chunks-before.json b/benchmarks/apple/fine-chunks-before.json new file mode 100644 index 00000000..3664edcc --- /dev/null +++ b/benchmarks/apple/fine-chunks-before.json @@ -0,0 +1,20 @@ +{ + "rows": 4096, + "columns": 16, + "chunks": { + "i": 1, + "j": 1 + }, + "seconds": 0.031371750053949654, + "samples": [ + 0.0313179170479998, + 0.03183329198509455, + 0.031980833038687706, + 0.0329570829635486, + 0.031632416998036206, + 0.031359917018562555, + 0.031371750053949654, + 0.030526499962434173, + 0.030797166051343083 + ] +} diff --git a/benchmarks/apple/framework-gpt2-1024-forward-jax_eager.json b/benchmarks/apple/framework-gpt2-1024-forward-jax_eager.json new file mode 100644 index 00000000..eb0725b0 --- /dev/null +++ b/benchmarks/apple/framework-gpt2-1024-forward-jax_eager.json @@ -0,0 +1,62 @@ +{ + "backend": "jax_eager", + "context": 1024, + "parameters": 124337664, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": true, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 1.035269625019282, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 1, + "seconds": 0.4574369170004502, + "loss": 10.804215487348133, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.48019329202361405, + "loss": 10.841585972493611, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 3, + "seconds": 0.46009145909920335, + "loss": 10.872348942052431, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 4, + "seconds": 0.45806304190773517, + "loss": 10.87314253629185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 0.0806786670582369, + "warm_seconds": 0.4639461775077507, + "five_step_seconds": 2.891054335050285, + "five_steps_with_setup_compile_seconds": 2.9717330021085218 +} diff --git a/benchmarks/apple/framework-gpt2-1024-forward-jax_jit.json b/benchmarks/apple/framework-gpt2-1024-forward-jax_jit.json new file mode 100644 index 00000000..473c3d37 --- /dev/null +++ b/benchmarks/apple/framework-gpt2-1024-forward-jax_jit.json @@ -0,0 +1,62 @@ +{ + "backend": "jax_jit", + "context": 1024, + "parameters": 124337664, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": true, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.4770496250130236, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 1, + "seconds": 0.32251091592479497, + "loss": 10.804215487348133, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.3207364999689162, + "loss": 10.841585972493611, + "check": "ok", + "max_scaled_error": 1.6384658516817357e-16 + }, + { + "step": 3, + "seconds": 0.3164084159070626, + "loss": 10.872348942052431, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 4, + "seconds": 0.3200405000243336, + "loss": 10.87314253629185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.1769480420043692, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 0.2612458330113441, + "warm_seconds": 0.31992408295627683, + "five_step_seconds": 1.756745956838131, + "five_steps_with_setup_compile_seconds": 2.017991789849475 +} diff --git a/benchmarks/apple/framework-gpt2-1024-forward-torch_cpu.json b/benchmarks/apple/framework-gpt2-1024-forward-torch_cpu.json new file mode 100644 index 00000000..ddd783db --- /dev/null +++ b/benchmarks/apple/framework-gpt2-1024-forward-torch_cpu.json @@ -0,0 +1,66 @@ +{ + "backend": "torch_cpu", + "context": 1024, + "parameters": 124337664, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": true, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.38666091696359217, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 1.647358279840836e-16 + }, + { + "step": 1, + "seconds": 0.32166445802431554, + "loss": 10.804215487348133, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.35894229100085795, + "loss": 10.841585972493611, + "check": "ok", + "max_scaled_error": 1.6384658516817357e-16 + }, + { + "step": 3, + "seconds": 0.3645232500275597, + "loss": 10.872348942052431, + "check": "ok", + "max_scaled_error": 1.633829864059641e-16 + }, + { + "step": 4, + "seconds": 0.3359561249380931, + "loss": 10.87314253629185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "cpu" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 2.160730791045353, + "warm_seconds": 0.3452715309977066, + "five_step_seconds": 1.7677470409544185, + "five_steps_with_setup_compile_seconds": 3.9284778319997713 +} diff --git a/benchmarks/apple/framework-gpt2-1024-forward-torch_mps.json b/benchmarks/apple/framework-gpt2-1024-forward-torch_mps.json new file mode 100644 index 00000000..71d51b56 --- /dev/null +++ b/benchmarks/apple/framework-gpt2-1024-forward-torch_mps.json @@ -0,0 +1,66 @@ +{ + "backend": "torch_mps", + "context": 1024, + "parameters": 124337664, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": true, + "rtol": 3e-05, + "atol": 3e-05, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.2647716250503436, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 2.1740594775497947e-08 + }, + { + "step": 1, + "seconds": 0.02600145898759365, + "loss": 10.804215487348133, + "check": "ok", + "max_scaled_error": 5.195634407695925e-09 + }, + { + "step": 2, + "seconds": 0.03451741603203118, + "loss": 10.841585972493611, + "check": "ok", + "max_scaled_error": 1.29577409940046e-08 + }, + { + "step": 3, + "seconds": 0.030054125003516674, + "loss": 10.872348942052431, + "check": "ok", + "max_scaled_error": 1.4408297738376235e-08 + }, + { + "step": 4, + "seconds": 0.034106959006749094, + "loss": 10.87314253629185, + "check": "ok", + "max_scaled_error": 2.7026244535502437e-08 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "mps" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 2.078200125019066, + "warm_seconds": 0.03116998975747265, + "five_step_seconds": 0.38945158408023417, + "five_steps_with_setup_compile_seconds": 2.4676517090993 +} diff --git a/benchmarks/apple/framework-gpt2-16-jax_eager.json b/benchmarks/apple/framework-gpt2-16-jax_eager.json new file mode 100644 index 00000000..d3f228f9 --- /dev/null +++ b/benchmarks/apple/framework-gpt2-16-jax_eager.json @@ -0,0 +1,62 @@ +{ + "backend": "jax_eager", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 2.9766601249575615, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 1, + "seconds": 0.3764039580710232, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.31518250005319715, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 3, + "seconds": 0.3376008750637993, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 4, + "seconds": 0.30860195797868073, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 0.11831612489186227, + "warm_seconds": 0.3344473227916751, + "five_step_seconds": 4.314449416124262, + "five_steps_with_setup_compile_seconds": 4.432765541016124 +} diff --git a/benchmarks/apple/framework-gpt2-16-jax_jit.json b/benchmarks/apple/framework-gpt2-16-jax_jit.json new file mode 100644 index 00000000..2c80a677 --- /dev/null +++ b/benchmarks/apple/framework-gpt2-16-jax_jit.json @@ -0,0 +1,62 @@ +{ + "backend": "jax_jit", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.25373025005683303, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 3.4880647602577323e-16 + }, + { + "step": 1, + "seconds": 0.1414729580283165, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 3.5790513615769393e-16 + }, + { + "step": 2, + "seconds": 0.1349089159630239, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 3.5737750783812467e-16 + }, + { + "step": 3, + "seconds": 0.1352096670307219, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 3.571484214546241e-16 + }, + { + "step": 4, + "seconds": 0.13392283394932747, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 3.5720036936486486e-16 + } + ], + "compile_seconds": 3.0000361249549314, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 3.084211624925956, + "warm_seconds": 0.13637859374284744, + "five_step_seconds": 0.7992446250282228, + "five_steps_with_setup_compile_seconds": 3.883456249954179 +} diff --git a/benchmarks/apple/framework-gpt2-16-jax_separate.json b/benchmarks/apple/framework-gpt2-16-jax_separate.json new file mode 100644 index 00000000..1aebe79f --- /dev/null +++ b/benchmarks/apple/framework-gpt2-16-jax_separate.json @@ -0,0 +1,62 @@ +{ + "backend": "jax_separate", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 3.103274374967441, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 1, + "seconds": 0.4117213749559596, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.3880377080058679, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 3, + "seconds": 0.3956522080115974, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 4, + "seconds": 0.3818153750617057, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 0.08685445797163993, + "warm_seconds": 0.39430666650878266, + "five_step_seconds": 4.680501041002572, + "five_steps_with_setup_compile_seconds": 4.7673554989742115 +} diff --git a/benchmarks/apple/framework-gpt2-16-torch_cpu.json b/benchmarks/apple/framework-gpt2-16-torch_cpu.json new file mode 100644 index 00000000..513e1272 --- /dev/null +++ b/benchmarks/apple/framework-gpt2-16-torch_cpu.json @@ -0,0 +1,66 @@ +{ + "backend": "torch_cpu", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.1971114999614656, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 1.6506898020374621e-15 + }, + { + "step": 1, + "seconds": 0.09883254196029156, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 1.6277329480048546e-15 + }, + { + "step": 2, + "seconds": 0.09388387494254857, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 1.6253333233185944e-15 + }, + { + "step": 3, + "seconds": 0.0943373340414837, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 1.6047216744810507e-15 + }, + { + "step": 4, + "seconds": 0.09407875000033528, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 1.6049550842695296e-15 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "cpu" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 3.5439498749328777, + "warm_seconds": 0.09528312523616478, + "five_step_seconds": 0.5782440009061247, + "five_steps_with_setup_compile_seconds": 4.122193875839002 +} diff --git a/benchmarks/apple/framework-gpt2-16-torch_mps.json b/benchmarks/apple/framework-gpt2-16-torch_mps.json new file mode 100644 index 00000000..10ac7dba --- /dev/null +++ b/benchmarks/apple/framework-gpt2-16-torch_mps.json @@ -0,0 +1,66 @@ +{ + "backend": "torch_mps", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.876074375002645, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 2.466638482282098e-07 + }, + { + "step": 1, + "seconds": 0.025007624994032085, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 2.4429602051829147e-07 + }, + { + "step": 2, + "seconds": 0.0235599169973284, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 2.4332607268461294e-07 + }, + { + "step": 3, + "seconds": 0.02224191604182124, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 2.5501415047442646e-07 + }, + { + "step": 4, + "seconds": 0.02418895799200982, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 2.5291695020020007e-07 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "mps" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 2.113612666958943, + "warm_seconds": 0.023749604006297886, + "five_step_seconds": 0.9710727910278365, + "five_steps_with_setup_compile_seconds": 3.0846854579867795 +} diff --git a/benchmarks/apple/framework-runs.json b/benchmarks/apple/framework-runs.json new file mode 100644 index 00000000..74f4133b --- /dev/null +++ b/benchmarks/apple/framework-runs.json @@ -0,0 +1,182 @@ +[ + { + "case": "gpt2-16", + "backend": "jax_separate", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "-u", + "benchmarks/apple/compare_nanogpt_frameworks.py", + "--backend", + "jax_separate", + "--context", + "16", + "--steps", + "5", + "--output", + "benchmarks/apple/framework-gpt2-16-jax_separate.json" + ], + "peak_rss_gib": 7.794158935546875, + "status": "passed", + "exit_code": 0, + "wall_seconds": 11.596209582989104 + }, + { + "case": "gpt2-16", + "backend": "jax_jit", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "-u", + "benchmarks/apple/compare_nanogpt_frameworks.py", + "--backend", + "jax_jit", + "--context", + "16", + "--steps", + "5", + "--output", + "benchmarks/apple/framework-gpt2-16-jax_jit.json" + ], + "peak_rss_gib": 7.4196929931640625, + "status": "passed", + "exit_code": 0, + "wall_seconds": 12.743993417010643 + }, + { + "case": "gpt2-16", + "backend": "torch_cpu", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "-u", + "benchmarks/apple/compare_nanogpt_frameworks.py", + "--backend", + "torch_cpu", + "--context", + "16", + "--steps", + "5", + "--output", + "benchmarks/apple/framework-gpt2-16-torch_cpu.json", + "--nanogpt-source", + "/private/tmp/xql-upstream-nanogpt" + ], + "peak_rss_gib": 8.059890747070312, + "status": "passed", + "exit_code": 0, + "wall_seconds": 14.431298207957298 + }, + { + "case": "gpt2-16", + "backend": "torch_mps", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "-u", + "benchmarks/apple/compare_nanogpt_frameworks.py", + "--backend", + "torch_mps", + "--context", + "16", + "--steps", + "5", + "--output", + "benchmarks/apple/framework-gpt2-16-torch_mps.json", + "--nanogpt-source", + "/private/tmp/xql-upstream-nanogpt" + ], + "peak_rss_gib": 6.3382415771484375, + "status": "passed", + "exit_code": 0, + "wall_seconds": 13.586361584020779 + }, + { + "case": "gpt2-1024-forward", + "backend": "jax_eager", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "-u", + "benchmarks/apple/compare_nanogpt_frameworks.py", + "--backend", + "jax_eager", + "--context", + "1024", + "--steps", + "5", + "--output", + "benchmarks/apple/framework-gpt2-1024-forward-jax_eager.json", + "--forward-only" + ], + "peak_rss_gib": 4.4201507568359375, + "status": "passed", + "exit_code": 0, + "wall_seconds": 7.088339874986559 + }, + { + "case": "gpt2-1024-forward", + "backend": "jax_jit", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "-u", + "benchmarks/apple/compare_nanogpt_frameworks.py", + "--backend", + "jax_jit", + "--context", + "1024", + "--steps", + "5", + "--output", + "benchmarks/apple/framework-gpt2-1024-forward-jax_jit.json", + "--forward-only" + ], + "peak_rss_gib": 5.3981475830078125, + "status": "passed", + "exit_code": 0, + "wall_seconds": 6.573438124964014 + }, + { + "case": "gpt2-1024-forward", + "backend": "torch_cpu", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "-u", + "benchmarks/apple/compare_nanogpt_frameworks.py", + "--backend", + "torch_cpu", + "--context", + "1024", + "--steps", + "5", + "--output", + "benchmarks/apple/framework-gpt2-1024-forward-torch_cpu.json", + "--forward-only", + "--nanogpt-source", + "/private/tmp/xql-upstream-nanogpt" + ], + "peak_rss_gib": 4.853973388671875, + "status": "passed", + "exit_code": 0, + "wall_seconds": 8.588310541934334 + }, + { + "case": "gpt2-1024-forward", + "backend": "torch_mps", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "-u", + "benchmarks/apple/compare_nanogpt_frameworks.py", + "--backend", + "torch_mps", + "--context", + "1024", + "--steps", + "5", + "--output", + "benchmarks/apple/framework-gpt2-1024-forward-torch_mps.json", + "--forward-only", + "--nanogpt-source", + "/private/tmp/xql-upstream-nanogpt" + ], + "peak_rss_gib": 4.019256591796875, + "status": "passed", + "exit_code": 0, + "wall_seconds": 7.17828550003469 + } +] diff --git a/benchmarks/apple/initial-attention.csv b/benchmarks/apple/initial-attention.csv new file mode 100644 index 00000000..253d52bb --- /dev/null +++ b/benchmarks/apple/initial-attention.csv @@ -0,0 +1,19 @@ +demo,step,host,tokens,seconds,check,operators +attention,forward (window),datafusion,16,0.002329750102944672,ok (2e-16),24 +attention,gradient (window),datafusion,16,0.02357399999164045,ok (7e-16), +attention,forward (group by),datafusion,16,0.002127207932062447,ok (2e-16),36 +attention,gradient (group by),datafusion,16,0.017093332950025797,ok (9e-16), +attention,forward (window),einfold,16,0.002680541016161442,ok (2e-16),1 +attention,gradient (window),einfold,16,0.05557191697880626,ok (8e-16), +attention,forward (group by),einfold,16,0.003269708016887307,ok (2e-16),1 +attention,gradient (group by),einfold,16,0.05157554196193814,ok (9e-16), +attention,forward (window),datafusion,64,0.006661833031103015,ok (3e-16),24 +attention,gradient (window),datafusion,64,0.03245445899665356,ok (1e-15), +attention,forward (group by),datafusion,64,0.010099084000103176,ok (2e-16),36 +attention,gradient (group by),datafusion,64,0.027367457980290055,ok (1e-15), +attention,forward (window),einfold,64,0.002633958007209003,ok (3e-16),1 +attention,gradient (window),einfold,64,0.06518874992616475,ok (2e-15), +attention,forward (group by),einfold,64,0.003470374969765544,ok (3e-16),1 +attention,gradient (group by),einfold,64,0.06482166599016637,ok (1e-15), +attention,nadaraya-watson (256 sources),datafusion,64,0.004530707956291735,ok (1e-15), +attention,nadaraya-watson (256 sources),einfold,64,0.004136291914619505,ok (4e-16), diff --git a/benchmarks/apple/m5-ultra-crossover-results.json b/benchmarks/apple/m5-ultra-crossover-results.json new file mode 100644 index 00000000..1e8224aa --- /dev/null +++ b/benchmarks/apple/m5-ultra-crossover-results.json @@ -0,0 +1,615 @@ +{ + "environment": { + "platform": "macOS-27.0.1-arm64-arm-64bit", + "python": "3.12.14", + "hardware": "Apple M5 Ultra", + "gpu": { + "device_name": "Apple M5 Ultra", + "max_recommended_working_set_size": 239143780352, + "memory_size": 274877906944, + "architecture": "applegpu_g17d", + "max_buffer_length": 179357827072, + "resource_limit": 499000 + }, + "versions": { + "datafusion": "55.0.0", + "einfold": "0.2.0+xqlmlx.1", + "ddxdb": "0.4.6", + "mlx": "0.32.4", + "numpy": "2.5.4", + "pyarrow": "26.0.0" + }, + "min_gpu_ops": 268435456, + "matrix_features": 512, + "matrix_outputs": 256, + "reps": 7, + "warmups": 1, + "MLX_ENABLE_TF32": "0" + }, + "results": [ + { + "seconds": 0.03467987501062453, + "samples": [ + 0.033546874998137355, + 0.0353211669716984, + 0.0324287919793278, + 0.034489499987103045, + 0.040166874998249114, + 0.03479254199191928, + 0.03467987501062453 + ], + "max_abs_error": 2.1316282072803006e-13, + "task": "matmul", + "size": 256, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.007995583931915462, + "samples": [ + 0.008144416962750256, + 0.008228708989918232, + 0.008120957994833589, + 0.007995583931915462, + 0.007956666988320649, + 0.007608499960042536, + 0.007501750020310283 + ], + "max_abs_error": 5.684341886080802e-14, + "task": "matmul", + "size": 256, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.004163082921877503, + "samples": [ + 0.0041704170871526, + 0.004373166942968965, + 0.0041763330809772015, + 0.004068666021339595, + 0.0040285419672727585, + 0.004163082921877503, + 0.004004749935120344 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 256, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.0041097080102190375, + "samples": [ + 0.004159207921475172, + 0.004081082995980978, + 0.004188625025562942, + 0.004015292041003704, + 0.004403333994559944, + 0.0041097080102190375, + 0.0038869590498507023 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 256, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.10724899999331683, + "samples": [ + 0.10916600003838539, + 0.1043576670344919, + 0.10095504194032401, + 0.10750695795286447, + 0.10724899999331683, + 0.10815141699276865, + 0.10638300003483891 + ], + "max_abs_error": 1.9895196601282805e-13, + "task": "matmul", + "size": 1024, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.014330249978229403, + "samples": [ + 0.014330249978229403, + 0.015335666947066784, + 0.015169582911767066, + 0.01465595792979002, + 0.014212082955054939, + 0.01396304089576006, + 0.013570832903496921 + ], + "max_abs_error": 8.526512829121202e-14, + "task": "matmul", + "size": 1024, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.010507583967410028, + "samples": [ + 0.0104450419312343, + 0.01008116698358208, + 0.010507583967410028, + 0.011345290928147733, + 0.010775750037282705, + 0.011266167042776942, + 0.010281207971274853 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 1024, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.01004716707393527, + "samples": [ + 0.010107749956659973, + 0.010021792026236653, + 0.010461625060997903, + 0.009937916067428887, + 0.010314290993846953, + 0.01004716707393527, + 0.009976707980968058 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 1024, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.17732704093214124, + "samples": [ + 0.18663854198530316, + 0.1822330829454586, + 0.17732704093214124, + 0.18499387498013675, + 0.17668391694314778, + 0.17427162500098348, + 0.17005962505936623 + ], + "max_abs_error": 1.5543122344752192e-15, + "task": "attention", + "size": 1024, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.0059549580328166485, + "samples": [ + 0.006014915998093784, + 0.006213083979673684, + 0.005966707947663963, + 0.005903749959543347, + 0.005476790945976973, + 0.005503042018972337, + 0.0059549580328166485 + ], + "max_abs_error": 9.43689570931383e-16, + "task": "attention", + "size": 1024, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.00558520806953311, + "samples": [ + 0.005524040898308158, + 0.005882166093215346, + 0.0058734159683808684, + 0.00558520806953311, + 0.00539245898835361, + 0.005502167041413486, + 0.005755291902460158 + ], + "max_abs_error": 9.43689570931383e-16, + "task": "attention", + "size": 1024, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 0.005566375097259879, + "samples": [ + 0.005494083045050502, + 0.005757124978117645, + 0.00565466599073261, + 0.005847083986736834, + 0.005566375097259879, + 0.00536629103589803, + 0.005513833952136338 + ], + "max_abs_error": 9.43689570931383e-16, + "task": "attention", + "size": 1024, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 3.975829084054567, + "samples": [ + 3.98318570794072, + 4.027776000089943, + 3.878668999997899, + 3.9884126250399277, + 3.8961958750151098, + 3.975829084054567, + 3.9732770409900695 + ], + "max_abs_error": 1.3322676295501878e-15, + "task": "attention", + "size": 4096, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.01771312498021871, + "samples": [ + 0.01791345898527652, + 0.01771312498021871, + 0.017213375074788928, + 0.017485540942288935, + 0.017303750035353005, + 0.018094124970957637, + 0.018920292030088603 + ], + "max_abs_error": 7.355227538141662e-16, + "task": "attention", + "size": 4096, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.018658083048649132, + "samples": [ + 0.01868212502449751, + 0.020146082970313728, + 0.01820754201617092, + 0.018027208978310227, + 0.018658083048649132, + 0.017602000036276877, + 0.01931470795534551 + ], + "max_abs_error": 7.355227538141662e-16, + "task": "attention", + "size": 4096, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 0.01881379191763699, + "samples": [ + 0.02049695805180818, + 0.017754125059582293, + 0.01961679197847843, + 0.01771912502590567, + 0.01981704100035131, + 0.01881379191763699, + 0.01661945809610188 + ], + "max_abs_error": 7.233715347276792e-07, + "task": "attention", + "size": 4096, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "gpu_attention": 8 + } + }, + { + "seconds": 0.015278792008757591, + "samples": [ + 0.020454333978705108, + 0.01889808406122029, + 0.015303082996979356, + 0.015278792008757591, + 0.01498049998190254, + 0.014423292013816535, + 0.013541916967369616 + ], + "max_abs_error": 1.5916157281026244e-12, + "task": "gradient", + "size": 256, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.011964416946284473, + "samples": [ + 0.012386833084747195, + 0.01104100001975894, + 0.011964416946284473, + 0.012264874996617436, + 0.011707832920365036, + 0.012131207971833646, + 0.011208917014300823 + ], + "max_abs_error": 1.2505552149377763e-12, + "task": "gradient", + "size": 256, + "host": "datafusion", + "mode": "prepared" + }, + { + "seconds": 0.014963667024858296, + "samples": [ + 0.014843666926026344, + 0.014963667024858296, + 0.015572416945360601, + 0.01481816591694951, + 0.01418062497396022, + 0.020724667003378272, + 0.024070666986517608 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.012564625008963048, + "samples": [ + 0.016409624950028956, + 0.019632874988019466, + 0.017652499955147505, + 0.012564625008963048, + 0.011405666009522974, + 0.009934375062584877, + 0.00972333294339478 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold", + "mode": "prepared" + }, + { + "seconds": 0.015661209006793797, + "samples": [ + 0.015037416946142912, + 0.018905958044342697, + 0.016543834004551172, + 0.015661209006793797, + 0.015325959073379636, + 0.015762958908453584, + 0.015376000083051622 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_blas", + "mode": "sql", + "kernels": {} + }, + { + "seconds": 0.011225791997276247, + "samples": [ + 0.010004541021771729, + 0.011958457995206118, + 0.014068415970541537, + 0.015039875055663288, + 0.011225791997276247, + 0.010373125085607171, + 0.010503250057809055 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_blas", + "mode": "prepared", + "kernels": {} + }, + { + "seconds": 0.014979791012592614, + "samples": [ + 0.01518812496215105, + 0.014865915989503264, + 0.015802042093127966, + 0.014979791012592614, + 0.014761707978323102, + 0.014714707969687879, + 0.01855849998537451 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_mlx", + "mode": "sql", + "kernels": {} + }, + { + "seconds": 0.01324024994391948, + "samples": [ + 0.013317707926034927, + 0.011940250056795776, + 0.014842999982647598, + 0.019714541966095567, + 0.01324024994391948, + 0.012721375096589327, + 0.011978624970652163 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_mlx", + "mode": "prepared", + "kernels": {} + }, + { + "seconds": 0.014702541986480355, + "samples": [ + 0.014702541986480355, + 0.014687165967188776, + 0.016257000039331615, + 0.01736079202964902, + 0.015119750052690506, + 0.014639667002484202, + 0.013788875075988472 + ], + "max_abs_error": 1.546140993013978e-11, + "task": "gradient", + "size": 2048, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.01308862492442131, + "samples": [ + 0.013046541018411517, + 0.012211333028972149, + 0.013755292049609125, + 0.013860749895684421, + 0.012673666002228856, + 0.01308862492442131, + 0.015432916930876672 + ], + "max_abs_error": 1.546140993013978e-11, + "task": "gradient", + "size": 2048, + "host": "datafusion", + "mode": "prepared" + }, + { + "seconds": 0.023178542032837868, + "samples": [ + 0.021692708018235862, + 0.021561875008046627, + 0.022291457979008555, + 0.024925041012465954, + 0.025611499906517565, + 0.023557625012472272, + 0.023178542032837868 + ], + "max_abs_error": 2.7284841053187847e-12, + "task": "gradient", + "size": 2048, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.01802716590464115, + "samples": [ + 0.01802716590464115, + 0.01790883403737098, + 0.017985665937885642, + 0.017709832987748086, + 0.018435708014294505, + 0.018110375036485493, + 0.01805950002744794 + ], + "max_abs_error": 2.7284841053187847e-12, + "task": "gradient", + "size": 2048, + "host": "einfold", + "mode": "prepared" + }, + { + "seconds": 0.019701458048075438, + "samples": [ + 0.024547167005948722, + 0.02257395803462714, + 0.019701458048075438, + 0.019287458970211446, + 0.019367750035598874, + 0.019317583995871246, + 0.020750290947034955 + ], + "max_abs_error": 0.0, + "task": "gradient", + "size": 2048, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "cpu_matmul": 16 + } + }, + { + "seconds": 0.014643292059190571, + "samples": [ + 0.014517624978907406, + 0.014182458980940282, + 0.015024124993942678, + 0.014643292059190571, + 0.014446416986174881, + 0.015658959047868848, + 0.015430875006131828 + ], + "max_abs_error": 0.0, + "task": "gradient", + "size": 2048, + "host": "einfold_blas", + "mode": "prepared", + "kernels": { + "cpu_matmul": 32 + } + }, + { + "seconds": 0.02528562501538545, + "samples": [ + 0.021554250037297606, + 0.021690874942578375, + 0.030212791985832155, + 0.02878812502603978, + 0.025615999940782785, + 0.024190915981307626, + 0.02528562501538545 + ], + "max_abs_error": 0.0, + "task": "gradient", + "size": 2048, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "cpu_matmul": 16 + } + }, + { + "seconds": 0.017236208077520132, + "samples": [ + 0.017457042005844414, + 0.017236208077520132, + 0.016433584038168192, + 0.017437457921914756, + 0.017230959027074277, + 0.01700900006107986, + 0.01772133307531476 + ], + "max_abs_error": 0.0, + "task": "gradient", + "size": 2048, + "host": "einfold_mlx", + "mode": "prepared", + "kernels": { + "cpu_matmul": 32 + } + } + ] +} diff --git a/benchmarks/apple/m5-ultra-large-results.json b/benchmarks/apple/m5-ultra-large-results.json new file mode 100644 index 00000000..a11bcdf7 --- /dev/null +++ b/benchmarks/apple/m5-ultra-large-results.json @@ -0,0 +1,308 @@ +{ + "environment": { + "platform": "macOS-27.0.1-arm64-arm-64bit", + "python": "3.12.14", + "hardware": "Apple M5 Ultra", + "gpu": { + "device_name": "Apple M5 Ultra", + "max_recommended_working_set_size": 239143780352, + "memory_size": 274877906944, + "architecture": "applegpu_g17d", + "max_buffer_length": 179357827072, + "resource_limit": 499000 + }, + "versions": { + "datafusion": "55.0.0", + "einfold": "0.2.0+xqlmlx.1", + "ddxdb": "0.4.6", + "mlx": "0.32.4", + "numpy": "2.5.4", + "pyarrow": "26.0.0" + }, + "min_gpu_ops": 0, + "matrix_features": 1024, + "matrix_outputs": 1024, + "reps": 7, + "warmups": 1, + "MLX_ENABLE_TF32": "0" + }, + "results": [ + { + "seconds": 0.0641314999666065, + "samples": [ + 0.0641314999666065, + 0.06548804196063429, + 0.06414304196368903, + 0.06296700006350875, + 0.06397216697223485, + 0.06500491604674608, + 0.06377370900008827 + ], + "max_abs_error": 9.947598300641403e-14, + "task": "matmul", + "size": 1024, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.033744666958227754, + "samples": [ + 0.033744666958227754, + 0.03341629193164408, + 0.03395483398344368, + 0.03451108396984637, + 0.03360816603526473, + 0.03271808393765241, + 0.034363833954557776 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 1024, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.03517441696021706, + "samples": [ + 0.03748345794156194, + 0.03411979100201279, + 0.035450957948341966, + 0.03579208301380277, + 0.03318416594993323, + 0.03428708401042968, + 0.03517441696021706 + ], + "max_abs_error": 0.0002157680848924315, + "task": "matmul", + "size": 1024, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "gpu_matmul": 8 + } + }, + { + "seconds": 0.036204333999194205, + "samples": [ + 0.03822545800358057, + 0.04263283300679177, + 0.036204333999194205, + 0.035412250086665154, + 0.03545145899988711, + 0.03853862499818206, + 0.034787624957971275 + ], + "max_abs_error": 1.1102230246251565e-15, + "task": "attention", + "size": 8192, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.038157875067554414, + "samples": [ + 0.038157875067554414, + 0.03861979197245091, + 0.0366627499461174, + 0.03984958305954933, + 0.03523704200051725, + 0.036340667051263154, + 0.03840645798482001 + ], + "max_abs_error": 1.1102230246251565e-15, + "task": "attention", + "size": 8192, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 0.033864042023196816, + "samples": [ + 0.03383962495718151, + 0.03333904198370874, + 0.029663499910384417, + 0.035098375054076314, + 0.03506312507670373, + 0.033864042023196816, + 0.03408912499435246 + ], + "max_abs_error": 4.7658891388913105e-07, + "task": "attention", + "size": 8192, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "gpu_attention": 8 + } + }, + { + "seconds": 0.09504425001796335, + "samples": [ + 0.10317474999465048, + 0.09833612502552569, + 0.09502670797519386, + 0.09316825005225837, + 0.09478995797690004, + 0.09504425001796335, + 0.10191924998071045 + ], + "max_abs_error": 7.771561172376096e-16, + "task": "attention", + "size": 16384, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.0979778339387849, + "samples": [ + 0.10023287497460842, + 0.095370874973014, + 0.1038307499838993, + 0.09671766695100814, + 0.0979778339387849, + 0.09868874994572252, + 0.09621483297087252 + ], + "max_abs_error": 7.771561172376096e-16, + "task": "attention", + "size": 16384, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 0.07129174994770437, + "samples": [ + 0.07760083302855492, + 0.06743962492328137, + 0.07418670889455825, + 0.07069333398249, + 0.07129174994770437, + 0.06834658409934491, + 0.07403929205611348 + ], + "max_abs_error": 6.084088360336359e-07, + "task": "attention", + "size": 16384, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "gpu_attention": 8 + } + }, + { + "seconds": 0.015220083994790912, + "samples": [ + 0.015611832961440086, + 0.014741832972504199, + 0.015435041976161301, + 0.015220083994790912, + 0.014909958001226187, + 0.015370499924756587, + 0.014580749906599522 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.009485499933362007, + "samples": [ + 0.009304750012233853, + 0.00958825007546693, + 0.00925179198384285, + 0.00994008406996727, + 0.009583958075381815, + 0.009308708948083222, + 0.009485499933362007 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold", + "mode": "prepared" + }, + { + "seconds": 0.014062332920730114, + "samples": [ + 0.014269749983213842, + 0.014037708984687924, + 0.014856624999083579, + 0.014387291972525418, + 0.013784916023723781, + 0.014062332920730114, + 0.013960666954517365 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_blas", + "mode": "sql", + "kernels": {} + }, + { + "seconds": 0.00944916601292789, + "samples": [ + 0.009144334006123245, + 0.009522875072434545, + 0.009269790956750512, + 0.009878166951239109, + 0.00962150003761053, + 0.009053791989572346, + 0.00944916601292789 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_blas", + "mode": "prepared", + "kernels": {} + }, + { + "seconds": 0.014310125028714538, + "samples": [ + 0.01411020802333951, + 0.013880624901503325, + 0.014773207949474454, + 0.015137208043597639, + 0.015393750043585896, + 0.014310125028714538, + 0.014199291937984526 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_mlx", + "mode": "sql", + "kernels": {} + }, + { + "seconds": 0.009435249958187342, + "samples": [ + 0.00913699995726347, + 0.009435249958187342, + 0.00932283396832645, + 0.009557249955832958, + 0.00979891698807478, + 0.009187916992232203, + 0.009511125041171908 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_mlx", + "mode": "prepared", + "kernels": {} + } + ] +} diff --git a/benchmarks/apple/m5-ultra-results.json b/benchmarks/apple/m5-ultra-results.json new file mode 100644 index 00000000..e040e888 --- /dev/null +++ b/benchmarks/apple/m5-ultra-results.json @@ -0,0 +1,615 @@ +{ + "environment": { + "platform": "macOS-27.0.1-arm64-arm-64bit", + "python": "3.12.14", + "hardware": "Apple M5 Ultra", + "gpu": { + "device_name": "Apple M5 Ultra", + "max_recommended_working_set_size": 239143780352, + "memory_size": 274877906944, + "architecture": "applegpu_g17d", + "max_buffer_length": 179357827072, + "resource_limit": 499000 + }, + "versions": { + "datafusion": "55.0.0", + "einfold": "0.2.0+xqlmlx.1", + "ddxdb": "0.4.6", + "mlx": "0.32.4", + "numpy": "2.5.4", + "pyarrow": "26.0.0" + }, + "min_gpu_ops": 8589934592, + "matrix_features": 512, + "matrix_outputs": 256, + "reps": 7, + "warmups": 1, + "MLX_ENABLE_TF32": "0" + }, + "results": [ + { + "seconds": 0.03650950000155717, + "samples": [ + 0.03348529106006026, + 0.036792583065107465, + 0.04583529208321124, + 0.0355317909270525, + 0.03605679189786315, + 0.03731225000228733, + 0.03650950000155717 + ], + "max_abs_error": 2.2737367544323206e-13, + "task": "matmul", + "size": 256, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.008697833982296288, + "samples": [ + 0.008978625060990453, + 0.008697833982296288, + 0.008847834076732397, + 0.008254749933257699, + 0.008819458074867725, + 0.008130542002618313, + 0.00807441701181233 + ], + "max_abs_error": 5.684341886080802e-14, + "task": "matmul", + "size": 256, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.004421874997206032, + "samples": [ + 0.004272458027116954, + 0.0046836669789627194, + 0.004357667057774961, + 0.004421874997206032, + 0.004841624991968274, + 0.004426667001098394, + 0.004104375024326146 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 256, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.00477541598957032, + "samples": [ + 0.004699875018559396, + 0.004995000082999468, + 0.00477541598957032, + 0.005321125034242868, + 0.004750791005790234, + 0.004687209031544626, + 0.004977000062353909 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 256, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.10892904200591147, + "samples": [ + 0.11270404199603945, + 0.10927825001999736, + 0.10898987506516278, + 0.10700170800555497, + 0.10892904200591147, + 0.10439729201607406, + 0.10878999996930361 + ], + "max_abs_error": 1.9895196601282805e-13, + "task": "matmul", + "size": 1024, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.014433832955546677, + "samples": [ + 0.01396329200360924, + 0.015116750029847026, + 0.014701291918754578, + 0.014386125025339425, + 0.014433832955546677, + 0.015078250085934997, + 0.01432854204904288 + ], + "max_abs_error": 8.526512829121202e-14, + "task": "matmul", + "size": 1024, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.010543291922658682, + "samples": [ + 0.011198207968845963, + 0.010911333956755698, + 0.010634000063873827, + 0.010543291922658682, + 0.01049029198475182, + 0.00997225008904934, + 0.009718374931253493 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 1024, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.010593250044621527, + "samples": [ + 0.009694750071503222, + 0.009566167020238936, + 0.010295124957337976, + 0.010593250044621527, + 0.01088545797392726, + 0.010960582992993295, + 0.011202791007235646 + ], + "max_abs_error": 0.0, + "task": "matmul", + "size": 1024, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "cpu_matmul": 8 + } + }, + { + "seconds": 0.18128250003792346, + "samples": [ + 0.18128250003792346, + 0.1720742080360651, + 0.17686283309012651, + 0.17444599990267307, + 0.18400837504304945, + 0.18428154103457928, + 0.18216116691473871 + ], + "max_abs_error": 1.5543122344752192e-15, + "task": "attention", + "size": 1024, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.006125832907855511, + "samples": [ + 0.006579625071026385, + 0.006125832907855511, + 0.0060667089419439435, + 0.006518083973787725, + 0.005998375010676682, + 0.005825707921758294, + 0.00681787496432662 + ], + "max_abs_error": 9.43689570931383e-16, + "task": "attention", + "size": 1024, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.006251499988138676, + "samples": [ + 0.007852416019886732, + 0.006660750019364059, + 0.006132833077572286, + 0.005827917018905282, + 0.006506415898911655, + 0.005928749917075038, + 0.006251499988138676 + ], + "max_abs_error": 9.43689570931383e-16, + "task": "attention", + "size": 1024, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 0.00605787499807775, + "samples": [ + 0.00605787499807775, + 0.005858666030690074, + 0.0059556670021265745, + 0.006402292056009173, + 0.010383334010839462, + 0.006959790945984423, + 0.005962750059552491 + ], + "max_abs_error": 9.43689570931383e-16, + "task": "attention", + "size": 1024, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 3.9853448329959065, + "samples": [ + 4.000599583960138, + 3.9540730000007898, + 3.989653833094053, + 3.9341937500284985, + 3.8349077919265255, + 3.988171082921326, + 3.9853448329959065 + ], + "max_abs_error": 1.2212453270876722e-15, + "task": "attention", + "size": 4096, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.01738054200541228, + "samples": [ + 0.017086333013139665, + 0.01761212502606213, + 0.01642574998550117, + 0.01988620893098414, + 0.01851162500679493, + 0.01738054200541228, + 0.016933250008150935 + ], + "max_abs_error": 7.355227538141662e-16, + "task": "attention", + "size": 4096, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.016877375077456236, + "samples": [ + 0.016877375077456236, + 0.016703082947060466, + 0.019046709057874978, + 0.01838266698177904, + 0.018183042062446475, + 0.016388374962843955, + 0.016587499994784594 + ], + "max_abs_error": 7.355227538141662e-16, + "task": "attention", + "size": 4096, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 0.016592834028415382, + "samples": [ + 0.016592834028415382, + 0.01608016702812165, + 0.01668666605837643, + 0.016350624966435134, + 0.016515207942575216, + 0.017156208981759846, + 0.017644708044826984 + ], + "max_abs_error": 7.355227538141662e-16, + "task": "attention", + "size": 4096, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "declined": 8 + } + }, + { + "seconds": 0.013483166927471757, + "samples": [ + 0.01355858298484236, + 0.013061458012089133, + 0.01350962498690933, + 0.013925750041380525, + 0.01288475003093481, + 0.01293837500270456, + 0.013483166927471757 + ], + "max_abs_error": 1.1368683772161603e-12, + "task": "gradient", + "size": 256, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.011435291962698102, + "samples": [ + 0.01139879203401506, + 0.011435291962698102, + 0.011211875011213124, + 0.011817499995231628, + 0.01146824995521456, + 0.011608625063672662, + 0.011374249937944114 + ], + "max_abs_error": 1.3642420526593924e-12, + "task": "gradient", + "size": 256, + "host": "datafusion", + "mode": "prepared" + }, + { + "seconds": 0.014973292010836303, + "samples": [ + 0.016161291976459324, + 0.01573212502989918, + 0.015628542052581906, + 0.014455042080953717, + 0.014967041905038059, + 0.014463625033386052, + 0.014973292010836303 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.00943570805247873, + "samples": [ + 0.009883707971312106, + 0.009506791946478188, + 0.009324791957624257, + 0.00943570805247873, + 0.009935207897797227, + 0.00913345895241946, + 0.00936245801858604 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold", + "mode": "prepared" + }, + { + "seconds": 0.014592000050470233, + "samples": [ + 0.014608165947720408, + 0.015230666031129658, + 0.014773291070014238, + 0.014592000050470233, + 0.014099624939262867, + 0.013886124943383038, + 0.014357707928866148 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_blas", + "mode": "sql", + "kernels": {} + }, + { + "seconds": 0.009846749948337674, + "samples": [ + 0.009846749948337674, + 0.0093158750096336, + 0.009217499988153577, + 0.009454415994696319, + 0.01688945898786187, + 0.011865749955177307, + 0.009849041933193803 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_blas", + "mode": "prepared", + "kernels": {} + }, + { + "seconds": 0.014478959026746452, + "samples": [ + 0.014816417009569705, + 0.015438708011060953, + 0.01553770899772644, + 0.01425537490285933, + 0.014160916907712817, + 0.014478959026746452, + 0.014105041977018118 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_mlx", + "mode": "sql", + "kernels": {} + }, + { + "seconds": 0.009423832991160452, + "samples": [ + 0.00941945903468877, + 0.009150334051810205, + 0.00967295910231769, + 0.009423832991160452, + 0.009763958980329335, + 0.009287916938774288, + 0.009468415984883904 + ], + "max_abs_error": 3.410605131648481e-13, + "task": "gradient", + "size": 256, + "host": "einfold_mlx", + "mode": "prepared", + "kernels": {} + }, + { + "seconds": 0.013122082920745015, + "samples": [ + 0.012607499957084656, + 0.013019790989346802, + 0.015714749926701188, + 0.012630583951249719, + 0.013707792037166655, + 0.013512082980014384, + 0.013122082920745015 + ], + "max_abs_error": 1.546140993013978e-11, + "task": "gradient", + "size": 2048, + "host": "datafusion", + "mode": "sql" + }, + { + "seconds": 0.011634624912403524, + "samples": [ + 0.011254124925471842, + 0.011634624912403524, + 0.01137483399361372, + 0.012385208043269813, + 0.012090083095245063, + 0.012346082949079573, + 0.010972667019814253 + ], + "max_abs_error": 1.546140993013978e-11, + "task": "gradient", + "size": 2048, + "host": "datafusion", + "mode": "prepared" + }, + { + "seconds": 0.020306667080149055, + "samples": [ + 0.020838707918301225, + 0.021198792033828795, + 0.020268957945518196, + 0.01982695807237178, + 0.020452709053643048, + 0.020306667080149055, + 0.019674874958582222 + ], + "max_abs_error": 2.7284841053187847e-12, + "task": "gradient", + "size": 2048, + "host": "einfold", + "mode": "sql" + }, + { + "seconds": 0.015128874918445945, + "samples": [ + 0.015505333081819117, + 0.015128874918445945, + 0.015168542042374611, + 0.014946250012144446, + 0.014669416937977076, + 0.014902999973855913, + 0.015216333093121648 + ], + "max_abs_error": 2.7284841053187847e-12, + "task": "gradient", + "size": 2048, + "host": "einfold", + "mode": "prepared" + }, + { + "seconds": 0.019468375016003847, + "samples": [ + 0.01951533299870789, + 0.019849624950438738, + 0.019468375016003847, + 0.01920754194725305, + 0.019126832950860262, + 0.018846082966774702, + 0.01947341696359217 + ], + "max_abs_error": 0.0, + "task": "gradient", + "size": 2048, + "host": "einfold_blas", + "mode": "sql", + "kernels": { + "cpu_matmul": 16 + } + }, + { + "seconds": 0.0144174579763785, + "samples": [ + 0.014441750012338161, + 0.0144174579763785, + 0.015712332911789417, + 0.014626209042035043, + 0.0140356250340119, + 0.01381333393510431, + 0.013865999993868172 + ], + "max_abs_error": 0.0, + "task": "gradient", + "size": 2048, + "host": "einfold_blas", + "mode": "prepared", + "kernels": { + "cpu_matmul": 32 + } + }, + { + "seconds": 0.01954283402301371, + "samples": [ + 0.021776458946987987, + 0.022029958083294332, + 0.020606374950148165, + 0.01954283402301371, + 0.019152999971993268, + 0.0195115830283612, + 0.019392916932702065 + ], + "max_abs_error": 0.0, + "task": "gradient", + "size": 2048, + "host": "einfold_mlx", + "mode": "sql", + "kernels": { + "cpu_matmul": 16 + } + }, + { + "seconds": 0.01404854201246053, + "samples": [ + 0.014109708019532263, + 0.013971665990538895, + 0.01468774990644306, + 0.01411841600202024, + 0.013925750041380525, + 0.013960332958959043, + 0.01404854201246053 + ], + "max_abs_error": 0.0, + "task": "gradient", + "size": 2048, + "host": "einfold_mlx", + "mode": "prepared", + "kernels": { + "cpu_matmul": 32 + } + } + ] +} diff --git a/benchmarks/apple/nanogpt-5-steps-all-variants.csv b/benchmarks/apple/nanogpt-5-steps-all-variants.csv new file mode 100644 index 00000000..7cde05c5 --- /dev/null +++ b/benchmarks/apple/nanogpt-5-steps-all-variants.csv @@ -0,0 +1,81 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,load_seconds,model_seconds,table_registers,query_preparations,gradient_preparations,inplace_updates,gpu_matmul,cpu_matmul,gpu_attention,declined +nanogpt,"loss, step 0",datafusion,4,1.0686229170532897,ok (4e-16),4.179251,9882,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 0",datafusion,4,0.3961218750337139,ok (5e-16),,,,,0.003927670302800834,0.024913375033065677,,,,,,,, +nanogpt,"loss, step 1",datafusion,4,1.0534565839916468,ok (2e-16),4.104661,9882,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 1",datafusion,4,0.1625057499622926,ok (5e-16),,,,,0.005973211140371859,0.047230042051523924,,,,,,,, +nanogpt,"loss, step 2",datafusion,4,1.0999285000143573,ok (0e+00),3.883857,9882,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 2",datafusion,4,0.16057641606312245,ok (6e-16),,,,,0.007929836283437908,0.0724585420684889,,,,,,,, +nanogpt,"loss, step 3",datafusion,4,1.0528932920424268,ok (1e-16),3.641763,9882,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 3",datafusion,4,0.16286754107568413,ok (6e-16),,,,,0.009877295116893947,0.09487400006037205,,,,,,,, +nanogpt,"loss, step 4",datafusion,4,1.055990333086811,ok (5e-16),3.842234,9882,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 4",datafusion,4,0.16142054204829037,ok (8e-16),,,,,0.01193312811665237,0.11751883407123387,,,,,,,, +nanogpt,"loss, step 0",einfold,4,1.1594368750229478,ok (2e-16),4.179251,1,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 0",einfold,4,0.5700416659237817,ok (4e-16),,,,,0.00349158130120486,0.022942875046283007,,,,,,,, +nanogpt,"loss, step 1",einfold,4,1.1623720839852467,ok (0e+00),4.104661,1,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 1",einfold,4,0.33289270801469684,ok (4e-16),,,,,0.005380875198170543,0.04651916597504169,,,,,,,, +nanogpt,"loss, step 2",einfold,4,1.1755689999554306,ok (1e-16),3.883857,1,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 2",einfold,4,0.3288239169633016,ok (5e-16),,,,,0.007296708063222468,0.07072183303534985,,,,,,,, +nanogpt,"loss, step 3",einfold,4,1.1617839999962598,ok (1e-16),3.641763,1,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 3",einfold,4,0.33115795894991606,ok (5e-16),,,,,0.00917091709561646,0.09490029094740748,,,,,,,, +nanogpt,"loss, step 4",einfold,4,1.1724911669734865,ok (3e-16),3.842234,1,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 4",einfold,4,0.33088800008408725,ok (8e-16),,,,,0.01106328924652189,0.11939912394154817,,,,,,,, +nanogpt,"loss, step 0",duckdb,4,0.02867420797701925,ok (2e-16),4.179251,126,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 0",duckdb,4,0.29887858394067734,ok (5e-16),,,,,0.010591790894977748,0.0010317090200260282,,,,,,,, +nanogpt,"loss, step 1",duckdb,4,0.02431754197459668,ok (0e+00),4.104661,126,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 1",duckdb,4,0.2568598750513047,ok (5e-16),,,,,0.016889666207134724,0.001983791938982904,,,,,,,, +nanogpt,"loss, step 2",duckdb,4,0.024851958034560084,ok (0e+00),3.883857,126,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 2",duckdb,4,0.2529260829323903,ok (6e-16),,,,,0.02282862539868802,0.0029568750178441405,,,,,,,, +nanogpt,"loss, step 3",duckdb,4,0.02433566702529788,ok (0e+00),3.641763,126,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 3",duckdb,4,0.2544355830177665,ok (6e-16),,,,,0.028621625271625817,0.003920124960131943,,,,,,,, +nanogpt,"loss, step 4",duckdb,4,0.024451749981381,ok (0e+00),3.842234,126,1e-09,1e-09,,,,,,,,,, +nanogpt,"train (sgd), step 4",duckdb,4,0.2519476660527289,ok (1e-15),,,,,0.034349376102909446,0.0049009999493137,,,,,,,, +nanogpt,"loss, step 0",datafusion_prepared,4,2.1832260839873925,ok (4e-16),4.179251,9882,1e-09,1e-09,,,13,1,,,,,, +nanogpt,"train (sgd), step 0",datafusion_prepared,4,0.40616029198281467,ok (5e-16),,,,,0.003930752049200237,0.022789500071667135,13,1,1,11,,,, +nanogpt,"loss, step 1",datafusion_prepared,4,0.9771634170319885,ok (2e-16),4.104661,9882,1e-09,1e-09,,,13,1,1,12,,,, +nanogpt,"train (sgd), step 1",datafusion_prepared,4,0.12461866706144065,ok (5e-16),,,,,0.005682999966666102,0.022810125024989247,13,1,1,23,,,, +nanogpt,"loss, step 2",datafusion_prepared,4,0.9772000409429893,ok (1e-16),3.883857,9882,1e-09,1e-09,,,13,1,1,24,,,, +nanogpt,"train (sgd), step 2",datafusion_prepared,4,0.13093287497758865,ok (9e-16),,,,,0.0074820860754698515,0.022829458001069725,13,1,1,35,,,, +nanogpt,"loss, step 3",datafusion_prepared,4,0.9754911670461297,ok (1e-16),3.641763,9882,1e-09,1e-09,,,13,1,1,36,,,, +nanogpt,"train (sgd), step 3",datafusion_prepared,4,0.12260366708505899,ok (7e-16),,,,,0.009205922018736601,0.022852165973745286,13,1,1,47,,,, +nanogpt,"loss, step 4",datafusion_prepared,4,0.9794474589871243,ok (5e-16),3.842234,9882,1e-09,1e-09,,,13,1,1,48,,,, +nanogpt,"train (sgd), step 4",datafusion_prepared,4,0.12534429098013788,ok (1e-15),,,,,0.010954088880680501,0.0228714159457013,13,1,1,59,,,, +nanogpt,"loss, step 0",cpu_optimized,4,2.409379499964416,ok (2e-16),4.179251,1,1e-09,1e-09,,,13,1,,,,,, +nanogpt,"train (sgd), step 0",cpu_optimized,4,0.5740339169278741,ok (3e-16),,,,,0.003814624040387571,0.02370245906058699,13,1,1,11,,,, +nanogpt,"loss, step 1",cpu_optimized,4,1.0812175000319257,ok (2e-16),4.104661,1,1e-09,1e-09,,,13,1,1,12,,,, +nanogpt,"train (sgd), step 1",cpu_optimized,4,0.25947070901747793,ok (3e-16),,,,,0.005555543000809848,0.02372129214927554,13,1,1,23,,,, +nanogpt,"loss, step 2",cpu_optimized,4,1.0774768339470029,ok (1e-16),3.883857,1,1e-09,1e-09,,,13,1,1,24,,,, +nanogpt,"train (sgd), step 2",cpu_optimized,4,0.2592378749977797,ok (4e-16),,,,,0.007279127836227417,0.02374212513677776,13,1,1,35,,,, +nanogpt,"loss, step 3",cpu_optimized,4,1.0851524999598041,ok (1e-16),3.641763,1,1e-09,1e-09,,,13,1,1,36,,,, +nanogpt,"train (sgd), step 3",cpu_optimized,4,0.26522787497378886,ok (5e-16),,,,,0.00898762617725879,0.023761417134664953,13,1,1,47,,,, +nanogpt,"loss, step 4",cpu_optimized,4,1.08284512499813,ok (3e-16),3.842234,1,1e-09,1e-09,,,13,1,1,48,,,, +nanogpt,"train (sgd), step 4",cpu_optimized,4,0.26300137501675636,ok (9e-16),,,,,0.010736083262600005,0.02378150017466396,13,1,1,59,,,, +nanogpt,"loss, step 0",cpu_forced,4,2.362867624964565,ok (2e-16),4.179251,1,1e-09,1e-09,,,13,1,,,0,378,0,0 +nanogpt,"train (sgd), step 0",cpu_forced,4,0.5587617499986663,ok (3e-16),,,,,0.004180459305644035,0.02603362500667572,13,1,1,11,0,403,0,0 +nanogpt,"loss, step 1",cpu_forced,4,1.0796167920343578,ok (0e+00),4.104661,1,1e-09,1e-09,,,13,1,1,12,0,781,0,0 +nanogpt,"train (sgd), step 1",cpu_forced,4,0.261068458086811,ok (3e-16),,,,,0.0059157488867640495,0.026055250084027648,13,1,1,23,0,806,0,0 +nanogpt,"loss, step 2",cpu_forced,4,1.0719282500213012,ok (1e-16),3.883857,1,1e-09,1e-09,,,13,1,1,24,0,1184,0,0 +nanogpt,"train (sgd), step 2",cpu_forced,4,0.2637048749020323,ok (6e-16),,,,,0.007647874765098095,0.026077875052578747,13,1,1,35,0,1209,0,0 +nanogpt,"loss, step 3",cpu_forced,4,1.078952332958579,ok (1e-16),3.641763,1,1e-09,1e-09,,,13,1,1,36,0,1587,0,0 +nanogpt,"train (sgd), step 3",cpu_forced,4,0.25885908398777246,ok (5e-16),,,,,0.009369082748889923,0.02609775005839765,13,1,1,47,0,1612,0,0 +nanogpt,"loss, step 4",cpu_forced,4,1.0935512910364196,ok (3e-16),3.842234,1,1e-09,1e-09,,,13,1,1,48,0,1990,0,0 +nanogpt,"train (sgd), step 4",cpu_forced,4,0.25716737494803965,ok (8e-16),,,,,0.01108320860657841,0.026117125060409307,13,1,1,59,0,2015,0,0 +nanogpt,"loss, step 0",gpu_optimized,4,2.407174542080611,ok (2e-16),4.179251,1,3e-05,3e-05,,,13,1,,,,,, +nanogpt,"train (sgd), step 0",gpu_optimized,4,0.5606161670293659,ok (3e-16),,,,,0.0041911660227924585,0.023232624982483685,13,1,1,11,,,, +nanogpt,"loss, step 1",gpu_optimized,4,1.0832417090423405,ok (0e+00),4.104661,1,3e-05,3e-05,,,13,1,1,12,,,, +nanogpt,"train (sgd), step 1",gpu_optimized,4,0.260167375090532,ok (4e-16),,,,,0.005934665212407708,0.023253499995917082,13,1,1,23,,,, +nanogpt,"loss, step 2",gpu_optimized,4,1.0810972499893978,ok (1e-16),3.883857,1,3e-05,3e-05,,,13,1,1,24,,,, +nanogpt,"train (sgd), step 2",gpu_optimized,4,0.26968591706827283,ok (5e-16),,,,,0.007713287021033466,0.023271125042811036,13,1,1,35,,,, +nanogpt,"loss, step 3",gpu_optimized,4,1.0888637909665704,ok (1e-16),3.641763,1,3e-05,3e-05,,,13,1,1,36,,,, +nanogpt,"train (sgd), step 3",gpu_optimized,4,0.26123370800632983,ok (5e-16),,,,,0.009434079169295728,0.023294582962989807,13,1,1,47,,,, +nanogpt,"loss, step 4",gpu_optimized,4,1.079754958045669,ok (3e-16),3.842234,1,3e-05,3e-05,,,13,1,1,48,,,, +nanogpt,"train (sgd), step 4",gpu_optimized,4,0.25898429099470377,ok (8e-16),,,,,0.011198204243555665,0.023313790909014642,13,1,1,59,,,, +nanogpt,"loss, step 0",gpu_forced,4,2.6045536249876022,ok (3e-10),4.179251,1,3e-05,3e-05,,,13,1,,,378,,, +nanogpt,"train (sgd), step 0",gpu_forced,4,0.743422916973941,ok (1e-07),,,,,0.004090919974260032,0.022802916006185114,13,1,1,11,403,,, +nanogpt,"loss, step 1",gpu_forced,4,1.19075020798482,ok (9e-10),4.104661,1,3e-05,3e-05,,,13,1,1,12,781,,, +nanogpt,"train (sgd), step 1",gpu_forced,4,0.3084587079938501,ok (2e-07),,,,,0.005839210934937,0.02282529103104025,13,1,1,23,806,,, +nanogpt,"loss, step 2",gpu_forced,4,1.1888486669631675,ok (3e-10),3.883857,1,3e-05,3e-05,,,13,1,1,24,1184,,, +nanogpt,"train (sgd), step 2",gpu_forced,4,0.29137099999934435,ok (2e-07),,,,,0.007563128252513707,0.022846041014418006,13,1,1,35,1209,,, +nanogpt,"loss, step 3",gpu_forced,4,1.165826000040397,ok (5e-09),3.641763,1,3e-05,3e-05,,,13,1,1,36,1587,,, +nanogpt,"train (sgd), step 3",gpu_forced,4,0.2927575410576537,ok (2e-07),,,,,0.009313585818745196,0.02286595804616809,13,1,1,47,1612,,, +nanogpt,"loss, step 4",gpu_forced,4,1.1657407499151304,ok (5e-10),3.842234,1,3e-05,3e-05,,,13,1,1,48,1990,,, +nanogpt,"train (sgd), step 4",gpu_forced,4,0.31906829203944653,ok (3e-07),,,,,0.011052169720642269,0.022886125021614134,13,1,1,59,2015,,, diff --git a/benchmarks/apple/nanogpt-5-steps-comparison.log b/benchmarks/apple/nanogpt-5-steps-comparison.log new file mode 100644 index 00000000..2ebe3fa9 --- /dev/null +++ b/benchmarks/apple/nanogpt-5-steps-comparison.log @@ -0,0 +1,146 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + loss, step 0 datafusion batch=4 1.069s + train (sgd), step 0 datafusion batch=4 0.396s + loss, step 1 datafusion batch=4 1.053s + train (sgd), step 1 datafusion batch=4 0.163s + loss, step 2 datafusion batch=4 1.100s + train (sgd), step 2 datafusion batch=4 0.161s + loss, step 3 datafusion batch=4 1.053s + train (sgd), step 3 datafusion batch=4 0.163s + loss, step 4 datafusion batch=4 1.056s + train (sgd), step 4 datafusion batch=4 0.161s + loss, step 0 einfold batch=4 1.159s + train (sgd), step 0 einfold batch=4 0.570s + loss, step 1 einfold batch=4 1.162s + train (sgd), step 1 einfold batch=4 0.333s + loss, step 2 einfold batch=4 1.176s + train (sgd), step 2 einfold batch=4 0.329s + loss, step 3 einfold batch=4 1.162s + train (sgd), step 3 einfold batch=4 0.331s + loss, step 4 einfold batch=4 1.172s + train (sgd), step 4 einfold batch=4 0.331s + loss, step 0 duckdb batch=4 0.029s + train (sgd), step 0 duckdb batch=4 0.299s + loss, step 1 duckdb batch=4 0.024s + train (sgd), step 1 duckdb batch=4 0.257s + loss, step 2 duckdb batch=4 0.025s + train (sgd), step 2 duckdb batch=4 0.253s + loss, step 3 duckdb batch=4 0.024s + train (sgd), step 3 duckdb batch=4 0.254s + loss, step 4 duckdb batch=4 0.024s + train (sgd), step 4 duckdb batch=4 0.252s + loss, step 0 datafusion_prepared batch=4 2.183s + train (sgd), step 0 datafusion_prepared batch=4 0.406s + loss, step 1 datafusion_prepared batch=4 0.977s + train (sgd), step 1 datafusion_prepared batch=4 0.125s + loss, step 2 datafusion_prepared batch=4 0.977s + train (sgd), step 2 datafusion_prepared batch=4 0.131s + loss, step 3 datafusion_prepared batch=4 0.975s + train (sgd), step 3 datafusion_prepared batch=4 0.123s + loss, step 4 datafusion_prepared batch=4 0.979s + train (sgd), step 4 datafusion_prepared batch=4 0.125s + loss, step 0 cpu_optimized batch=4 2.409s + train (sgd), step 0 cpu_optimized batch=4 0.574s + loss, step 1 cpu_optimized batch=4 1.081s + train (sgd), step 1 cpu_optimized batch=4 0.259s + loss, step 2 cpu_optimized batch=4 1.077s + train (sgd), step 2 cpu_optimized batch=4 0.259s + loss, step 3 cpu_optimized batch=4 1.085s + train (sgd), step 3 cpu_optimized batch=4 0.265s + loss, step 4 cpu_optimized batch=4 1.083s + train (sgd), step 4 cpu_optimized batch=4 0.263s + loss, step 0 gpu_optimized batch=4 2.407s + train (sgd), step 0 gpu_optimized batch=4 0.561s + loss, step 1 gpu_optimized batch=4 1.083s + train (sgd), step 1 gpu_optimized batch=4 0.260s + loss, step 2 gpu_optimized batch=4 1.081s + train (sgd), step 2 gpu_optimized batch=4 0.270s + loss, step 3 gpu_optimized batch=4 1.089s + train (sgd), step 3 gpu_optimized batch=4 0.261s + loss, step 4 gpu_optimized batch=4 1.080s + train (sgd), step 4 gpu_optimized batch=4 0.259s + loss, step 0 gpu_forced batch=4 2.605s + train (sgd), step 0 gpu_forced batch=4 0.743s + loss, step 1 gpu_forced batch=4 1.191s + train (sgd), step 1 gpu_forced batch=4 0.308s + loss, step 2 gpu_forced batch=4 1.189s + train (sgd), step 2 gpu_forced batch=4 0.291s + loss, step 3 gpu_forced batch=4 1.166s + train (sgd), step 3 gpu_forced batch=4 0.293s + loss, step 4 gpu_forced batch=4 1.166s + train (sgd), step 4 gpu_forced batch=4 0.319s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | load_seconds | model_seconds | table_registers | query_preparations | gradient_preparations | inplace_updates | gpu_matmul | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | datafusion | 4 | 1.069 | ok (4e-16) | 4.179 | 9882 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | datafusion | 4 | 0.396 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.004 | 0.025 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 1 | datafusion | 4 | 1.053 | ok (2e-16) | 4.105 | 9882 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 1 | datafusion | 4 | 0.163 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.006 | 0.047 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 2 | datafusion | 4 | 1.100 | ok (0e+00) | 3.884 | 9882 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 2 | datafusion | 4 | 0.161 | ok (6e-16) | n/a | n/a | n/a | n/a | 0.008 | 0.072 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 3 | datafusion | 4 | 1.053 | ok (1e-16) | 3.642 | 9882 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 3 | datafusion | 4 | 0.163 | ok (6e-16) | n/a | n/a | n/a | n/a | 0.010 | 0.095 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 4 | datafusion | 4 | 1.056 | ok (5e-16) | 3.842 | 9882 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 4 | datafusion | 4 | 0.161 | ok (8e-16) | n/a | n/a | n/a | n/a | 0.012 | 0.118 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 0 | einfold | 4 | 1.159 | ok (2e-16) | 4.179 | 1 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | einfold | 4 | 0.570 | ok (4e-16) | n/a | n/a | n/a | n/a | 0.003 | 0.023 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 1 | einfold | 4 | 1.162 | ok (0e+00) | 4.105 | 1 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 1 | einfold | 4 | 0.333 | ok (4e-16) | n/a | n/a | n/a | n/a | 0.005 | 0.047 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 2 | einfold | 4 | 1.176 | ok (1e-16) | 3.884 | 1 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 2 | einfold | 4 | 0.329 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.007 | 0.071 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 3 | einfold | 4 | 1.162 | ok (1e-16) | 3.642 | 1 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 3 | einfold | 4 | 0.331 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.009 | 0.095 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 4 | einfold | 4 | 1.172 | ok (3e-16) | 3.842 | 1 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 4 | einfold | 4 | 0.331 | ok (8e-16) | n/a | n/a | n/a | n/a | 0.011 | 0.119 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 0 | duckdb | 4 | 0.029 | ok (2e-16) | 4.179 | 126 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | duckdb | 4 | 0.299 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.011 | 0.001 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 1 | duckdb | 4 | 0.024 | ok (0e+00) | 4.105 | 126 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 1 | duckdb | 4 | 0.257 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.017 | 0.002 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 2 | duckdb | 4 | 0.025 | ok (0e+00) | 3.884 | 126 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 2 | duckdb | 4 | 0.253 | ok (6e-16) | n/a | n/a | n/a | n/a | 0.023 | 0.003 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 3 | duckdb | 4 | 0.024 | ok (0e+00) | 3.642 | 126 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 3 | duckdb | 4 | 0.254 | ok (6e-16) | n/a | n/a | n/a | n/a | 0.029 | 0.004 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 4 | duckdb | 4 | 0.024 | ok (0e+00) | 3.842 | 126 | 0.000 | 0.000 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 4 | duckdb | 4 | 0.252 | ok (1e-15) | n/a | n/a | n/a | n/a | 0.034 | 0.005 | n/a | n/a | n/a | n/a | n/a | +| nanogpt | loss, step 0 | datafusion_prepared | 4 | 2.183 | ok (4e-16) | 4.179 | 9882 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | datafusion_prepared | 4 | 0.406 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.004 | 0.023 | 13 | 1 | 1 | 11 | n/a | +| nanogpt | loss, step 1 | datafusion_prepared | 4 | 0.977 | ok (2e-16) | 4.105 | 9882 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 12 | n/a | +| nanogpt | train (sgd), step 1 | datafusion_prepared | 4 | 0.125 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.006 | 0.023 | 13 | 1 | 1 | 23 | n/a | +| nanogpt | loss, step 2 | datafusion_prepared | 4 | 0.977 | ok (1e-16) | 3.884 | 9882 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 24 | n/a | +| nanogpt | train (sgd), step 2 | datafusion_prepared | 4 | 0.131 | ok (9e-16) | n/a | n/a | n/a | n/a | 0.007 | 0.023 | 13 | 1 | 1 | 35 | n/a | +| nanogpt | loss, step 3 | datafusion_prepared | 4 | 0.975 | ok (1e-16) | 3.642 | 9882 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 36 | n/a | +| nanogpt | train (sgd), step 3 | datafusion_prepared | 4 | 0.123 | ok (7e-16) | n/a | n/a | n/a | n/a | 0.009 | 0.023 | 13 | 1 | 1 | 47 | n/a | +| nanogpt | loss, step 4 | datafusion_prepared | 4 | 0.979 | ok (5e-16) | 3.842 | 9882 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 48 | n/a | +| nanogpt | train (sgd), step 4 | datafusion_prepared | 4 | 0.125 | ok (1e-15) | n/a | n/a | n/a | n/a | 0.011 | 0.023 | 13 | 1 | 1 | 59 | n/a | +| nanogpt | loss, step 0 | cpu_optimized | 4 | 2.409 | ok (2e-16) | 4.179 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | cpu_optimized | 4 | 0.574 | ok (3e-16) | n/a | n/a | n/a | n/a | 0.004 | 0.024 | 13 | 1 | 1 | 11 | n/a | +| nanogpt | loss, step 1 | cpu_optimized | 4 | 1.081 | ok (2e-16) | 4.105 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 12 | n/a | +| nanogpt | train (sgd), step 1 | cpu_optimized | 4 | 0.259 | ok (3e-16) | n/a | n/a | n/a | n/a | 0.006 | 0.024 | 13 | 1 | 1 | 23 | n/a | +| nanogpt | loss, step 2 | cpu_optimized | 4 | 1.077 | ok (1e-16) | 3.884 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 24 | n/a | +| nanogpt | train (sgd), step 2 | cpu_optimized | 4 | 0.259 | ok (4e-16) | n/a | n/a | n/a | n/a | 0.007 | 0.024 | 13 | 1 | 1 | 35 | n/a | +| nanogpt | loss, step 3 | cpu_optimized | 4 | 1.085 | ok (1e-16) | 3.642 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 36 | n/a | +| nanogpt | train (sgd), step 3 | cpu_optimized | 4 | 0.265 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.009 | 0.024 | 13 | 1 | 1 | 47 | n/a | +| nanogpt | loss, step 4 | cpu_optimized | 4 | 1.083 | ok (3e-16) | 3.842 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 48 | n/a | +| nanogpt | train (sgd), step 4 | cpu_optimized | 4 | 0.263 | ok (9e-16) | n/a | n/a | n/a | n/a | 0.011 | 0.024 | 13 | 1 | 1 | 59 | n/a | +| nanogpt | loss, step 0 | gpu_optimized | 4 | 2.407 | ok (2e-16) | 4.179 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | gpu_optimized | 4 | 0.561 | ok (3e-16) | n/a | n/a | n/a | n/a | 0.004 | 0.023 | 13 | 1 | 1 | 11 | n/a | +| nanogpt | loss, step 1 | gpu_optimized | 4 | 1.083 | ok (0e+00) | 4.105 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 12 | n/a | +| nanogpt | train (sgd), step 1 | gpu_optimized | 4 | 0.260 | ok (4e-16) | n/a | n/a | n/a | n/a | 0.006 | 0.023 | 13 | 1 | 1 | 23 | n/a | +| nanogpt | loss, step 2 | gpu_optimized | 4 | 1.081 | ok (1e-16) | 3.884 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 24 | n/a | +| nanogpt | train (sgd), step 2 | gpu_optimized | 4 | 0.270 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.008 | 0.023 | 13 | 1 | 1 | 35 | n/a | +| nanogpt | loss, step 3 | gpu_optimized | 4 | 1.089 | ok (1e-16) | 3.642 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 36 | n/a | +| nanogpt | train (sgd), step 3 | gpu_optimized | 4 | 0.261 | ok (5e-16) | n/a | n/a | n/a | n/a | 0.009 | 0.023 | 13 | 1 | 1 | 47 | n/a | +| nanogpt | loss, step 4 | gpu_optimized | 4 | 1.080 | ok (3e-16) | 3.842 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 48 | n/a | +| nanogpt | train (sgd), step 4 | gpu_optimized | 4 | 0.259 | ok (8e-16) | n/a | n/a | n/a | n/a | 0.011 | 0.023 | 13 | 1 | 1 | 59 | n/a | +| nanogpt | loss, step 0 | gpu_forced | 4 | 2.605 | ok (3e-10) | 4.179 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | n/a | n/a | 378 | +| nanogpt | train (sgd), step 0 | gpu_forced | 4 | 0.743 | ok (1e-07) | n/a | n/a | n/a | n/a | 0.004 | 0.023 | 13 | 1 | 1 | 11 | 403 | +| nanogpt | loss, step 1 | gpu_forced | 4 | 1.191 | ok (9e-10) | 4.105 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 12 | 781 | +| nanogpt | train (sgd), step 1 | gpu_forced | 4 | 0.308 | ok (2e-07) | n/a | n/a | n/a | n/a | 0.006 | 0.023 | 13 | 1 | 1 | 23 | 806 | +| nanogpt | loss, step 2 | gpu_forced | 4 | 1.189 | ok (3e-10) | 3.884 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 24 | 1184 | +| nanogpt | train (sgd), step 2 | gpu_forced | 4 | 0.291 | ok (2e-07) | n/a | n/a | n/a | n/a | 0.008 | 0.023 | 13 | 1 | 1 | 35 | 1209 | +| nanogpt | loss, step 3 | gpu_forced | 4 | 1.166 | ok (5e-09) | 3.642 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 36 | 1587 | +| nanogpt | train (sgd), step 3 | gpu_forced | 4 | 0.293 | ok (2e-07) | n/a | n/a | n/a | n/a | 0.009 | 0.023 | 13 | 1 | 1 | 47 | 1612 | +| nanogpt | loss, step 4 | gpu_forced | 4 | 1.166 | ok (5e-10) | 3.842 | 1 | 0.000 | 0.000 | n/a | n/a | 13 | 1 | 1 | 48 | 1990 | +| nanogpt | train (sgd), step 4 | gpu_forced | 4 | 0.319 | ok (3e-07) | n/a | n/a | n/a | n/a | 0.011 | 0.023 | 13 | 1 | 1 | 59 | 2015 | + +wrote benchmarks/apple/nanogpt-5-steps-comparison.csv diff --git a/benchmarks/apple/nanogpt-5-steps-cpu-forced.log b/benchmarks/apple/nanogpt-5-steps-cpu-forced.log new file mode 100644 index 00000000..d2749e36 --- /dev/null +++ b/benchmarks/apple/nanogpt-5-steps-cpu-forced.log @@ -0,0 +1,26 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + loss, step 0 cpu_forced batch=4 2.363s + train (sgd), step 0 cpu_forced batch=4 0.559s + loss, step 1 cpu_forced batch=4 1.080s + train (sgd), step 1 cpu_forced batch=4 0.261s + loss, step 2 cpu_forced batch=4 1.072s + train (sgd), step 2 cpu_forced batch=4 0.264s + loss, step 3 cpu_forced batch=4 1.079s + train (sgd), step 3 cpu_forced batch=4 0.259s + loss, step 4 cpu_forced batch=4 1.094s + train (sgd), step 4 cpu_forced batch=4 0.257s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | cpu_forced | 4 | 2.363 | ok (2e-16) | 4.179 | 1 | 1e-09 | 1e-09 | 13 | 1 | 378 | 0 | 0 | 0 | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | cpu_forced | 4 | 0.559 | ok (3e-16) | n/a | n/a | n/a | n/a | 13 | 1 | 403 | 0 | 0 | 0 | 0.004 | 0.026 | 1 | 11 | +| nanogpt | loss, step 1 | cpu_forced | 4 | 1.080 | ok (0e+00) | 4.105 | 1 | 1e-09 | 1e-09 | 13 | 1 | 781 | 0 | 0 | 0 | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | cpu_forced | 4 | 0.261 | ok (3e-16) | n/a | n/a | n/a | n/a | 13 | 1 | 806 | 0 | 0 | 0 | 0.006 | 0.026 | 1 | 23 | +| nanogpt | loss, step 2 | cpu_forced | 4 | 1.072 | ok (1e-16) | 3.884 | 1 | 1e-09 | 1e-09 | 13 | 1 | 1184 | 0 | 0 | 0 | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | cpu_forced | 4 | 0.264 | ok (6e-16) | n/a | n/a | n/a | n/a | 13 | 1 | 1209 | 0 | 0 | 0 | 0.008 | 0.026 | 1 | 35 | +| nanogpt | loss, step 3 | cpu_forced | 4 | 1.079 | ok (1e-16) | 3.642 | 1 | 1e-09 | 1e-09 | 13 | 1 | 1587 | 0 | 0 | 0 | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | cpu_forced | 4 | 0.259 | ok (5e-16) | n/a | n/a | n/a | n/a | 13 | 1 | 1612 | 0 | 0 | 0 | 0.009 | 0.026 | 1 | 47 | +| nanogpt | loss, step 4 | cpu_forced | 4 | 1.094 | ok (3e-16) | 3.842 | 1 | 1e-09 | 1e-09 | 13 | 1 | 1990 | 0 | 0 | 0 | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | cpu_forced | 4 | 0.257 | ok (8e-16) | n/a | n/a | n/a | n/a | 13 | 1 | 2015 | 0 | 0 | 0 | 0.011 | 0.026 | 1 | 59 | + +wrote benchmarks/apple/nanogpt-5-steps-cpu-forced.csv diff --git a/benchmarks/apple/nanogpt-5-steps-summary.json b/benchmarks/apple/nanogpt-5-steps-summary.json new file mode 100644 index 00000000..e9f3e25f --- /dev/null +++ b/benchmarks/apple/nanogpt-5-steps-summary.json @@ -0,0 +1,319 @@ +{ + "steps": 5, + "optimizer": "sgd", + "batch": 4, + "layers": 1, + "embed": 16, + "heads": 2, + "context": 16, + "parameters": 4416, + "cpu_target_partitions": 30, + "MLX_ENABLE_TF32": "0", + "samples_per_step": 1, + "versions": { + "datafusion": "55.0.0", + "einfold": "0.2.0+xqlmlx.1", + "ddxdb": "0.4.6", + "mlx": "0.32.4", + "duckdb": "1.5.6", + "jax": "0.12.0.1" + }, + "hosts": { + "datafusion": { + "loss_seconds": [ + 1.0686229170532897, + 1.0534565839916468, + 1.0999285000143573, + 1.0528932920424268, + 1.055990333086811 + ], + "train_seconds": [ + 0.3961218750337139, + 0.1625057499622926, + 0.16057641606312245, + 0.16286754107568413, + 0.16142054204829037 + ], + "loss_total_seconds": 5.3308916261885315, + "train_total_seconds": 1.0434921241831034, + "load_seconds": 0.01193312811665237, + "model_seconds": 0.11751883407123387, + "measured_total_seconds": 6.503835712559521, + "warm_step_mean_seconds": 1.2274097395711578, + "max_reported_scaled_error": 8e-16, + "reference_loss": [ + "4.179251", + "4.104661", + "3.883857", + "3.641763", + "3.842234" + ], + "table_registers": 0, + "inplace_updates": 0, + "query_preparations": 0, + "gradient_preparations": 0, + "cpu_matmul": null, + "gpu_matmul": 0, + "gpu_attention": null + }, + "einfold": { + "loss_seconds": [ + 1.1594368750229478, + 1.1623720839852467, + 1.1755689999554306, + 1.1617839999962598, + 1.1724911669734865 + ], + "train_seconds": [ + 0.5700416659237817, + 0.33289270801469684, + 0.3288239169633016, + 0.33115795894991606, + 0.33088800008408725 + ], + "loss_total_seconds": 5.8316531259333715, + "train_total_seconds": 1.8938042499357834, + "load_seconds": 0.01106328924652189, + "model_seconds": 0.11939912394154817, + "measured_total_seconds": 7.855919789057225, + "warm_step_mean_seconds": 1.4989947087306064, + "max_reported_scaled_error": 8e-16, + "reference_loss": [ + "4.179251", + "4.104661", + "3.883857", + "3.641763", + "3.842234" + ], + "table_registers": 0, + "inplace_updates": 0, + "query_preparations": 0, + "gradient_preparations": 0, + "cpu_matmul": null, + "gpu_matmul": 0, + "gpu_attention": null + }, + "duckdb": { + "loss_seconds": [ + 0.02867420797701925, + 0.02431754197459668, + 0.024851958034560084, + 0.02433566702529788, + 0.024451749981381 + ], + "train_seconds": [ + 0.29887858394067734, + 0.2568598750513047, + 0.2529260829323903, + 0.2544355830177665, + 0.2519476660527289 + ], + "loss_total_seconds": 0.1266311249928549, + "train_total_seconds": 1.3150477909948677, + "load_seconds": 0.034349376102909446, + "model_seconds": 0.0049009999493137, + "measured_total_seconds": 1.4809292920399457, + "warm_step_mean_seconds": 0.2785315310175065, + "max_reported_scaled_error": 1e-15, + "reference_loss": [ + "4.179251", + "4.104661", + "3.883857", + "3.641763", + "3.842234" + ], + "table_registers": 0, + "inplace_updates": 0, + "query_preparations": 0, + "gradient_preparations": 0, + "cpu_matmul": null, + "gpu_matmul": 0, + "gpu_attention": null + }, + "datafusion_prepared": { + "loss_seconds": [ + 2.1832260839873925, + 0.9771634170319885, + 0.9772000409429893, + 0.9754911670461297, + 0.9794474589871243 + ], + "train_seconds": [ + 0.40616029198281467, + 0.12461866706144065, + 0.13093287497758865, + 0.12260366708505899, + 0.12534429098013788 + ], + "loss_total_seconds": 6.092528167995624, + "train_total_seconds": 0.9096597920870408, + "load_seconds": 0.010954088880680501, + "model_seconds": 0.0228714159457013, + "measured_total_seconds": 7.036013464909047, + "warm_step_mean_seconds": 1.1032003960281145, + "max_reported_scaled_error": 1e-15, + "reference_loss": [ + "4.179251", + "4.104661", + "3.883857", + "3.641763", + "3.842234" + ], + "table_registers": 13, + "inplace_updates": 59, + "query_preparations": 1, + "gradient_preparations": 1, + "cpu_matmul": null, + "gpu_matmul": 0, + "gpu_attention": null + }, + "cpu_optimized": { + "loss_seconds": [ + 2.409379499964416, + 1.0812175000319257, + 1.0774768339470029, + 1.0851524999598041, + 1.08284512499813 + ], + "train_seconds": [ + 0.5740339169278741, + 0.25947070901747793, + 0.2592378749977797, + 0.26522787497378886, + 0.26300137501675636 + ], + "loss_total_seconds": 6.736071458901279, + "train_total_seconds": 1.620971750933677, + "load_seconds": 0.010736083262600005, + "model_seconds": 0.02378150017466396, + "measured_total_seconds": 8.39156079327222, + "warm_step_mean_seconds": 1.3434074482356664, + "max_reported_scaled_error": 9e-16, + "reference_loss": [ + "4.179251", + "4.104661", + "3.883857", + "3.641763", + "3.842234" + ], + "table_registers": 13, + "inplace_updates": 59, + "query_preparations": 1, + "gradient_preparations": 1, + "cpu_matmul": 0, + "gpu_matmul": 0, + "gpu_attention": 0 + }, + "cpu_forced": { + "loss_seconds": [ + 2.362867624964565, + 1.0796167920343578, + 1.0719282500213012, + 1.078952332958579, + 1.0935512910364196 + ], + "train_seconds": [ + 0.5587617499986663, + 0.261068458086811, + 0.2637048749020323, + 0.25885908398777246, + 0.25716737494803965 + ], + "loss_total_seconds": 6.686916291015223, + "train_total_seconds": 1.5995615419233218, + "load_seconds": 0.01108320860657841, + "model_seconds": 0.026117125060409307, + "measured_total_seconds": 8.323678166605532, + "warm_step_mean_seconds": 1.3412121144938283, + "max_reported_scaled_error": 8e-16, + "reference_loss": [ + "4.179251", + "4.104661", + "3.883857", + "3.641763", + "3.842234" + ], + "table_registers": 13, + "inplace_updates": 59, + "query_preparations": 1, + "gradient_preparations": 1, + "cpu_matmul": 2015, + "gpu_matmul": 0, + "gpu_attention": 0 + }, + "gpu_optimized": { + "loss_seconds": [ + 2.407174542080611, + 1.0832417090423405, + 1.0810972499893978, + 1.0888637909665704, + 1.079754958045669 + ], + "train_seconds": [ + 0.5606161670293659, + 0.260167375090532, + 0.26968591706827283, + 0.26123370800632983, + 0.25898429099470377 + ], + "loss_total_seconds": 6.740132250124589, + "train_total_seconds": 1.6106874581892043, + "load_seconds": 0.011198204243555665, + "model_seconds": 0.023313790909014642, + "measured_total_seconds": 8.385331703466363, + "warm_step_mean_seconds": 1.345757249800954, + "max_reported_scaled_error": 8e-16, + "reference_loss": [ + "4.179251", + "4.104661", + "3.883857", + "3.641763", + "3.842234" + ], + "table_registers": 13, + "inplace_updates": 59, + "query_preparations": 1, + "gradient_preparations": 1, + "cpu_matmul": 0, + "gpu_matmul": 0, + "gpu_attention": 0 + }, + "gpu_forced": { + "loss_seconds": [ + 2.6045536249876022, + 1.19075020798482, + 1.1888486669631675, + 1.165826000040397, + 1.1657407499151304 + ], + "train_seconds": [ + 0.743422916973941, + 0.3084587079938501, + 0.29137099999934435, + 0.2927575410576537, + 0.31906829203944653 + ], + "loss_total_seconds": 7.315719249891117, + "train_total_seconds": 1.9550784580642357, + "load_seconds": 0.011052169720642269, + "model_seconds": 0.022886125021614134, + "measured_total_seconds": 9.30473600269761, + "warm_step_mean_seconds": 1.4807052914984524, + "max_reported_scaled_error": 3e-07, + "reference_loss": [ + "4.179251", + "4.104661", + "3.883857", + "3.641763", + "3.842234" + ], + "table_registers": 13, + "inplace_updates": 59, + "query_preparations": 1, + "gradient_preparations": 1, + "cpu_matmul": 0, + "gpu_matmul": 2015, + "gpu_attention": 0 + } + } +} diff --git a/benchmarks/apple/nanogpt-5-steps.csv b/benchmarks/apple/nanogpt-5-steps.csv new file mode 100644 index 00000000..bf42e770 --- /dev/null +++ b/benchmarks/apple/nanogpt-5-steps.csv @@ -0,0 +1,31 @@ +demo,step,host,batch,seconds,check,loss,operators +nanogpt,"loss, step 0",datafusion,4,1.0857603329932317,ok (4e-16),4.179251,9882 +nanogpt,"train (sgd), step 0",datafusion,4,0.401873541995883,ok (5e-16),, +nanogpt,"loss, step 1",datafusion,4,1.067169375019148,ok (2e-16),4.104661,9882 +nanogpt,"train (sgd), step 1",datafusion,4,0.16367095906753093,ok (6e-16),, +nanogpt,"loss, step 2",datafusion,4,1.06832433398813,ok (1e-16),3.883857,9882 +nanogpt,"train (sgd), step 2",datafusion,4,0.17810562497470528,ok (6e-16),, +nanogpt,"loss, step 3",datafusion,4,1.1186981670325622,ok (1e-16),3.641763,9882 +nanogpt,"train (sgd), step 3",datafusion,4,0.17602383298799396,ok (6e-16),, +nanogpt,"loss, step 4",datafusion,4,1.1557306670583785,ok (5e-16),3.842234,9882 +nanogpt,"train (sgd), step 4",datafusion,4,0.18496695801150054,ok (9e-16),, +nanogpt,"loss, step 0",einfold,4,1.2271647920133546,ok (2e-16),4.179251,1 +nanogpt,"train (sgd), step 0",einfold,4,0.6224314590217546,ok (3e-16),, +nanogpt,"loss, step 1",einfold,4,1.2855452080257237,ok (0e+00),4.104661,1 +nanogpt,"train (sgd), step 1",einfold,4,0.34117912501096725,ok (3e-16),, +nanogpt,"loss, step 2",einfold,4,1.183204917004332,ok (1e-16),3.883857,1 +nanogpt,"train (sgd), step 2",einfold,4,0.3368055419996381,ok (7e-16),, +nanogpt,"loss, step 3",einfold,4,1.1981873749755323,ok (1e-16),3.641763,1 +nanogpt,"train (sgd), step 3",einfold,4,0.3686611250741407,ok (5e-16),, +nanogpt,"loss, step 4",einfold,4,1.218982332968153,ok (3e-16),3.842234,1 +nanogpt,"train (sgd), step 4",einfold,4,0.33348158304579556,ok (8e-16),, +nanogpt,"loss, step 0",duckdb,4,0.028204041998833418,ok (0e+00),4.179251,126 +nanogpt,"train (sgd), step 0",duckdb,4,0.30877691705245525,ok (6e-16),, +nanogpt,"loss, step 1",duckdb,4,0.02514837495982647,ok (2e-16),4.104661,126 +nanogpt,"train (sgd), step 1",duckdb,4,0.2552210410358384,ok (6e-16),, +nanogpt,"loss, step 2",duckdb,4,0.02464066701941192,ok (1e-16),3.883857,126 +nanogpt,"train (sgd), step 2",duckdb,4,0.2527270830469206,ok (5e-16),, +nanogpt,"loss, step 3",duckdb,4,0.02492891694419086,ok (0e+00),3.641763,126 +nanogpt,"train (sgd), step 3",duckdb,4,0.25987891701515764,ok (5e-16),, +nanogpt,"loss, step 4",duckdb,4,0.024324666941538453,ok (0e+00),3.842234,126 +nanogpt,"train (sgd), step 4",duckdb,4,0.255693459068425,ok (1e-15),, diff --git a/benchmarks/apple/nanogpt-5-steps.md b/benchmarks/apple/nanogpt-5-steps.md new file mode 100644 index 00000000..d84cf969 --- /dev/null +++ b/benchmarks/apple/nanogpt-5-steps.md @@ -0,0 +1,101 @@ +# nanoGPT benchmark 2: five-step CPU/GPU comparison + +Every variant ran **`--steps 5`**, SGD, batch 4, one layer, 16 embedding +channels, two heads, context 16, 4,416 parameters and the same Tiny +Shakespeare batches. This keeps the model and SQL from +[`02_nanogpt.py`](../ml/02_nanogpt.py); JAX checks the loss and every +parameter table after each update. Original inputs are copied into the +resident hosts so in-place updates cannot mutate another host's initial +weights or batch data. The five reference losses are identical across +all eight variants. + +Machine: Apple M5 Ultra, 30 CPU target partitions, 64 GPU cores, 256 GiB. +MLX 0.32.4 with `MLX_ENABLE_TF32=0`; patched EinFold 0.2.0+xqlmlx.1, +DataFusion 55.0.0, ddxdb 0.4.6 and DuckDB 1.5.6. The seven-way comparison +ran in one process; the extra forced CPU-kernel control ran separately. +One sample per step, as requested by the script's default `--reps 1`. +These are not seven-repetition kernel microbenchmarks. + +The measured five-step total adds the five loss queries, five training +queries, cumulative table load/update time and model-view creation time. +It includes first-use preparation/compilation. It excludes JAX reference +work, correctness checking, result-to-Dataset reconstruction, operator +inspection and console/reporting time, so it is **not wall-clock runtime**. +The warm step column averages paired loss + training timings for steps +1–4, after the initial step. Each step uses a different batch. + +| Variant | Five loss queries | Five training queries | Measured five-step total | Warm step mean | Completed GPU matrix kernels | +|---|---:|---:|---:|---:|---:| +| DataFusion baseline | 5.331 s | 1.043 s | 6.504 s | 1227.4 ms | 0 | +| EinFold baseline | 5.832 s | 1.894 s | 7.856 s | 1499.0 ms | 0 | +| DuckDB baseline | 0.127 s | 1.315 s | 1.481 s | 278.5 ms | 0 | +| Prepared DataFusion CPU | 6.093 s | 0.910 s | 7.036 s | 1103.2 ms | 0 | +| Prepared EinFold + CPU auto | 6.736 s | 1.621 s | 8.392 s | 1343.4 ms | 0 | +| Prepared EinFold + CPU forced | 6.687 s | 1.600 s | 8.324 s | 1341.2 ms | 0 | +| Prepared EinFold + GPU auto | 6.740 s | 1.611 s | 8.385 s | 1345.8 ms | 0 | +| Prepared EinFold + GPU forced | 7.316 s | 1.955 s | 9.305 s | 1480.7 ms | 2015 | + +## What the variants actually execute + +- Baselines recreate the model view and register updated parameter tables + as the original demo does. They use the current checkout's registration + fixes, but do not enable prepared training or the new kernel bridge. +- `datafusion_prepared` registers tables once, writes updated values into + their existing buffers, keeps the unchanged model view, caches the + forward `Context.prepare` result and reuses `ad.prepare` for all eleven + parameter statements sharing the loss. +- `cpu_optimized` adds the float64 CPU matrix backend and EinFold planning + with the normal operation threshold. At this tiny model size, no new + CPU backend kernel passes the threshold. +- `gpu_optimized` uses the same prepared/resident path and normal GPU + dispatch thresholds. **It executes zero GPU kernels on this model**. + It should not be reported as a GPU speedup. +- `cpu_forced` lowers the operation threshold to zero for float64 CPU + backend calls. It completes **2,015 CPU matrix kernels**. +- `gpu_forced` lowers the CPU/Python and GPU thresholds to zero. It + completes **2,015 actual MLX GPU matrix kernels**. No GPU fused-attention + kernels were recorded. This exercises the GPU bridge while exposing + dispatch and buffer-conversion costs of tiny products. + +The resident hosts each recorded 13 initial table registrations, 59 +in-place updates, one forward-query preparation and one shared gradient +program preparation. The original loop instead reloads the tables and +recreates its view after each batch. + +## Findings + +- Warm prepared DataFusion steps improved **1227.4 → 1103.2 ms (1.11×)**. Warm training-query time alone improved **161.8 → 125.9 ms**. +- Over only five steps, prepared DataFusion's initial planning cost outweighed the warm gains: **6.504 s baseline versus 7.036 s prepared**. The first prepared loss took 2.183 s; later forward losses still took about 0.977 s. +- Forced GPU execution was **10.4% slower** per warm step than the forced float64 CPU backend. The CPU/GPU bridge and physical-plan work outweigh GPU arithmetic savings at these shapes. +- The automatic GPU threshold correctly keeps this small model on CPU. + The cached optimized logical plan does not remove the remaining large + physical-planning cost of the forward loss. This benchmark does not + reproduce the large-attention GPU gains from the separate tensor tests. +- DuckDB remains the fastest complete five-step variant in this run. +- **All 80 loss/update records passed their JAX checks.** Float64 hosts use + `rtol=atol=1e-9`. GPU-enabled hosts use explicit float32 permission and + `rtol=atol=3e-5`. Forced GPU's largest reported error relative to the + reference's maximum magnitude was about **3e-7**, versus about **1e-15** + for float64 controls. Every individual array also passed the elementwise + tolerance check. Reference losses were 4.179251, 4.104661, 3.883857, + 3.641763 and 3.842234 (different batches). + +## Reproduce and inspect + +```sh +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python benchmarks/ml/02_nanogpt.py \ + --steps 5 --hosts datafusion einfold duckdb datafusion_prepared \ + cpu_optimized gpu_optimized gpu_forced \ + --csv benchmarks/apple/nanogpt-5-steps-comparison.csv + +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python benchmarks/ml/02_nanogpt.py \ + --steps 5 --hosts cpu_forced \ + --csv benchmarks/apple/nanogpt-5-steps-cpu-forced.csv +``` + +[Combined CSV](nanogpt-5-steps-all-variants.csv), +[JSON summary](nanogpt-5-steps-summary.json), +[seven-way log](nanogpt-5-steps-comparison.log), +[forced CPU log](nanogpt-5-steps-cpu-forced.log). +The earlier original-default-host run is also retained in +[nanogpt-5-steps.csv](nanogpt-5-steps.csv). diff --git a/benchmarks/apple/nanogpt-adamw-diagnostic.json b/benchmarks/apple/nanogpt-adamw-diagnostic.json new file mode 100644 index 00000000..646d4ffc --- /dev/null +++ b/benchmarks/apple/nanogpt-adamw-diagnostic.json @@ -0,0 +1,83 @@ +{ + "learning_rate": 0.001, + "hosts": { + "shared_cpu": { + "gradient_max_absolute": { + "wte": 1.249000902703301e-16, + "wpe": 1.3877787807814457e-17, + "ln_1": 1.3010426069826053e-18, + "wq": 5.692061405548898e-19, + "wk": 6.505213034913027e-19, + "wv": 3.469446951953614e-17, + "wo": 4.163336342344337e-17, + "ln_2": 1.734723475976807e-18, + "w_fc": 2.42861286636753e-17, + "w_proj": 5.204170427930421e-17, + "ln_f": 2.6020852139652106e-18 + }, + "statistics": { + "table_registers": 24, + "gradient_preparations": 1, + "query_preparations": 11, + "cpu_matmul": 50, + "gpu_matmul": 0, + "gpu_attention": 0, + "declined": 0, + "physical_plans": 87, + "plan_executions": 163, + "stage_buffer_allocations": 54, + "stage_buffer_writes": 54, + "forward_stages": 1, + "planning_seconds": 0.8369682069169357, + "execution_seconds": 4.5019768761703745 + }, + "wpe": { + "gradient_sql": 1.541975823364026e-08, + "gradient_reference": 1.5419758236706518e-08, + "update_sql": 0.04069803292556577, + "update_reference": 0.04069803292553674, + "update_from_sql_gradient": 0.04069803292558585 + }, + "optimizer_vs_same_gradient_max_absolute": 2.996214387707141e-14 + }, + "shared_gpu": { + "gradient_max_absolute": { + "wte": 2.948455140455719e-08, + "wpe": 7.4002049228055755e-09, + "ln_1": 4.71347003094813e-10, + "wq": 1.626235499887184e-10, + "wk": 1.5224943631124074e-10, + "wv": 7.87050620509877e-09, + "wo": 1.0137126929543605e-08, + "ln_2": 1.1144436918436584e-09, + "w_fc": 8.032833453963661e-09, + "w_proj": 1.3967929500535714e-08, + "ln_f": 8.510604197287455e-10 + }, + "statistics": { + "table_registers": 24, + "gradient_preparations": 1, + "query_preparations": 11, + "cpu_matmul": 0, + "gpu_matmul": 50, + "gpu_attention": 0, + "declined": 0, + "physical_plans": 87, + "plan_executions": 163, + "stage_buffer_allocations": 54, + "stage_buffer_writes": 54, + "forward_stages": 1, + "planning_seconds": 0.895146002410911, + "execution_seconds": 4.649418214336038 + }, + "wpe": { + "gradient_sql": 1.3021668260323674e-08, + "gradient_reference": 1.5419758236706518e-08, + "update_sql": 0.040739296722165294, + "update_reference": 0.04069803292553674, + "update_from_sql_gradient": 0.04073929672214913 + }, + "optimizer_vs_same_gradient_max_absolute": 1.6160683902199935e-14 + } + } +} diff --git a/benchmarks/apple/nanogpt-adamw-failed-demo.log b/benchmarks/apple/nanogpt-adamw-failed-demo.log new file mode 100644 index 00000000..f3eaf908 --- /dev/null +++ b/benchmarks/apple/nanogpt-adamw-failed-demo.log @@ -0,0 +1,39 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=adamw, context=64, heads=4, embed=128, forward_only=False + loss, step 0 cpu_forced batch=12 7.931s + train (adamw), step 0 cpu_forced batch=12 3.832s + loss, step 1 cpu_forced batch=12 7.739s + train (adamw), step 1 cpu_forced batch=12 3.465s + loss, step 2 cpu_forced batch=12 3.930s + train (adamw), step 2 cpu_forced batch=12 1.621s + loss, step 3 cpu_forced batch=12 1.156s + train (adamw), step 3 cpu_forced batch=12 1.128s + loss, step 4 cpu_forced batch=12 1.157s + train (adamw), step 4 cpu_forced batch=12 1.025s + loss, step 0 gpu_forced batch=12 2.794s + train (adamw), step 0 gpu_forced batch=12 1.442s +Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 779, in + main() + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/02_nanogpt.py", line 738, in main + assert_close( + File "/Users/cromwellian/xql/xarray-sql/benchmarks/ml/_bench.py", line 66, in assert_close + np.testing.assert_allclose( + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 1780, in assert_allclose + assert_array_compare(compare, actual, desired, err_msg=str(err_msg), + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/numpy/testing/_private/utils.py", line 988, in assert_array_compare + raise AssertionError(msg) +AssertionError: +Not equal to tolerance rtol=3e-05, atol=3e-05 +wpe.w, step 0 +Mismatched elements: 1 / 8192 (0.0122%) +Mismatch at index: + [48, 53]: 0.04073929672215308 (ACTUAL), 0.04069803292553674 (DESIRED) +Max absolute difference among violations: 4.12637966e-05 +Max relative difference among violations: 0.0010139 + ACTUAL: array([[-0.041635, -0.017197, 0.020464, ..., 0.023298, 0.022704, + 0.000739], + [ 0.002403, 0.030883, -0.02728 , ..., -0.008039, -0.007208,... + DESIRED: array([[-0.041635, -0.017197, 0.020464, ..., 0.023298, 0.022704, + 0.000739], + [ 0.002403, 0.030883, -0.02728 , ..., -0.008039, -0.007208,... diff --git a/benchmarks/apple/nanogpt-framework-summary.json b/benchmarks/apple/nanogpt-framework-summary.json new file mode 100644 index 00000000..127b365d --- /dev/null +++ b/benchmarks/apple/nanogpt-framework-summary.json @@ -0,0 +1,882 @@ +{ + "native": [ + { + "file": "framework-gpt2-1024-forward-jax_eager.json", + "backend": "jax_eager", + "context": 1024, + "parameters": 124337664, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": true, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 1.035269625019282, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 1, + "seconds": 0.4574369170004502, + "loss": 10.804215487348133, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.48019329202361405, + "loss": 10.841585972493611, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 3, + "seconds": 0.46009145909920335, + "loss": 10.872348942052431, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 4, + "seconds": 0.45806304190773517, + "loss": 10.87314253629185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 0.0806786670582369, + "warm_seconds": 0.4639461775077507, + "five_step_seconds": 2.891054335050285, + "five_steps_with_setup_compile_seconds": 2.9717330021085218 + }, + { + "file": "framework-gpt2-1024-forward-jax_jit.json", + "backend": "jax_jit", + "context": 1024, + "parameters": 124337664, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": true, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.4770496250130236, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 1, + "seconds": 0.32251091592479497, + "loss": 10.804215487348133, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.3207364999689162, + "loss": 10.841585972493611, + "check": "ok", + "max_scaled_error": 1.6384658516817357e-16 + }, + { + "step": 3, + "seconds": 0.3164084159070626, + "loss": 10.872348942052431, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 4, + "seconds": 0.3200405000243336, + "loss": 10.87314253629185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.1769480420043692, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 0.2612458330113441, + "warm_seconds": 0.31992408295627683, + "five_step_seconds": 1.756745956838131, + "five_steps_with_setup_compile_seconds": 2.017991789849475 + }, + { + "file": "framework-gpt2-1024-forward-torch_cpu.json", + "backend": "torch_cpu", + "context": 1024, + "parameters": 124337664, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": true, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.38666091696359217, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 1.647358279840836e-16 + }, + { + "step": 1, + "seconds": 0.32166445802431554, + "loss": 10.804215487348133, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.35894229100085795, + "loss": 10.841585972493611, + "check": "ok", + "max_scaled_error": 1.6384658516817357e-16 + }, + { + "step": 3, + "seconds": 0.3645232500275597, + "loss": 10.872348942052431, + "check": "ok", + "max_scaled_error": 1.633829864059641e-16 + }, + { + "step": 4, + "seconds": 0.3359561249380931, + "loss": 10.87314253629185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "cpu" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 2.160730791045353, + "warm_seconds": 0.3452715309977066, + "five_step_seconds": 1.7677470409544185, + "five_steps_with_setup_compile_seconds": 3.9284778319997713 + }, + { + "file": "framework-gpt2-1024-forward-torch_mps.json", + "backend": "torch_mps", + "context": 1024, + "parameters": 124337664, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": true, + "rtol": 3e-05, + "atol": 3e-05, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.2647716250503436, + "loss": 10.783063169305695, + "check": "ok", + "max_scaled_error": 2.1740594775497947e-08 + }, + { + "step": 1, + "seconds": 0.02600145898759365, + "loss": 10.804215487348133, + "check": "ok", + "max_scaled_error": 5.195634407695925e-09 + }, + { + "step": 2, + "seconds": 0.03451741603203118, + "loss": 10.841585972493611, + "check": "ok", + "max_scaled_error": 1.29577409940046e-08 + }, + { + "step": 3, + "seconds": 0.030054125003516674, + "loss": 10.872348942052431, + "check": "ok", + "max_scaled_error": 1.4408297738376235e-08 + }, + { + "step": 4, + "seconds": 0.034106959006749094, + "loss": 10.87314253629185, + "check": "ok", + "max_scaled_error": 2.7026244535502437e-08 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "mps" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 2.078200125019066, + "warm_seconds": 0.03116998975747265, + "five_step_seconds": 0.38945158408023417, + "five_steps_with_setup_compile_seconds": 2.4676517090993 + }, + { + "file": "framework-gpt2-16-jax_eager.json", + "backend": "jax_eager", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 2.9766601249575615, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 1, + "seconds": 0.3764039580710232, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.31518250005319715, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 3, + "seconds": 0.3376008750637993, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 4, + "seconds": 0.30860195797868073, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 0.11831612489186227, + "warm_seconds": 0.3344473227916751, + "five_step_seconds": 4.314449416124262, + "five_steps_with_setup_compile_seconds": 4.432765541016124 + }, + { + "file": "framework-gpt2-16-jax_jit.json", + "backend": "jax_jit", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.25373025005683303, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 3.4880647602577323e-16 + }, + { + "step": 1, + "seconds": 0.1414729580283165, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 3.5790513615769393e-16 + }, + { + "step": 2, + "seconds": 0.1349089159630239, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 3.5737750783812467e-16 + }, + { + "step": 3, + "seconds": 0.1352096670307219, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 3.571484214546241e-16 + }, + { + "step": 4, + "seconds": 0.13392283394932747, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 3.5720036936486486e-16 + } + ], + "compile_seconds": 3.0000361249549314, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 3.084211624925956, + "warm_seconds": 0.13637859374284744, + "five_step_seconds": 0.7992446250282228, + "five_steps_with_setup_compile_seconds": 3.883456249954179 + }, + { + "file": "framework-gpt2-16-jax_separate.json", + "backend": "jax_separate", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 3.103274374967441, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 1, + "seconds": 0.4117213749559596, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 2, + "seconds": 0.3880377080058679, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 3, + "seconds": 0.3956522080115974, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 0.0 + }, + { + "step": 4, + "seconds": 0.3818153750617057, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 0.0 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "devices": [ + "cpu:0" + ], + "setup_seconds_including_compile": 0.08685445797163993, + "warm_seconds": 0.39430666650878266, + "five_step_seconds": 4.680501041002572, + "five_steps_with_setup_compile_seconds": 4.7673554989742115 + }, + { + "file": "framework-gpt2-16-torch_cpu.json", + "backend": "torch_cpu", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 1e-09, + "atol": 1e-09, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.1971114999614656, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 1.6506898020374621e-15 + }, + { + "step": 1, + "seconds": 0.09883254196029156, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 1.6277329480048546e-15 + }, + { + "step": 2, + "seconds": 0.09388387494254857, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 1.6253333233185944e-15 + }, + { + "step": 3, + "seconds": 0.0943373340414837, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 1.6047216744810507e-15 + }, + { + "step": 4, + "seconds": 0.09407875000033528, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 1.6049550842695296e-15 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "cpu" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 3.5439498749328777, + "warm_seconds": 0.09528312523616478, + "five_step_seconds": 0.5782440009061247, + "five_steps_with_setup_compile_seconds": 4.122193875839002 + }, + { + "file": "framework-gpt2-16-torch_mps.json", + "backend": "torch_mps", + "context": 16, + "parameters": 123563520, + "steps": 5, + "batch": 1, + "learning_rate": 0.01, + "optimizer": "sgd", + "gelu": "tanh", + "forward_only": false, + "rtol": 3e-05, + "atol": 3e-05, + "python": "3.12.14", + "jax_version": "0.12.0.1", + "reference_sha256": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "rows": [ + { + "step": 0, + "seconds": 0.876074375002645, + "loss": 11.218448458220333, + "check": "ok", + "max_scaled_error": 2.466638482282098e-07 + }, + { + "step": 1, + "seconds": 0.025007624994032085, + "loss": 9.443567400904483, + "check": "ok", + "max_scaled_error": 2.4429602051829147e-07 + }, + { + "step": 2, + "seconds": 0.0235599169973284, + "loss": 7.050589630335465, + "check": "ok", + "max_scaled_error": 2.4332607268461294e-07 + }, + { + "step": 3, + "seconds": 0.02224191604182124, + "loss": 8.57805561725344, + "check": "ok", + "max_scaled_error": 2.5501415047442646e-07 + }, + { + "step": 4, + "seconds": 0.02418895799200982, + "loss": 6.7450402840490185, + "check": "ok", + "max_scaled_error": 2.5291695020020007e-07 + } + ], + "compile_seconds": 0.0, + "scope": "One loss forward and, for training, its backward and SGD update; synchronization included; validation and input/state setup excluded.", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "nanogpt_model_sha256": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "torch_version": "2.15.0", + "devices": [ + "mps" + ], + "torch_threads": 30, + "setup_seconds_including_compile": 2.113612666958943, + "warm_seconds": 0.023749604006297886, + "five_step_seconds": 0.9710727910278365, + "five_steps_with_setup_compile_seconds": 3.0846854579867795 + } + ], + "sql": [ + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-16-shared_gpu.log", + "csv": "nanogpt-shared-v3-gpt2-16-shared_gpu.csv", + "peak_rss_gib": 40.02874755859375, + "status": "passed", + "started_utc": "2026-11-01T10:54:32.541752+00:00", + "exit_code": 0, + "wall_seconds": 846.1769028330455, + "checked_rows": 10, + "case": "gpt2-16", + "loss_warm_seconds": 1.3720009690150619, + "loss_all_seconds": 7.187280835118145, + "train_warm_seconds": 162.27428429198335, + "train_all_seconds": 822.448966667871, + "warm_seconds": 163.6462852609984, + "all_api_seconds": 829.6362475029891, + "measured_total_seconds": 829.9728821297176, + "config": { + "layers": "12", + "embed": "768", + "heads": "12", + "context": "16", + "vocab": "50257", + "parameters": "123563520", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 710.0, + "gpu_attention": 60.0, + "physical_plans": 7148.0, + "plan_executions": 9020.0, + "planning_seconds": 131.42177372181322, + "execution_seconds": 504.44527716143057, + "stage_storage_seconds": 169.1986960049253, + "key_cache_bytes": 2892210304.0, + "key_copy_bytes_avoided": 6005381504.0 + } + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-16-shared_cpu.log", + "csv": "nanogpt-shared-v3-gpt2-16-shared_cpu.csv", + "peak_rss_gib": 38.71354675292969, + "status": "passed", + "started_utc": "2026-11-01T11:08:38.723798+00:00", + "exit_code": 0, + "wall_seconds": 846.0968177500181, + "checked_rows": 10, + "case": "gpt2-16", + "loss_warm_seconds": 1.1682787809986621, + "loss_all_seconds": 6.23576199891977, + "train_warm_seconds": 162.86555686450447, + "train_all_seconds": 823.4401903749676, + "warm_seconds": 164.03383564550313, + "all_api_seconds": 829.6759523738874, + "measured_total_seconds": 829.991804452613, + "config": { + "layers": "12", + "embed": "768", + "heads": "12", + "context": "16", + "vocab": "50257", + "parameters": "123563520", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 710.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0, + "physical_plans": 7148.0, + "plan_executions": 9020.0, + "planning_seconds": 130.0049377027899, + "execution_seconds": 505.680548057775, + "stage_storage_seconds": 169.5953018902801, + "key_cache_bytes": 2892210304.0, + "key_copy_bytes_avoided": 6005381504.0 + } + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.csv", + "--forward-only", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-forward-shared_cpu.log", + "csv": "nanogpt-shared-v3-gpt2-forward-shared_cpu.csv", + "peak_rss_gib": 15.54058837890625, + "status": "passed", + "started_utc": "2026-11-01T11:22:44.844329+00:00", + "exit_code": 0, + "wall_seconds": 38.22789254190866, + "checked_rows": 5, + "case": "gpt2-forward", + "loss_warm_seconds": 6.572556927247206, + "loss_all_seconds": 33.19323650095612, + "warm_seconds": 6.572556927247206, + "all_api_seconds": 33.19323650095612, + "measured_total_seconds": 33.268904172233306, + "config": { + "layers": "12", + "embed": "768", + "heads": "12", + "context": "1024", + "vocab": "50257", + "parameters": "124337664", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 365.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0, + "physical_plans": 65.0, + "plan_executions": 325.0, + "planning_seconds": 0.3588349998462945, + "execution_seconds": 20.173762080725282, + "stage_storage_seconds": 12.39298641658388, + "key_cache_bytes": 1254006784.0, + "key_copy_bytes_avoided": 1914986496.0 + } + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.csv", + "--forward-only", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-forward-shared_gpu.log", + "csv": "nanogpt-shared-v3-gpt2-forward-shared_gpu.csv", + "peak_rss_gib": 15.64544677734375, + "status": "passed", + "started_utc": "2026-11-01T11:23:23.073032+00:00", + "exit_code": 0, + "wall_seconds": 39.74374524992891, + "checked_rows": 5, + "case": "gpt2-forward", + "loss_warm_seconds": 6.857087958022021, + "loss_all_seconds": 34.839587540132925, + "warm_seconds": 6.857087958022021, + "all_api_seconds": 34.839587540132925, + "measured_total_seconds": 34.920842704945244, + "config": { + "layers": "12", + "embed": "768", + "heads": "12", + "context": "1024", + "vocab": "50257", + "parameters": "124337664", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 365.0, + "gpu_attention": 60.0, + "physical_plans": 65.0, + "plan_executions": 325.0, + "planning_seconds": 0.372420794214122, + "execution_seconds": 21.33640320203267, + "stage_storage_seconds": 12.807666914421134, + "key_cache_bytes": 1254006784.0, + "key_copy_bytes_avoided": 1914986496.0 + } + } + ], + "notes": [ + "Native training uses one forward/backward and SGD; jax_separate recomputes backward after a separate forward to match SQL call structure.", + "JAX runs on CPU; PyTorch CPU uses float64 with 30 threads; PyTorch MPS uses float32. SQL MLX retains float64 Arrow storage around float32 GPU kernels.", + "All cases use five steps, batch 1, 12 layers, 12 heads, width 768, vocabulary 50257, identical NumPy seed-0 weights, seed-1 Shakespeare batches and SGD lr=0.01.", + "Device synchronization is inside each timer; JAX reference loss and every updated weight are checked outside timers. Native parameter readback for checks is excluded; SQL query outputs are collected to Arrow inside SQL timings.", + "nanoGPT model is configured with bias=False/dropout=0 and tanh GELU to match SQL; stock bench.py uses AdamW, exact GELU, larger batch/context and CUDA." + ], + "ratios": { + "gpt2-forward:shared_cpu/jax_eager": 14.16663666150672, + "gpt2-forward:shared_gpu/jax_eager": 14.779921228917694, + "gpt2-forward:shared_cpu/jax_jit": 20.54411429897092, + "gpt2-forward:shared_gpu/jax_jit": 21.433484765068968, + "gpt2-forward:shared_cpu/torch_cpu": 19.035907502292343, + "gpt2-forward:shared_gpu/torch_cpu": 19.859986539311773, + "gpt2-forward:shared_cpu/torch_mps": 210.8616967277479, + "gpt2-forward:shared_gpu/torch_mps": 219.9900613178133, + "gpt2-16:shared_gpu/jax_eager": 489.303618563969, + "gpt2-16:shared_cpu/jax_eager": 490.4623971162079, + "gpt2-16:shared_gpu/jax_jit": 1199.9411400997897, + "gpt2-16:shared_cpu/jax_jit": 1202.7828645513225, + "gpt2-16:shared_gpu/jax_separate": 415.0228721972506, + "gpt2-16:shared_cpu/jax_separate": 416.005737609941, + "gpt2-16:shared_gpu/torch_cpu": 1717.4739478306526, + "gpt2-16:shared_cpu/torch_cpu": 1721.5413037611404, + "gpt2-16:shared_gpu/torch_mps": 6890.484793666577, + "gpt2-16:shared_cpu/torch_mps": 6906.802976672995 + } +} diff --git a/benchmarks/apple/nanogpt-frameworks.md b/benchmarks/apple/nanogpt-frameworks.md new file mode 100644 index 00000000..7de93b54 --- /dev/null +++ b/benchmarks/apple/nanogpt-frameworks.md @@ -0,0 +1,141 @@ +# Native JAX and nanoGPT comparison on the M5 Ultra + +The native frameworks are substantially faster than the SQL training path. +These local measurements use the same model dimensions, initial weights, +Shakespeare batches, loss and SGD update as the SQL benchmark, on the same +Apple M5 Ultra with serial execution and no game contention. + +[nanoGPT](https://github.com/karpathy/nanoGPT) is Andrej Karpathy's PyTorch +implementation of GPT-2. Its [bench.py](https://github.com/karpathy/nanoGPT/blob/3adf61e154c3fe3fca428ad6bc3818b27a3b8291/bench.py) +provides a training benchmark: 12 layers, 12 heads, width 768, batch 12, +context 1024, AdamW, optional compilation, 10 warmup steps and 20 timed +steps. Its vocabulary is padded to 50304 and its timing loop uses CUDA +synchronization. Those defaults differ from our five-step SQL run. + +The local harness imports the upstream model at revision +`3adf61e154c3fe3fca428ad6bc3818b27a3b8291`. It sets vocabulary 50257, +batch 1, no biases/dropout, SGD at 0.01 and tanh GELU to match SQL. Upstream +normally uses exact GELU and its benchmark uses AdamW. These measurements +describe a matched configuration of the actual nanoGPT model, rather than +an unmodified execution of stock `bench.py`. + +## Parameter count and checks + +**The training model has 123,563,520 trainable parameters.** The harness +computes the number from the allocated NumPy arrays. PyTorch independently +checks its model's parameter count against it. + +| Parameter group | Elements | +|---|---:| +| Tied token/output embedding | 38,597,376 | +| Positional embedding, context 16 | 12,288 | +| Q, K, V and output projections, 12 layers | 28,311,552 | +| MLP expansion and projection, 12 layers | 56,623,104 | +| LayerNorm gains | 19,200 | +| **Total** | **123,563,520** | + +Each training step computes a loss, backpropagates through all 12 layers +and updates every parameter array. Timers wait for both loss and parameter +updates to finish. Loss and every updated parameter are checked against +the SQL demo's float64 JAX reference outside the timer, after every step. +All **45 native training/forward checkpoint records pass** across nine +runs. CPU uses `rtol=atol=1e-9`; GPU uses `3e-5`. + +Batch 1 and context 16 mean 16 target positions per training step, or 80 +across five steps. The output projection computes all 50257 classes. +This is a full-parameter, short-context training benchmark. Context 1024 +adds positional weights, giving 124,337,664 parameters. + +## Five-step training at context 16 + +Warm time averages steps 1–4. The five-step column includes initial model +setup and explicit JIT compilation where applicable, and excludes imports, +common input/weight generation and correctness validation. JAX JIT takes +3.00 seconds to compile its training step; that cost is included below. + +| Implementation | Device / arithmetic | Warm step | Five steps + setup/compilation | +|---|---|---:|---:| +| SQL shared CPU | CPU / float64 | 164.034 s | 829.992 s | +| SQL shared MLX | GPU kernels float32, Arrow state float64 | 163.646 s | 829.973 s | +| Native JAX, separate loss/backward calls | CPU / float64 | 0.3943 s | 4.767 s | +| Native JAX, eager value-and-gradient | CPU / float64 | 0.3344 s | 4.433 s | +| Native JAX, compiled training step | CPU / float64 | 0.1364 s | 3.883 s | +| nanoGPT PyTorch, eager | CPU / float64, 30 threads | 0.0953 s | 4.122 s | +| nanoGPT PyTorch, eager | Apple MPS GPU / float32 | 0.02375 s | 3.085 s | + +Relative to SQL shared CPU, the separate-call JAX case is about 416 times +faster per warm step, compiled JAX about 1203 times, and native PyTorch CPU +about 1722 times. Native PyTorch MPS is about 6890 times faster than SQL +shared MLX per warm step. Including first-use setup reduces these ratios: +five steps plus setup take about 830 s in SQL versus 3–5 s in native runs. +Four warm observations are not a confidence interval. + +Native training normally reuses one forward pass for its backward pass. +The SQL interface separately evaluates a loss query and a gradient/update +program. `jax_separate` reproduces that two-call structure and remains +hundreds of times faster. Native state stays in framework arrays between +steps. SQL timings include collecting updated parameters into Arrow; +native parameter readback for correctness checking is outside its timers. +PyTorch MPS keeps float32 state, while the SQL GPU bridge retains float64 +Arrow storage. These ratios include those execution/storage differences; +they measure complete implementations, with different output interfaces. + +## Context 1024: forward-only comparison + +All five losses cover all 1024 target positions and all 50257 output +classes. There are no updates, matching the existing SQL forward run. + +| Implementation | Warm forward loss | +|---|---:| +| SQL shared CPU | 6.5726 s | +| SQL shared MLX | 6.8571 s | +| Native JAX CPU, eager | 0.4639 s | +| Native JAX CPU, JIT | 0.3199 s | +| nanoGPT PyTorch CPU float64 | 0.3453 s | +| nanoGPT PyTorch MPS float32 | 0.03117 s | + +SQL shared MLX is about 220 times slower than native PyTorch MPS on this +forward loss. Full-context SQL backward performance remains unestablished +within the configured memory bound. + +## Interpretation and reproduction + +The SQL path remains far behind native array training. Integer coordinate +processing, Arrow storage/conversion, planning and large relational +backward intermediates dominate. GPU matrix and attention kernels alone +do not remove those costs. The training comparison exposes a much larger +gap than the forward-only comparison. See the +[SQL stage profile and results](nanogpt-shared.md). + +Versions: JAX/jaxlib 0.12.0.1, PyTorch 2.15.0, NumPy 2.5.4. JAX devices +are CPU-only in this environment. PyTorch was installed in a temporary +environment, leaving the checkout's dependencies unchanged. PyTorch CPU +uses 30 threads. MPS timing calls `torch.mps.synchronize()` and JAX timing +blocks on all returned state. + +```sh +git clone https://github.com/karpathy/nanoGPT.git /private/tmp/xql-upstream-nanogpt +git -C /private/tmp/xql-upstream-nanogpt checkout 3adf61e154c3fe3fca428ad6bc3818b27a3b8291 +uv venv --python "$PWD/.venv/bin/python" /private/tmp/xql-framework-venv +uv pip install --python /private/tmp/xql-framework-venv/bin/python torch==2.15.0 + +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD:$PWD/.venv/lib/python3.12/site-packages" \ + /private/tmp/xql-framework-venv/bin/python benchmarks/apple/compare_nanogpt_frameworks.py \ + --backend jax_jit --context 16 --steps 5 \ + --output benchmarks/apple/local-jax-jit.json + +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD:$PWD/.venv/lib/python3.12/site-packages" \ + /private/tmp/xql-framework-venv/bin/python benchmarks/apple/compare_nanogpt_frameworks.py \ + --backend torch_mps --context 16 --steps 5 \ + --nanogpt-source /private/tmp/xql-upstream-nanogpt \ + --output benchmarks/apple/local-torch-mps.json +``` + +Use `--context 1024 --forward-only` for the larger forward comparison. +The harness also supports `jax_eager`, `jax_separate` and `torch_cpu`. +[All results and ratios](nanogpt-framework-summary.json), +[serial run records](framework-runs.json), +[JAX training result](framework-gpt2-16-jax_jit.json), +[PyTorch MPS training result](framework-gpt2-16-torch_mps.json). +Source revisions/hashes, losses, errors and per-step timings are retained +alongside each run's JSON and log. diff --git a/benchmarks/apple/nanogpt-plan-breakdown.json b/benchmarks/apple/nanogpt-plan-breakdown.json new file mode 100644 index 00000000..ca19fde9 --- /dev/null +++ b/benchmarks/apple/nanogpt-plan-breakdown.json @@ -0,0 +1,122 @@ +{ + "datafusion": { + "stages": [ + { + "sql_dataframe_seconds": 0.01304174994584173, + "physical_planning_seconds": 0.9503352920291945, + "execute_existing_plan_seconds": 0.2082049580058083, + "loss": 4.1792514509261265 + }, + { + "sql_dataframe_seconds": 0.015322582912631333, + "physical_planning_seconds": 0.9564947080798447, + "execute_existing_plan_seconds": 0.1604719590395689, + "loss": 4.1792514509261265 + }, + { + "sql_dataframe_seconds": 0.013111292035318911, + "physical_planning_seconds": 0.9356817919760942, + "execute_existing_plan_seconds": 0.1561511670006439, + "loss": 4.1792514509261265 + }, + { + "sql_dataframe_seconds": 0.012777541996911168, + "physical_planning_seconds": 0.9527803340461105, + "execute_existing_plan_seconds": 0.16131829202640802, + "loss": 4.1792514509261265 + } + ], + "physical_operators": 9882, + "operator_types": { + "ProjectionExec": 2229, + "AggregateExec": 1050, + "HashJoinExec": 2988, + "WindowAggExec": 81, + "SortExec": 81, + "DataSourceExec": 2989, + "FilterExec": 464 + } + }, + "duckdb": { + "stages": [ + { + "relation_binding_seconds": 0.0006599589250981808, + "execute_relation_including_physical_planning_seconds": 0.030375082977116108, + "loss": 4.179251450926128 + }, + { + "relation_binding_seconds": 0.0008465000428259373, + "execute_relation_including_physical_planning_seconds": 0.02674108394421637, + "loss": 4.179251450926128 + }, + { + "relation_binding_seconds": 0.0008734579896554351, + "execute_relation_including_physical_planning_seconds": 0.024813542026095092, + "loss": 4.179251450926127 + }, + { + "relation_binding_seconds": 0.000853791949339211, + "execute_relation_including_physical_planning_seconds": 0.023691624985076487, + "loss": 4.179251450926128 + } + ], + "physical_operators": 126, + "operator_types": { + "CTE": 9, + "UNGROUPED_AGGREGATE": 1, + "PROJECTION": 31, + "HASH_JOIN": 28, + "HASH_GROUP_BY": 17, + "WINDOW": 2, + "CTE_SCAN": 24, + "ARROW_SCAN": 14 + }, + "ctes": [ + { + "CTE Name": "predicted", + "Table Index": "0", + "Estimated Cardinality": "1" + }, + { + "CTE Name": "x0", + "Table Index": "4", + "Estimated Cardinality": "0" + }, + { + "CTE Name": "l0_ln1_mean", + "Table Index": "5", + "Estimated Cardinality": "0" + }, + { + "CTE Name": "l0_ln1", + "Table Index": "7", + "Estimated Cardinality": "0" + }, + { + "CTE Name": "l0_weights", + "Table Index": "12", + "Estimated Cardinality": "0" + }, + { + "CTE Name": "l0_mid", + "Table Index": "16", + "Estimated Cardinality": "0" + }, + { + "CTE Name": "l0_ln2_mean", + "Table Index": "17", + "Estimated Cardinality": "0" + }, + { + "CTE Name": "x1", + "Table Index": "23", + "Estimated Cardinality": "0" + }, + { + "CTE Name": "final_mean", + "Table Index": "24", + "Estimated Cardinality": "0" + } + ] + } +} diff --git a/benchmarks/apple/nanogpt-shared-summary.json b/benchmarks/apple/nanogpt-shared-summary.json new file mode 100644 index 00000000..2f134429 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-summary.json @@ -0,0 +1,1045 @@ +{ + "method": "Warm mean of paired loss and train API timings at steps 1-4; step 0 excluded. Forward-only timings contain losses only. Failed runs excluded from performance comparisons.", + "runs": [ + { + "host": "datafusion", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "datafusion", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-datafusion.log", + "csv": "nanogpt-shared-v3-tiny-datafusion.csv", + "peak_rss_gib": 1.084442138671875, + "status": "passed", + "started_utc": "2026-11-01T10:48:19.089914+00:00", + "exit_code": 0, + "wall_seconds": 14.230142958112992, + "checked_rows": 10, + "case": "tiny", + "loss_warm_seconds": 1.0925903437309898, + "loss_all_seconds": 5.550554624991491, + "train_warm_seconds": 0.16717918749782257, + "train_all_seconds": 1.1127010000636801, + "warm_seconds": 1.2597695312288124, + "all_api_seconds": 6.6632556250551715, + "measured_total_seconds": 6.794837421388365, + "config": { + "layers": "1", + "embed": "16", + "heads": "2", + "context": "16", + "vocab": "65", + "parameters": "4416", + "batch": "4", + "optimizer": "sgd", + "sgd_learning_rate": "0.5", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": {} + }, + { + "host": "duckdb", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "duckdb", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-duckdb.log", + "csv": "nanogpt-shared-v3-tiny-duckdb.csv", + "peak_rss_gib": 0.7633819580078125, + "status": "passed", + "started_utc": "2026-11-01T10:48:33.320634+00:00", + "exit_code": 0, + "wall_seconds": 4.258987625013106, + "checked_rows": 10, + "case": "tiny", + "loss_warm_seconds": 0.02809041674481705, + "loss_all_seconds": 0.14293133397586644, + "train_warm_seconds": 0.27745356247760355, + "train_all_seconds": 1.415508957928978, + "warm_seconds": 0.3055439792224206, + "all_api_seconds": 1.5584402919048443, + "measured_total_seconds": 1.6013755867024884, + "config": { + "layers": "1", + "embed": "16", + "heads": "2", + "context": "16", + "vocab": "65", + "parameters": "4416", + "batch": "4", + "optimizer": "sgd", + "sgd_learning_rate": "0.5", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": {} + }, + { + "host": "cpu_forced", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "cpu_forced", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-cpu_forced.log", + "csv": "nanogpt-shared-v3-tiny-cpu_forced.csv", + "peak_rss_gib": 0.940399169921875, + "status": "passed", + "started_utc": "2026-11-01T10:48:37.580129+00:00", + "exit_code": 0, + "wall_seconds": 5.64062604191713, + "checked_rows": 10, + "case": "tiny", + "loss_warm_seconds": 0.049882802268257365, + "loss_all_seconds": 1.428845834103413, + "train_warm_seconds": 0.26416770822834224, + "train_all_seconds": 1.61255974881351, + "warm_seconds": 0.3140505104965996, + "all_api_seconds": 3.041405582916923, + "measured_total_seconds": 3.0762215801514685, + "config": { + "layers": "1", + "embed": "16", + "heads": "2", + "context": "16", + "vocab": "65", + "parameters": "4416", + "batch": "4", + "optimizer": "sgd", + "sgd_learning_rate": "0.5", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 1015.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0 + } + }, + { + "host": "gpu_forced", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "gpu_forced", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-gpu_forced.log", + "csv": "nanogpt-shared-v3-tiny-gpu_forced.csv", + "peak_rss_gib": 0.952362060546875, + "status": "passed", + "started_utc": "2026-11-01T10:48:43.221551+00:00", + "exit_code": 0, + "wall_seconds": 5.972789291990921, + "checked_rows": 10, + "case": "tiny", + "loss_warm_seconds": 0.08501578099094331, + "loss_all_seconds": 1.600211248965934, + "train_warm_seconds": 0.2860841774963774, + "train_all_seconds": 1.7828770430060104, + "warm_seconds": 0.3710999584873207, + "all_api_seconds": 3.3830882919719443, + "measured_total_seconds": 3.4174957488430664, + "config": { + "layers": "1", + "embed": "16", + "heads": "2", + "context": "16", + "vocab": "65", + "parameters": "4416", + "batch": "4", + "optimizer": "sgd", + "sgd_learning_rate": "0.5", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 1015.0, + "gpu_attention": 200.0 + } + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-shared_cpu.log", + "csv": "nanogpt-shared-v3-tiny-shared_cpu.csv", + "peak_rss_gib": 0.628387451171875, + "status": "passed", + "started_utc": "2026-11-01T10:48:49.194917+00:00", + "exit_code": 0, + "wall_seconds": 4.014455707976595, + "checked_rows": 10, + "case": "tiny", + "loss_warm_seconds": 0.0059946770197711885, + "loss_all_seconds": 0.07138104108162224, + "train_warm_seconds": 0.19978481272119097, + "train_all_seconds": 1.287024250952527, + "warm_seconds": 0.20577948974096216, + "all_api_seconds": 1.3584052920341492, + "measured_total_seconds": 1.370951619115658, + "config": { + "layers": "1", + "embed": "16", + "heads": "2", + "context": "16", + "vocab": "65", + "parameters": "4416", + "batch": "4", + "optimizer": "sgd", + "sgd_learning_rate": "0.5", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 160.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0, + "physical_plans": 138.0, + "plan_executions": 430.0, + "planning_seconds": 0.6167673374293372, + "execution_seconds": 0.5663185446755961, + "stage_storage_seconds": 0.036390537046827376, + "key_cache_bytes": 571392.0, + "key_copy_bytes_avoided": 4918144.0 + } + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-shared_gpu.log", + "csv": "nanogpt-shared-v3-tiny-shared_gpu.csv", + "peak_rss_gib": 0.6777496337890625, + "status": "passed", + "started_utc": "2026-11-01T10:48:53.209970+00:00", + "exit_code": 0, + "wall_seconds": 4.1090395000064746, + "checked_rows": 10, + "case": "tiny", + "loss_warm_seconds": 0.010802604490891099, + "loss_all_seconds": 0.10134358401410282, + "train_warm_seconds": 0.2128371979633812, + "train_all_seconds": 1.4291370417922735, + "warm_seconds": 0.22363980245427229, + "all_api_seconds": 1.5304806258063763, + "measured_total_seconds": 1.5432380823185667, + "config": { + "layers": "1", + "embed": "16", + "heads": "2", + "context": "16", + "vocab": "65", + "parameters": "4416", + "batch": "4", + "optimizer": "sgd", + "sgd_learning_rate": "0.5", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 160.0, + "gpu_attention": 5.0, + "physical_plans": 138.0, + "plan_executions": 430.0, + "planning_seconds": 0.6424464656738564, + "execution_seconds": 0.6964817652478814, + "stage_storage_seconds": 0.03924583981279284, + "key_cache_bytes": 571392.0, + "key_copy_bytes_avoided": 4918144.0 + } + }, + { + "host": "cpu_forced", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "cpu_forced", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-demo-cpu_forced.log", + "csv": "nanogpt-shared-v3-demo-cpu_forced.csv", + "peak_rss_gib": 2.0538330078125, + "status": "passed", + "started_utc": "2026-11-01T10:48:57.343307+00:00", + "exit_code": 0, + "wall_seconds": 6.822247041971423, + "checked_rows": 10, + "case": "demo", + "loss_warm_seconds": 0.15433374998974614, + "loss_all_seconds": 1.994750665850006, + "train_warm_seconds": 0.39687466676696204, + "train_all_seconds": 2.2844609580934048, + "warm_seconds": 0.5512084167567082, + "all_api_seconds": 4.279211623943411, + "measured_total_seconds": 4.314747250638902, + "config": { + "layers": "1", + "embed": "128", + "heads": "4", + "context": "64", + "vocab": "65", + "parameters": "213504", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 1015.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0 + } + }, + { + "host": "gpu_forced", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "gpu_forced", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-demo-gpu_forced.log", + "csv": "nanogpt-shared-v3-demo-gpu_forced.csv", + "peak_rss_gib": 2.1047821044921875, + "status": "passed", + "started_utc": "2026-11-01T10:49:04.166131+00:00", + "exit_code": 0, + "wall_seconds": 7.168097125017084, + "checked_rows": 10, + "case": "demo", + "loss_warm_seconds": 0.18047364600352012, + "loss_all_seconds": 2.2357670429628342, + "train_warm_seconds": 0.42031162526109256, + "train_all_seconds": 2.4041642510565, + "warm_seconds": 0.6007852712646127, + "all_api_seconds": 4.639931294019334, + "measured_total_seconds": 4.675838371971622, + "config": { + "layers": "1", + "embed": "128", + "heads": "4", + "context": "64", + "vocab": "65", + "parameters": "213504", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 1015.0, + "gpu_attention": 200.0 + } + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-demo-shared_cpu.log", + "csv": "nanogpt-shared-v3-demo-shared_cpu.csv", + "peak_rss_gib": 1.028533935546875, + "status": "passed", + "started_utc": "2026-11-01T10:49:11.334747+00:00", + "exit_code": 0, + "wall_seconds": 4.974934917059727, + "checked_rows": 10, + "case": "demo", + "loss_warm_seconds": 0.020543166989227757, + "loss_all_seconds": 0.141694750986062, + "train_warm_seconds": 0.410445937741315, + "train_all_seconds": 2.3422849599737674, + "warm_seconds": 0.43098910473054275, + "all_api_seconds": 2.4839797109598294, + "measured_total_seconds": 2.497708463575691, + "config": { + "layers": "1", + "embed": "128", + "heads": "4", + "context": "64", + "vocab": "65", + "parameters": "213504", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 160.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0, + "physical_plans": 138.0, + "plan_executions": 430.0, + "planning_seconds": 0.6375595483696088, + "execution_seconds": 1.5532573777018115, + "stage_storage_seconds": 0.11537673976272345, + "key_cache_bytes": 6406656.0, + "key_copy_bytes_avoided": 39252480.0 + } + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-demo-shared_gpu.log", + "csv": "nanogpt-shared-v3-demo-shared_gpu.csv", + "peak_rss_gib": 1.0576171875, + "status": "passed", + "started_utc": "2026-11-01T10:49:16.310295+00:00", + "exit_code": 0, + "wall_seconds": 5.374738791957498, + "checked_rows": 10, + "case": "demo", + "loss_warm_seconds": 0.02553020851337351, + "loss_all_seconds": 0.1757639170391485, + "train_warm_seconds": 0.4564060004777275, + "train_all_seconds": 2.585350918932818, + "warm_seconds": 0.481936208991101, + "all_api_seconds": 2.7611148359719664, + "measured_total_seconds": 2.7760195907903835, + "config": { + "layers": "1", + "embed": "128", + "heads": "4", + "context": "64", + "vocab": "65", + "parameters": "213504", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 160.0, + "gpu_attention": 5.0, + "physical_plans": 138.0, + "plan_executions": 430.0, + "planning_seconds": 0.669295079424046, + "execution_seconds": 1.7730445764027536, + "stage_storage_seconds": 0.12421796540729702, + "key_cache_bytes": 6406656.0, + "key_copy_bytes_avoided": 39252480.0 + } + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "shakespeare", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-shakespeare-shared_cpu.log", + "csv": "nanogpt-shared-v3-shakespeare-shared_cpu.csv", + "peak_rss_gib": 9.9989013671875, + "status": "passed", + "started_utc": "2026-11-01T10:49:21.710678+00:00", + "exit_code": 0, + "wall_seconds": 153.34261495794635, + "checked_rows": 10, + "case": "shakespeare", + "loss_warm_seconds": 0.556670270714676, + "loss_all_seconds": 2.8871656658593565, + "train_warm_seconds": 28.699999969045166, + "train_all_seconds": 145.48821391817182, + "warm_seconds": 29.256670239759842, + "all_api_seconds": 148.37537958403118, + "measured_total_seconds": 148.43053996062372, + "config": { + "layers": "6", + "embed": "384", + "heads": "6", + "context": "256", + "vocab": "65", + "parameters": "10745088", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 410.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0, + "physical_plans": 1795.0, + "plan_executions": 2775.0, + "planning_seconds": 19.42623228521552, + "execution_seconds": 100.87469962309115, + "stage_storage_seconds": 23.41542024689261, + "key_cache_bytes": 263093248.0, + "key_copy_bytes_avoided": 5296482304.0 + } + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "shakespeare", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-shakespeare-shared_gpu.log", + "csv": "nanogpt-shared-v3-shakespeare-shared_gpu.csv", + "peak_rss_gib": 10.218841552734375, + "status": "passed", + "started_utc": "2026-11-01T10:51:55.054654+00:00", + "exit_code": 0, + "wall_seconds": 157.46115179196931, + "checked_rows": 10, + "case": "shakespeare", + "loss_warm_seconds": 0.5914996457868256, + "loss_all_seconds": 3.0487270831363276, + "train_warm_seconds": 29.345046656235354, + "train_all_seconds": 149.26009399991017, + "warm_seconds": 29.93654630202218, + "all_api_seconds": 152.3088210830465, + "measured_total_seconds": 152.36542546015698, + "config": { + "layers": "6", + "embed": "384", + "heads": "6", + "context": "256", + "vocab": "65", + "parameters": "10745088", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 410.0, + "gpu_attention": 30.0, + "physical_plans": 1795.0, + "plan_executions": 2775.0, + "planning_seconds": 20.114735280629247, + "execution_seconds": 103.3905114326626, + "stage_storage_seconds": 23.975200915709138, + "key_cache_bytes": 263093248.0, + "key_copy_bytes_avoided": 5296482304.0 + } + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-16-shared_gpu.log", + "csv": "nanogpt-shared-v3-gpt2-16-shared_gpu.csv", + "peak_rss_gib": 40.02874755859375, + "status": "passed", + "started_utc": "2026-11-01T10:54:32.541752+00:00", + "exit_code": 0, + "wall_seconds": 846.1769028330455, + "checked_rows": 10, + "case": "gpt2-16", + "loss_warm_seconds": 1.3720009690150619, + "loss_all_seconds": 7.187280835118145, + "train_warm_seconds": 162.27428429198335, + "train_all_seconds": 822.448966667871, + "warm_seconds": 163.6462852609984, + "all_api_seconds": 829.6362475029891, + "measured_total_seconds": 829.9728821297176, + "config": { + "layers": "12", + "embed": "768", + "heads": "12", + "context": "16", + "vocab": "50257", + "parameters": "123563520", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 710.0, + "gpu_attention": 60.0, + "physical_plans": 7148.0, + "plan_executions": 9020.0, + "planning_seconds": 131.42177372181322, + "execution_seconds": 504.44527716143057, + "stage_storage_seconds": 169.1986960049253, + "key_cache_bytes": 2892210304.0, + "key_copy_bytes_avoided": 6005381504.0 + } + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-16-shared_cpu.log", + "csv": "nanogpt-shared-v3-gpt2-16-shared_cpu.csv", + "peak_rss_gib": 38.71354675292969, + "status": "passed", + "started_utc": "2026-11-01T11:08:38.723798+00:00", + "exit_code": 0, + "wall_seconds": 846.0968177500181, + "checked_rows": 10, + "case": "gpt2-16", + "loss_warm_seconds": 1.1682787809986621, + "loss_all_seconds": 6.23576199891977, + "train_warm_seconds": 162.86555686450447, + "train_all_seconds": 823.4401903749676, + "warm_seconds": 164.03383564550313, + "all_api_seconds": 829.6759523738874, + "measured_total_seconds": 829.991804452613, + "config": { + "layers": "12", + "embed": "768", + "heads": "12", + "context": "16", + "vocab": "50257", + "parameters": "123563520", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 710.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0, + "physical_plans": 7148.0, + "plan_executions": 9020.0, + "planning_seconds": 130.0049377027899, + "execution_seconds": 505.680548057775, + "stage_storage_seconds": 169.5953018902801, + "key_cache_bytes": 2892210304.0, + "key_copy_bytes_avoided": 6005381504.0 + } + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.csv", + "--forward-only", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-forward-shared_cpu.log", + "csv": "nanogpt-shared-v3-gpt2-forward-shared_cpu.csv", + "peak_rss_gib": 15.54058837890625, + "status": "passed", + "started_utc": "2026-11-01T11:22:44.844329+00:00", + "exit_code": 0, + "wall_seconds": 38.22789254190866, + "checked_rows": 5, + "case": "gpt2-forward", + "loss_warm_seconds": 6.572556927247206, + "loss_all_seconds": 33.19323650095612, + "warm_seconds": 6.572556927247206, + "all_api_seconds": 33.19323650095612, + "measured_total_seconds": 33.268904172233306, + "config": { + "layers": "12", + "embed": "768", + "heads": "12", + "context": "1024", + "vocab": "50257", + "parameters": "124337664", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "1e-09", + "atol": "1e-09" + }, + "counters": { + "cpu_matmul": 365.0, + "gpu_matmul": 0.0, + "gpu_attention": 0.0, + "physical_plans": 65.0, + "plan_executions": 325.0, + "planning_seconds": 0.3588349998462945, + "execution_seconds": 20.173762080725282, + "stage_storage_seconds": 12.39298641658388, + "key_cache_bytes": 1254006784.0, + "key_copy_bytes_avoided": 1914986496.0 + } + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.csv", + "--forward-only", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-forward-shared_gpu.log", + "csv": "nanogpt-shared-v3-gpt2-forward-shared_gpu.csv", + "peak_rss_gib": 15.64544677734375, + "status": "passed", + "started_utc": "2026-11-01T11:23:23.073032+00:00", + "exit_code": 0, + "wall_seconds": 39.74374524992891, + "checked_rows": 5, + "case": "gpt2-forward", + "loss_warm_seconds": 6.857087958022021, + "loss_all_seconds": 34.839587540132925, + "warm_seconds": 6.857087958022021, + "all_api_seconds": 34.839587540132925, + "measured_total_seconds": 34.920842704945244, + "config": { + "layers": "12", + "embed": "768", + "heads": "12", + "context": "1024", + "vocab": "50257", + "parameters": "124337664", + "batch": "1", + "optimizer": "sgd", + "sgd_learning_rate": "0.01", + "rtol": "3e-05", + "atol": "3e-05" + }, + "counters": { + "cpu_matmul": 0.0, + "gpu_matmul": 365.0, + "gpu_attention": 60.0, + "physical_plans": 65.0, + "plan_executions": 325.0, + "planning_seconds": 0.372420794214122, + "execution_seconds": 21.33640320203267, + "stage_storage_seconds": 12.807666914421134, + "key_cache_bytes": 1254006784.0, + "key_copy_bytes_avoided": 1914986496.0 + } + } + ], + "source_sha256": { + "xarray_sql/shared.py": "3f8ad3f0366d5a5677a487d169ed17818e70993141ae1be6ff4d5bda039bac8c", + "xarray_sql/mlx.py": "9f92065ab6a97102c280582409917a55ccadbff7df5dfefcdd232e22bf3583de", + "xarray_sql/sql.py": "5b99950302641bd8ea2190245f808855cc9df9f64a0b473baaac79fb4eb4d499", + "src/shared.rs": "1d982d19389a63863f5140a5bb53977792eb3e72aea6d83ca3e9a15366e74dc5", + "benchmarks/ml/02_nanogpt.py": "80a5a55b39161f7e45fa717ec2d60aabaeee3edf90d19ef8ad736afa3c678841", + "benchmarks/ml/_hosts.py": "a92210a6ee0451900a77e9fe8b61403dbec4dd960d306a714cb0ee4fb103ba53", + "benchmarks/apple/run_nanogpt_sizes.py": "a32ab028f8faf9384dd2c5e7957c6565fcd61a5217f58819604fc1447f36fa3c", + "accelerators/einfold_mlx/dense-backend.patch": "837964f53844563b7a4cfee1d26d65cf424492932912e73c2bac328da4a60ec1" + }, + "speedups": { + "demo:cpu_forced/shared_cpu": 1.2789381696814062, + "demo:gpu_forced/shared_gpu": 1.2466074556263653, + "demo:shared_cpu/shared_gpu": 0.8942866227727267, + "tiny:datafusion/shared_cpu": 6.121939231235467, + "tiny:duckdb/shared_cpu": 1.4848126001626463, + "tiny:cpu_forced/shared_cpu": 1.5261506911691265, + "tiny:gpu_forced/shared_gpu": 1.6593645425134, + "tiny:shared_cpu/shared_gpu": 0.9201380410941741, + "gpt2-forward:shared_cpu/shared_gpu": 0.9585055591357924, + "gpt2-16:shared_cpu/shared_gpu": 1.0023682198705985, + "shakespeare:shared_cpu/shared_gpu": 0.9772894289340113 + }, + "environment": { + "hardware": "Apple M5 Ultra", + "cpu_cores": 30, + "gpu_cores": 64, + "memory_gib": 256, + "platform": "macOS-27.0.1-arm64-arm-64bit", + "python": "3.12.14", + "native_datafusion": "55.2.0", + "MLX_ENABLE_TF32": "0", + "cpu_partitions": 30, + "versions": { + "datafusion": "55.0.0", + "ddxdb": "0.4.6", + "einfold": "0.2.0+xqlmlx.3", + "mlx": "0.32.4", + "numpy": "2.5.4", + "pyarrow": "26.0.0", + "xarray": "2026.9.0", + "duckdb": "1.5.6" + } + }, + "source_hashes_recorded_utc": "2026-11-01T11:27:03.183697+00:00", + "source_hashes_scope": "Final numerical implementation; source code was unchanged between the final correctness suite and these timing runs.", + "native_extension_sha256": "b08b896466b75d6fadbb081e6f95c113d05dad5323ead815ae86e3acf05aa0b9", + "validation": { + "python_passed": 637, + "python_skipped": 277, + "integration_deselected": 102, + "native_passed": 60, + "native_ignored": 4, + "native_excluded": 1, + "benchmark_checked_records": 150 + } +} diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.csv b/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.csv new file mode 100644 index 00000000..ad1b4e9a --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",cpu_forced,1,1.3774156658910215,ok (0e+00),4.185923,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,178,0,0,40,,,, +nanogpt,"train (sgd), step 0",cpu_forced,1,0.6969622910255566,ok (1e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,203,0,0,40,0.005011956789530814,0.022940042079426348,1,11 +nanogpt,"loss, step 1",cpu_forced,1,0.1559582919580862,ok (2e-16),4.111482,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,381,0,0,80,,,1,12 +nanogpt,"train (sgd), step 1",cpu_forced,1,0.39953866705764085,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,406,0,0,80,0.006960206548683345,0.022970542078837752,1,23 +nanogpt,"loss, step 2",cpu_forced,1,0.1518029160797596,ok (2e-16),4.147195,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,584,0,0,120,,,1,24 +nanogpt,"train (sgd), step 2",cpu_forced,1,0.39952995895873755,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,609,0,0,120,0.008780081290751696,0.022990667028352618,1,35 +nanogpt,"loss, step 3",cpu_forced,1,0.15667470800690353,ok (0e+00),4.065422,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,787,0,0,160,,,1,36 +nanogpt,"train (sgd), step 3",cpu_forced,1,0.393339250003919,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,812,0,0,160,0.010600750567391515,0.023009917000308633,1,47 +nanogpt,"loss, step 4",cpu_forced,1,0.15289908391423523,ok (0e+00),4.03082,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,990,0,0,200,,,1,48 +nanogpt,"train (sgd), step 4",cpu_forced,1,0.39509079104755074,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,1015,0,0,200,0.012504501733928919,0.023031124961562455,1,59 diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.log b/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.log new file mode 100644 index 00000000..3d29c12e --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=64, heads=4, embed=128, forward_only=False + loss, step 0 cpu_forced batch=1 1.377s + train (sgd), step 0 cpu_forced batch=1 0.697s + loss, step 1 cpu_forced batch=1 0.156s + train (sgd), step 1 cpu_forced batch=1 0.400s + loss, step 2 cpu_forced batch=1 0.152s + train (sgd), step 2 cpu_forced batch=1 0.400s + loss, step 3 cpu_forced batch=1 0.157s + train (sgd), step 3 cpu_forced batch=1 0.393s + loss, step 4 cpu_forced batch=1 0.153s + train (sgd), step 4 cpu_forced batch=1 0.395s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | cpu_forced | 1 | 1.377 | ok (0e+00) | 4.186 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 178 | 0 | 0 | 40 | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | cpu_forced | 1 | 0.697 | ok (1e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 203 | 0 | 0 | 40 | 0.005 | 0.023 | 1 | 11 | +| nanogpt | loss, step 1 | cpu_forced | 1 | 0.156 | ok (2e-16) | 4.111 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 381 | 0 | 0 | 80 | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | cpu_forced | 1 | 0.400 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 406 | 0 | 0 | 80 | 0.007 | 0.023 | 1 | 23 | +| nanogpt | loss, step 2 | cpu_forced | 1 | 0.152 | ok (2e-16) | 4.147 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 584 | 0 | 0 | 120 | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | cpu_forced | 1 | 0.400 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 609 | 0 | 0 | 120 | 0.009 | 0.023 | 1 | 35 | +| nanogpt | loss, step 3 | cpu_forced | 1 | 0.157 | ok (0e+00) | 4.065 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 787 | 0 | 0 | 160 | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | cpu_forced | 1 | 0.393 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 812 | 0 | 0 | 160 | 0.011 | 0.023 | 1 | 47 | +| nanogpt | loss, step 4 | cpu_forced | 1 | 0.153 | ok (0e+00) | 4.031 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 990 | 0 | 0 | 200 | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | cpu_forced | 1 | 0.395 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 1015 | 0 | 0 | 200 | 0.013 | 0.023 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.csv b/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.csv new file mode 100644 index 00000000..75d027ad --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",gpu_forced,1,1.5138724589487538,ok (3e-09),4.185923,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,178,40,0,,,, +nanogpt,"train (sgd), step 0",gpu_forced,1,0.7229177500121295,ok (7e-09),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,203,40,0,0.005215372075326741,0.023196999914944172,1,11 +nanogpt,"loss, step 1",gpu_forced,1,0.1805570829892531,ok (3e-10),4.111482,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,381,80,0,,,1,12 +nanogpt,"train (sgd), step 1",gpu_forced,1,0.42121950001455843,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,406,80,0,0.0070047477493062615,0.023220749921165407,1,23 +nanogpt,"loss, step 2",gpu_forced,1,0.18294504203367978,ok (3e-09),4.147195,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,584,120,0,,,1,24 +nanogpt,"train (sgd), step 2",gpu_forced,1,0.42531804204918444,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,609,120,0,0.008877455838955939,0.02324062492698431,1,35 +nanogpt,"loss, step 3",gpu_forced,1,0.1787009589606896,ok (2e-09),4.065422,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,787,160,0,,,1,36 +nanogpt,"train (sgd), step 3",gpu_forced,1,0.41308645892422646,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,812,160,0,0.010756538948044181,0.02326087502297014,1,47 +nanogpt,"loss, step 4",gpu_forced,1,0.17969150003045797,ok (2e-09),4.03082,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,990,200,0,,,1,48 +nanogpt,"train (sgd), step 4",gpu_forced,1,0.4216225000564009,ok (2e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,1015,200,0,0.012626537005417049,0.023280540946871042,1,59 diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.log b/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.log new file mode 100644 index 00000000..de2a9317 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=64, heads=4, embed=128, forward_only=False + loss, step 0 gpu_forced batch=1 1.514s + train (sgd), step 0 gpu_forced batch=1 0.723s + loss, step 1 gpu_forced batch=1 0.181s + train (sgd), step 1 gpu_forced batch=1 0.421s + loss, step 2 gpu_forced batch=1 0.183s + train (sgd), step 2 gpu_forced batch=1 0.425s + loss, step 3 gpu_forced batch=1 0.179s + train (sgd), step 3 gpu_forced batch=1 0.413s + loss, step 4 gpu_forced batch=1 0.180s + train (sgd), step 4 gpu_forced batch=1 0.422s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | gpu_forced | 1 | 1.514 | ok (3e-09) | 4.186 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 178 | 40 | 0 | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | gpu_forced | 1 | 0.723 | ok (7e-09) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 203 | 40 | 0 | 0.005 | 0.023 | 1 | 11 | +| nanogpt | loss, step 1 | gpu_forced | 1 | 0.181 | ok (3e-10) | 4.111 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 381 | 80 | 0 | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | gpu_forced | 1 | 0.421 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 406 | 80 | 0 | 0.007 | 0.023 | 1 | 23 | +| nanogpt | loss, step 2 | gpu_forced | 1 | 0.183 | ok (3e-09) | 4.147 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 584 | 120 | 0 | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | gpu_forced | 1 | 0.425 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 609 | 120 | 0 | 0.009 | 0.023 | 1 | 35 | +| nanogpt | loss, step 3 | gpu_forced | 1 | 0.179 | ok (2e-09) | 4.065 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 787 | 160 | 0 | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | gpu_forced | 1 | 0.413 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 812 | 160 | 0 | 0.011 | 0.023 | 1 | 47 | +| nanogpt | loss, step 4 | gpu_forced | 1 | 0.180 | ok (2e-09) | 4.031 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 990 | 200 | 0 | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | gpu_forced | 1 | 0.422 | ok (2e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 1015 | 200 | 0 | 0.013 | 0.023 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-runs.json b/benchmarks/apple/nanogpt-shared-v3-demo-runs.json new file mode 100644 index 00000000..b169d60c --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-runs.json @@ -0,0 +1,126 @@ +[ + { + "host": "cpu_forced", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "cpu_forced", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-cpu_forced.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-demo-cpu_forced.log", + "csv": "nanogpt-shared-v3-demo-cpu_forced.csv", + "peak_rss_gib": 2.0538330078125, + "status": "passed", + "started_utc": "2026-11-01T10:48:57.343307+00:00", + "exit_code": 0, + "wall_seconds": 6.822247041971423, + "checked_rows": 10 + }, + { + "host": "gpu_forced", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "gpu_forced", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-gpu_forced.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-demo-gpu_forced.log", + "csv": "nanogpt-shared-v3-demo-gpu_forced.csv", + "peak_rss_gib": 2.1047821044921875, + "status": "passed", + "started_utc": "2026-11-01T10:49:04.166131+00:00", + "exit_code": 0, + "wall_seconds": 7.168097125017084, + "checked_rows": 10 + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-demo-shared_cpu.log", + "csv": "nanogpt-shared-v3-demo-shared_cpu.csv", + "peak_rss_gib": 1.028533935546875, + "status": "passed", + "started_utc": "2026-11-01T10:49:11.334747+00:00", + "exit_code": 0, + "wall_seconds": 4.974934917059727, + "checked_rows": 10 + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "demo", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-demo-shared_gpu.log", + "csv": "nanogpt-shared-v3-demo-shared_gpu.csv", + "peak_rss_gib": 1.0576171875, + "status": "passed", + "started_utc": "2026-11-01T10:49:16.310295+00:00", + "exit_code": 0, + "wall_seconds": 5.374738791957498, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.csv b/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.csv new file mode 100644 index 00000000..b9edd85d --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,stage_buffer_writes,stage_replacements_keys_or_nulls +nanogpt,"loss, step 0",shared_cpu,1,0.05952208302915096,ok (0e+00),4.185923,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,7,0,0,1,10,10,9,16,691712,10,0.03477808309253305,0.019424956873990595,0.0010527089470997453,297472,,,,,, +nanogpt,"train (sgd), step 0",shared_cpu,1,0.7005012090085074,ok (1e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,32,0,0,1,86,86,63,208,13300224,10,0.32974554400425404,0.3028500401414931,0.02451970591209829,5620224,0.004958042758516967,0.0008636669954285026,1,11,, +nanogpt,"loss, step 1",shared_cpu,1,0.020905375014990568,ok (2e-16),4.111482,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,39,0,0,2,86,96,63,208,13300224,10,0.32974554400425404,0.32258108200039715,0.025313622900284827,5620224,,,1,12,9, +nanogpt,"train (sgd), step 1",shared_cpu,1,0.39800829195883125,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,64,0,0,2,99,172,73,307,19788288,10,0.40431675186846405,0.6063004543539137,0.04733012092765421,5816832,0.006836917717009783,0.0008967920439317822,1,23,53,10 +nanogpt,"loss, step 2",shared_cpu,1,0.021257083979435265,ok (2e-16),4.147195,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,71,0,0,3,99,182,73,307,19788288,10,0.40431675186846405,0.6263974122703075,0.04802757874131203,5816832,,,1,24,62,10 +nanogpt,"train (sgd), step 2",shared_cpu,1,0.41999837500043213,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,96,0,0,3,112,258,83,406,26276352,10,0.4836999209364876,0.9264447934692726,0.06990620295982808,6013440,0.008945833425968885,0.0009305001003667712,1,35,106,20 +nanogpt,"loss, step 3",shared_cpu,1,0.020761874970048666,ok (0e+00),4.065422,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,103,0,0,4,112,268,83,406,26276352,10,0.4836999209364876,0.9461334196384996,0.07060595403891057,6013440,,,1,36,115,20 +nanogpt,"train (sgd), step 3",shared_cpu,1,0.4101320420159027,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,128,0,0,4,125,344,93,505,32764416,10,0.5606922972947359,1.2386524591129273,0.09279836714267731,6210048,0.0108249606564641,0.0009597500320523977,1,47,159,30 +nanogpt,"loss, step 4",shared_cpu,1,0.019248333992436528,ok (0e+00),4.03082,,1e-09,1e-09,1,128,4,64,65,213504,sgd,0.01,False,13,1,135,0,0,5,125,354,93,505,32764416,10,0.5606922972947359,1.2568477501627058,0.0934464072342962,6210048,,,1,48,168,30 +nanogpt,"train (sgd), step 4",shared_cpu,1,0.4136450419900939,ok (2e-16),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,160,0,0,5,138,430,103,604,39252480,10,0.6375595483696088,1.5532573777018115,0.11537673976272345,6406656,0.012739294557832181,0.0009894580580294132,1,59,212,40 diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.log b/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.log new file mode 100644 index 00000000..f192fef2 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=64, heads=4, embed=128, forward_only=False + loss, step 0 shared_cpu batch=1 0.060s + train (sgd), step 0 shared_cpu batch=1 0.701s + loss, step 1 shared_cpu batch=1 0.021s + train (sgd), step 1 shared_cpu batch=1 0.398s + loss, step 2 shared_cpu batch=1 0.021s + train (sgd), step 2 shared_cpu batch=1 0.420s + loss, step 3 shared_cpu batch=1 0.021s + train (sgd), step 3 shared_cpu batch=1 0.410s + loss, step 4 shared_cpu batch=1 0.019s + train (sgd), step 4 shared_cpu batch=1 0.414s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | stage_buffer_writes | stage_replacements_keys_or_nulls | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_cpu | 1 | 0.060 | ok (0e+00) | 4.186 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 7 | 0 | 0 | 1 | 10 | 10 | 9 | 16 | 691712 | 10 | 0.035 | 0.019 | 0.001 | 297472 | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_cpu | 1 | 0.701 | ok (1e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 32 | 0 | 0 | 1 | 86 | 86 | 63 | 208 | 13300224 | 10 | 0.330 | 0.303 | 0.025 | 5620224 | 0.005 | 0.001 | 1 | 11 | n/a | n/a | +| nanogpt | loss, step 1 | shared_cpu | 1 | 0.021 | ok (2e-16) | 4.111 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 39 | 0 | 0 | 2 | 86 | 96 | 63 | 208 | 13300224 | 10 | 0.330 | 0.323 | 0.025 | 5620224 | n/a | n/a | 1 | 12 | 9 | n/a | +| nanogpt | train (sgd), step 1 | shared_cpu | 1 | 0.398 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 64 | 0 | 0 | 2 | 99 | 172 | 73 | 307 | 19788288 | 10 | 0.404 | 0.606 | 0.047 | 5816832 | 0.007 | 0.001 | 1 | 23 | 53 | 10 | +| nanogpt | loss, step 2 | shared_cpu | 1 | 0.021 | ok (2e-16) | 4.147 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 71 | 0 | 0 | 3 | 99 | 182 | 73 | 307 | 19788288 | 10 | 0.404 | 0.626 | 0.048 | 5816832 | n/a | n/a | 1 | 24 | 62 | 10 | +| nanogpt | train (sgd), step 2 | shared_cpu | 1 | 0.420 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 96 | 0 | 0 | 3 | 112 | 258 | 83 | 406 | 26276352 | 10 | 0.484 | 0.926 | 0.070 | 6013440 | 0.009 | 0.001 | 1 | 35 | 106 | 20 | +| nanogpt | loss, step 3 | shared_cpu | 1 | 0.021 | ok (0e+00) | 4.065 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 103 | 0 | 0 | 4 | 112 | 268 | 83 | 406 | 26276352 | 10 | 0.484 | 0.946 | 0.071 | 6013440 | n/a | n/a | 1 | 36 | 115 | 20 | +| nanogpt | train (sgd), step 3 | shared_cpu | 1 | 0.410 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 128 | 0 | 0 | 4 | 125 | 344 | 93 | 505 | 32764416 | 10 | 0.561 | 1.239 | 0.093 | 6210048 | 0.011 | 0.001 | 1 | 47 | 159 | 30 | +| nanogpt | loss, step 4 | shared_cpu | 1 | 0.019 | ok (0e+00) | 4.031 | n/a | 1e-09 | 1e-09 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 135 | 0 | 0 | 5 | 125 | 354 | 93 | 505 | 32764416 | 10 | 0.561 | 1.257 | 0.093 | 6210048 | n/a | n/a | 1 | 48 | 168 | 30 | +| nanogpt | train (sgd), step 4 | shared_cpu | 1 | 0.414 | ok (2e-16) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 160 | 0 | 0 | 5 | 138 | 430 | 103 | 604 | 39252480 | 10 | 0.638 | 1.553 | 0.115 | 6406656 | 0.013 | 0.001 | 1 | 59 | 212 | 40 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-shared_cpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.csv b/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.csv new file mode 100644 index 00000000..83de2c8f --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,stage_buffer_writes,stage_replacements_keys_or_nulls +nanogpt,"loss, step 0",shared_gpu,1,0.07364308298565447,ok (3e-09),4.185923,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,7,1,0,10,10,9,16,691712,10,0.0351487488951534,0.03315420786384493,0.0010580827947705984,297472,,,,,, +nanogpt,"train (sgd), step 0",shared_gpu,1,0.7597269170219079,ok (7e-09),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,32,1,0,86,86,63,208,13300224,10,0.34443778544664383,0.3500732510583475,0.02551862201653421,5620224,0.005821418599225581,0.0008630419615656137,1,11,, +nanogpt,"loss, step 1",shared_gpu,1,0.02672970900312066,ok (3e-10),4.111482,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,39,2,0,86,96,63,208,13300224,10,0.34443778544664383,0.3754327918868512,0.02641387307085097,5620224,,,1,12,9, +nanogpt,"train (sgd), step 1",shared_gpu,1,0.4349845419637859,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,64,2,0,99,172,73,307,19788288,10,0.4219699113164097,0.6915260396199301,0.04900387616362423,5816832,0.0077131298603489995,0.0008990420028567314,1,23,53,10 +nanogpt,"loss, step 2",shared_gpu,1,0.025595374987460673,ok (3e-09),4.147195,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,71,3,0,99,182,73,307,19788288,10,0.4219699113164097,0.7159901646664366,0.049688293249346316,5816832,,,1,24,62,10 +nanogpt,"train (sgd), step 2",shared_gpu,1,0.4285022090189159,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,96,3,0,112,258,83,406,26276352,10,0.49719836947042495,1.028531331801787,0.07187491795048118,6013440,0.009621838806197047,0.0009297090582549572,1,35,106,20 +nanogpt,"loss, step 3",shared_gpu,1,0.023657625075429678,ok (2e-09),4.065422,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,103,4,0,112,268,83,406,26276352,10,0.49719836947042495,1.0511117066489533,0.07251737697515637,6013440,,,1,36,115,20 +nanogpt,"train (sgd), step 3",shared_gpu,1,0.4487860419321805,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,128,4,0,125,344,93,505,32764416,10,0.5785856615984812,1.374327371828258,0.09689504560083151,6210048,0.011780338711105287,0.000958126038312912,1,47,159,30 +nanogpt,"loss, step 4",shared_gpu,1,0.026138124987483025,ok (2e-09),4.03082,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,135,5,0,125,354,93,505,32764416,10,0.5785856615984812,1.3992769548203796,0.09761183767113835,6210048,,,1,48,168,30 +nanogpt,"train (sgd), step 4",shared_gpu,1,0.5133512089960277,ok (2e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,160,5,0,138,430,103,604,39252480,10,0.669295079424046,1.7730445764027536,0.12421796540729702,6406656,0.013911170768551528,0.0009935840498656034,1,59,212,40 diff --git a/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.log b/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.log new file mode 100644 index 00000000..096640ca --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=64, heads=4, embed=128, forward_only=False + loss, step 0 shared_gpu batch=1 0.074s + train (sgd), step 0 shared_gpu batch=1 0.760s + loss, step 1 shared_gpu batch=1 0.027s + train (sgd), step 1 shared_gpu batch=1 0.435s + loss, step 2 shared_gpu batch=1 0.026s + train (sgd), step 2 shared_gpu batch=1 0.429s + loss, step 3 shared_gpu batch=1 0.024s + train (sgd), step 3 shared_gpu batch=1 0.449s + loss, step 4 shared_gpu batch=1 0.026s + train (sgd), step 4 shared_gpu batch=1 0.513s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | stage_buffer_writes | stage_replacements_keys_or_nulls | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_gpu | 1 | 0.074 | ok (3e-09) | 4.186 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 7 | 1 | 0 | 10 | 10 | 9 | 16 | 691712 | 10 | 0.035 | 0.033 | 0.001 | 297472 | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_gpu | 1 | 0.760 | ok (7e-09) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 32 | 1 | 0 | 86 | 86 | 63 | 208 | 13300224 | 10 | 0.344 | 0.350 | 0.026 | 5620224 | 0.006 | 0.001 | 1 | 11 | n/a | n/a | +| nanogpt | loss, step 1 | shared_gpu | 1 | 0.027 | ok (3e-10) | 4.111 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 39 | 2 | 0 | 86 | 96 | 63 | 208 | 13300224 | 10 | 0.344 | 0.375 | 0.026 | 5620224 | n/a | n/a | 1 | 12 | 9 | n/a | +| nanogpt | train (sgd), step 1 | shared_gpu | 1 | 0.435 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 64 | 2 | 0 | 99 | 172 | 73 | 307 | 19788288 | 10 | 0.422 | 0.692 | 0.049 | 5816832 | 0.008 | 0.001 | 1 | 23 | 53 | 10 | +| nanogpt | loss, step 2 | shared_gpu | 1 | 0.026 | ok (3e-09) | 4.147 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 71 | 3 | 0 | 99 | 182 | 73 | 307 | 19788288 | 10 | 0.422 | 0.716 | 0.050 | 5816832 | n/a | n/a | 1 | 24 | 62 | 10 | +| nanogpt | train (sgd), step 2 | shared_gpu | 1 | 0.429 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 96 | 3 | 0 | 112 | 258 | 83 | 406 | 26276352 | 10 | 0.497 | 1.029 | 0.072 | 6013440 | 0.010 | 0.001 | 1 | 35 | 106 | 20 | +| nanogpt | loss, step 3 | shared_gpu | 1 | 0.024 | ok (2e-09) | 4.065 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 103 | 4 | 0 | 112 | 268 | 83 | 406 | 26276352 | 10 | 0.497 | 1.051 | 0.073 | 6013440 | n/a | n/a | 1 | 36 | 115 | 20 | +| nanogpt | train (sgd), step 3 | shared_gpu | 1 | 0.449 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 128 | 4 | 0 | 125 | 344 | 93 | 505 | 32764416 | 10 | 0.579 | 1.374 | 0.097 | 6210048 | 0.012 | 0.001 | 1 | 47 | 159 | 30 | +| nanogpt | loss, step 4 | shared_gpu | 1 | 0.026 | ok (2e-09) | 4.031 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 135 | 5 | 0 | 125 | 354 | 93 | 505 | 32764416 | 10 | 0.579 | 1.399 | 0.098 | 6210048 | n/a | n/a | 1 | 48 | 168 | 30 | +| nanogpt | train (sgd), step 4 | shared_gpu | 1 | 0.513 | ok (2e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 160 | 5 | 0 | 138 | 430 | 103 | 604 | 39252480 | 10 | 0.669 | 1.773 | 0.124 | 6406656 | 0.014 | 0.001 | 1 | 59 | 212 | 40 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-demo-shared_gpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-16-runs.json b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-runs.json new file mode 100644 index 00000000..1b83b644 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-runs.json @@ -0,0 +1,72 @@ +[ + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-16-shared_gpu.log", + "csv": "nanogpt-shared-v3-gpt2-16-shared_gpu.csv", + "peak_rss_gib": 40.02874755859375, + "status": "passed", + "started_utc": "2026-11-01T10:54:32.541752+00:00", + "exit_code": 0, + "wall_seconds": 846.1769028330455, + "checked_rows": 10 + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.csv", + "--context", + "16", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-16-shared_cpu.log", + "csv": "nanogpt-shared-v3-gpt2-16-shared_cpu.csv", + "peak_rss_gib": 38.71354675292969, + "status": "passed", + "started_utc": "2026-11-01T11:08:38.723798+00:00", + "exit_code": 0, + "wall_seconds": 846.0968177500181, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.csv b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.csv new file mode 100644 index 00000000..c8aff733 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,deferred_stages,union_reduction_graphs,reduction_stage_executions,stage_buffer_writes,stage_replacements_keys_or_nulls,stage_replacements_rows +nanogpt,"loss, step 0",shared_cpu,1,1.5626468749251217,ok (0e+00),11.218448,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,73,0,0,12,65,65,64,159,29921664,65,0.3532896207179874,1.1356185004115105,0.027896497631445527,19593856,,,,,,,,,, +nanogpt,"train (sgd), step 0",shared_cpu,1,171.97796291694976,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,142,0,0,12,1804,1804,1781,7356,4976716160,65,30.148012829246,101.750006427872,34.1254796134308,2890244224,0.12705116590950638,0.005362250027246773,1,11,175,19,1291,,, +nanogpt,"loss, step 1",shared_cpu,1,1.144399291020818,ok (2e-16),9.443567,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,215,0,0,24,1804,1869,1781,7356,4976716160,65,30.148012829246,102.85664480482228,34.14672348613385,2890244224,,,1,12,175,19,1291,64,, +nanogpt,"train (sgd), step 1",shared_cpu,1,164.39309562498238,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,284,0,0,24,3140,3608,1949,10036,5232945536,65,55.20476774126291,203.69975529285148,68.47792956058402,2890760320,0.17494299914687872,0.005453457939438522,1,23,350,38,2582,1613,69,99 +nanogpt,"loss, step 2",shared_cpu,1,1.1623300410574302,ok (1e-15),7.05059,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,357,0,0,36,3140,3673,1949,10036,5232945536,65,55.20476774126291,204.82587458507624,68.49839135527145,2890760320,,,1,24,350,38,2582,1677,69,99 +nanogpt,"train (sgd), step 2",shared_cpu,1,160.3256674159784,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,426,0,0,36,4476,5412,2117,12717,5489236352,65,79.55557916406542,303.47940241149627,101.69690532481764,2891214976,0.219828290049918,0.005530416965484619,1,35,525,57,3873,3226,237,99 +nanogpt,"loss, step 3",shared_cpu,1,1.1383747919462621,ok (2e-16),8.578056,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,499,0,0,48,4476,5477,2117,12717,5489236352,65,79.55557916406542,304.56935041118413,101.73197549092583,2891214976,,,1,36,525,57,3873,3290,237,99 +nanogpt,"train (sgd), step 3",shared_cpu,1,161.9083900420228,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,568,0,0,48,5812,7216,2285,15398,5747947904,65,104.50293072347995,403.45681396708824,135.57881917792838,2891681920,0.2646699551260099,0.005610875901766121,1,47,700,76,5164,4839,306,198 +nanogpt,"loss, step 4",shared_cpu,1,1.228010999970138,ok (5e-16),6.74504,,1e-09,1e-09,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,641,0,0,60,5812,7281,2285,15398,5747947904,65,104.50293072347995,404.63065463048406,135.61787184141576,2891681920,,,1,48,700,76,5164,4903,306,198 +nanogpt,"train (sgd), step 4",shared_cpu,1,164.8350743750343,ok (2e-15),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,710,0,0,60,7148,9020,2453,18078,6005381504,65,130.0049377027899,505.680548057775,169.5953018902801,2892210304,0.310160327819176,0.005691750906407833,1,59,875,95,6455,6452,375,297 diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.log b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.log new file mode 100644 index 00000000..76183d47 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 123,563,520 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=16, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 shared_cpu batch=1 1.563s + train (sgd), step 0 shared_cpu batch=1 171.978s + loss, step 1 shared_cpu batch=1 1.144s + train (sgd), step 1 shared_cpu batch=1 164.393s + loss, step 2 shared_cpu batch=1 1.162s + train (sgd), step 2 shared_cpu batch=1 160.326s + loss, step 3 shared_cpu batch=1 1.138s + train (sgd), step 3 shared_cpu batch=1 161.908s + loss, step 4 shared_cpu batch=1 1.228s + train (sgd), step 4 shared_cpu batch=1 164.835s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | deferred_stages | union_reduction_graphs | reduction_stage_executions | stage_buffer_writes | stage_replacements_keys_or_nulls | stage_replacements_rows | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_cpu | 1 | 1.563 | ok (0e+00) | 11.218 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 73 | 0 | 0 | 12 | 65 | 65 | 64 | 159 | 29921664 | 65 | 0.353 | 1.136 | 0.028 | 19593856 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_cpu | 1 | 171.978 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 142 | 0 | 0 | 12 | 1804 | 1804 | 1781 | 7356 | 4976716160 | 65 | 30.148 | 101.750 | 34.125 | 2890244224 | 0.127 | 0.005 | 1 | 11 | 175 | 19 | 1291 | n/a | n/a | n/a | +| nanogpt | loss, step 1 | shared_cpu | 1 | 1.144 | ok (2e-16) | 9.444 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 215 | 0 | 0 | 24 | 1804 | 1869 | 1781 | 7356 | 4976716160 | 65 | 30.148 | 102.857 | 34.147 | 2890244224 | n/a | n/a | 1 | 12 | 175 | 19 | 1291 | 64 | n/a | n/a | +| nanogpt | train (sgd), step 1 | shared_cpu | 1 | 164.393 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 284 | 0 | 0 | 24 | 3140 | 3608 | 1949 | 10036 | 5232945536 | 65 | 55.205 | 203.700 | 68.478 | 2890760320 | 0.175 | 0.005 | 1 | 23 | 350 | 38 | 2582 | 1613 | 69 | 99 | +| nanogpt | loss, step 2 | shared_cpu | 1 | 1.162 | ok (1e-15) | 7.051 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 357 | 0 | 0 | 36 | 3140 | 3673 | 1949 | 10036 | 5232945536 | 65 | 55.205 | 204.826 | 68.498 | 2890760320 | n/a | n/a | 1 | 24 | 350 | 38 | 2582 | 1677 | 69 | 99 | +| nanogpt | train (sgd), step 2 | shared_cpu | 1 | 160.326 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 426 | 0 | 0 | 36 | 4476 | 5412 | 2117 | 12717 | 5489236352 | 65 | 79.556 | 303.479 | 101.697 | 2891214976 | 0.220 | 0.006 | 1 | 35 | 525 | 57 | 3873 | 3226 | 237 | 99 | +| nanogpt | loss, step 3 | shared_cpu | 1 | 1.138 | ok (2e-16) | 8.578 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 499 | 0 | 0 | 48 | 4476 | 5477 | 2117 | 12717 | 5489236352 | 65 | 79.556 | 304.569 | 101.732 | 2891214976 | n/a | n/a | 1 | 36 | 525 | 57 | 3873 | 3290 | 237 | 99 | +| nanogpt | train (sgd), step 3 | shared_cpu | 1 | 161.908 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 568 | 0 | 0 | 48 | 5812 | 7216 | 2285 | 15398 | 5747947904 | 65 | 104.503 | 403.457 | 135.579 | 2891681920 | 0.265 | 0.006 | 1 | 47 | 700 | 76 | 5164 | 4839 | 306 | 198 | +| nanogpt | loss, step 4 | shared_cpu | 1 | 1.228 | ok (5e-16) | 6.745 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 641 | 0 | 0 | 60 | 5812 | 7281 | 2285 | 15398 | 5747947904 | 65 | 104.503 | 404.631 | 135.618 | 2891681920 | n/a | n/a | 1 | 48 | 700 | 76 | 5164 | 4903 | 306 | 198 | +| nanogpt | train (sgd), step 4 | shared_cpu | 1 | 164.835 | ok (2e-15) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 710 | 0 | 0 | 60 | 7148 | 9020 | 2453 | 18078 | 6005381504 | 65 | 130.005 | 505.681 | 169.595 | 2892210304 | 0.310 | 0.006 | 1 | 59 | 875 | 95 | 6455 | 6452 | 375 | 297 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_cpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.csv b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.csv new file mode 100644 index 00000000..9f468cf4 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,deferred_stages,union_reduction_graphs,reduction_stage_executions,stage_buffer_writes,stage_replacements_keys_or_nulls,stage_replacements_rows +nanogpt,"loss, step 0",shared_gpu,1,1.6992769590578973,ok (1e-08),11.218448,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,73,12,0,65,65,64,159,29921664,65,0.3258923740359023,1.3052752176299691,0.025083831511437893,19593856,,,,,,,,,, +nanogpt,"train (sgd), step 0",shared_gpu,1,173.3518294999376,ok (2e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,142,12,0,1804,1804,1781,7356,4976716160,65,31.2222567202989,102.51599211723078,33.68474707752466,2890244224,0.14581520785577595,0.004968416993506253,1,11,175,19,1291,,, +nanogpt,"loss, step 1",shared_gpu,1,1.403824417036958,ok (4e-08),9.443567,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,215,24,0,1804,1869,1781,7356,4976716160,65,31.2222567202989,103.88103378762025,33.70870261825621,2890244224,,,1,12,175,19,1291,64,, +nanogpt,"train (sgd), step 1",shared_gpu,1,160.88270816695876,ok (2e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,284,24,0,3140,3608,1949,10036,5232945536,65,55.939812661730684,202.61187011376023,67.01114168728236,2890760320,0.19210025074426085,0.005057167029008269,1,23,350,38,2582,1613,69,99 +nanogpt,"loss, step 2",shared_gpu,1,1.3760372919496149,ok (1e-08),7.05059,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,357,36,0,3140,3673,1949,10036,5232945536,65,55.939812661730684,203.9302918666508,67.05127823108342,2890760320,,,1,24,350,38,2582,1677,69,99 +nanogpt,"train (sgd), step 2",shared_gpu,1,162.62384758295957,ok (2e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,426,36,0,4476,5412,2117,12717,5489236352,65,80.69579961337149,303.72874302533455,100.95708299044054,2891214976,0.24007600254844874,0.005138583946973085,1,35,525,57,3873,3226,237,99 +nanogpt,"loss, step 3",shared_gpu,1,1.4009044170379639,ok (1e-07),8.578056,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,499,48,0,4476,5477,2117,12717,5489236352,65,80.69579961337149,305.0742852302501,100.99592144554481,2891214976,,,1,36,525,57,3873,3290,237,99 +nanogpt,"train (sgd), step 3",shared_gpu,1,162.9218788340222,ok (2e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,568,48,0,5812,7216,2285,15398,5747947904,65,106.0572794227628,404.30129503342323,135.0784483748721,2891681920,0.28602766687981784,0.0052178759360685945,1,47,700,76,5164,4839,306,198 +nanogpt,"loss, step 4",shared_gpu,1,1.3072377500357106,ok (2e-07),6.74504,,3e-05,3e-05,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,641,60,0,5812,7281,2285,15398,5747947904,65,106.0572794227628,405.5521910339594,135.1154667106457,2891681920,,,1,48,700,76,5164,4903,306,198 +nanogpt,"train (sgd), step 4",shared_gpu,1,162.66870258399285,ok (2e-07),,,,,12,768,12,16,50257,123563520,sgd,0.01,False,13,1,0,710,60,0,7148,9020,2453,18078,6005381504,65,131.42177372181322,504.44527716143057,169.1986960049253,2892210304,0.33133512584026903,0.005299500888213515,1,59,875,95,6455,6452,375,297 diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.log b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.log new file mode 100644 index 00000000..d9317d8f --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.log @@ -0,0 +1,28 @@ + 12-layer GPT, vocab 50257, 123,563,520 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=16, heads=12, embed=768, forward_only=False + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 shared_gpu batch=1 1.699s + train (sgd), step 0 shared_gpu batch=1 173.352s + loss, step 1 shared_gpu batch=1 1.404s + train (sgd), step 1 shared_gpu batch=1 160.883s + loss, step 2 shared_gpu batch=1 1.376s + train (sgd), step 2 shared_gpu batch=1 162.624s + loss, step 3 shared_gpu batch=1 1.401s + train (sgd), step 3 shared_gpu batch=1 162.922s + loss, step 4 shared_gpu batch=1 1.307s + train (sgd), step 4 shared_gpu batch=1 162.669s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | deferred_stages | union_reduction_graphs | reduction_stage_executions | stage_buffer_writes | stage_replacements_keys_or_nulls | stage_replacements_rows | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_gpu | 1 | 1.699 | ok (1e-08) | 11.218 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 73 | 12 | 0 | 65 | 65 | 64 | 159 | 29921664 | 65 | 0.326 | 1.305 | 0.025 | 19593856 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_gpu | 1 | 173.352 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 142 | 12 | 0 | 1804 | 1804 | 1781 | 7356 | 4976716160 | 65 | 31.222 | 102.516 | 33.685 | 2890244224 | 0.146 | 0.005 | 1 | 11 | 175 | 19 | 1291 | n/a | n/a | n/a | +| nanogpt | loss, step 1 | shared_gpu | 1 | 1.404 | ok (4e-08) | 9.444 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 215 | 24 | 0 | 1804 | 1869 | 1781 | 7356 | 4976716160 | 65 | 31.222 | 103.881 | 33.709 | 2890244224 | n/a | n/a | 1 | 12 | 175 | 19 | 1291 | 64 | n/a | n/a | +| nanogpt | train (sgd), step 1 | shared_gpu | 1 | 160.883 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 284 | 24 | 0 | 3140 | 3608 | 1949 | 10036 | 5232945536 | 65 | 55.940 | 202.612 | 67.011 | 2890760320 | 0.192 | 0.005 | 1 | 23 | 350 | 38 | 2582 | 1613 | 69 | 99 | +| nanogpt | loss, step 2 | shared_gpu | 1 | 1.376 | ok (1e-08) | 7.051 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 357 | 36 | 0 | 3140 | 3673 | 1949 | 10036 | 5232945536 | 65 | 55.940 | 203.930 | 67.051 | 2890760320 | n/a | n/a | 1 | 24 | 350 | 38 | 2582 | 1677 | 69 | 99 | +| nanogpt | train (sgd), step 2 | shared_gpu | 1 | 162.624 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 426 | 36 | 0 | 4476 | 5412 | 2117 | 12717 | 5489236352 | 65 | 80.696 | 303.729 | 100.957 | 2891214976 | 0.240 | 0.005 | 1 | 35 | 525 | 57 | 3873 | 3226 | 237 | 99 | +| nanogpt | loss, step 3 | shared_gpu | 1 | 1.401 | ok (1e-07) | 8.578 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 499 | 48 | 0 | 4476 | 5477 | 2117 | 12717 | 5489236352 | 65 | 80.696 | 305.074 | 100.996 | 2891214976 | n/a | n/a | 1 | 36 | 525 | 57 | 3873 | 3290 | 237 | 99 | +| nanogpt | train (sgd), step 3 | shared_gpu | 1 | 162.922 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 568 | 48 | 0 | 5812 | 7216 | 2285 | 15398 | 5747947904 | 65 | 106.057 | 404.301 | 135.078 | 2891681920 | 0.286 | 0.005 | 1 | 47 | 700 | 76 | 5164 | 4839 | 306 | 198 | +| nanogpt | loss, step 4 | shared_gpu | 1 | 1.307 | ok (2e-07) | 6.745 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 641 | 60 | 0 | 5812 | 7281 | 2285 | 15398 | 5747947904 | 65 | 106.057 | 405.552 | 135.115 | 2891681920 | n/a | n/a | 1 | 48 | 700 | 76 | 5164 | 4903 | 306 | 198 | +| nanogpt | train (sgd), step 4 | shared_gpu | 1 | 162.669 | ok (2e-07) | n/a | n/a | n/a | n/a | 12 | 768 | 12 | 16 | 50257 | 123563520 | sgd | 0.010 | False | 13 | 1 | 0 | 710 | 60 | 0 | 7148 | 9020 | 2453 | 18078 | 6005381504 | 65 | 131.422 | 504.445 | 169.199 | 2892210304 | 0.331 | 0.005 | 1 | 59 | 875 | 95 | 6455 | 6452 | 375 | 297 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-16-shared_gpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-runs.json b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-runs.json new file mode 100644 index 00000000..540b743f --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-runs.json @@ -0,0 +1,70 @@ +[ + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.csv", + "--forward-only", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-forward-shared_cpu.log", + "csv": "nanogpt-shared-v3-gpt2-forward-shared_cpu.csv", + "peak_rss_gib": 15.54058837890625, + "status": "passed", + "started_utc": "2026-11-01T11:22:44.844329+00:00", + "exit_code": 0, + "wall_seconds": 38.22789254190866, + "checked_rows": 5 + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "gpt2", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.csv", + "--forward-only", + "--vocab-size", + "50257" + ], + "contended": false, + "log": "nanogpt-shared-v3-gpt2-forward-shared_gpu.log", + "csv": "nanogpt-shared-v3-gpt2-forward-shared_gpu.csv", + "peak_rss_gib": 15.64544677734375, + "status": "passed", + "started_utc": "2026-11-01T11:23:23.073032+00:00", + "exit_code": 0, + "wall_seconds": 39.74374524992891, + "checked_rows": 5 + } +] diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.csv b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.csv new file mode 100644 index 00000000..bec2093b --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.csv @@ -0,0 +1,6 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,inplace_updates,stage_buffer_writes +nanogpt,"loss, step 0",shared_cpu,1,6.903008791967295,ok (2e-16),10.783063,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,73,0,0,12,65,65,64,159,1914986496,65,0.3588349998462945,3.7845330436248332,2.671604332048446,1254006784,0.06907412817236036,0.005020042066462338,, +nanogpt,"loss, step 1",shared_cpu,1,6.688245082972571,ok (0e+00),10.804215,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,146,0,0,24,65,130,64,159,1914986496,65,0.3588349998462945,7.945391705841757,5.14123300020583,1254006784,0.0694134192308411,0.005090501043014228,1,64 +nanogpt,"loss, step 2",shared_cpu,1,6.671622333000414,ok (3e-16),10.841586,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,219,0,0,36,65,195,64,159,1914986496,65,0.3588349998462945,12.136758621316403,7.575714293401688,1254006784,0.06971462815999985,0.005166250979527831,2,128 +nanogpt,"loss, step 3",shared_cpu,1,6.493429458932951,ok (2e-16),10.872349,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,292,0,0,48,65,260,64,159,1914986496,65,0.3588349998462945,16.16713270219043,10.000846037408337,1254006784,0.07004275324288756,0.005251250928267837,3,192 +nanogpt,"loss, step 4",shared_cpu,1,6.436930834082887,ok (0e+00),10.873143,,1e-09,1e-09,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,365,0,0,60,65,325,64,159,1914986496,65,0.3588349998462945,20.173762080725282,12.39298641658388,1254006784,0.0703430452849716,0.00532462599221617,4,256 diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.log b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.log new file mode 100644 index 00000000..6b17da9c --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.log @@ -0,0 +1,18 @@ + 12-layer GPT, vocab 50257, 124,337,664 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=1024, heads=12, embed=768, forward_only=True + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 shared_cpu batch=1 6.903s + loss, step 1 shared_cpu batch=1 6.688s + loss, step 2 shared_cpu batch=1 6.672s + loss, step 3 shared_cpu batch=1 6.493s + loss, step 4 shared_cpu batch=1 6.437s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | inplace_updates | stage_buffer_writes | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_cpu | 1 | 6.903 | ok (2e-16) | 10.783 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 73 | 0 | 0 | 12 | 65 | 65 | 64 | 159 | 1914986496 | 65 | 0.359 | 3.785 | 2.672 | 1254006784 | 0.069 | 0.005 | n/a | n/a | +| nanogpt | loss, step 1 | shared_cpu | 1 | 6.688 | ok (0e+00) | 10.804 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 146 | 0 | 0 | 24 | 65 | 130 | 64 | 159 | 1914986496 | 65 | 0.359 | 7.945 | 5.141 | 1254006784 | 0.069 | 0.005 | 1 | 64 | +| nanogpt | loss, step 2 | shared_cpu | 1 | 6.672 | ok (3e-16) | 10.842 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 219 | 0 | 0 | 36 | 65 | 195 | 64 | 159 | 1914986496 | 65 | 0.359 | 12.137 | 7.576 | 1254006784 | 0.070 | 0.005 | 2 | 128 | +| nanogpt | loss, step 3 | shared_cpu | 1 | 6.493 | ok (2e-16) | 10.872 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 292 | 0 | 0 | 48 | 65 | 260 | 64 | 159 | 1914986496 | 65 | 0.359 | 16.167 | 10.001 | 1254006784 | 0.070 | 0.005 | 3 | 192 | +| nanogpt | loss, step 4 | shared_cpu | 1 | 6.437 | ok (0e+00) | 10.873 | n/a | 1e-09 | 1e-09 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 365 | 0 | 0 | 60 | 65 | 325 | 64 | 159 | 1914986496 | 65 | 0.359 | 20.174 | 12.393 | 1254006784 | 0.070 | 0.005 | 4 | 256 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_cpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.csv b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.csv new file mode 100644 index 00000000..1786dc22 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.csv @@ -0,0 +1,6 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,inplace_updates,stage_buffer_writes +nanogpt,"loss, step 0",shared_gpu,1,7.411235708044842,ok (5e-10),10.783063,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,0,73,12,0,65,65,64,159,1914986496,65,0.372420794214122,4.180155749898404,2.763059668126516,1254006784,0.07428483304101974,0.005400249967351556,, +nanogpt,"loss, step 1",shared_gpu,1,6.95124583307188,ok (7e-10),10.804215,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,0,146,24,0,65,130,64,159,1914986496,65,0.372420794214122,8.532764205359854,5.303040168597363,1254006784,0.07462554099038243,0.005471041891723871,1,64 +nanogpt,"loss, step 2",shared_gpu,1,6.704163665999658,ok (3e-09),10.841586,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,0,219,36,0,65,195,64,159,1914986496,65,0.372420794214122,12.710171454586089,7.774752877303399,1254006784,0.07490658201277256,0.0055522918701171875,2,128 +nanogpt,"loss, step 3",shared_gpu,1,6.807813833002001,ok (4e-09),10.872349,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,0,292,48,0,65,260,64,159,1914986496,65,0.372420794214122,16.98152495105751,10.267630003974773,1254006784,0.07524299900978804,0.005638041882775724,3,192 +nanogpt,"loss, step 4",shared_gpu,1,6.9651285000145435,ok (6e-10),10.873143,,3e-05,3e-05,12,768,12,1024,50257,124337664,sgd,0.01,True,13,1,0,365,60,0,65,325,64,159,1914986496,65,0.372420794214122,21.33640320203267,12.807666914421134,1254006784,0.07554033200722188,0.005714832805097103,4,256 diff --git a/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.log b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.log new file mode 100644 index 00000000..a631194a --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.log @@ -0,0 +1,18 @@ + 12-layer GPT, vocab 50257, 124,337,664 parameters: the loss is 1047 lines of SQL + steps=5, optimizer=sgd, context=1024, heads=12, embed=768, forward_only=True + Padded vocabulary: inputs are character IDs, without GPT-2 BPE tokenization + loss, step 0 shared_gpu batch=1 7.411s + loss, step 1 shared_gpu batch=1 6.951s + loss, step 2 shared_gpu batch=1 6.704s + loss, step 3 shared_gpu batch=1 6.808s + loss, step 4 shared_gpu batch=1 6.965s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | inplace_updates | stage_buffer_writes | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_gpu | 1 | 7.411 | ok (5e-10) | 10.783 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 0 | 73 | 12 | 0 | 65 | 65 | 64 | 159 | 1914986496 | 65 | 0.372 | 4.180 | 2.763 | 1254006784 | 0.074 | 0.005 | n/a | n/a | +| nanogpt | loss, step 1 | shared_gpu | 1 | 6.951 | ok (7e-10) | 10.804 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 0 | 146 | 24 | 0 | 65 | 130 | 64 | 159 | 1914986496 | 65 | 0.372 | 8.533 | 5.303 | 1254006784 | 0.075 | 0.005 | 1 | 64 | +| nanogpt | loss, step 2 | shared_gpu | 1 | 6.704 | ok (3e-09) | 10.842 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 0 | 219 | 36 | 0 | 65 | 195 | 64 | 159 | 1914986496 | 65 | 0.372 | 12.710 | 7.775 | 1254006784 | 0.075 | 0.006 | 2 | 128 | +| nanogpt | loss, step 3 | shared_gpu | 1 | 6.808 | ok (4e-09) | 10.872 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 0 | 292 | 48 | 0 | 65 | 260 | 64 | 159 | 1914986496 | 65 | 0.372 | 16.982 | 10.268 | 1254006784 | 0.075 | 0.006 | 3 | 192 | +| nanogpt | loss, step 4 | shared_gpu | 1 | 6.965 | ok (6e-10) | 10.873 | n/a | 3e-05 | 3e-05 | 12 | 768 | 12 | 1024 | 50257 | 124337664 | sgd | 0.010 | True | 13 | 1 | 0 | 365 | 60 | 0 | 65 | 325 | 64 | 159 | 1914986496 | 65 | 0.372 | 21.336 | 12.808 | 1254006784 | 0.076 | 0.006 | 4 | 256 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-gpt2-forward-shared_gpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-shakespeare-runs.json b/benchmarks/apple/nanogpt-shared-v3-shakespeare-runs.json new file mode 100644 index 00000000..9e46939c --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-shakespeare-runs.json @@ -0,0 +1,64 @@ +[ + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "shakespeare", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-shakespeare-shared_cpu.log", + "csv": "nanogpt-shared-v3-shakespeare-shared_cpu.csv", + "peak_rss_gib": 9.9989013671875, + "status": "passed", + "started_utc": "2026-11-01T10:49:21.710678+00:00", + "exit_code": 0, + "wall_seconds": 153.34261495794635, + "checked_rows": 10 + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "shakespeare", + "--steps", + "5", + "--sizes", + "1", + "--optimizer", + "sgd", + "--learning-rate", + "0.01", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-shakespeare-shared_gpu.log", + "csv": "nanogpt-shared-v3-shakespeare-shared_gpu.csv", + "peak_rss_gib": 10.218841552734375, + "status": "passed", + "started_utc": "2026-11-01T10:51:55.054654+00:00", + "exit_code": 0, + "wall_seconds": 157.46115179196931, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.csv b/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.csv new file mode 100644 index 00000000..8621b9d6 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,deferred_stages,union_reduction_graphs,reduction_stage_executions,stage_buffer_writes,stage_replacements_keys_or_nulls,stage_replacements_rows +nanogpt,"loss, step 0",shared_cpu,1,0.6604845830006525,ok (0e+00),4.243064,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,37,0,0,6,35,35,34,81,42915840,35,0.16071724786888808,0.3618839969858527,0.11359791830182076,2762752,,,,,,,,,, +nanogpt,"train (sgd), step 0",shared_cpu,1,30.68821404199116,ok (2e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,82,0,0,6,555,555,532,2354,2018056192,35,5.334270742139779,19.859300299547613,4.659096539602615,249680896,0.02764891821425408,0.0027456670068204403,1,11,31,6,276,,, +nanogpt,"loss, step 1",shared_cpu,1,0.51197587500792,ok (1e-16),3.670806,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,119,0,0,12,555,590,532,2354,2018056192,35,5.334270742139779,20.25614563410636,4.766956499661319,249680896,,,1,12,31,6,276,34,, +nanogpt,"train (sgd), step 1",shared_cpu,1,28.097506750025786,ok (3e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,164,0,0,12,865,1110,628,3483,2836817920,35,8.73113919980824,39.702524170745164,9.27473694679793,253017088,0.03357916802633554,0.0028095010202378035,1,23,62,12,552,436,45,51 +nanogpt,"loss, step 2",shared_cpu,1,0.5451391659444198,ok (0e+00),3.681532,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,201,0,0,18,865,1145,628,3483,2836817920,35,8.73113919980824,40.12857500615064,9.385910529526882,253017088,,,1,24,62,12,552,470,45,51 +nanogpt,"train (sgd), step 2",shared_cpu,1,29.01373270805925,ok (3e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,246,0,0,18,1175,1665,724,4612,3658344448,35,12.278070572763681,60.15693916846067,14.037566698389128,256408576,0.039670710219070315,0.0028653350891545415,1,35,93,18,828,872,90,102 +nanogpt,"loss, step 3",shared_cpu,1,0.5919682089006528,ok (1e-16),3.366265,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,283,0,0,24,1175,1700,724,4612,3658344448,35,12.278070572763681,60.623913920717314,14.154495742521249,256408576,,,1,36,93,18,828,906,90,102 +nanogpt,"train (sgd), step 3",shared_cpu,1,28.57278045907151,ok (3e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,328,0,0,24,1485,2220,820,5741,4478027776,35,15.80578744923696,80.34330729558133,18.688923750421964,259763200,0.04589975031558424,0.002918877173215151,1,47,124,24,1104,1308,135,153 +nanogpt,"loss, step 4",shared_cpu,1,0.5775978330057114,ok (1e-16),3.36678,,1e-09,1e-09,6,384,6,256,65,10745088,sgd,0.01,False,13,1,365,0,0,30,1485,2255,820,5741,4478027776,35,15.80578744923696,80.79399025253952,18.807512791012414,259763200,,,1,48,124,24,1104,1342,135,153 +nanogpt,"train (sgd), step 4",shared_cpu,1,29.115979959024116,ok (3e-16),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,410,0,0,30,1795,2775,916,6870,5296482304,35,19.42623228521552,100.87469962309115,23.41542024689261,263093248,0.05219091637991369,0.002969460212625563,1,59,155,30,1380,1744,180,204 diff --git a/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.log b/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.log new file mode 100644 index 00000000..61813c2d --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.log @@ -0,0 +1,27 @@ + 6-layer GPT, vocab 65, 10,745,088 parameters: the loss is 543 lines of SQL + steps=5, optimizer=sgd, context=256, heads=6, embed=384, forward_only=False + loss, step 0 shared_cpu batch=1 0.660s + train (sgd), step 0 shared_cpu batch=1 30.688s + loss, step 1 shared_cpu batch=1 0.512s + train (sgd), step 1 shared_cpu batch=1 28.098s + loss, step 2 shared_cpu batch=1 0.545s + train (sgd), step 2 shared_cpu batch=1 29.014s + loss, step 3 shared_cpu batch=1 0.592s + train (sgd), step 3 shared_cpu batch=1 28.573s + loss, step 4 shared_cpu batch=1 0.578s + train (sgd), step 4 shared_cpu batch=1 29.116s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | deferred_stages | union_reduction_graphs | reduction_stage_executions | stage_buffer_writes | stage_replacements_keys_or_nulls | stage_replacements_rows | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_cpu | 1 | 0.660 | ok (0e+00) | 4.243 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 37 | 0 | 0 | 6 | 35 | 35 | 34 | 81 | 42915840 | 35 | 0.161 | 0.362 | 0.114 | 2762752 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_cpu | 1 | 30.688 | ok (2e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 82 | 0 | 0 | 6 | 555 | 555 | 532 | 2354 | 2018056192 | 35 | 5.334 | 19.859 | 4.659 | 249680896 | 0.028 | 0.003 | 1 | 11 | 31 | 6 | 276 | n/a | n/a | n/a | +| nanogpt | loss, step 1 | shared_cpu | 1 | 0.512 | ok (1e-16) | 3.671 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 119 | 0 | 0 | 12 | 555 | 590 | 532 | 2354 | 2018056192 | 35 | 5.334 | 20.256 | 4.767 | 249680896 | n/a | n/a | 1 | 12 | 31 | 6 | 276 | 34 | n/a | n/a | +| nanogpt | train (sgd), step 1 | shared_cpu | 1 | 28.098 | ok (3e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 164 | 0 | 0 | 12 | 865 | 1110 | 628 | 3483 | 2836817920 | 35 | 8.731 | 39.703 | 9.275 | 253017088 | 0.034 | 0.003 | 1 | 23 | 62 | 12 | 552 | 436 | 45 | 51 | +| nanogpt | loss, step 2 | shared_cpu | 1 | 0.545 | ok (0e+00) | 3.682 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 201 | 0 | 0 | 18 | 865 | 1145 | 628 | 3483 | 2836817920 | 35 | 8.731 | 40.129 | 9.386 | 253017088 | n/a | n/a | 1 | 24 | 62 | 12 | 552 | 470 | 45 | 51 | +| nanogpt | train (sgd), step 2 | shared_cpu | 1 | 29.014 | ok (3e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 246 | 0 | 0 | 18 | 1175 | 1665 | 724 | 4612 | 3658344448 | 35 | 12.278 | 60.157 | 14.038 | 256408576 | 0.040 | 0.003 | 1 | 35 | 93 | 18 | 828 | 872 | 90 | 102 | +| nanogpt | loss, step 3 | shared_cpu | 1 | 0.592 | ok (1e-16) | 3.366 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 283 | 0 | 0 | 24 | 1175 | 1700 | 724 | 4612 | 3658344448 | 35 | 12.278 | 60.624 | 14.154 | 256408576 | n/a | n/a | 1 | 36 | 93 | 18 | 828 | 906 | 90 | 102 | +| nanogpt | train (sgd), step 3 | shared_cpu | 1 | 28.573 | ok (3e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 328 | 0 | 0 | 24 | 1485 | 2220 | 820 | 5741 | 4478027776 | 35 | 15.806 | 80.343 | 18.689 | 259763200 | 0.046 | 0.003 | 1 | 47 | 124 | 24 | 1104 | 1308 | 135 | 153 | +| nanogpt | loss, step 4 | shared_cpu | 1 | 0.578 | ok (1e-16) | 3.367 | n/a | 1e-09 | 1e-09 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 365 | 0 | 0 | 30 | 1485 | 2255 | 820 | 5741 | 4478027776 | 35 | 15.806 | 80.794 | 18.808 | 259763200 | n/a | n/a | 1 | 48 | 124 | 24 | 1104 | 1342 | 135 | 153 | +| nanogpt | train (sgd), step 4 | shared_cpu | 1 | 29.116 | ok (3e-16) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 410 | 0 | 0 | 30 | 1795 | 2775 | 916 | 6870 | 5296482304 | 35 | 19.426 | 100.875 | 23.415 | 263093248 | 0.052 | 0.003 | 1 | 59 | 155 | 30 | 1380 | 1744 | 180 | 204 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_cpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.csv b/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.csv new file mode 100644 index 00000000..0a13306c --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,deferred_stages,union_reduction_graphs,reduction_stage_executions,stage_buffer_writes,stage_replacements_keys_or_nulls,stage_replacements_rows +nanogpt,"loss, step 0",shared_gpu,1,0.6827284999890253,ok (5e-09),4.243064,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,37,6,0,35,35,34,81,42915840,35,0.15273854020051658,0.3946002065204084,0.11164350190665573,2762752,,,,,,,,,, +nanogpt,"train (sgd), step 0",shared_gpu,1,31.879907374968752,ok (1e-08),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,82,6,0,555,555,532,2354,2018056192,35,5.636024660663679,20.57021320494823,4.798318377113901,249680896,0.0288813728839159,0.002718250034376979,1,11,31,6,276,,, +nanogpt,"loss, step 1",shared_gpu,1,0.5666102920658886,ok (6e-09),3.670806,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,119,12,0,555,590,532,2354,2018056192,35,5.636024660663679,21.017905494896695,4.909002291271463,249680896,,,1,12,31,6,276,34,, +nanogpt,"train (sgd), step 1",shared_gpu,1,29.327827792032622,ok (1e-08),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,164,12,0,865,1110,628,3483,2836817920,35,9.226754650007933,41.261208712356165,9.615783666609786,253017088,0.034734625020064414,0.002781375078484416,1,23,62,12,552,436,45,51 +nanogpt,"loss, step 2",shared_gpu,1,0.6047885000007227,ok (3e-09),3.681532,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,201,18,0,865,1145,628,3483,2836817920,35,9.226754650007933,41.74008700624108,9.732747918227687,253017088,,,1,24,62,12,552,470,45,51 +nanogpt,"train (sgd), step 2",shared_gpu,1,29.023384082945995,ok (1e-08),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,246,18,0,1175,1665,724,4612,3658344448,35,12.733951197820716,61.7860451687593,14.413165081175976,256408576,0.04104074905626476,0.0028365420876070857,1,35,93,18,828,872,90,102 +nanogpt,"loss, step 3",shared_gpu,1,0.6068668750813231,ok (8e-10),3.366265,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,283,24,0,1175,1700,724,4612,3658344448,35,12.733951197820716,62.27113512577489,14.526576167205349,256408576,,,1,36,93,18,828,906,90,102 +nanogpt,"train (sgd), step 3",shared_gpu,1,30.370450207963586,ok (1e-08),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,328,24,0,1485,2220,820,5741,4478027776,35,16.574972821515985,83.14150327350944,19.329121914692223,259763200,0.047586916014552116,0.002893292112275958,1,47,124,24,1104,1308,135,153 +nanogpt,"loss, step 4",shared_gpu,1,0.5877329159993678,ok (8e-09),3.36678,,3e-05,3e-05,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,365,30,0,1485,2255,820,5741,4478027776,35,16.574972821515985,83.60956593963783,19.440723289735615,259763200,,,1,48,124,24,1104,1342,135,153 +nanogpt,"train (sgd), step 4",shared_gpu,1,28.658524541999213,ok (1e-08),,,,,6,384,6,256,65,10745088,sgd,0.01,False,13,1,0,410,30,0,1795,2775,916,6870,5296482304,35,20.114735280629247,103.3905114326626,23.975200915709138,263093248,0.0536419179989025,0.002962459111586213,1,59,155,30,1380,1744,180,204 diff --git a/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.log b/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.log new file mode 100644 index 00000000..c1c1a0bf --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.log @@ -0,0 +1,27 @@ + 6-layer GPT, vocab 65, 10,745,088 parameters: the loss is 543 lines of SQL + steps=5, optimizer=sgd, context=256, heads=6, embed=384, forward_only=False + loss, step 0 shared_gpu batch=1 0.683s + train (sgd), step 0 shared_gpu batch=1 31.880s + loss, step 1 shared_gpu batch=1 0.567s + train (sgd), step 1 shared_gpu batch=1 29.328s + loss, step 2 shared_gpu batch=1 0.605s + train (sgd), step 2 shared_gpu batch=1 29.023s + loss, step 3 shared_gpu batch=1 0.607s + train (sgd), step 3 shared_gpu batch=1 30.370s + loss, step 4 shared_gpu batch=1 0.588s + train (sgd), step 4 shared_gpu batch=1 28.659s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | deferred_stages | union_reduction_graphs | reduction_stage_executions | stage_buffer_writes | stage_replacements_keys_or_nulls | stage_replacements_rows | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_gpu | 1 | 0.683 | ok (5e-09) | 4.243 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 37 | 6 | 0 | 35 | 35 | 34 | 81 | 42915840 | 35 | 0.153 | 0.395 | 0.112 | 2762752 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_gpu | 1 | 31.880 | ok (1e-08) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 82 | 6 | 0 | 555 | 555 | 532 | 2354 | 2018056192 | 35 | 5.636 | 20.570 | 4.798 | 249680896 | 0.029 | 0.003 | 1 | 11 | 31 | 6 | 276 | n/a | n/a | n/a | +| nanogpt | loss, step 1 | shared_gpu | 1 | 0.567 | ok (6e-09) | 3.671 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 119 | 12 | 0 | 555 | 590 | 532 | 2354 | 2018056192 | 35 | 5.636 | 21.018 | 4.909 | 249680896 | n/a | n/a | 1 | 12 | 31 | 6 | 276 | 34 | n/a | n/a | +| nanogpt | train (sgd), step 1 | shared_gpu | 1 | 29.328 | ok (1e-08) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 164 | 12 | 0 | 865 | 1110 | 628 | 3483 | 2836817920 | 35 | 9.227 | 41.261 | 9.616 | 253017088 | 0.035 | 0.003 | 1 | 23 | 62 | 12 | 552 | 436 | 45 | 51 | +| nanogpt | loss, step 2 | shared_gpu | 1 | 0.605 | ok (3e-09) | 3.682 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 201 | 18 | 0 | 865 | 1145 | 628 | 3483 | 2836817920 | 35 | 9.227 | 41.740 | 9.733 | 253017088 | n/a | n/a | 1 | 24 | 62 | 12 | 552 | 470 | 45 | 51 | +| nanogpt | train (sgd), step 2 | shared_gpu | 1 | 29.023 | ok (1e-08) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 246 | 18 | 0 | 1175 | 1665 | 724 | 4612 | 3658344448 | 35 | 12.734 | 61.786 | 14.413 | 256408576 | 0.041 | 0.003 | 1 | 35 | 93 | 18 | 828 | 872 | 90 | 102 | +| nanogpt | loss, step 3 | shared_gpu | 1 | 0.607 | ok (8e-10) | 3.366 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 283 | 24 | 0 | 1175 | 1700 | 724 | 4612 | 3658344448 | 35 | 12.734 | 62.271 | 14.527 | 256408576 | n/a | n/a | 1 | 36 | 93 | 18 | 828 | 906 | 90 | 102 | +| nanogpt | train (sgd), step 3 | shared_gpu | 1 | 30.370 | ok (1e-08) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 328 | 24 | 0 | 1485 | 2220 | 820 | 5741 | 4478027776 | 35 | 16.575 | 83.142 | 19.329 | 259763200 | 0.048 | 0.003 | 1 | 47 | 124 | 24 | 1104 | 1308 | 135 | 153 | +| nanogpt | loss, step 4 | shared_gpu | 1 | 0.588 | ok (8e-09) | 3.367 | n/a | 3e-05 | 3e-05 | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 365 | 30 | 0 | 1485 | 2255 | 820 | 5741 | 4478027776 | 35 | 16.575 | 83.610 | 19.441 | 259763200 | n/a | n/a | 1 | 48 | 124 | 24 | 1104 | 1342 | 135 | 153 | +| nanogpt | train (sgd), step 4 | shared_gpu | 1 | 28.659 | ok (1e-08) | n/a | n/a | n/a | n/a | 6 | 384 | 6 | 256 | 65 | 10745088 | sgd | 0.010 | False | 13 | 1 | 0 | 410 | 30 | 0 | 1795 | 2775 | 916 | 6870 | 5296482304 | 35 | 20.115 | 103.391 | 23.975 | 263093248 | 0.054 | 0.003 | 1 | 59 | 155 | 30 | 1380 | 1744 | 180 | 204 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-shakespeare-shared_gpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.csv b/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.csv new file mode 100644 index 00000000..df2d8bf3 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",cpu_forced,4,1.2293146250303835,ok (2e-16),4.179251,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,178,0,0,40,,,, +nanogpt,"train (sgd), step 0",cpu_forced,4,0.555888915900141,ok (3e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,203,0,0,40,0.004390998277813196,0.023162584053352475,1,11 +nanogpt,"loss, step 1",cpu_forced,4,0.05024325009435415,ok (0e+00),4.104661,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,381,0,0,80,,,1,12 +nanogpt,"train (sgd), step 1",cpu_forced,4,0.2640987909398973,ok (4e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,406,0,0,80,0.00627616629935801,0.023184001096524298,1,23 +nanogpt,"loss, step 2",cpu_forced,4,0.05006787495221943,ok (1e-16),3.883857,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,584,0,0,120,,,1,24 +nanogpt,"train (sgd), step 2",cpu_forced,4,0.26585491688456386,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,609,0,0,120,0.008079332299530506,0.023212334141135216,1,35 +nanogpt,"loss, step 3",cpu_forced,4,0.04632487497292459,ok (1e-16),3.641763,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,787,0,0,160,,,1,36 +nanogpt,"train (sgd), step 3",cpu_forced,4,0.2592986250529066,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,812,0,0,160,0.00981253816280514,0.023231125087477267,1,47 +nanogpt,"loss, step 4",cpu_forced,4,0.05289520905353129,ok (3e-16),3.842234,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,990,0,0,200,,,1,48 +nanogpt,"train (sgd), step 4",cpu_forced,4,0.2674185000360012,ok (1e-15),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,1015,0,0,200,0.011564747081138194,0.023251250153407454,1,59 diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.log b/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.log new file mode 100644 index 00000000..e139ebae --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 cpu_forced batch=4 1.229s + train (sgd), step 0 cpu_forced batch=4 0.556s + loss, step 1 cpu_forced batch=4 0.050s + train (sgd), step 1 cpu_forced batch=4 0.264s + loss, step 2 cpu_forced batch=4 0.050s + train (sgd), step 2 cpu_forced batch=4 0.266s + loss, step 3 cpu_forced batch=4 0.046s + train (sgd), step 3 cpu_forced batch=4 0.259s + loss, step 4 cpu_forced batch=4 0.053s + train (sgd), step 4 cpu_forced batch=4 0.267s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | cpu_forced | 4 | 1.229 | ok (2e-16) | 4.179 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 178 | 0 | 0 | 40 | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | cpu_forced | 4 | 0.556 | ok (3e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 203 | 0 | 0 | 40 | 0.004 | 0.023 | 1 | 11 | +| nanogpt | loss, step 1 | cpu_forced | 4 | 0.050 | ok (0e+00) | 4.105 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 381 | 0 | 0 | 80 | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | cpu_forced | 4 | 0.264 | ok (4e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 406 | 0 | 0 | 80 | 0.006 | 0.023 | 1 | 23 | +| nanogpt | loss, step 2 | cpu_forced | 4 | 0.050 | ok (1e-16) | 3.884 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 584 | 0 | 0 | 120 | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | cpu_forced | 4 | 0.266 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 609 | 0 | 0 | 120 | 0.008 | 0.023 | 1 | 35 | +| nanogpt | loss, step 3 | cpu_forced | 4 | 0.046 | ok (1e-16) | 3.642 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 787 | 0 | 0 | 160 | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | cpu_forced | 4 | 0.259 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 812 | 0 | 0 | 160 | 0.010 | 0.023 | 1 | 47 | +| nanogpt | loss, step 4 | cpu_forced | 4 | 0.053 | ok (3e-16) | 3.842 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 990 | 0 | 0 | 200 | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | cpu_forced | 4 | 0.267 | ok (1e-15) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 1015 | 0 | 0 | 200 | 0.012 | 0.023 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.csv b/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.csv new file mode 100644 index 00000000..cd161c1d --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,load_seconds,model_seconds +nanogpt,"loss, step 0",datafusion,4,1.180193250067532,ok (4e-16),4.179251,9882,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 0",datafusion,4,0.44398425007238984,ok (5e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.004402041551657021,0.027535583009012043 +nanogpt,"loss, step 1",datafusion,4,1.1881296660285443,ok (2e-16),4.104661,9882,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 1",datafusion,4,0.18606933299452066,ok (5e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.0064596674637869,0.05075858300551772 +nanogpt,"loss, step 2",datafusion,4,1.0919131670380011,ok (1e-16),3.883857,9882,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 2",datafusion,4,0.1615661249961704,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.008429794339463115,0.07419408299028873 +nanogpt,"loss, step 3",datafusion,4,1.0477787499548867,ok (1e-16),3.641763,9882,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 3",datafusion,4,0.16016441700048745,ok (8e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.01036438054870814,0.09697204199619591 +nanogpt,"loss, step 4",datafusion,4,1.0425397919025272,ok (5e-16),3.842234,9882,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 4",datafusion,4,0.16091687500011176,ok (1e-15),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.01233108842279762,0.11925070791039616 diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.log b/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.log new file mode 100644 index 00000000..6f5bd7b7 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 datafusion batch=4 1.180s + train (sgd), step 0 datafusion batch=4 0.444s + loss, step 1 datafusion batch=4 1.188s + train (sgd), step 1 datafusion batch=4 0.186s + loss, step 2 datafusion batch=4 1.092s + train (sgd), step 2 datafusion batch=4 0.162s + loss, step 3 datafusion batch=4 1.048s + train (sgd), step 3 datafusion batch=4 0.160s + loss, step 4 datafusion batch=4 1.043s + train (sgd), step 4 datafusion batch=4 0.161s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | load_seconds | model_seconds | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | datafusion | 4 | 1.180 | ok (4e-16) | 4.179 | 9882 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 0 | datafusion | 4 | 0.444 | ok (5e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.004 | 0.028 | +| nanogpt | loss, step 1 | datafusion | 4 | 1.188 | ok (2e-16) | 4.105 | 9882 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 1 | datafusion | 4 | 0.186 | ok (5e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.006 | 0.051 | +| nanogpt | loss, step 2 | datafusion | 4 | 1.092 | ok (1e-16) | 3.884 | 9882 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 2 | datafusion | 4 | 0.162 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.008 | 0.074 | +| nanogpt | loss, step 3 | datafusion | 4 | 1.048 | ok (1e-16) | 3.642 | 9882 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 3 | datafusion | 4 | 0.160 | ok (8e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.010 | 0.097 | +| nanogpt | loss, step 4 | datafusion | 4 | 1.043 | ok (5e-16) | 3.842 | 9882 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 4 | datafusion | 4 | 0.161 | ok (1e-15) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.012 | 0.119 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.csv b/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.csv new file mode 100644 index 00000000..39bd6ae2 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,load_seconds,model_seconds +nanogpt,"loss, step 0",duckdb,4,0.030569666996598244,ok (0e+00),4.179251,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 0",duckdb,4,0.3056947080185637,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.011522499844431877,0.0011709999525919557 +nanogpt,"loss, step 1",duckdb,4,0.027087333030067384,ok (0e+00),4.104661,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 1",duckdb,4,0.29074604203924537,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.017679126700386405,0.0021719170035794377 +nanogpt,"loss, step 2",duckdb,4,0.026691499981097877,ok (1e-16),3.883857,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 2",duckdb,4,0.28412749990820885,ok (5e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.024700419744476676,0.003432541969232261 +nanogpt,"loss, step 3",duckdb,4,0.03180316695943475,ok (1e-16),3.641763,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 3",duckdb,4,0.27657245798036456,ok (7e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.03182517073582858,0.004535124986432493 +nanogpt,"loss, step 4",duckdb,4,0.026779667008668184,ok (1e-16),3.842234,126,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,, +nanogpt,"train (sgd), step 4",duckdb,4,0.25836824998259544,ok (1e-15),,,,,1,16,2,16,65,4416,sgd,0.5,False,0.03732700285036117,0.005608291947282851 diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.log b/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.log new file mode 100644 index 00000000..2dfe13d8 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 duckdb batch=4 0.031s + train (sgd), step 0 duckdb batch=4 0.306s + loss, step 1 duckdb batch=4 0.027s + train (sgd), step 1 duckdb batch=4 0.291s + loss, step 2 duckdb batch=4 0.027s + train (sgd), step 2 duckdb batch=4 0.284s + loss, step 3 duckdb batch=4 0.032s + train (sgd), step 3 duckdb batch=4 0.277s + loss, step 4 duckdb batch=4 0.027s + train (sgd), step 4 duckdb batch=4 0.258s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | load_seconds | model_seconds | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | duckdb | 4 | 0.031 | ok (0e+00) | 4.179 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 0 | duckdb | 4 | 0.306 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.012 | 0.001 | +| nanogpt | loss, step 1 | duckdb | 4 | 0.027 | ok (0e+00) | 4.105 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 1 | duckdb | 4 | 0.291 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.018 | 0.002 | +| nanogpt | loss, step 2 | duckdb | 4 | 0.027 | ok (1e-16) | 3.884 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 2 | duckdb | 4 | 0.284 | ok (5e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.025 | 0.003 | +| nanogpt | loss, step 3 | duckdb | 4 | 0.032 | ok (1e-16) | 3.642 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 3 | duckdb | 4 | 0.277 | ok (7e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.032 | 0.005 | +| nanogpt | loss, step 4 | duckdb | 4 | 0.027 | ok (1e-16) | 3.842 | 126 | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | n/a | n/a | +| nanogpt | train (sgd), step 4 | duckdb | 4 | 0.258 | ok (1e-15) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 0.037 | 0.006 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.csv b/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.csv new file mode 100644 index 00000000..1d831e02 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,load_seconds,model_seconds,gradient_preparations,inplace_updates +nanogpt,"loss, step 0",gpu_forced,4,1.2601481250021607,ok (5e-10),4.179251,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,178,40,0,,,, +nanogpt,"train (sgd), step 0",gpu_forced,4,0.6385403330205008,ok (1e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,203,40,0,0.004239418660290539,0.023072166019119322,1,11 +nanogpt,"loss, step 1",gpu_forced,4,0.08971595799084753,ok (1e-09),4.104661,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,381,80,0,,,1,12 +nanogpt,"train (sgd), step 1",gpu_forced,4,0.27746470901183784,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,406,80,0,0.005962001625448465,0.023096458055078983,1,23 +nanogpt,"loss, step 2",gpu_forced,4,0.08255058305803686,ok (7e-10),3.883857,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,584,120,0,,,1,24 +nanogpt,"train (sgd), step 2",gpu_forced,4,0.27421466703526676,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,609,120,0,0.007719710934907198,0.02311579103115946,1,35 +nanogpt,"loss, step 3",gpu_forced,4,0.08232533291447908,ok (4e-09),3.641763,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,787,160,0,,,1,36 +nanogpt,"train (sgd), step 3",gpu_forced,4,0.2763963749166578,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,812,160,0,0.009452790720388293,0.023135791067034006,1,47 +nanogpt,"loss, step 4",gpu_forced,4,0.08547125000040978,ok (2e-09),3.842234,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,990,200,0,,,1,48 +nanogpt,"train (sgd), step 4",gpu_forced,4,0.3162609590217471,ok (3e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,1015,200,0,0.011253248783759773,0.02315420808736235,1,59 diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.log b/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.log new file mode 100644 index 00000000..17485a11 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 gpu_forced batch=4 1.260s + train (sgd), step 0 gpu_forced batch=4 0.639s + loss, step 1 gpu_forced batch=4 0.090s + train (sgd), step 1 gpu_forced batch=4 0.277s + loss, step 2 gpu_forced batch=4 0.083s + train (sgd), step 2 gpu_forced batch=4 0.274s + loss, step 3 gpu_forced batch=4 0.082s + train (sgd), step 3 gpu_forced batch=4 0.276s + loss, step 4 gpu_forced batch=4 0.085s + train (sgd), step 4 gpu_forced batch=4 0.316s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | load_seconds | model_seconds | gradient_preparations | inplace_updates | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | gpu_forced | 4 | 1.260 | ok (5e-10) | 4.179 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 178 | 40 | 0 | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | gpu_forced | 4 | 0.639 | ok (1e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 203 | 40 | 0 | 0.004 | 0.023 | 1 | 11 | +| nanogpt | loss, step 1 | gpu_forced | 4 | 0.090 | ok (1e-09) | 4.105 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 381 | 80 | 0 | n/a | n/a | 1 | 12 | +| nanogpt | train (sgd), step 1 | gpu_forced | 4 | 0.277 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 406 | 80 | 0 | 0.006 | 0.023 | 1 | 23 | +| nanogpt | loss, step 2 | gpu_forced | 4 | 0.083 | ok (7e-10) | 3.884 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 584 | 120 | 0 | n/a | n/a | 1 | 24 | +| nanogpt | train (sgd), step 2 | gpu_forced | 4 | 0.274 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 609 | 120 | 0 | 0.008 | 0.023 | 1 | 35 | +| nanogpt | loss, step 3 | gpu_forced | 4 | 0.082 | ok (4e-09) | 3.642 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 787 | 160 | 0 | n/a | n/a | 1 | 36 | +| nanogpt | train (sgd), step 3 | gpu_forced | 4 | 0.276 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 812 | 160 | 0 | 0.009 | 0.023 | 1 | 47 | +| nanogpt | loss, step 4 | gpu_forced | 4 | 0.085 | ok (2e-09) | 3.842 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 990 | 200 | 0 | n/a | n/a | 1 | 48 | +| nanogpt | train (sgd), step 4 | gpu_forced | 4 | 0.316 | ok (3e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 1015 | 200 | 0 | 0.011 | 0.023 | 1 | 59 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-runs.json b/benchmarks/apple/nanogpt-shared-v3-tiny-runs.json new file mode 100644 index 00000000..d73f6129 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-runs.json @@ -0,0 +1,188 @@ +[ + { + "host": "datafusion", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "datafusion", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-datafusion.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-datafusion.log", + "csv": "nanogpt-shared-v3-tiny-datafusion.csv", + "peak_rss_gib": 1.084442138671875, + "status": "passed", + "started_utc": "2026-11-01T10:48:19.089914+00:00", + "exit_code": 0, + "wall_seconds": 14.230142958112992, + "checked_rows": 10 + }, + { + "host": "duckdb", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "duckdb", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-duckdb.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-duckdb.log", + "csv": "nanogpt-shared-v3-tiny-duckdb.csv", + "peak_rss_gib": 0.7633819580078125, + "status": "passed", + "started_utc": "2026-11-01T10:48:33.320634+00:00", + "exit_code": 0, + "wall_seconds": 4.258987625013106, + "checked_rows": 10 + }, + { + "host": "cpu_forced", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "cpu_forced", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-cpu_forced.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-cpu_forced.log", + "csv": "nanogpt-shared-v3-tiny-cpu_forced.csv", + "peak_rss_gib": 0.940399169921875, + "status": "passed", + "started_utc": "2026-11-01T10:48:37.580129+00:00", + "exit_code": 0, + "wall_seconds": 5.64062604191713, + "checked_rows": 10 + }, + { + "host": "gpu_forced", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "gpu_forced", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-gpu_forced.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-gpu_forced.log", + "csv": "nanogpt-shared-v3-tiny-gpu_forced.csv", + "peak_rss_gib": 0.952362060546875, + "status": "passed", + "started_utc": "2026-11-01T10:48:43.221551+00:00", + "exit_code": 0, + "wall_seconds": 5.972789291990921, + "checked_rows": 10 + }, + { + "host": "shared_cpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_cpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-shared_cpu.log", + "csv": "nanogpt-shared-v3-tiny-shared_cpu.csv", + "peak_rss_gib": 0.628387451171875, + "status": "passed", + "started_utc": "2026-11-01T10:48:49.194917+00:00", + "exit_code": 0, + "wall_seconds": 4.014455707976595, + "checked_rows": 10 + }, + { + "host": "shared_gpu", + "command": [ + "/Users/cromwellian/xql/xarray-sql/.venv/bin/python", + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + "tiny", + "--steps", + "5", + "--sizes", + "4", + "--optimizer", + "sgd", + "--learning-rate", + "0.5", + "--hosts", + "shared_gpu", + "--csv", + "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.csv" + ], + "contended": false, + "log": "nanogpt-shared-v3-tiny-shared_gpu.log", + "csv": "nanogpt-shared-v3-tiny-shared_gpu.csv", + "peak_rss_gib": 0.6777496337890625, + "status": "passed", + "started_utc": "2026-11-01T10:48:53.209970+00:00", + "exit_code": 0, + "wall_seconds": 4.1090395000064746, + "checked_rows": 10 + } +] diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.csv b/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.csv new file mode 100644 index 00000000..62b06ef4 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,stage_buffer_writes,stage_replacements_keys_or_nulls +nanogpt,"loss, step 0",shared_cpu,4,0.04740233300253749,ok (2e-16),4.179251,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,7,0,0,1,10,10,9,16,175616,10,0.03529512509703636,0.006938831880688667,0.0011270000832155347,125440,,,,,, +nanogpt,"train (sgd), step 0",shared_cpu,4,0.48788500006776303,ok (3e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,32,0,0,1,86,86,63,206,1706880,10,0.31684083584696054,0.11455008445773274,0.009130209917202592,440320,0.004517540684901178,0.0008482909761369228,1,11,, +nanogpt,"loss, step 1",shared_cpu,4,0.0069417080376297235,ok (0e+00),4.104661,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,39,0,0,2,86,96,63,206,1706880,10,0.31684083584696054,0.12070783460512757,0.009767459705471992,440320,,,1,12,9, +nanogpt,"train (sgd), step 1",shared_cpu,4,0.19213799992576241,ok (4e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,64,0,0,2,99,172,73,304,2509696,10,0.388820503721945,0.22379191650543362,0.015906415414065123,473088,0.006231039413250983,0.0008797489572316408,1,23,53,10 +nanogpt,"loss, step 2",shared_cpu,4,0.005637416034005582,ok (1e-16),3.883857,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,71,0,0,3,99,182,73,304,2509696,10,0.388820503721945,0.22882729256525636,0.01639454031828791,473088,,,1,24,62,10 +nanogpt,"train (sgd), step 2",shared_cpu,4,0.1880880839889869,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,96,0,0,3,112,258,83,402,3312512,10,0.4594011284643784,0.3305220474721864,0.02222141798119992,505856,0.008042955305427313,0.0009070408996194601,1,35,106,20 +nanogpt,"loss, step 3",shared_cpu,4,0.005523583968169987,ok (1e-16),3.641763,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,103,0,0,4,112,268,83,402,3312512,10,0.4594011284643784,0.33542921347543597,0.02272412576712668,505856,,,1,36,115,20 +nanogpt,"train (sgd), step 3",shared_cpu,4,0.1911064579617232,ok (6e-16),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,128,0,0,4,125,344,93,500,4115328,10,0.5305807555560023,0.43915771308820695,0.028686666628345847,538624,0.009837454184889793,0.0009330828906968236,1,47,159,30 +nanogpt,"loss, step 4",shared_cpu,4,0.005876000039279461,ok (3e-16),3.842234,,1e-09,1e-09,1,16,2,16,65,4416,sgd,0.5,False,13,1,135,0,0,5,125,354,93,500,4115328,10,0.5305807555560023,0.44439329486340284,0.029205709462985396,538624,,,1,48,168,30 +nanogpt,"train (sgd), step 4",shared_cpu,4,0.22780670900829136,ok (1e-15),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,160,0,0,5,138,430,103,598,4918144,10,0.6167673374293372,0.5663185446755961,0.036390537046827376,571392,0.011585077154450119,0.000961249927058816,1,59,212,40 diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.log b/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.log new file mode 100644 index 00000000..a7bd7ae5 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 shared_cpu batch=4 0.047s + train (sgd), step 0 shared_cpu batch=4 0.488s + loss, step 1 shared_cpu batch=4 0.007s + train (sgd), step 1 shared_cpu batch=4 0.192s + loss, step 2 shared_cpu batch=4 0.006s + train (sgd), step 2 shared_cpu batch=4 0.188s + loss, step 3 shared_cpu batch=4 0.006s + train (sgd), step 3 shared_cpu batch=4 0.191s + loss, step 4 shared_cpu batch=4 0.006s + train (sgd), step 4 shared_cpu batch=4 0.228s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | stage_buffer_writes | stage_replacements_keys_or_nulls | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_cpu | 4 | 0.047 | ok (2e-16) | 4.179 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 7 | 0 | 0 | 1 | 10 | 10 | 9 | 16 | 175616 | 10 | 0.035 | 0.007 | 0.001 | 125440 | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_cpu | 4 | 0.488 | ok (3e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 32 | 0 | 0 | 1 | 86 | 86 | 63 | 206 | 1706880 | 10 | 0.317 | 0.115 | 0.009 | 440320 | 0.005 | 0.001 | 1 | 11 | n/a | n/a | +| nanogpt | loss, step 1 | shared_cpu | 4 | 0.007 | ok (0e+00) | 4.105 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 39 | 0 | 0 | 2 | 86 | 96 | 63 | 206 | 1706880 | 10 | 0.317 | 0.121 | 0.010 | 440320 | n/a | n/a | 1 | 12 | 9 | n/a | +| nanogpt | train (sgd), step 1 | shared_cpu | 4 | 0.192 | ok (4e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 64 | 0 | 0 | 2 | 99 | 172 | 73 | 304 | 2509696 | 10 | 0.389 | 0.224 | 0.016 | 473088 | 0.006 | 0.001 | 1 | 23 | 53 | 10 | +| nanogpt | loss, step 2 | shared_cpu | 4 | 0.006 | ok (1e-16) | 3.884 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 71 | 0 | 0 | 3 | 99 | 182 | 73 | 304 | 2509696 | 10 | 0.389 | 0.229 | 0.016 | 473088 | n/a | n/a | 1 | 24 | 62 | 10 | +| nanogpt | train (sgd), step 2 | shared_cpu | 4 | 0.188 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 96 | 0 | 0 | 3 | 112 | 258 | 83 | 402 | 3312512 | 10 | 0.459 | 0.331 | 0.022 | 505856 | 0.008 | 0.001 | 1 | 35 | 106 | 20 | +| nanogpt | loss, step 3 | shared_cpu | 4 | 0.006 | ok (1e-16) | 3.642 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 103 | 0 | 0 | 4 | 112 | 268 | 83 | 402 | 3312512 | 10 | 0.459 | 0.335 | 0.023 | 505856 | n/a | n/a | 1 | 36 | 115 | 20 | +| nanogpt | train (sgd), step 3 | shared_cpu | 4 | 0.191 | ok (6e-16) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 128 | 0 | 0 | 4 | 125 | 344 | 93 | 500 | 4115328 | 10 | 0.531 | 0.439 | 0.029 | 538624 | 0.010 | 0.001 | 1 | 47 | 159 | 30 | +| nanogpt | loss, step 4 | shared_cpu | 4 | 0.006 | ok (3e-16) | 3.842 | n/a | 1e-09 | 1e-09 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 135 | 0 | 0 | 5 | 125 | 354 | 93 | 500 | 4115328 | 10 | 0.531 | 0.444 | 0.029 | 538624 | n/a | n/a | 1 | 48 | 168 | 30 | +| nanogpt | train (sgd), step 4 | shared_cpu | 4 | 0.228 | ok (1e-15) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 160 | 0 | 0 | 5 | 138 | 430 | 103 | 598 | 4918144 | 10 | 0.617 | 0.566 | 0.036 | 571392 | 0.012 | 0.001 | 1 | 59 | 212 | 40 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-shared_cpu.csv diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.csv b/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.csv new file mode 100644 index 00000000..cea5de43 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,stage_buffer_writes,stage_replacements_keys_or_nulls +nanogpt,"loss, step 0",shared_gpu,4,0.05813316605053842,ok (5e-10),4.179251,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,7,1,0,10,10,9,16,175616,10,0.035397707019001245,0.017396375071257353,0.0011400420917198062,125440,,,,,, +nanogpt,"train (sgd), step 0",shared_gpu,4,0.5777882499387488,ok (1e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,32,1,0,86,86,63,206,1706880,10,0.35364033619407564,0.16184416192118078,0.011007625726051629,440320,0.004639498423784971,0.0009422500152140856,1,11,, +nanogpt,"loss, step 1",shared_gpu,4,0.01436633407138288,ok (1e-09),4.104661,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,39,2,0,86,96,63,206,1706880,10,0.35364033619407564,0.1753199144732207,0.011724751559086144,440320,,,1,12,9, +nanogpt,"train (sgd), step 1",shared_gpu,4,0.21114599995780736,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,64,2,0,99,172,73,304,2509696,10,0.4251532113412395,0.297862381557934,0.01809529901947826,473088,0.006414997391402721,0.0009780001128092408,1,23,53,10 +nanogpt,"loss, step 2",shared_gpu,4,0.008756291936151683,ok (7e-10),3.883857,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,71,3,0,99,182,73,304,2509696,10,0.4251532113412395,0.30596663150936365,0.018626047880388796,473088,,,1,24,62,10 +nanogpt,"train (sgd), step 2",shared_gpu,4,0.2122394589241594,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,96,3,0,112,258,83,402,3312512,10,0.49925179756246507,0.4274676761124283,0.024857794516719878,505856,0.008171080378815532,0.0010059591149911284,1,35,106,20 +nanogpt,"loss, step 3",shared_gpu,4,0.011361208045855165,ok (4e-09),3.641763,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,103,4,0,112,268,83,402,3312512,10,0.49925179756246507,0.43803921912331134,0.025500044226646423,505856,,,1,36,115,20 +nanogpt,"train (sgd), step 3",shared_gpu,4,0.20790537493303418,ok (2e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,128,4,0,125,344,93,500,4115328,10,0.5701148813823238,0.5586874283617362,0.03164112754166126,538624,0.009931456297636032,0.0010321681620553136,1,47,159,30 +nanogpt,"loss, step 4",shared_gpu,4,0.008726583910174668,ok (2e-09),3.842234,,3e-05,3e-05,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,135,5,0,125,354,93,500,4115328,10,0.5701148813823238,0.5667331813601777,0.03218816837761551,538624,,,1,48,168,30 +nanogpt,"train (sgd), step 4",shared_gpu,4,0.2200579580385238,ok (3e-07),,,,,1,16,2,16,65,4416,sgd,0.5,False,13,1,0,160,5,0,138,430,103,598,4918144,10,0.6424464656738564,0.6964817652478814,0.03924583981279284,571392,0.011698538437485695,0.0010589180747047067,1,59,212,40 diff --git a/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.log b/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.log new file mode 100644 index 00000000..1e6f4278 --- /dev/null +++ b/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 4,416 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=16, heads=2, embed=16, forward_only=False + loss, step 0 shared_gpu batch=4 0.058s + train (sgd), step 0 shared_gpu batch=4 0.578s + loss, step 1 shared_gpu batch=4 0.014s + train (sgd), step 1 shared_gpu batch=4 0.211s + loss, step 2 shared_gpu batch=4 0.009s + train (sgd), step 2 shared_gpu batch=4 0.212s + loss, step 3 shared_gpu batch=4 0.011s + train (sgd), step 3 shared_gpu batch=4 0.208s + loss, step 4 shared_gpu batch=4 0.009s + train (sgd), step 4 shared_gpu batch=4 0.220s + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | stage_buffer_writes | stage_replacements_keys_or_nulls | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_gpu | 4 | 0.058 | ok (5e-10) | 4.179 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 7 | 1 | 0 | 10 | 10 | 9 | 16 | 175616 | 10 | 0.035 | 0.017 | 0.001 | 125440 | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_gpu | 4 | 0.578 | ok (1e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 32 | 1 | 0 | 86 | 86 | 63 | 206 | 1706880 | 10 | 0.354 | 0.162 | 0.011 | 440320 | 0.005 | 0.001 | 1 | 11 | n/a | n/a | +| nanogpt | loss, step 1 | shared_gpu | 4 | 0.014 | ok (1e-09) | 4.105 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 39 | 2 | 0 | 86 | 96 | 63 | 206 | 1706880 | 10 | 0.354 | 0.175 | 0.012 | 440320 | n/a | n/a | 1 | 12 | 9 | n/a | +| nanogpt | train (sgd), step 1 | shared_gpu | 4 | 0.211 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 64 | 2 | 0 | 99 | 172 | 73 | 304 | 2509696 | 10 | 0.425 | 0.298 | 0.018 | 473088 | 0.006 | 0.001 | 1 | 23 | 53 | 10 | +| nanogpt | loss, step 2 | shared_gpu | 4 | 0.009 | ok (7e-10) | 3.884 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 71 | 3 | 0 | 99 | 182 | 73 | 304 | 2509696 | 10 | 0.425 | 0.306 | 0.019 | 473088 | n/a | n/a | 1 | 24 | 62 | 10 | +| nanogpt | train (sgd), step 2 | shared_gpu | 4 | 0.212 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 96 | 3 | 0 | 112 | 258 | 83 | 402 | 3312512 | 10 | 0.499 | 0.427 | 0.025 | 505856 | 0.008 | 0.001 | 1 | 35 | 106 | 20 | +| nanogpt | loss, step 3 | shared_gpu | 4 | 0.011 | ok (4e-09) | 3.642 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 103 | 4 | 0 | 112 | 268 | 83 | 402 | 3312512 | 10 | 0.499 | 0.438 | 0.026 | 505856 | n/a | n/a | 1 | 36 | 115 | 20 | +| nanogpt | train (sgd), step 3 | shared_gpu | 4 | 0.208 | ok (2e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 128 | 4 | 0 | 125 | 344 | 93 | 500 | 4115328 | 10 | 0.570 | 0.559 | 0.032 | 538624 | 0.010 | 0.001 | 1 | 47 | 159 | 30 | +| nanogpt | loss, step 4 | shared_gpu | 4 | 0.009 | ok (2e-09) | 3.842 | n/a | 3e-05 | 3e-05 | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 135 | 5 | 0 | 125 | 354 | 93 | 500 | 4115328 | 10 | 0.570 | 0.567 | 0.032 | 538624 | n/a | n/a | 1 | 48 | 168 | 30 | +| nanogpt | train (sgd), step 4 | shared_gpu | 4 | 0.220 | ok (3e-07) | n/a | n/a | n/a | n/a | 1 | 16 | 2 | 16 | 65 | 4416 | sgd | 0.500 | False | 13 | 1 | 0 | 160 | 5 | 0 | 138 | 430 | 103 | 598 | 4918144 | 10 | 0.642 | 0.696 | 0.039 | 571392 | 0.012 | 0.001 | 1 | 59 | 212 | 40 | + +wrote /Users/cromwellian/xql/xarray-sql/benchmarks/apple/nanogpt-shared-v3-tiny-shared_gpu.csv diff --git a/benchmarks/apple/nanogpt-shared.md b/benchmarks/apple/nanogpt-shared.md new file mode 100644 index 00000000..d10ccd2a --- /dev/null +++ b/benchmarks/apple/nanogpt-shared.md @@ -0,0 +1,256 @@ +# Shared SQL execution with CPU and MLX kernels + +This comparison combines shared common table expression (CTE) execution +with [EinFold](https://github.com/xqlsystems/einfold), a tensor optimizer for +SQL, and [MLX](https://github.com/ml-explore/mlx), Apple's array framework. +[ddx](https://github.com/xqlsystems/ddx) still differentiates the original +SQL model. The implementation follows SQL → DataFusion → EinFold → MLX, +including numerical operators generated by SQL differentiation. + +The measured machine is an Apple M5 Ultra with 30 CPU cores, 64 GPU cores +and 256 GiB memory. These final `nanogpt-shared-v3-*` runs took place after +game contention ended, one host per process and one benchmark at a time. +No builds or correctness suites ran concurrently with these timings. +Versions: Python 3.12.14, DataFusion Python 55.0.0, ddxdb 0.4.6, MLX 0.32.4 +and the local EinFold 0.2.0+xqlmlx.3 patch, with native DataFusion 55.2.0. +CPU partition count is 30. + +## Execution changes + +- Materialize reused, reachable forward CTEs once. Preserve the attention + score/softmax/value region so EinFold can fuse it across batch/head keys. +- Retain forward physical plans and prepared ddx programs across training + steps. Reset execution state through EinFold's reference codec, including + DataFusion 55's FFI child-replacement path. Disable four dynamic-filter + settings on the explicit shared executor so cached bounds cannot become + stale after writes. +- Reuse owned Arrow value buffers. Share immutable integer/string keys + using exact equality checks and a bounded cache. Retain verified canonical + ordering so downstream operators need fewer sorts. Changed domains, + schemas and row counts invalidate affected physical plans. +- Keep join-only ddx stages as logical views until their consumers reduce + them. Large backward joins otherwise materialize billions of rows before + performing a matrix contraction. Very large cotangent sums execute their + reduced branches sequentially. Those compound stages rebind transient + ddx providers each call; ordinary stages retain physical plans. +- Dispatch validated dense arithmetic through float64 CPU matrix products + or explicitly permitted float32 MLX matrix/attention kernels. EinFold + retains SQL key, NULL, duplicate-row and output-domain reconstruction. + +The [bridge documentation](../../accelerators/einfold_mlx/README.md) +describes installation and the public experimental `SharedExecutor` API. +Stage boundaries still exchange Arrow buffers. This is not a GPU-resident +training graph, and MLX does not supply the derivatives. + +## Timing and correctness method + +Every completed training run uses **`--steps 5`** and checks the loss and +every updated parameter against the same JAX float64 reference after every +step. Each host starts from identical seeds and batches. CPU checks use +`rtol=atol=1e-9`; GPU checks explicitly permit float32 arithmetic with +`rtol=atol=3e-5`. `MLX_ENABLE_TF32=0` is set before importing MLX. Every GPU +callback synchronizes before returning Arrow data. + +`cpu_forced` and `gpu_forced` retain parameters and prepared queries but +expand the forward CTEs normally. `shared_cpu` and `shared_gpu` add shared +execution. Both pairs set the kernel thresholds to zero to compare actual +CPU/GPU dispatch rather than an automatic policy that can stay on CPU. +Shared hosts use a 4 GiB per-backend working estimate and 4 GiB coordinate +cache budget; library defaults are 512 MiB and 256 MiB respectively. The +plain DataFusion/DuckDB controls also use the checkout's registration fixes. +Each ratio includes the planning, storage and dispatch changes between +the compared final implementations. + +Warm time is the mean paired loss + training API time for steps 1–4; +step 0 is excluded. Each of those four steps uses a different batch. These +are single measurements per step, not a statistical confidence interval. +The measured five-step total includes all five loss/training API calls and +cumulative Dataset loading and model-view setup. It excludes JAX reference +work, parameter validation, result-to-Dataset conversion and reporting. +Runner wall time includes those costs and process startup. Forward-only +results time five checked losses and perform no backward pass or updates. + +Sampled peak RSS is measured once per second. It is not a hard allocator +limit or a measurement of all Metal driver memory. The runner terminates +only its own child process group when a configured memory/time bound is +crossed. Failed or incomplete runs contribute no training speedup. + +## Final measurements + + + +| Model / mode | Host | Warm loss | Warm training | Warm combined | Five-step measured total | Peak sampled RSS | +|---|---|---:|---:|---:|---:|---:| +| 4,416, ctx 16 / training | `datafusion` | 1.093 s | 0.167 s | 1.260 s | 6.795 s | 1.1 GiB | +| 4,416, ctx 16 / training | `duckdb` | 0.028 s | 0.277 s | 0.306 s | 1.601 s | 0.8 GiB | +| 4,416, ctx 16 / training | `cpu_forced` | 0.050 s | 0.264 s | 0.314 s | 3.076 s | 0.9 GiB | +| 4,416, ctx 16 / training | `gpu_forced` | 0.085 s | 0.286 s | 0.371 s | 3.417 s | 1.0 GiB | +| 4,416, ctx 16 / training | `shared_cpu` | 0.006 s | 0.200 s | 0.206 s | 1.371 s | 0.6 GiB | +| 4,416, ctx 16 / training | `shared_gpu` | 0.011 s | 0.213 s | 0.224 s | 1.543 s | 0.7 GiB | +| 213,504, ctx 64 / training | `cpu_forced` | 0.154 s | 0.397 s | 0.551 s | 4.315 s | 2.1 GiB | +| 213,504, ctx 64 / training | `gpu_forced` | 0.180 s | 0.420 s | 0.601 s | 4.676 s | 2.1 GiB | +| 213,504, ctx 64 / training | `shared_cpu` | 0.021 s | 0.410 s | 0.431 s | 2.498 s | 1.0 GiB | +| 213,504, ctx 64 / training | `shared_gpu` | 0.026 s | 0.456 s | 0.482 s | 2.776 s | 1.1 GiB | +| 10,745,088, ctx 256 / training | `shared_cpu` | 0.557 s | 28.700 s | 29.257 s | 148.431 s | 10.0 GiB | +| 10,745,088, ctx 256 / training | `shared_gpu` | 0.591 s | 29.345 s | 29.937 s | 152.365 s | 10.2 GiB | +| 123,563,520, ctx 16 / training | `shared_gpu` | 1.372 s | 162.274 s | 163.646 s | 829.973 s | 40.0 GiB | +| 123,563,520, ctx 16 / training | `shared_cpu` | 1.168 s | 162.866 s | 164.034 s | 829.992 s | 38.7 GiB | +| 124,337,664, ctx 1024 / forward | `shared_cpu` | 6.573 s | — | 6.573 s | 33.269 s | 15.5 GiB | +| 124,337,664, ctx 1024 / forward | `shared_gpu` | 6.857 s | — | 6.857 s | 34.921 s | 15.6 GiB | + + + +Sharing improves the tiny warm CPU step **314 → 206 ms (1.53×)** and GPU +step **371 → 224 ms (1.66×)** against the corresponding unshared prepared +kernel hosts. Shared CPU is **6.12×** faster than plain DataFusion and +**1.48×** faster than DuckDB for that complete warm step. Its five-step +measured total, including preparation/loading, is 1.37 s versus 6.80 s for +DataFusion. These controls all use the final checkout. + +At 213K parameters, sharing improves the CPU step **551 → 431 ms (1.28×)** +and GPU step **601 → 482 ms (1.25×)**. At 10.7M parameters, shared CPU/GPU +take **29.26/29.94 s**. GPU remains slower than CPU on these configurations. +The tiny shared GPU host completes **160 matrix and 5 attention kernels**, +versus **1,015 matrix and 200 attention kernels** without shared CTEs. +The 10.7M GPU run completes **410 matrix and 30 attention kernels**. The +improvement from sharing comes from avoiding repeated SQL work and GPU +dispatches. + +The 123.6M-parameter, context-16 model completes all five training steps on +both hosts: **164.03 s CPU versus 163.65 s GPU** per warm combined step, +effectively equal in this run. First training queries take 171.98/173.35 s. +Each host takes about 830 s for the measured five-step total and 846 s wall +time. CPU/GPU sampled peak RSS is 38.7/40.0 GiB. The GPU executes 710 matrix +and 60 attention kernels across the run. + +At the full context of 1024, the 124.3M-parameter model passes five forward +checks on each host: **6.57 s CPU versus 6.86 s GPU** per warm loss. The GPU +completes 365 matrix and 60 attention kernels. These are forward-only +results; full-context training performance remains unestablished. Across +all 16 final runs, **all 150 loss/update checkpoint records pass**. + +[Machine-readable summary](nanogpt-shared-summary.json) and run records: +[tiny](nanogpt-shared-v3-tiny-runs.json), +[213K](nanogpt-shared-v3-demo-runs.json), +[10.7M](nanogpt-shared-v3-shakespeare-runs.json), +[123.6M short context](nanogpt-shared-v3-gpt2-16-runs.json), +[124.3M forward](nanogpt-shared-v3-gpt2-forward-runs.json). + +## What profiling explains + +The original tiny forward loss expands to **9,882 DataFusion physical +operators**, including 2,988 joins and 1,050 aggregates. DuckDB retains nine +shared CTEs and uses 126 operators. A separate CPU timing breakdown found +about 0.94–0.96 s in DataFusion physical planning, 0.013 s in SQL/DataFrame +construction and 0.16–0.21 s executing an existing forward plan. DuckDB's +forward execution including physical planning took 0.024–0.030 s. See the +[raw planning breakdown](nanogpt-plan-breakdown.json). Operator counts are +not comparable for the opaque EinFold FFI nodes, so accelerated hosts leave +that field empty and record completed kernel/stage counters instead. + +For the checked five-step 10.7M CPU run, shared-executor counters record +19.4 s planning, 23.4 s stage sorting/storage and 100.9 s execution. These +cover the loss and backward stages together; execution includes native CPU +work and buffer conversion as well as numerical kernels. The GPU run +records 20.1 s, 24.0 s and 103.4 s respectively. GPU dispatch changes the +arithmetic inside a stage, while those surrounding costs remain. The +coordinate cache avoids 5.30 GB of additional key copies and retains +263 MB of immutable keys at the end of that CPU run. + +Shared execution addresses the repeated work that made DuckDB faster on +the tiny forward pass. GPU arithmetic alone cannot remove SQL planning, +Arrow coordinate processing, stage storage or many small dispatches. The +[earlier CPU profile](cpu-profile.txt) and [overhead/thread sweep](README.md#training-overheads-and-cpu-parallelism) +also motivated prepared gradients, direct registration and arithmetic +chunk counts. Increasing the partition count above the existing 30-core +default did not consistently improve EinFold, so this comparison uses 30. + +A separate five-step 213K GPU [CPU profile](shared-profile.txt), collected +after all timing runs, records 0.81 s in DataFusion execution entry calls, +0.94 s in record-stream iteration, 0.70 s in physical planning and 0.15 s +in the MLX callback. These cumulative costs include nested native work; +they cannot be added as independent phases. JAX reference compilation also +appears in this whole-process profile and is excluded from the benchmark +API timings. The [profile log](shared-profile-demo.log) and +[checked rows](shared-profile-demo.csv) are kept separately from the final +performance table. + +An additional contraction-input staging experiment did not resolve the +full-vocabulary backward memory limit and increased the 10.7M model's +first CPU training query from approximately 25 s to 63 s. It was removed. +Sequential reduction staging is reserved for at least 32 UNION branches; +applying it to smaller backward programs also added unnecessary work. + +## Precision and size limits + +GPU float32 preserves the SQL and ddx formulas, with explicitly different +rounding from float64. It does not provide arbitrary float64 equivalence. +An AdamW test at 213,504 parameters, batch 12, failed the unchanged GPU +tolerance: `wpe.w[48,53]` differed by 4.1264e-5. The float64 reference +gradient there was 1.54198e-8 and the GPU gradient was 1.30217e-8. AdamW's +division by small gradient moments amplified that difference. Applying +AdamW to identical gradients agreed within 3.0e-14 on CPU and 1.7e-14 on +GPU, isolating the discrepancy to numerical gradient precision. See the +[diagnostic](nanogpt-adamw-diagnostic.json) and +[failed checked run](nanogpt-adamw-failed-demo.log). The final comparisons +use SGD; no tolerance was relaxed to make AdamW pass. + +The GPT-2 preset follows [nanoGPT's GPT-2 dimensions](https://github.com/karpathy/nanoGPT/blob/master/model.py): +12 layers, 12 heads, 768 channels, context 1024 and vocabulary 50257. +This SQL demo omits biases and dropout and uses tied token/output weights. +The 124,337,664-parameter count includes positional embeddings. Character +inputs from Tiny Shakespeare are padded to the larger vocabulary; these +runs do not use GPT-2 BPE, pretrained weights or GPT-2 quality evaluations. +The six-layer configuration follows +[nanoGPT's Shakespeare character model](https://github.com/karpathy/nanoGPT/blob/master/config/train_shakespeare_char.py) +dimensions, with batch 1 and no dropout here. + +The 123,649,536-parameter, context-128 forward loss passed, but backward +execution crossed the 96 GiB sampled-RSS limit. Its ~2.2 s forward timing +does not establish training performance. Successive development attempts +and their exact commands remain in `nanogpt-final-gpt2-128*-runs.json`. +The remaining expensive path includes reconstruction of normalized residual +operands and the tied vocabulary-weight gradient. More GPU arithmetic +dispatch cannot by itself eliminate those relational intermediates. +Reducing context to 16 lets the same 12-layer, full-vocabulary parameter +model complete five checked training updates within approximately 40 GiB +sampled RSS. This establishes short-context training correctness and cost, +while the full context of 1024 is checked only for forward execution. + +## Validation and reproduction + +The final local suite passed **637 tests**, with 277 skipped and 102 +integration tests deselected. Coverage includes SQL bags and NULLs, +coordinate/schema invalidation, repeated changed parameters, snapshots, +batched/head-separated attention, grad/VJP/JVP and typed ddx refusals. +[Python test log](shared-correctness.log). EinFold's native kernel and +planner suites passed 60 tests; four remain ignored upstream, and the +unchanged plain-DataFusion ARM NaN-sign assumption is excluded. See +[kernel tests](shared-native-kernel-tests.log) and +[planner tests](shared-native-planner-tests.log). Ruff checks pass; focused +mypy checks with missing third-party stubs ignored also pass. The native +patch applies cleanly to its pinned revision. + +```sh +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python benchmarks/apple/run_nanogpt_sizes.py \ + --preset tiny --steps 5 --batch 4 --learning-rate 0.5 \ + --hosts datafusion duckdb cpu_forced gpu_forced shared_cpu shared_gpu \ + --prefix local-tiny + +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python benchmarks/apple/run_nanogpt_sizes.py \ + --preset shakespeare --steps 5 --batch 1 --learning-rate 0.01 \ + --hosts shared_cpu shared_gpu --prefix local-shakespeare + +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python benchmarks/apple/run_nanogpt_sizes.py \ + --preset gpt2 --steps 5 --batch 1 --forward-only \ + --hosts shared_cpu shared_gpu --prefix local-gpt2-forward +``` + +The checked CSVs, complete logs, status records and exact commands use the +`nanogpt-shared-v3-` prefix alongside this report. The summary retains failed +statuses separately and includes source hashes to identify the measured +uncommitted implementation. + +The subsequent [native framework comparison](nanogpt-frameworks.md) runs +the same model directly through JAX and upstream nanoGPT's PyTorch model. +It measures the substantial remaining gap to native array training. diff --git a/benchmarks/apple/overheads.py b/benchmarks/apple/overheads.py new file mode 100644 index 00000000..f318f005 --- /dev/null +++ b/benchmarks/apple/overheads.py @@ -0,0 +1,216 @@ +"""Measure training overhead and CPU partition counts independently of GPU math. + +Run from the checkout root with ``python -m benchmarks.apple.overheads``. +Times include checked Arrow results. Registration and in-place writes have +separate measurements. References use NumPy float64 arithmetic. +""" + +from __future__ import annotations + +import argparse +import json +import os +import platform +import statistics +import time +from pathlib import Path + +import numpy as np +import xarray as xr +from einfold.datafusion import EinFold + +from xarray_sql import Context, ad + +_PRODUCT = """SELECT a.i,b.k,SUM(a.v*b.v) AS v FROM a JOIN b ON a.j=b.j +GROUP BY a.i,b.k ORDER BY a.i,b.k""" +_SHARED = """WITH p AS ( +SELECT a.i,b.k,SUM(a.v*b.v) AS v FROM a JOIN b ON a.j=b.j GROUP BY a.i,b.k) +SELECT x.i,x.k,x.v+y.v AS v FROM p x JOIN p y ON x.i=y.i AND x.k=y.k +ORDER BY x.i,x.k""" +_LOSS = """WITH p AS ( +SELECT a.i,b.k,SUM(a.v*b.v) AS v FROM a JOIN b ON a.j=b.j GROUP BY a.i,b.k), +loss AS (SELECT SUM(v*v)/2 AS l FROM p) +SELECT j,k,grad(loss,v) AS v FROM b ORDER BY j,k""" +_ATTENTION = """WITH scores AS ( +SELECT q.i,k.j,SUM(q.v*k.v)/8.0 AS s FROM q JOIN k ON q.d=k.d +WHERE q.i>=k.j GROUP BY q.i,k.j), +weights AS (SELECT i,j,exp(s-MAX(s) OVER (PARTITION BY i)) AS w FROM scores), +norm AS (SELECT i,SUM(w) AS z FROM weights GROUP BY i) +SELECT w.i,v.d,SUM(w.w/n.z*v.v) AS v FROM weights w +JOIN v ON w.j=v.j JOIN norm n ON w.i=n.i GROUP BY w.i,v.d ORDER BY w.i,v.d""" + + +def _dataset(a, dims): + return xr.Dataset( + {"v": (dims, a)}, + coords={d: np.arange(n) for d, n in zip(dims, a.shape)}, + ) + + +def _time(run, check, reps): + check(run()) + times = [] + for _ in range(reps): + start = time.perf_counter() + got = run() + times.append(time.perf_counter() - start) + check(got) + return {"seconds": statistics.median(times), "samples": times} + + +def _check(want): + def check(rows): + np.testing.assert_allclose( + rows["v"].to_numpy().reshape(want.shape), want, rtol=1e-9, atol=1e-9 + ) + + return check + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--reps", type=int, default=9) + parser.add_argument( + "--threads", nargs="+", type=int, default=[1, 10, 20, 30, 60] + ) + parser.add_argument("--thread-tokens", type=int, default=2048) + parser.add_argument("--label", default="current") + parser.add_argument("--skip-thread-sweep", action="store_true") + parser.add_argument("--only-thread-sweep", action="store_true") + args = parser.parse_args() + report = { + "environment": { + "cpu_count": os.cpu_count(), + "platform": platform.platform(), + "reps": args.reps, + "label": args.label, + }, + "results": [], + } + + def record(task, run, check, **fields): + result = {"task": task, **fields, **_time(run, check, args.reps)} + report["results"].append(result) + print(task, fields, f"{result['seconds'] * 1000:.3f} ms", flush=True) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n") + + rng = np.random.default_rng(804) + if not args.only_thread_sweep: + for rows in (32, 4096, 65536): + data = _dataset(np.ones((rows, 16)), ("i", "j")) + ctx = Context() + + def register(ctx=ctx, data=data): + ctx.deregister_table("w") + ctx.from_dataset("w", data) + + record("register_dataset", register, lambda _: None, rows=rows) + query = ctx.prepare("SELECT SUM(v) AS v FROM w") + replacement = data * 2 + record( + "prepared_write_and_sum", + lambda query=query, replacement=replacement: query( + w=replacement + ), + _check(np.array([rows * 32.0])), + rows=rows, + ) + + def replace_and_sum(ctx=ctx, data=replacement): + ctx.deregister_table("w") + ctx.from_dataset("w", data) + return ctx.sql("SELECT SUM(v) AS v FROM w").to_arrow_table() + + record( + "register_and_sum", + replace_and_sum, + _check(np.array([rows * 32.0])), + rows=rows, + ) + data = _dataset(np.ones((4096, 16)), ("i", "j")) + ctx = Context() + + def fine_chunks(): + ctx.deregister_table("w") + ctx.from_dataset("w", data, chunks={"i": 1, "j": 1}) + + record( + "register_fine_chunks", + fine_chunks, + lambda _: None, + rows=4096, + columns=16, + ) + assert ctx.sql("SELECT SUM(v) AS v FROM w").to_pylist() == [ + {"v": 65536.0} + ] + rng = np.random.default_rng(804) + a, b = rng.normal(size=(128, 32)), rng.normal(size=(32, 16)) + for host in ("datafusion", "einfold"): + ctx = Context() + if host == "einfold": + ctx = ctx.with_extensions(EinFold()) + ctx.from_dataset("a", _dataset(a, ("i", "j"))) + ctx.from_dataset("b", _dataset(b, ("j", "k"))) + for name, sql, want in [ + ("product", _PRODUCT, a @ b), + ("repeated_cte", _SHARED, 2 * (a @ b)), + ]: + record( + "sql", + lambda ctx=ctx, sql=sql: ctx.sql(sql).to_arrow_table(), + _check(want), + host=host, + query=name, + ) + prepared = ctx.prepare(sql) + record( + "prepared", prepared, _check(want), host=host, query=name + ) + prepared = ctx.prepare(_LOSS) + record( + "gradient_context_prepared", + prepared, + _check(a.T @ (a @ b)), + host=host, + ) + prepared = ad.prepare(ctx, [_LOSS]) + record( + "gradient_prepared", + lambda prepared=prepared: prepared()[0], + _check(a.T @ (a @ b)), + host=host, + ) + if args.skip_thread_sweep: + return + n = args.thread_tokens + q, k, v = (rng.normal(size=(n, 64)) for _ in range(3)) + score = q @ k.T / 8 + score[np.triu_indices(n, 1)] = -np.inf + weights = np.exp(score - np.max(score, axis=1, keepdims=True)) + want = weights / weights.sum(axis=1, keepdims=True) @ v + for threads in args.threads: + for host in ("datafusion", "einfold"): + ctx = Context(threads=threads) + if host == "einfold": + ctx = ctx.with_extensions(EinFold()) + for name, values, dims in [ + ("q", q, ("i", "d")), + ("k", k, ("j", "d")), + ("v", v, ("j", "d")), + ]: + ctx.from_dataset(name, _dataset(values, dims)) + record( + "thread_sweep", + lambda ctx=ctx: ctx.sql(_ATTENTION).to_arrow_table(), + _check(want), + host=host, + threads=threads, + tokens=n, + ) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/apple/profile_dense_execution.py b/benchmarks/apple/profile_dense_execution.py new file mode 100644 index 00000000..6d464e2a --- /dev/null +++ b/benchmarks/apple/profile_dense_execution.py @@ -0,0 +1,118 @@ +"""Profile one warm compact SQL loss/backward call, outside benchmarks. + +GPU wait time includes pending device computation. NumPy view creation is +reported separately; these are host-observed phases, not GPU kernel events. +The profile leaves input parameters unchanged and is not a training run. +""" + +from __future__ import annotations + +import argparse +import cProfile +import importlib.util +import io +import json +import os +import pstats +import sys +import time +from pathlib import Path + +import mlx.core as mx +import numpy as np + +_ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(_ROOT / "benchmarks/ml")) +from _hosts import DenseCPU, DenseGPU + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--gpu", action="store_true") + parser.add_argument("--context", type=int, default=16) + parser.add_argument("--cache-gib", type=float, default=8) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args() + os.environ["XQL_DENSE_CACHE_BYTES"] = str(int(args.cache_gib * (1 << 30))) + spec = importlib.util.spec_from_file_location( + "dense_profile_model", _ROOT / "benchmarks/ml/02_nanogpt.py" + ) + n = importlib.util.module_from_spec(spec) + spec.loader.exec_module(n) + n.LAYERS, n.EMBED, n.HEADS, n.HEAD, n.HIDDEN, n.CONTEXT = ( + 12, + 768, + 12, + 64, + 3072, + args.context, + ) + host = DenseGPU() if args.gpu else DenseCPU() + parameters = n.init_params(50257, np.random.default_rng(0)) + for name, values in parameters.items(): + host.load(name, n.as_table(name, values)) + host.load( + "batch", next(n.batches(n.shakespeare(), 1, np.random.default_rng(1))) + ) + host.execute(n.model()) + queries = n.step_statements(0.01) + host.query(n.loss_statement()) + host.query_all(queries) + phases = { + "input_conversion_seconds": 0.0, + "device_wait_seconds": 0.0, + "numpy_view_seconds": 0.0, + } + if args.gpu: + runtime = host.executor._runtime + array, numpy = runtime.array, runtime.numpy + + def import_array(value): + start = time.perf_counter() + result = array(value) + phases["input_conversion_seconds"] += time.perf_counter() - start + return result + + def export_array(value): + if runtime.owns(value): + start = time.perf_counter() + mx.eval(value) + phases["device_wait_seconds"] += time.perf_counter() - start + start = time.perf_counter() + result = numpy(value) + phases["numpy_view_seconds"] += time.perf_counter() - start + return result + + runtime.array, runtime.numpy = import_array, export_array + profiler = cProfile.Profile() + start = time.perf_counter() + profiler.enable() + host.query(n.loss_statement()) + rows = host.query_all(queries) + profiler.disable() + elapsed = time.perf_counter() - start + profiler.dump_stats(str(args.output.with_suffix(".pstats"))) + report = io.StringIO() + pstats.Stats(profiler, stream=report).strip_dirs().sort_stats( + "cumtime" + ).print_stats(40) + args.output.with_suffix(".txt").write_text(report.getvalue()) + record = { + "diagnostic": True, + "host": host.name, + "context": args.context, + "parameters": sum(v.size for v in parameters.values()), + "cache_gib": args.cache_gib, + "paired_seconds": elapsed, + "public_output_bytes": sum(t.nbytes for t in rows), + "phases": phases, + "statistics": host.measurements(), + } + args.output.with_suffix(".json").write_text( + json.dumps(record, indent=2) + "\n" + ) + print(json.dumps(record, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/apple/profile_nanogpt_stages.py b/benchmarks/apple/profile_nanogpt_stages.py new file mode 100644 index 00000000..b9d1a857 --- /dev/null +++ b/benchmarks/apple/profile_nanogpt_stages.py @@ -0,0 +1,108 @@ +"""Attribute one SQL nanoGPT backward pass to stages and physical plans. + +Diagnostic timings include instrumentation and are separate from performance +comparisons. Saved SQL is ddx's generated program, not a substitute model. +""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import sys +import time +from pathlib import Path + +import numpy as np +from ddxdb._ddxdb import to_sql + +_ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(_ROOT / "benchmarks/ml")) +from _hosts import SharedCPU, SharedGPU + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--gpu", action="store_true") + parser.add_argument("--context", type=int, default=16) + parser.add_argument("--output", type=Path, required=True) + args = parser.parse_args() + spec = importlib.util.spec_from_file_location( + "xql_profile_nanogpt", _ROOT / "benchmarks/ml/02_nanogpt.py" + ) + n = importlib.util.module_from_spec(spec) + spec.loader.exec_module(n) + n.LAYERS, n.EMBED, n.HEADS, n.HEAD, n.HIDDEN, n.CONTEXT = ( + 12, + 768, + 12, + 64, + 3072, + args.context, + ) + host = SharedGPU() if args.gpu else SharedCPU() + for name, values in n.init_params(50257, np.random.default_rng(0)).items(): + host.load(name, n.as_table(name, values)) + host.load( + "batch", next(n.batches(n.shakespeare(), 1, np.random.default_rng(1))) + ) + host.execute(n.model()) + backend = host.executor._backend + records = [] + materialize = backend.materialize_compiled + compile_plan = backend.compile + + def save(): + args.output.write_text( + json.dumps( + { + "host": host.name, + "context": args.context, + "instrumented": True, + "stages": records, + "statistics": host.measurements(), + }, + indent=2, + ) + + "\n" + ) + + def compile(plan): + start = time.perf_counter() + compiled = compile_plan(plan) + compiled._profile_compile_seconds = time.perf_counter() - start + return compiled + + def execute(name, plan, compiled): + print("BEGIN", name, flush=True) + start = time.perf_counter() + materialize(name, plan, compiled) + elapsed = time.perf_counter() - start + record = { + "name": name, + "seconds": elapsed, + "compile_seconds": getattr(compiled, "_profile_compile_seconds", 0), + "deferred": compiled.deferred, + "graph": compiled.graph is not None, + "rows": None if compiled.batch is None else compiled.batch.num_rows, + "bytes": None if compiled.batch is None else compiled.batch.nbytes, + } + if elapsed > 0.2: + record["sql"] = to_sql(plan, "datafusion") + if compiled.plan is not None: + record["physical"] = compiled.plan.display_indent() + records.append(record) + print("END", name, elapsed, record["rows"], flush=True) + save() + + backend.compile = compile + backend.materialize_compiled = execute + print("FORWARD", host.query(n.loss_statement()).to_pydict(), flush=True) + host.query_all(n.step_statements(0.01)) + save() + for r in sorted(records, key=lambda r: r["seconds"], reverse=True)[:20]: + print(r["name"], r["seconds"], r["rows"], flush=True) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/apple/publication-sources.json b/benchmarks/apple/publication-sources.json new file mode 100644 index 00000000..848f38c2 --- /dev/null +++ b/benchmarks/apple/publication-sources.json @@ -0,0 +1,41 @@ +{ + "xarray-sql": { + "head": "8982f419793508f057475a3a6e43ba34fb29b99a", + "branch": "codex/resident-sql-training" + }, + "ddx": { + "head": "953b78ecfc8bce2b1c1a71f0d4c7300444d00cc7", + "branch": "codex/prepared-primal-values" + }, + "einfold": { + "head": "9feb8e5a7efaecf6322afc436d0cac60814ef842", + "branch": "codex/compact-array-programs" + }, + "note": "Publication commits; performance records retain their original build versions and source hashes. The EinFold publication stack includes the subsequently merged upstream runtime changes.", + "source_sha256": { + "xarray_sql/training.py": "9f56ce5b65615f3aefae6a740e9c991207b89c6dd7610513f02cf95f2a6b3997", + "xarray_sql/dense.py": "c5c0e431f6c872022eff1089000e5cd2c67038910a13d23e248d62ad525ebc86", + "xarray_sql/_dense_mlx.py": "21e8f37e393e0bb026f31fc0f85375a9af2b323456cd4949d1c690474fda1385", + "benchmarks/ml/03_shakespeare.py": "5d17a05d2821d1717c73ecb23c226d710a4f01edb5d9c8aeb291cae4972800ec", + "benchmarks/ml/_shakespeare_torch.py": "f341fc1ff6c3c70c368b78fca2efa21cbbe9521b00e14a3996aed602b7d8d9e0", + "accelerators/einfold_mlx/build.py": "5b64b5f85aea72e9b6c223c6e984ae2ff92edd5144f664ca769ab8375700200c", + "accelerators/ddx_math/build.py": "31c472ab6e63b746775ee0b6238c03cc3e8c6a123eb7a479a248a1ff7ef618ca" + }, + "pull_requests": { + "xarray-sql": [ + 301, + 302, + 303, + 305, + 304 + ], + "ddx": [ + 256, + 257 + ], + "einfold": [ + 156, + 157 + ] + } +} diff --git a/benchmarks/apple/publication-upstream-python.log b/benchmarks/apple/publication-upstream-python.log new file mode 100644 index 00000000..a4616aaf --- /dev/null +++ b/benchmarks/apple/publication-upstream-python.log @@ -0,0 +1,3 @@ +..............................................................sss..s.... [ 53%] +................................................ss............ [100%] +128 passed, 6 skipped in 6.40s diff --git a/benchmarks/apple/resident-final-merged-correctness.log b/benchmarks/apple/resident-final-merged-correctness.log new file mode 100644 index 00000000..f7b6edcc --- /dev/null +++ b/benchmarks/apple/resident-final-merged-correctness.log @@ -0,0 +1,88 @@ +.............sssssssss...sssssssss...ssssssssss..ssssssssss..ssssssssss. [ 7%] +.ssssssssss..sssssssss.....ssssssssssssssssss....sssssssss...sssssssss.. [ 14%] +...ssssssssssssssssss....sssssssss...sssssssss...sssssssss...sssssssss.. [ 21%] +.sssssssss...sssssssss...sssssssss...sssssssss...sssssssss..ssssssssssss [ 29%] +sssssssssssssssssssssssssssssssssssssssssssssssssssssssssss............. [ 36%] +........................................................................ [ 43%] +........................................................................ [ 50%] +........................................................................ [ 58%] +.................................................sss.....s.............. [ 65%] +........................................................................ [ 72%] +........................................................................ [ 79%] +........................................................................ [ 87%] +........................................................................ [ 94%] +....................................................... [100%] +=============================== warnings summary =============================== +tests/test_df.py:688 + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:688: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT"), np.datetime64("2000-01-01")], + +tests/test_adbc_backend.py: 28 warnings +tests/test_flight_sql.py: 10 warnings + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py:371: Warning: Cannot disable autocommit; conn will not be DB-API 2.0 compliant + warnings.warn( + +tests/test_arrow_dataset.py::test_unchunked_zarr_scans_by_storage_chunks + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:260: ZarrUserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future. + warnings.warn( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/tests/test_coord_lookup.py:111: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + template = xr.Dataset( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/xarray_sql/ds.py:707: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + return xr.Dataset(data_vars=data_vars, coords=coords_arg) + +tests/test_df.py::test_partition_metadata_preserves_coordinate_dtype[float16-values9-expected9] + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:639: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"v": (["x"], np.arange(len(coord)))}, coords={"x": coord}) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'z' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'u' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'v' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_flight_sql.py::test_shutdown_does_not_wait_forever_on_an_unread_result + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:63: PytestUnraisableExceptionWarning: Exception ignored in: + + Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 1053, in __del__ + self.close() + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 851, in close + self._stmt.close() + File "adbc_driver_manager/_lib.pyx", line 1555, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 1561, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 262, in adbc_driver_manager._lib.check_error + adbc_driver_manager.OperationalError: IO: [FlightSQL] connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:53472: connect: connection refused" (Unavailable; ClosePreparedStatement). Vendor code: 14 + + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) + +tests/test_lazy_roundtrip.py: 32 warnings + /Users/cromwellian/xql/xarray-sql/xarray_sql/lazyscan.py:63: DeprecationWarning: `is_in` with a collection of the same datatype is ambiguous and deprecated. + Please use `implode` to return to previous behavior. + + See https://github.com/pola-rs/polars/issues/22149 for more information. + return lf.collect(engine="streaming") + +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place + /Users/cromwellian/xql/xarray-sql/tests/test_prepare.py:169: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT") if d is None else np.datetime64(d) for d in days], + +tests/test_reader.py::TestFilterPushdown::test_float_coordinate_types_prune_with_natural_literal[float16] + /Users/cromwellian/xql/xarray-sql/tests/test_reader.py:1067: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"value": (["x"], np.arange(8))}, coords={"x": coord}) + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +714 passed, 277 skipped, 102 deselected, 83 warnings in 57.09s diff --git a/benchmarks/apple/resident-publication-correctness.log b/benchmarks/apple/resident-publication-correctness.log new file mode 100644 index 00000000..990d3eeb --- /dev/null +++ b/benchmarks/apple/resident-publication-correctness.log @@ -0,0 +1,88 @@ +.............sssssssss...sssssssss...ssssssssss..ssssssssss..ssssssssss. [ 7%] +.ssssssssss..sssssssss.....ssssssssssssssssss....sssssssss...sssssssss.. [ 14%] +...ssssssssssssssssss....sssssssss...sssssssss...sssssssss...sssssssss.. [ 21%] +.sssssssss...sssssssss...sssssssss...sssssssss...sssssssss..ssssssssssss [ 29%] +sssssssssssssssssssssssssssssssssssssssssssssssssssssssssss............. [ 36%] +........................................................................ [ 43%] +........................................................................ [ 50%] +........................................................................ [ 58%] +.................................................sss.....s.............. [ 65%] +........................................................................ [ 72%] +........................................................................ [ 79%] +........................................................................ [ 87%] +........................................................................ [ 94%] +....................................................... [100%] +=============================== warnings summary =============================== +tests/test_df.py:688 + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:688: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT"), np.datetime64("2000-01-01")], + +tests/test_adbc_backend.py: 28 warnings +tests/test_flight_sql.py: 10 warnings + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py:371: Warning: Cannot disable autocommit; conn will not be DB-API 2.0 compliant + warnings.warn( + +tests/test_arrow_dataset.py::test_unchunked_zarr_scans_by_storage_chunks + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:260: ZarrUserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future. + warnings.warn( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/tests/test_coord_lookup.py:111: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + template = xr.Dataset( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/xarray_sql/ds.py:707: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + return xr.Dataset(data_vars=data_vars, coords=coords_arg) + +tests/test_df.py::test_partition_metadata_preserves_coordinate_dtype[float16-values9-expected9] + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:639: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"v": (["x"], np.arange(len(coord)))}, coords={"x": coord}) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'z' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'u' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'v' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_flight_sql.py::test_shutdown_does_not_wait_forever_on_an_unread_result + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:63: PytestUnraisableExceptionWarning: Exception ignored in: + + Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 1053, in __del__ + self.close() + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 851, in close + self._stmt.close() + File "adbc_driver_manager/_lib.pyx", line 1555, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 1561, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 262, in adbc_driver_manager._lib.check_error + adbc_driver_manager.OperationalError: IO: [FlightSQL] connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:52812: connect: connection refused" (Unavailable; ClosePreparedStatement). Vendor code: 14 + + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) + +tests/test_lazy_roundtrip.py: 32 warnings + /Users/cromwellian/xql/xarray-sql/xarray_sql/lazyscan.py:63: DeprecationWarning: `is_in` with a collection of the same datatype is ambiguous and deprecated. + Please use `implode` to return to previous behavior. + + See https://github.com/pola-rs/polars/issues/22149 for more information. + return lf.collect(engine="streaming") + +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place + /Users/cromwellian/xql/xarray-sql/tests/test_prepare.py:169: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT") if d is None else np.datetime64(d) for d in days], + +tests/test_reader.py::TestFilterPushdown::test_float_coordinate_types_prune_with_natural_literal[float16] + /Users/cromwellian/xql/xarray-sql/tests/test_reader.py:1067: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"value": (["x"], np.arange(8))}, coords={"x": coord}) + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +714 passed, 277 skipped, 102 deselected, 83 warnings in 75.87s (0:01:15) diff --git a/benchmarks/apple/resident-summary.json b/benchmarks/apple/resident-summary.json new file mode 100644 index 00000000..c7302516 --- /dev/null +++ b/benchmarks/apple/resident-summary.json @@ -0,0 +1,74 @@ +{ + "date": "2026-11-01", + "device": "Apple M5 Ultra, 30 CPU cores, 64 GPU cores, 256 GiB unified memory", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "parameters": 10745088, + "batch_tokens": 3072, + "arithmetic": "float32; MLX_ENABLE_TF32=0; compile_training and gpu_dropout enabled", + "packages": { + "xarray-sql": "0.6.0", + "ddxdb": "0.4.6+xqlmath.3", + "einfold": "0.2.0+xqlmlx.6", + "mlx": "0.32.4", + "datafusion": "55.0.0", + "numpy": "2.5.4", + "pyarrow": "26.0.0", + "xarray": "2026.9.0" + }, + "ddx_base": "af71683f6678bb093e600f948c212490a4d8b74b", + "einfold_base": "39b25210d3b39027ef91584383fb114dfa8a4a48", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "baseline_gpu_seconds": 2.02414754149504, + "native_mps_100_warm_seconds": 0.026962120810316667, + "native_evidence": "benchmarks/apple/shakespeare-resident-torch-mps-100.json", + "scope": "Synchronized SQL/ddx forward, backward, clipping and AdamW, resident state publication and loss scalar. Evaluation, batch sampling/upload, oracle and checkpoint I/O excluded; GPU dropout is inside. First-step excludes prior evaluation and process startup. Caches may already be warm. Warm mean excludes first update. Native includes next-batch prefetch and uses its own RNG trajectory.", + "runs": [ + { + "metrics": "benchmarks/apple/shakespeare-resident-verified-gpu/metrics.jsonl", + "metrics_sha256": "412c14836eb1ba24c0a882625c52ea3d14320145054064b3e956c9a272bff2f8", + "updates": 5, + "verified_all_updates": true, + "first_step_seconds": 1.3801348749548197, + "warm_mean_seconds": 0.14594528125599027, + "warm_tokens_per_second": 21048.984753482127, + "speedup_over_arrow_gpu": 13.869222246005023, + "ratio_to_native_mps_100": 5.412974828009316, + "total_training_compilations": 1, + "timed_array_readback_bytes": 0, + "timed_training_exports": 0, + "status_reads_per_step": [ + 1 + ], + "loss_reads_per_step": [ + 1 + ] + }, + { + "metrics": "benchmarks/apple/shakespeare-resident-steady-gpu/metrics.jsonl", + "metrics_sha256": "ed5a12b7834319e30935b43144560158167241fba335762803f69c5bff6ebbe1", + "updates": 100, + "verified_all_updates": false, + "first_step_seconds": 1.4675100420136005, + "warm_mean_seconds": 0.10133358803601239, + "warm_tokens_per_second": 30315.713274736296, + "speedup_over_arrow_gpu": 19.975090004468107, + "ratio_to_native_mps_100": 3.758368592326704, + "total_training_compilations": 1, + "timed_array_readback_bytes": 0, + "timed_training_exports": 0, + "status_reads_per_step": [ + 1 + ], + "loss_reads_per_step": [ + 1 + ], + "first_rss_bytes": 2639101952, + "last_rss_bytes": 2644819968, + "last_mlx_memory": { + "active_bytes": 532354042, + "cache_bytes": 5081660545, + "peak_bytes": 4428988178 + } + } + ] +} diff --git a/benchmarks/apple/run_nanogpt_sizes.py b/benchmarks/apple/run_nanogpt_sizes.py new file mode 100644 index 00000000..92f9a46d --- /dev/null +++ b/benchmarks/apple/run_nanogpt_sizes.py @@ -0,0 +1,209 @@ +"""Run checked nanoGPT size comparisons in bounded, separate processes. + +Each host starts from identical seeds. Logs, checked CSV checkpoints, exact +commands, exit status and sampled peak resident memory are retained. The +limits apply only to these benchmark processes, never to other applications. +""" + +from __future__ import annotations + +import argparse +import csv +import json +import os +import signal +import subprocess +import sys +import time +from datetime import datetime, timezone +from pathlib import Path + +_ROOT = Path(__file__).resolve().parents[2] + + +def _stop(process): + try: + os.killpg(process.pid, signal.SIGTERM) + except ProcessLookupError: + return + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + os.killpg(process.pid, signal.SIGKILL) + process.wait() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--preset", + choices=["tiny", "demo", "shakespeare", "gpt2"], + required=True, + ) + parser.add_argument( + "--hosts", nargs="+", default=["shared_cpu", "shared_gpu"] + ) + parser.add_argument("--steps", type=int, default=5) + parser.add_argument("--batch", type=int, default=1) + parser.add_argument("--context", type=int) + parser.add_argument("--vocab-size", type=int) + parser.add_argument("--forward-only", action="store_true") + parser.add_argument("--learning-rate", type=float, default=0.01) + parser.add_argument("--optimizer", choices=["sgd", "adamw"], default="sgd") + parser.add_argument( + "--keep-going", + action="store_true", + help="run later hosts after a failure", + ) + parser.add_argument("--max-minutes", type=float, default=20) + parser.add_argument("--rss-limit-gib", type=float, default=96) + parser.add_argument( + "--dense-cache-gib", + type=float, + default=8, + help="computed array cache for compact hosts", + ) + parser.add_argument( + "--contended", + action="store_true", + help="record concurrent game/device load", + ) + parser.add_argument( + "--prefix", help="output basename (defaults to nanogpt-shared-PRESET)" + ) + args = parser.parse_args() + if ( + min( + args.steps, + args.batch, + args.max_minutes, + args.rss_limit_gib, + args.dense_cache_gib, + ) + <= 0 + ): + parser.error("steps, batch, timeout and memory limit must be positive") + output = Path(__file__).resolve().parent + prefix = args.prefix or f"nanogpt-shared-{args.preset}" + results = [] + for host in args.hosts: + stem = output / f"{prefix}-{host}" + csv_path, log_path = stem.with_suffix(".csv"), stem.with_suffix(".log") + command = [ + sys.executable, + "-u", + "benchmarks/ml/02_nanogpt.py", + "--preset", + args.preset, + "--steps", + str(args.steps), + "--sizes", + str(args.batch), + "--optimizer", + args.optimizer, + "--learning-rate", + str(args.learning_rate), + "--hosts", + host, + "--csv", + str(csv_path), + ] + if args.context is not None: + command.extend(["--context", str(args.context)]) + if args.forward_only: + command.append("--forward-only") + vocab = args.vocab_size or (50257 if args.preset == "gpt2" else None) + if vocab is not None: + command.extend(["--vocab-size", str(vocab)]) + # A checkpoint from an older process must never survive a failed run. + csv_path.unlink(missing_ok=True) + started = time.monotonic() + record = { + "host": host, + "command": command, + "contended": args.contended, + "dense_cache_gib": args.dense_cache_gib, + "log": log_path.name, + "csv": csv_path.name, + "peak_rss_gib": 0.0, + "status": "running", + "started_utc": datetime.now(timezone.utc).isoformat(), + } + print( + f"starting {host}: {args.preset}, steps={args.steps}, batch={args.batch}", + flush=True, + ) + with log_path.open("w") as log: + process = subprocess.Popen( + command, + cwd=_ROOT, + stdout=log, + stderr=subprocess.STDOUT, + start_new_session=True, + env={ + **os.environ, + "MLX_ENABLE_TF32": "0", + "XQL_DENSE_CACHE_BYTES": str( + int(args.dense_cache_gib * (1 << 30)) + ), + "PYTHONPATH": str(_ROOT) + + os.pathsep + + os.environ.get("PYTHONPATH", ""), + }, + ) + while process.poll() is None: + try: + process.wait(timeout=1) + except subprocess.TimeoutExpired: + sample = subprocess.run( + ["ps", "-o", "rss=", "-p", str(process.pid)], + capture_output=True, + text=True, + check=False, + ) + rss = sample.stdout.strip() + if rss: + record["peak_rss_gib"] = max( + record["peak_rss_gib"], int(rss) / (1 << 20) + ) + reason = None + if record["peak_rss_gib"] > args.rss_limit_gib: + reason = "rss_limit" + elif time.monotonic() - started > args.max_minutes * 60: + reason = "timeout" + if reason: + record["status"] = reason + _stop(process) + record["exit_code"] = process.returncode + record["wall_seconds"] = time.monotonic() - started + checked = ( + list(csv.DictReader(csv_path.open())) if csv_path.exists() else [] + ) + record["checked_rows"] = sum( + r["check"].startswith("ok (") for r in checked + ) + if record["status"] == "running": + record["status"] = ( + "passed" + if process.returncode == 0 + and record["checked_rows"] + == args.steps * (1 if args.forward_only else 2) + else "failed" + ) + results.append(record) + (output / f"{prefix}-runs.json").write_text( + json.dumps(results, indent=2) + "\n" + ) + print( + f"{host}: {record['status']}, {record['checked_rows']} checked rows, " + f"{record['wall_seconds']:.1f}s wall, sampled peak RSS {record['peak_rss_gib']:.1f} GiB", + flush=True, + ) + if record["status"] != "passed" and not args.keep_going: + break + if any(r["status"] != "passed" for r in results): + raise SystemExit(1) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/apple/shakespeare-correctness.log b/benchmarks/apple/shakespeare-correctness.log new file mode 100644 index 00000000..5d450958 --- /dev/null +++ b/benchmarks/apple/shakespeare-correctness.log @@ -0,0 +1,88 @@ +.............sssssssss...sssssssss...ssssssssss..ssssssssss..ssssssssss. [ 7%] +.ssssssssss..sssssssss.....ssssssssssssssssss....sssssssss...sssssssss.. [ 14%] +...ssssssssssssssssss....sssssssss...sssssssss...sssssssss...sssssssss.. [ 22%] +.sssssssss...sssssssss...sssssssss...sssssssss...sssssssss..ssssssssssss [ 29%] +sssssssssssssssssssssssssssssssssssssssssssssssssssssssssss............. [ 36%] +........................................................................ [ 44%] +........................................................................ [ 51%] +........................................................................ [ 58%] +.................................................sss.....s.............. [ 66%] +........................................................................ [ 73%] +........................................................................ [ 80%] +........................................................................ [ 88%] +........................................................................ [ 95%] +............................................. [100%] +=============================== warnings summary =============================== +tests/test_df.py:688 + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:688: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT"), np.datetime64("2000-01-01")], + +tests/test_adbc_backend.py: 28 warnings +tests/test_flight_sql.py: 10 warnings + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py:371: Warning: Cannot disable autocommit; conn will not be DB-API 2.0 compliant + warnings.warn( + +tests/test_arrow_dataset.py::test_unchunked_zarr_scans_by_storage_chunks + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:260: ZarrUserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future. + warnings.warn( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/tests/test_coord_lookup.py:111: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + template = xr.Dataset( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/xarray_sql/ds.py:707: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + return xr.Dataset(data_vars=data_vars, coords=coords_arg) + +tests/test_df.py::test_partition_metadata_preserves_coordinate_dtype[float16-values9-expected9] + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:639: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"v": (["x"], np.arange(len(coord)))}, coords={"x": coord}) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'z' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'u' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'v' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_flight_sql.py::test_shutdown_does_not_wait_forever_on_an_unread_result + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:63: PytestUnraisableExceptionWarning: Exception ignored in: + + Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 1053, in __del__ + self.close() + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 851, in close + self._stmt.close() + File "adbc_driver_manager/_lib.pyx", line 1555, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 1561, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 262, in adbc_driver_manager._lib.check_error + adbc_driver_manager.OperationalError: IO: [FlightSQL] connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:61529: connect: connection refused" (Unavailable; ClosePreparedStatement). Vendor code: 14 + + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) + +tests/test_lazy_roundtrip.py: 32 warnings + /Users/cromwellian/xql/xarray-sql/xarray_sql/lazyscan.py:63: DeprecationWarning: `is_in` with a collection of the same datatype is ambiguous and deprecated. + Please use `implode` to return to previous behavior. + + See https://github.com/pola-rs/polars/issues/22149 for more information. + return lf.collect(engine="streaming") + +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place + /Users/cromwellian/xql/xarray-sql/tests/test_prepare.py:169: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT") if d is None else np.datetime64(d) for d in days], + +tests/test_reader.py::TestFilterPushdown::test_float_coordinate_types_prune_with_natural_literal[float16] + /Users/cromwellian/xql/xarray-sql/tests/test_reader.py:1067: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"value": (["x"], np.arange(8))}, coords={"x": coord}) + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +704 passed, 277 skipped, 102 deselected, 83 warnings in 48.71s diff --git a/benchmarks/apple/shakespeare-matched-release-cpu-stable.log b/benchmarks/apple/shakespeare-matched-release-cpu-stable.log new file mode 100644 index 00000000..f3d41993 --- /dev/null +++ b/benchmarks/apple/shakespeare-matched-release-cpu-stable.log @@ -0,0 +1,6 @@ +number of parameters: 10.65M +{"backend": "dense_cpu", "parameters": 10745088, "vocabulary": 65, "config": {"backend": "dense_cpu", "data_dir": "/Users/cromwellian/nanoGPT/data/shakespeare_char", "out_dir": "benchmarks/apple/shakespeare-matched-release-cpu-stable", "resume": null, "steps": 5, "verify_nanogpt": "/Users/cromwellian/nanoGPT", "batch_size": 12, "block_size": 256, "n_layer": 6, "n_head": 6, "n_embd": 384, "gradient_accumulation_steps": 1, "max_iters": 2000, "lr_decay_iters": 2000, "warmup_iters": 100, "eval_interval": 200, "eval_iters": 20, "log_interval": 10, "seed": 1337, "dropout": 0.2, "learning_rate": 0.001, "min_lr": 0.0001, "beta1": 0.9, "beta2": 0.99, "weight_decay": 0.1, "grad_clip": 1.0, "cache_gib": 32.0}} +{"kind": "eval", "iteration": 0, "train": 4.1978681571745575, "val": 4.197463574385281} +{"kind": "train", "iteration": 0, "loss": 4.221166694066472, "lr": 9.900990099009901e-06, "seconds": 10.362358084064908, "tokens_per_second": 296.45761853415195, "verified": true} +{"kind": "train", "iteration": 4, "loss": 3.511723839465803, "lr": 4.950495049504951e-05, "seconds": 9.571594874956645, "tokens_per_second": 320.94964738192755, "verified": true} +{"checkpoint": "benchmarks/apple/shakespeare-matched-release-cpu-stable/last.npz", "next_iteration": 5, "measurements": {"table_registers": 55, "inplace_updates": 302, "dense_programs": 770, "dense_executions": 3265, "dense_cache_evictions": 3802, "dense_peak_cached_bytes": 34359736986, "dense_lowering_seconds": 61.90397226775531, "dense_call_seconds": 90.60476937738713, "key_cache_bytes": 0, "dense_last_decline": null}} diff --git a/benchmarks/apple/shakespeare-matched-release-gpu-stable.log b/benchmarks/apple/shakespeare-matched-release-gpu-stable.log new file mode 100644 index 00000000..7e5316c6 --- /dev/null +++ b/benchmarks/apple/shakespeare-matched-release-gpu-stable.log @@ -0,0 +1,6 @@ +number of parameters: 10.65M +{"backend": "dense_gpu", "parameters": 10745088, "vocabulary": 65, "config": {"backend": "dense_gpu", "data_dir": "/Users/cromwellian/nanoGPT/data/shakespeare_char", "out_dir": "benchmarks/apple/shakespeare-matched-release-gpu-stable", "resume": null, "steps": 5, "verify_nanogpt": "/Users/cromwellian/nanoGPT", "batch_size": 12, "block_size": 256, "n_layer": 6, "n_head": 6, "n_embd": 384, "gradient_accumulation_steps": 1, "max_iters": 2000, "lr_decay_iters": 2000, "warmup_iters": 100, "eval_interval": 200, "eval_iters": 20, "log_interval": 10, "seed": 1337, "dropout": 0.2, "learning_rate": 0.001, "min_lr": 0.0001, "beta1": 0.9, "beta2": 0.99, "weight_decay": 0.1, "grad_clip": 1.0, "cache_gib": 16}} +{"kind": "eval", "iteration": 0, "train": 4.197868084907531, "val": 4.197463512420654} +{"kind": "train", "iteration": 0, "loss": 4.221166610717773, "lr": 9.900990099009901e-06, "seconds": 2.52290066704154, "tokens_per_second": 1217.646037409137, "verified": true} +{"kind": "train", "iteration": 4, "loss": 3.511723756790161, "lr": 4.950495049504951e-05, "seconds": 1.8152677500620484, "tokens_per_second": 1692.3123323790635, "verified": true} +{"checkpoint": "benchmarks/apple/shakespeare-matched-release-gpu-stable/last.npz", "next_iteration": 5, "measurements": {"table_registers": 55, "inplace_updates": 302, "dense_programs": 770, "dense_executions": 3265, "dense_cache_evictions": 0, "dense_peak_cached_bytes": 10523102578, "dense_lowering_seconds": 7.730973405065015, "dense_call_seconds": 12.962130957050249, "key_cache_bytes": 0, "gpu_input_arrays": 6905, "gpu_input_bytes": 9938858000, "gpu_fused_programs": 73, "gpu_fused_executions": 9695, "gpu_contractions": 12560, "gpu_readbacks": 325, "gpu_readback_bytes": 859607295, "gpu_scatter_reductions": 260, "gpu_union_reductions": 245, "dense_last_decline": null}} diff --git a/benchmarks/apple/shakespeare-resident-steady-gpu.log b/benchmarks/apple/shakespeare-resident-steady-gpu.log new file mode 100644 index 00000000..4a1a8d6b --- /dev/null +++ b/benchmarks/apple/shakespeare-resident-steady-gpu.log @@ -0,0 +1,14 @@ +{"backend": "dense_gpu", "parameters": 10745088, "vocabulary": 65, "config": {"backend": "dense_gpu", "data_dir": "/Users/cromwellian/nanoGPT/data/shakespeare_char", "out_dir": "benchmarks/apple/shakespeare-resident-steady-gpu", "resume": null, "steps": 100, "verify_nanogpt": null, "batch_size": 12, "block_size": 256, "n_layer": 6, "n_head": 6, "n_embd": 384, "gradient_accumulation_steps": 1, "max_iters": 2000, "lr_decay_iters": 2000, "warmup_iters": 100, "eval_interval": 200, "eval_iters": 20, "log_interval": 10, "seed": 1337, "dropout": 0.2, "learning_rate": 0.001, "min_lr": 0.0001, "beta1": 0.9, "beta2": 0.99, "weight_decay": 0.1, "grad_clip": 1.0, "cache_gib": 16, "resident": true, "compile_training": true, "gpu_dropout": true, "profile_memory": true}} +{"kind": "eval", "iteration": 0, "train": 4.197868084907531, "val": 4.197463512420654} +{"kind": "train", "iteration": 0, "loss": 4.212620735168457, "lr": 9.900990099009901e-06, "seconds": 1.4675100420136005, "tokens_per_second": 2093.3417230895716, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 5456, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 1, "gpu_loss_readbacks": 1}, "rss_bytes": 2639101952, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532360235, "cache_bytes": 4957262112, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 10, "loss": 3.150667905807495, "lr": 0.0001089108910891089, "seconds": 0.09898816596250981, "tokens_per_second": 31034.012703735425, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2640576512, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5080004481, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 20, "loss": 2.8672802448272705, "lr": 0.00020792079207920794, "seconds": 0.10083274997305125, "tokens_per_second": 30466.291961897583, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2641068032, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081188737, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 30, "loss": 2.672114372253418, "lr": 0.00030693069306930693, "seconds": 0.1000648329500109, "tokens_per_second": 30700.09622196312, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2641952768, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081583233, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 40, "loss": 2.6329939365386963, "lr": 0.000405940594059406, "seconds": 0.0993269169703126, "tokens_per_second": 30928.17227900245, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2642952192, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081622145, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 50, "loss": 2.5712954998016357, "lr": 0.000504950495049505, "seconds": 0.09998720802832395, "tokens_per_second": 30723.9301964485, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2643378176, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081622145, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 60, "loss": 2.591752052307129, "lr": 0.000603960396039604, "seconds": 0.09947183297481388, "tokens_per_second": 30883.114426752607, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2643820544, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081622913, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 70, "loss": 2.568532705307007, "lr": 0.000702970297029703, "seconds": 0.10018975008279085, "tokens_per_second": 30661.819172734555, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2644099072, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081635201, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 80, "loss": 2.56632924079895, "lr": 0.000801980198019802, "seconds": 0.1003592920023948, "tokens_per_second": 30610.020643895088, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2644426752, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081660289, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 90, "loss": 2.562467336654663, "lr": 0.0009009900990099009, "seconds": 0.09958774992264807, "tokens_per_second": 30847.167471763223, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2644623360, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081660289, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 99, "loss": 2.5603625774383545, "lr": 0.0009900990099009901, "seconds": 0.10080158396158367, "tokens_per_second": 30475.711583766035, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2644819968, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081660545, "peak_bytes": 4428988178}} +{"checkpoint": "benchmarks/apple/shakespeare-resident-steady-gpu/last.npz", "next_iteration": 100, "measurements": {"dense_cache_evictions": 0, "dense_peak_cached_bytes": 11111962625, "dense_programs": 374, "dense_executions": 1739, "training_steps": 100, "training_exports": 22, "dense_call_seconds": 14.176663464051671, "dense_lowering_seconds": 1.7204898563213646, "key_cache_bytes": 0, "gpu_input_arrays": 3276, "gpu_input_bytes": 614011888, "gpu_status_readbacks": 383, "gpu_fused_programs": 91, "gpu_fused_executions": 8738, "gpu_contractions": 6760, "gpu_readbacks": 73, "gpu_readback_bytes": 128941216, "gpu_training_compilations": 1, "gpu_scatter_reductions": 156, "gpu_union_reductions": 146, "gpu_training_executions": 100, "gpu_loss_readbacks": 100, "dense_last_decline": null, "training_version": 100}} diff --git a/benchmarks/apple/shakespeare-resident-steady-gpu/metrics.jsonl b/benchmarks/apple/shakespeare-resident-steady-gpu/metrics.jsonl new file mode 100644 index 00000000..9f551fc1 --- /dev/null +++ b/benchmarks/apple/shakespeare-resident-steady-gpu/metrics.jsonl @@ -0,0 +1,101 @@ +{"kind": "eval", "iteration": 0, "train": 4.197868084907531, "val": 4.197463512420654} +{"kind": "train", "iteration": 0, "loss": 4.212620735168457, "lr": 9.900990099009901e-06, "seconds": 1.4675100420136005, "tokens_per_second": 2093.3417230895716, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 5456, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 1, "gpu_loss_readbacks": 1}, "rss_bytes": 2639101952, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532360235, "cache_bytes": 4957262112, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 1, "loss": 4.043956279754639, "lr": 1.9801980198019803e-05, "seconds": 0.10743695800192654, "tokens_per_second": 28593.51248520005, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 2, "loss": 3.8534018993377686, "lr": 2.9702970297029702e-05, "seconds": 0.09998241602443159, "tokens_per_second": 30725.402747312382, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 3, "loss": 3.6280620098114014, "lr": 3.9603960396039605e-05, "seconds": 0.10310329194180667, "tokens_per_second": 29795.36290396907, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 4, "loss": 3.5160000324249268, "lr": 4.950495049504951e-05, "seconds": 0.09976200002711266, "tokens_per_second": 30793.288017131894, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 5, "loss": 3.4093363285064697, "lr": 5.9405940594059404e-05, "seconds": 0.10158049990423024, "tokens_per_second": 30242.02482657864, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 6, "loss": 3.3297832012176514, "lr": 6.930693069306931e-05, "seconds": 0.09971095900982618, "tokens_per_second": 30809.05078545343, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 7, "loss": 3.3352365493774414, "lr": 7.920792079207921e-05, "seconds": 0.09968729200772941, "tokens_per_second": 30816.365237023467, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 8, "loss": 3.2835910320281982, "lr": 8.910891089108912e-05, "seconds": 0.09960079193115234, "tokens_per_second": 30843.128256685723, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 9, "loss": 3.3178670406341553, "lr": 9.900990099009902e-05, "seconds": 0.09860754106193781, "tokens_per_second": 31153.803927332512, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 10, "loss": 3.150667905807495, "lr": 0.0001089108910891089, "seconds": 0.09898816596250981, "tokens_per_second": 31034.012703735425, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2640576512, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5080004481, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 11, "loss": 3.1630632877349854, "lr": 0.00011881188118811881, "seconds": 0.10570841690059751, "tokens_per_second": 29061.072808315188, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 12, "loss": 3.044696092605591, "lr": 0.00012871287128712872, "seconds": 0.09900891699362546, "tokens_per_second": 31027.508362684, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 13, "loss": 3.0542805194854736, "lr": 0.00013861386138613863, "seconds": 0.09915012493729591, "tokens_per_second": 30983.319506080104, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 14, "loss": 3.052129030227661, "lr": 0.0001485148514851485, "seconds": 0.0989487919723615, "tokens_per_second": 31046.361848036253, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 15, "loss": 2.945159912109375, "lr": 0.00015841584158415842, "seconds": 0.09907404100522399, "tokens_per_second": 31007.113153263013, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 16, "loss": 2.8976709842681885, "lr": 0.00016831683168316833, "seconds": 0.09873395808972418, "tokens_per_second": 31113.915206441226, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 17, "loss": 2.8428804874420166, "lr": 0.00017821782178217824, "seconds": 0.0987067079404369, "tokens_per_second": 31122.504884407175, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 18, "loss": 2.8468990325927734, "lr": 0.00018811881188118812, "seconds": 0.09867204097099602, "tokens_per_second": 31133.439318469085, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 19, "loss": 2.7697153091430664, "lr": 0.00019801980198019803, "seconds": 0.09867095900699496, "tokens_per_second": 31133.78070828541, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 20, "loss": 2.8672802448272705, "lr": 0.00020792079207920794, "seconds": 0.10083274997305125, "tokens_per_second": 30466.291961897583, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2641068032, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081188737, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 21, "loss": 2.7437469959259033, "lr": 0.0002178217821782178, "seconds": 0.10559629101771861, "tokens_per_second": 29091.93088500174, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 22, "loss": 2.7415847778320312, "lr": 0.0002277227722772277, "seconds": 0.10223533399403095, "tokens_per_second": 30048.319695217702, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 23, "loss": 2.7400169372558594, "lr": 0.00023762376237623762, "seconds": 0.10320170794147998, "tokens_per_second": 29766.949222797382, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 24, "loss": 2.7396652698516846, "lr": 0.00024752475247524753, "seconds": 0.10632508399430662, "tokens_per_second": 28892.523613378908, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 25, "loss": 2.653057813644409, "lr": 0.00025742574257425744, "seconds": 0.1005377909168601, "tokens_per_second": 30555.674358713488, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 26, "loss": 2.726198434829712, "lr": 0.00026732673267326735, "seconds": 0.10114275000523776, "tokens_per_second": 30372.913529055855, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 27, "loss": 2.7244298458099365, "lr": 0.00027722772277227726, "seconds": 0.09972058399580419, "tokens_per_second": 30806.07710970943, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 28, "loss": 2.689089775085449, "lr": 0.00028712871287128717, "seconds": 0.09896858397405595, "tokens_per_second": 31040.153113692195, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 29, "loss": 2.6387815475463867, "lr": 0.000297029702970297, "seconds": 0.09875679202377796, "tokens_per_second": 31106.72123959176, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 30, "loss": 2.672114372253418, "lr": 0.00030693069306930693, "seconds": 0.1000648329500109, "tokens_per_second": 30700.09622196312, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2641952768, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081583233, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 31, "loss": 2.6514194011688232, "lr": 0.00031683168316831684, "seconds": 0.1220430419780314, "tokens_per_second": 25171.44730424682, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 32, "loss": 2.6505534648895264, "lr": 0.00032673267326732675, "seconds": 0.10159208299592137, "tokens_per_second": 30238.576761176675, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 33, "loss": 2.632946491241455, "lr": 0.00033663366336633666, "seconds": 0.108002542052418, "tokens_per_second": 28443.77494845477, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 34, "loss": 2.6333577632904053, "lr": 0.00034653465346534657, "seconds": 0.1124694999307394, "tokens_per_second": 27314.071831845868, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 35, "loss": 2.6442530155181885, "lr": 0.0003564356435643565, "seconds": 0.10284254094585776, "tokens_per_second": 29870.90723105799, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 36, "loss": 2.631610631942749, "lr": 0.00036633663366336634, "seconds": 0.10951733402907848, "tokens_per_second": 28050.354103619233, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 37, "loss": 2.6182329654693604, "lr": 0.00037623762376237625, "seconds": 0.10114591696765274, "tokens_per_second": 30371.962527982716, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 38, "loss": 2.6093592643737793, "lr": 0.00038613861386138616, "seconds": 0.10303666698746383, "tokens_per_second": 29814.629003612485, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 39, "loss": 2.5962469577789307, "lr": 0.00039603960396039607, "seconds": 0.10149237501900643, "tokens_per_second": 30268.28369544715, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 40, "loss": 2.6329939365386963, "lr": 0.000405940594059406, "seconds": 0.0993269169703126, "tokens_per_second": 30928.17227900245, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2642952192, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081622145, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 41, "loss": 2.6068613529205322, "lr": 0.0004158415841584159, "seconds": 0.10756920906715095, "tokens_per_second": 28558.35816439144, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 42, "loss": 2.6137149333953857, "lr": 0.0004257425742574258, "seconds": 0.09929041692521423, "tokens_per_second": 30939.541751686244, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 43, "loss": 2.6228692531585693, "lr": 0.0004356435643564356, "seconds": 0.10060733393765986, "tokens_per_second": 30534.553295125865, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 44, "loss": 2.6079769134521484, "lr": 0.0004455445544554455, "seconds": 0.09953858307562768, "tokens_per_second": 30862.404356971285, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 45, "loss": 2.659830331802368, "lr": 0.0004554455445544554, "seconds": 0.10031529201660305, "tokens_per_second": 30623.44671729169, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 46, "loss": 2.6509110927581787, "lr": 0.0004653465346534653, "seconds": 0.10027833399362862, "tokens_per_second": 30634.733123858892, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 47, "loss": 2.5837552547454834, "lr": 0.00047524752475247524, "seconds": 0.09974945907015353, "tokens_per_second": 30797.159489751924, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 48, "loss": 2.608916759490967, "lr": 0.00048514851485148515, "seconds": 0.09948458301369101, "tokens_per_second": 30879.156417404225, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 49, "loss": 2.6431188583374023, "lr": 0.0004950495049504951, "seconds": 0.10064554202836007, "tokens_per_second": 30522.961455504574, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 50, "loss": 2.5712954998016357, "lr": 0.000504950495049505, "seconds": 0.09998720802832395, "tokens_per_second": 30723.9301964485, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2643378176, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081622145, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 51, "loss": 2.567256212234497, "lr": 0.0005148514851485149, "seconds": 0.11082708300091326, "tokens_per_second": 27718.856409625845, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 52, "loss": 2.567842721939087, "lr": 0.0005247524752475248, "seconds": 0.10052033292595297, "tokens_per_second": 30560.981152568904, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 53, "loss": 2.575836181640625, "lr": 0.0005346534653465347, "seconds": 0.10056995903141797, "tokens_per_second": 30545.900879210956, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 54, "loss": 2.5699961185455322, "lr": 0.0005445544554455446, "seconds": 0.09905483305919915, "tokens_per_second": 31013.125812488615, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 55, "loss": 2.5795650482177734, "lr": 0.0005544554455445545, "seconds": 0.09918166708666831, "tokens_per_second": 30973.466067227746, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 56, "loss": 2.615065813064575, "lr": 0.0005643564356435644, "seconds": 0.09852466708980501, "tokens_per_second": 31180.008933193138, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 57, "loss": 2.5787503719329834, "lr": 0.0005742574257425743, "seconds": 0.1000528329750523, "tokens_per_second": 30703.77828048096, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 58, "loss": 2.5955376625061035, "lr": 0.0005841584158415842, "seconds": 0.10015062498860061, "tokens_per_second": 30673.79759586785, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 59, "loss": 2.5936315059661865, "lr": 0.000594059405940594, "seconds": 0.09952179191168398, "tokens_per_second": 30867.611414453877, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 60, "loss": 2.591752052307129, "lr": 0.000603960396039604, "seconds": 0.09947183297481388, "tokens_per_second": 30883.114426752607, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2643820544, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081622913, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 61, "loss": 2.554917335510254, "lr": 0.0006138613861386139, "seconds": 0.10571241599973291, "tokens_per_second": 29059.97342836022, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 62, "loss": 2.52793288230896, "lr": 0.0006237623762376238, "seconds": 0.09996033401694149, "tokens_per_second": 30732.190225368304, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 63, "loss": 2.5390989780426025, "lr": 0.0006336633663366337, "seconds": 0.10045291692949831, "tokens_per_second": 30581.491248840954, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 64, "loss": 2.5433952808380127, "lr": 0.0006435643564356436, "seconds": 0.09957745892461389, "tokens_per_second": 30850.355423567176, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 65, "loss": 2.5496139526367188, "lr": 0.0006534653465346535, "seconds": 0.1007777500199154, "tokens_per_second": 30482.919090701274, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 66, "loss": 2.588672399520874, "lr": 0.0006633663366336634, "seconds": 0.09978062508162111, "tokens_per_second": 30787.54014105531, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 67, "loss": 2.536708116531372, "lr": 0.0006732673267326733, "seconds": 0.09983704099431634, "tokens_per_second": 30770.142718621708, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 68, "loss": 2.568600654602051, "lr": 0.0006831683168316832, "seconds": 0.100950333988294, "tokens_per_second": 30430.805710422144, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 69, "loss": 2.5463650226593018, "lr": 0.0006930693069306931, "seconds": 0.10043899994343519, "tokens_per_second": 30585.728668446278, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 70, "loss": 2.568532705307007, "lr": 0.000702970297029703, "seconds": 0.10018975008279085, "tokens_per_second": 30661.819172734555, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2644099072, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081635201, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 71, "loss": 2.511798620223999, "lr": 0.000712871287128713, "seconds": 0.10634508403018117, "tokens_per_second": 28887.089873643374, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 72, "loss": 2.5312068462371826, "lr": 0.0007227722772277228, "seconds": 0.10140849999152124, "tokens_per_second": 30293.31860994739, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 73, "loss": 2.565162420272827, "lr": 0.0007326732673267327, "seconds": 0.09970800008159131, "tokens_per_second": 30809.96507287454, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 74, "loss": 2.546576499938965, "lr": 0.0007425742574257426, "seconds": 0.09958887507673353, "tokens_per_second": 30846.818960782664, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 75, "loss": 2.53102707862854, "lr": 0.0007524752475247525, "seconds": 0.09946474991738796, "tokens_per_second": 30885.31366691716, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 76, "loss": 2.534687042236328, "lr": 0.0007623762376237624, "seconds": 0.099621209083125, "tokens_per_second": 30836.807024061418, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 77, "loss": 2.5891096591949463, "lr": 0.0007722772277227723, "seconds": 0.1001677910098806, "tokens_per_second": 30668.540945432014, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 78, "loss": 2.5815138816833496, "lr": 0.0007821782178217822, "seconds": 0.10116445797029883, "tokens_per_second": 30366.396080547554, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 79, "loss": 2.5498015880584717, "lr": 0.0007920792079207921, "seconds": 0.10010870906990021, "tokens_per_second": 30686.640838161216, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 80, "loss": 2.56632924079895, "lr": 0.000801980198019802, "seconds": 0.1003592920023948, "tokens_per_second": 30610.020643895088, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2644426752, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081660289, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 81, "loss": 2.5726816654205322, "lr": 0.000811881188118812, "seconds": 0.10727233299985528, "tokens_per_second": 28637.39339018705, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 82, "loss": 2.533210277557373, "lr": 0.0008217821782178219, "seconds": 0.09941529203206301, "tokens_per_second": 30900.678730684926, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 83, "loss": 2.5415358543395996, "lr": 0.0008316831683168318, "seconds": 0.10071520903147757, "tokens_per_second": 30501.848028135213, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 84, "loss": 2.5572240352630615, "lr": 0.0008415841584158417, "seconds": 0.09983079205267131, "tokens_per_second": 30772.068785943266, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 85, "loss": 2.5534398555755615, "lr": 0.0008514851485148516, "seconds": 0.09913037496153265, "tokens_per_second": 30989.49238507454, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 86, "loss": 2.492457151412964, "lr": 0.0008613861386138615, "seconds": 0.09928104199934751, "tokens_per_second": 30942.463315606514, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 87, "loss": 2.567437171936035, "lr": 0.0008712871287128712, "seconds": 0.10042741696815938, "tokens_per_second": 30589.256328020274, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 88, "loss": 2.614652395248413, "lr": 0.0008811881188118811, "seconds": 0.10232199996244162, "tokens_per_second": 30022.868993252774, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 89, "loss": 2.549145460128784, "lr": 0.000891089108910891, "seconds": 0.09926562500186265, "tokens_per_second": 30947.269006187755, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 90, "loss": 2.562467336654663, "lr": 0.0009009900990099009, "seconds": 0.09958774992264807, "tokens_per_second": 30847.167471763223, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2644623360, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081660289, "peak_bytes": 4428988178}} +{"kind": "train", "iteration": 91, "loss": 2.573745012283325, "lr": 0.0009108910891089108, "seconds": 0.11584975000005215, "tokens_per_second": 26517.1051296927, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 92, "loss": 2.589726686477661, "lr": 0.0009207920792079207, "seconds": 0.09892899997066706, "tokens_per_second": 31052.57306665248, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 93, "loss": 2.521784543991089, "lr": 0.0009306930693069307, "seconds": 0.09855470806360245, "tokens_per_second": 31170.504792297488, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 94, "loss": 2.5659496784210205, "lr": 0.0009405940594059406, "seconds": 0.09876670793164521, "tokens_per_second": 31103.598209693086, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 95, "loss": 2.536386251449585, "lr": 0.0009504950495049505, "seconds": 0.09874108294025064, "tokens_per_second": 31111.670122748223, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 96, "loss": 2.549837827682495, "lr": 0.0009603960396039604, "seconds": 0.09891345805954188, "tokens_per_second": 31057.452244271764, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 97, "loss": 2.5956199169158936, "lr": 0.0009702970297029703, "seconds": 0.09956100000999868, "tokens_per_second": 30855.455446324224, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 98, "loss": 2.5719308853149414, "lr": 0.0009801980198019802, "seconds": 0.09971891692839563, "tokens_per_second": 30806.592115374526, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 99, "loss": 2.5603625774383545, "lr": 0.0009900990099009901, "seconds": 0.10080158396158367, "tokens_per_second": 30475.711583766035, "verified": false, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}, "rss_bytes": 2644819968, "peak_rss_bytes": 2721824768, "mlx_memory": {"active_bytes": 532354042, "cache_bytes": 5081660545, "peak_bytes": 4428988178}} diff --git a/benchmarks/apple/shakespeare-resident-torch-mps-100.json b/benchmarks/apple/shakespeare-resident-torch-mps-100.json new file mode 100644 index 00000000..70a6b12a --- /dev/null +++ b/benchmarks/apple/shakespeare-resident-torch-mps-100.json @@ -0,0 +1,792 @@ +{ + "backend": "torch_mps", + "date": "2026-11-01", + "python": "3.12.14", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "torch_version": "2.15.0", + "steps": 100, + "parameters": 10745088, + "batch_tokens": 3072, + "config": { + "out_dir": "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/shakespeare-resident-torch-mps-100-out", + "eval_interval": 200, + "log_interval": 1, + "eval_iters": 20, + "eval_only": false, + "always_save_checkpoint": false, + "init_from": "scratch", + "wandb_log": false, + "wandb_project": "shakespeare-char", + "wandb_run_name": "mini-gpt", + "dataset": "shakespeare_char", + "gradient_accumulation_steps": 1, + "batch_size": 12, + "block_size": 256, + "n_layer": 6, + "n_head": 6, + "n_embd": 384, + "dropout": 0.2, + "bias": false, + "learning_rate": 0.001, + "max_iters": 99, + "weight_decay": 0.1, + "beta1": 0.9, + "beta2": 0.99, + "grad_clip": 1.0, + "decay_lr": true, + "warmup_iters": 100, + "lr_decay_iters": 2000, + "min_lr": 0.0001, + "backend": "nccl", + "device": "mps", + "dtype": "float32", + "compile": false + }, + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "source_sha256": { + "train.py": "413bd97b40bb400a2a0d01230e2946f056c263ebac1b7281c2367d964a123085", + "model.py": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "config/train_shakespeare_char.py": "9b41cdfb2c917259d796a8184d835221a349c23cd18a0a63b46da431d95d9c05" + }, + "instrumentation_sha256": "85cfe28aeaf7d105149a4bee073825bb80c61262aa82c0a7999f95146c265df1", + "native_argv": [ + "train.py", + "config/train_shakespeare_char.py", + "--device=mps", + "--dtype=float32", + "--compile=False", + "--batch_size=12", + "--block_size=256", + "--n_layer=6", + "--n_head=6", + "--n_embd=384", + "--gradient_accumulation_steps=1", + "--max_iters=99", + "--lr_decay_iters=2000", + "--warmup_iters=100", + "--eval_interval=200", + "--eval_iters=20", + "--log_interval=1", + "--wandb_log=False", + "--out_dir=/Users/cromwellian/xql/xarray-sql/benchmarks/apple/shakespeare-resident-torch-mps-100-out" + ], + "rows": [ + { + "iteration": 0, + "lr": 9.900990099009901e-06, + "loss": 4.258163928985596, + "seconds": 0.085335582960397, + "tokens_per_second": 35999.05096360179 + }, + { + "iteration": 1, + "lr": 1.9801980198019803e-05, + "loss": 4.1329345703125, + "seconds": 0.029339374974370003, + "tokens_per_second": 104705.70701262746 + }, + { + "iteration": 2, + "lr": 2.9702970297029702e-05, + "loss": 3.9018430709838867, + "seconds": 0.024168374948203564, + "tokens_per_second": 127108.25641292617 + }, + { + "iteration": 3, + "lr": 3.9603960396039605e-05, + "loss": 3.6646385192871094, + "seconds": 0.02443512505851686, + "tokens_per_second": 125720.65797262023 + }, + { + "iteration": 4, + "lr": 4.950495049504951e-05, + "loss": 3.5102603435516357, + "seconds": 0.025529541075229645, + "tokens_per_second": 120331.18773845278 + }, + { + "iteration": 5, + "lr": 5.9405940594059404e-05, + "loss": 3.4128153324127197, + "seconds": 0.023800749913789332, + "tokens_per_second": 129071.56333843873 + }, + { + "iteration": 6, + "lr": 6.930693069306931e-05, + "loss": 3.349642753601074, + "seconds": 0.02932554204016924, + "tokens_per_second": 104755.09696605326 + }, + { + "iteration": 7, + "lr": 7.920792079207921e-05, + "loss": 3.3626480102539062, + "seconds": 0.02830391691531986, + "tokens_per_second": 108536.21458792653 + }, + { + "iteration": 8, + "lr": 8.910891089108912e-05, + "loss": 3.362469434738159, + "seconds": 0.023630207986570895, + "tokens_per_second": 130003.08764721094 + }, + { + "iteration": 9, + "lr": 9.900990099009902e-05, + "loss": 3.227586030960083, + "seconds": 0.025374541990458965, + "tokens_per_second": 121066.22461028448 + }, + { + "iteration": 10, + "lr": 0.0001089108910891089, + "loss": 3.189467668533325, + "seconds": 0.023909708950668573, + "tokens_per_second": 128483.37076533503 + }, + { + "iteration": 11, + "lr": 0.00011881188118811881, + "loss": 3.0526139736175537, + "seconds": 0.023747375002130866, + "tokens_per_second": 129361.66627782429 + }, + { + "iteration": 12, + "lr": 0.00012871287128712872, + "loss": 3.132127523422241, + "seconds": 0.024750292068347335, + "tokens_per_second": 124119.74741618184 + }, + { + "iteration": 13, + "lr": 0.00013861386138613863, + "loss": 2.9958336353302, + "seconds": 0.026189875090494752, + "tokens_per_second": 117297.23755402482 + }, + { + "iteration": 14, + "lr": 0.0001485148514851485, + "loss": 3.0269088745117188, + "seconds": 0.023989250068552792, + "tokens_per_second": 128057.35865945416 + }, + { + "iteration": 15, + "lr": 0.00015841584158415842, + "loss": 2.9712722301483154, + "seconds": 0.02486316696740687, + "tokens_per_second": 123556.2631271827 + }, + { + "iteration": 16, + "lr": 0.00016831683168316833, + "loss": 2.8759543895721436, + "seconds": 0.02432841609697789, + "tokens_per_second": 126272.09218037044 + }, + { + "iteration": 17, + "lr": 0.00017821782178217824, + "loss": 2.8801815509796143, + "seconds": 0.025599832995794713, + "tokens_per_second": 120000.7828373192 + }, + { + "iteration": 18, + "lr": 0.00018811881188118812, + "loss": 2.85270619392395, + "seconds": 0.024541541002690792, + "tokens_per_second": 125175.51361844713 + }, + { + "iteration": 19, + "lr": 0.00019801980198019803, + "loss": 2.8255856037139893, + "seconds": 0.029676083009690046, + "tokens_per_second": 103517.70477919572 + }, + { + "iteration": 20, + "lr": 0.00020792079207920794, + "loss": 2.802380323410034, + "seconds": 0.03061033389531076, + "tokens_per_second": 100358.26497373177 + }, + { + "iteration": 21, + "lr": 0.0002178217821782178, + "loss": 2.7515790462493896, + "seconds": 0.03349004208575934, + "tokens_per_second": 91728.7590183793 + }, + { + "iteration": 22, + "lr": 0.0002277227722772277, + "loss": 2.8213050365448, + "seconds": 0.029485124978236854, + "tokens_per_second": 104188.12883674265 + }, + { + "iteration": 23, + "lr": 0.00023762376237623762, + "loss": 2.795553207397461, + "seconds": 0.028092917054891586, + "tokens_per_second": 109351.40676197947 + }, + { + "iteration": 24, + "lr": 0.00024752475247524753, + "loss": 2.736558198928833, + "seconds": 0.03997095790691674, + "tokens_per_second": 76855.80133340784 + }, + { + "iteration": 25, + "lr": 0.00025742574257425744, + "loss": 2.737056016921997, + "seconds": 0.04282162501476705, + "tokens_per_second": 71739.45404782327 + }, + { + "iteration": 26, + "lr": 0.00026732673267326735, + "loss": 2.721369504928589, + "seconds": 0.050365042057819664, + "tokens_per_second": 60994.68747535856 + }, + { + "iteration": 27, + "lr": 0.00027722772277227726, + "loss": 2.666743040084839, + "seconds": 0.042536415974609554, + "tokens_per_second": 72220.47108608563 + }, + { + "iteration": 28, + "lr": 0.00028712871287128717, + "loss": 2.650385618209839, + "seconds": 0.03761962498538196, + "tokens_per_second": 81659.50620703162 + }, + { + "iteration": 29, + "lr": 0.000297029702970297, + "loss": 2.7341578006744385, + "seconds": 0.03282724996097386, + "tokens_per_second": 93580.790460733 + }, + { + "iteration": 30, + "lr": 0.00030693069306930693, + "loss": 2.6708500385284424, + "seconds": 0.03186145890504122, + "tokens_per_second": 96417.43051238431 + }, + { + "iteration": 31, + "lr": 0.00031683168316831684, + "loss": 2.629115343093872, + "seconds": 0.0295188749441877, + "tokens_per_second": 104069.00689163562 + }, + { + "iteration": 32, + "lr": 0.00032673267326732675, + "loss": 2.6278908252716064, + "seconds": 0.029290499980561435, + "tokens_per_second": 104880.42204942643 + }, + { + "iteration": 33, + "lr": 0.00033663366336633666, + "loss": 2.656036376953125, + "seconds": 0.028508458053693175, + "tokens_per_second": 107757.49408172684 + }, + { + "iteration": 34, + "lr": 0.00034653465346534657, + "loss": 2.6278536319732666, + "seconds": 0.02703737502451986, + "tokens_per_second": 113620.49744895876 + }, + { + "iteration": 35, + "lr": 0.0003564356435643565, + "loss": 2.720553398132324, + "seconds": 0.030423750053159893, + "tokens_per_second": 100973.74566357686 + }, + { + "iteration": 36, + "lr": 0.00036633663366336634, + "loss": 2.6574971675872803, + "seconds": 0.026332083041779697, + "tokens_per_second": 116663.76697680252 + }, + { + "iteration": 37, + "lr": 0.00037623762376237625, + "loss": 2.641315460205078, + "seconds": 0.026108750025741756, + "tokens_per_second": 117661.70333590008 + }, + { + "iteration": 38, + "lr": 0.00038613861386138616, + "loss": 2.589958429336548, + "seconds": 0.024314124952070415, + "tokens_per_second": 126346.31129253989 + }, + { + "iteration": 39, + "lr": 0.00039603960396039607, + "loss": 2.6001198291778564, + "seconds": 0.025928208022378385, + "tokens_per_second": 118480.99943307253 + }, + { + "iteration": 40, + "lr": 0.000405940594059406, + "loss": 2.6292078495025635, + "seconds": 0.02398479206021875, + "tokens_per_second": 128081.1604406289 + }, + { + "iteration": 41, + "lr": 0.0004158415841584159, + "loss": 2.581244468688965, + "seconds": 0.025166834006085992, + "tokens_per_second": 122065.41352230126 + }, + { + "iteration": 42, + "lr": 0.0004257425742574258, + "loss": 2.619795799255371, + "seconds": 0.024953874992206693, + "tokens_per_second": 123107.13269820466 + }, + { + "iteration": 43, + "lr": 0.0004356435643564356, + "loss": 2.6092331409454346, + "seconds": 0.024929958977736533, + "tokens_per_second": 123225.2328511018 + }, + { + "iteration": 44, + "lr": 0.0004455445544554455, + "loss": 2.5824902057647705, + "seconds": 0.025409167050383985, + "tokens_per_second": 120901.24772325331 + }, + { + "iteration": 45, + "lr": 0.0004554455445544554, + "loss": 2.5890729427337646, + "seconds": 0.025858583045192063, + "tokens_per_second": 118800.01292534794 + }, + { + "iteration": 46, + "lr": 0.0004653465346534653, + "loss": 2.6262547969818115, + "seconds": 0.02563291694968939, + "tokens_per_second": 119845.89994301158 + }, + { + "iteration": 47, + "lr": 0.00047524752475247524, + "loss": 2.5728628635406494, + "seconds": 0.024451375007629395, + "tokens_per_second": 125637.10625850142 + }, + { + "iteration": 48, + "lr": 0.00048514851485148515, + "loss": 2.5873005390167236, + "seconds": 0.025666375062428415, + "tokens_per_second": 119689.67150709688 + }, + { + "iteration": 49, + "lr": 0.0004950495049504951, + "loss": 2.591423988342285, + "seconds": 0.024755166028626263, + "tokens_per_second": 124095.3099020873 + }, + { + "iteration": 50, + "lr": 0.000504950495049505, + "loss": 2.574220895767212, + "seconds": 0.025892333942465484, + "tokens_per_second": 118645.1560074187 + }, + { + "iteration": 51, + "lr": 0.0005148514851485149, + "loss": 2.581940174102783, + "seconds": 0.024616874987259507, + "tokens_per_second": 124792.4442720661 + }, + { + "iteration": 52, + "lr": 0.0005247524752475248, + "loss": 2.588782548904419, + "seconds": 0.024097625049762428, + "tokens_per_second": 127481.44241003892 + }, + { + "iteration": 53, + "lr": 0.0005346534653465347, + "loss": 2.602567434310913, + "seconds": 0.02369095792528242, + "tokens_per_second": 129669.72503554344 + }, + { + "iteration": 54, + "lr": 0.0005445544554455446, + "loss": 2.5464887619018555, + "seconds": 0.024918792070820928, + "tokens_per_second": 123280.45401515305 + }, + { + "iteration": 55, + "lr": 0.0005544554455445545, + "loss": 2.552751302719116, + "seconds": 0.024100791895762086, + "tokens_per_second": 127464.69133822048 + }, + { + "iteration": 56, + "lr": 0.0005643564356435644, + "loss": 2.5507423877716064, + "seconds": 0.024907833081670105, + "tokens_per_second": 123334.69515100901 + }, + { + "iteration": 57, + "lr": 0.0005742574257425743, + "loss": 2.5519697666168213, + "seconds": 0.024275499978102744, + "tokens_per_second": 126547.34208444892 + }, + { + "iteration": 58, + "lr": 0.0005841584158415842, + "loss": 2.6325881481170654, + "seconds": 0.024804166983813047, + "tokens_per_second": 123850.15799985368 + }, + { + "iteration": 59, + "lr": 0.000594059405940594, + "loss": 2.573974370956421, + "seconds": 0.024829584057442844, + "tokens_per_second": 123723.37743930698 + }, + { + "iteration": 60, + "lr": 0.000603960396039604, + "loss": 2.5605356693267822, + "seconds": 0.025533707928843796, + "tokens_per_second": 120311.55085508588 + }, + { + "iteration": 61, + "lr": 0.0006138613861386139, + "loss": 2.5274195671081543, + "seconds": 0.029588250094093382, + "tokens_per_second": 103824.99776873438 + }, + { + "iteration": 62, + "lr": 0.0006237623762376238, + "loss": 2.5843403339385986, + "seconds": 0.023867083014920354, + "tokens_per_second": 128712.83843440603 + }, + { + "iteration": 63, + "lr": 0.0006336633663366337, + "loss": 2.551851511001587, + "seconds": 0.0252909999107942, + "tokens_per_second": 121466.13462636842 + }, + { + "iteration": 64, + "lr": 0.0006435643564356436, + "loss": 2.587202548980713, + "seconds": 0.024731875048018992, + "tokens_per_second": 124212.17534196078 + }, + { + "iteration": 65, + "lr": 0.0006534653465346535, + "loss": 2.557934522628784, + "seconds": 0.024531125091016293, + "tokens_per_second": 125228.66312091889 + }, + { + "iteration": 66, + "lr": 0.0006633663366336634, + "loss": 2.5685551166534424, + "seconds": 0.02445845794863999, + "tokens_per_second": 125600.722925822 + }, + { + "iteration": 67, + "lr": 0.0006732673267326733, + "loss": 2.548347234725952, + "seconds": 0.025308625074103475, + "tokens_per_second": 121381.54447368064 + }, + { + "iteration": 68, + "lr": 0.0006831683168316832, + "loss": 2.56166934967041, + "seconds": 0.025087208952754736, + "tokens_per_second": 122452.8406402369 + }, + { + "iteration": 69, + "lr": 0.0006930693069306931, + "loss": 2.588054895401001, + "seconds": 0.028669708990491927, + "tokens_per_second": 107151.41897738841 + }, + { + "iteration": 70, + "lr": 0.000702970297029703, + "loss": 2.5407092571258545, + "seconds": 0.028996167005971074, + "tokens_per_second": 105945.03747227674 + }, + { + "iteration": 71, + "lr": 0.000712871287128713, + "loss": 2.6150436401367188, + "seconds": 0.02628637501038611, + "tokens_per_second": 116866.62762690597 + }, + { + "iteration": 72, + "lr": 0.0007227722772277228, + "loss": 2.5520284175872803, + "seconds": 0.0248262919485569, + "tokens_per_second": 123739.7838696797 + }, + { + "iteration": 73, + "lr": 0.0007326732673267327, + "loss": 2.5438811779022217, + "seconds": 0.02685883396770805, + "tokens_per_second": 114375.77683727509 + }, + { + "iteration": 74, + "lr": 0.0007425742574257426, + "loss": 2.5333008766174316, + "seconds": 0.0267595408950001, + "tokens_per_second": 114800.1758346306 + }, + { + "iteration": 75, + "lr": 0.0007524752475247525, + "loss": 2.553542375564575, + "seconds": 0.027210625004954636, + "tokens_per_second": 112897.07602969928 + }, + { + "iteration": 76, + "lr": 0.0007623762376237624, + "loss": 2.5517117977142334, + "seconds": 0.025551665923558176, + "tokens_per_second": 120226.9945603692 + }, + { + "iteration": 77, + "lr": 0.0007722772277227723, + "loss": 2.530944585800171, + "seconds": 0.024524333071894944, + "tokens_per_second": 125263.34522509537 + }, + { + "iteration": 78, + "lr": 0.0007821782178217822, + "loss": 2.584386110305786, + "seconds": 0.025131833041086793, + "tokens_per_second": 122235.41334918702 + }, + { + "iteration": 79, + "lr": 0.0007920792079207921, + "loss": 2.581756353378296, + "seconds": 0.024838499957695603, + "tokens_per_second": 123678.96633178992 + }, + { + "iteration": 80, + "lr": 0.000801980198019802, + "loss": 2.5355255603790283, + "seconds": 0.025191667024046183, + "tokens_per_second": 121945.08593129968 + }, + { + "iteration": 81, + "lr": 0.000811881188118812, + "loss": 2.5468780994415283, + "seconds": 0.027794708032160997, + "tokens_per_second": 110524.6364324251 + }, + { + "iteration": 82, + "lr": 0.0008217821782178219, + "loss": 2.5307397842407227, + "seconds": 0.024225833010859787, + "tokens_per_second": 126806.78508032749 + }, + { + "iteration": 83, + "lr": 0.0008316831683168318, + "loss": 2.528714895248413, + "seconds": 0.02536545810289681, + "tokens_per_second": 121109.58089296911 + }, + { + "iteration": 84, + "lr": 0.0008415841584158417, + "loss": 2.5082244873046875, + "seconds": 0.02394929202273488, + "tokens_per_second": 128271.01515501058 + }, + { + "iteration": 85, + "lr": 0.0008514851485148516, + "loss": 2.5860579013824463, + "seconds": 0.025103791966103017, + "tokens_per_second": 122371.95098445844 + }, + { + "iteration": 86, + "lr": 0.0008613861386138615, + "loss": 2.5327844619750977, + "seconds": 0.02890670800115913, + "tokens_per_second": 106272.91076786799 + }, + { + "iteration": 87, + "lr": 0.0008712871287128712, + "loss": 2.5511598587036133, + "seconds": 0.025116708013229072, + "tokens_per_second": 122309.02228038663 + }, + { + "iteration": 88, + "lr": 0.0008811881188118811, + "loss": 2.556593179702759, + "seconds": 0.024580332916229963, + "tokens_per_second": 124977.96553323377 + }, + { + "iteration": 89, + "lr": 0.000891089108910891, + "loss": 2.5564515590667725, + "seconds": 0.025468707899563015, + "tokens_per_second": 120618.60429333789 + }, + { + "iteration": 90, + "lr": 0.0009009900990099009, + "loss": 2.521738052368164, + "seconds": 0.024684750009328127, + "tokens_per_second": 124449.30569842194 + }, + { + "iteration": 91, + "lr": 0.0009108910891089108, + "loss": 2.529806137084961, + "seconds": 0.023941874969750643, + "tokens_per_second": 128310.75276607691 + }, + { + "iteration": 92, + "lr": 0.0009207920792079207, + "loss": 2.5546865463256836, + "seconds": 0.025045207934454083, + "tokens_per_second": 122658.19505430915 + }, + { + "iteration": 93, + "lr": 0.0009306930693069307, + "loss": 2.4922850131988525, + "seconds": 0.024255166994407773, + "tokens_per_second": 126653.42608064809 + }, + { + "iteration": 94, + "lr": 0.0009405940594059406, + "loss": 2.5221784114837646, + "seconds": 0.02563604200258851, + "tokens_per_second": 119831.29063721363 + }, + { + "iteration": 95, + "lr": 0.0009504950495049505, + "loss": 2.5305659770965576, + "seconds": 0.028339667012915015, + "tokens_per_second": 108399.29765582713 + }, + { + "iteration": 96, + "lr": 0.0009603960396039604, + "loss": 2.5316760540008545, + "seconds": 0.024501917068846524, + "tokens_per_second": 125377.94456524216 + }, + { + "iteration": 97, + "lr": 0.0009702970297029703, + "loss": 2.528303384780884, + "seconds": 0.03040549997240305, + "tokens_per_second": 101034.35242927233 + }, + { + "iteration": 98, + "lr": 0.0009801980198019802, + "loss": 2.5204098224639893, + "seconds": 0.030852458090521395, + "tokens_per_second": 99570.67248861416 + }, + { + "iteration": 99, + "lr": 0.0009900990099009901, + "loss": 2.5874476432800293, + "seconds": 0.028243625070899725, + "tokens_per_second": 108767.90752916402 + } + ], + "warm_mean_seconds": 0.026962120810316667, + "warm_tokens_per_second": 113937.62462575064, + "attention": "original model.py scaled_dot_product_attention selection", + "scope": "Synchronized native forward, backward, gradient clipping, AdamW, zero_grad and next-batch prefetch. Evaluation, initialization, logging and checkpoint I/O excluded. Weights and optimizer state remain on MPS.", + "matching": "Same model, data split, optimizer, dropout and LR schedule as the SQL recipe; stock PyTorch RNGs, rather than SQL NumPy RNGs. max_iters=steps-1 stops native's inclusive loop after exactly steps updates. compile=False.", + "sql_comparison": [ + { + "backend": "dense_gpu", + "seconds": 2.02414754149504, + "sql_over_torch_ratio": 75.07375090169202 + }, + { + "backend": "dense_cpu", + "seconds": 9.746508520736825, + "sql_over_torch_ratio": 361.48894181230224 + } + ] +} diff --git a/benchmarks/apple/shakespeare-resident-verified-gpu.log b/benchmarks/apple/shakespeare-resident-verified-gpu.log new file mode 100644 index 00000000..911e4487 --- /dev/null +++ b/benchmarks/apple/shakespeare-resident-verified-gpu.log @@ -0,0 +1,6 @@ +number of parameters: 10.65M +{"backend": "dense_gpu", "parameters": 10745088, "vocabulary": 65, "config": {"backend": "dense_gpu", "data_dir": "/Users/cromwellian/nanoGPT/data/shakespeare_char", "out_dir": "benchmarks/apple/shakespeare-resident-verified-gpu", "resume": null, "steps": 5, "verify_nanogpt": "/Users/cromwellian/nanoGPT", "batch_size": 12, "block_size": 256, "n_layer": 6, "n_head": 6, "n_embd": 384, "gradient_accumulation_steps": 1, "max_iters": 2000, "lr_decay_iters": 2000, "warmup_iters": 100, "eval_interval": 200, "eval_iters": 20, "log_interval": 10, "seed": 1337, "dropout": 0.2, "learning_rate": 0.001, "min_lr": 0.0001, "beta1": 0.9, "beta2": 0.99, "weight_decay": 0.1, "grad_clip": 1.0, "cache_gib": 16, "resident": true, "compile_training": true, "gpu_dropout": true, "profile_memory": false}} +{"kind": "eval", "iteration": 0, "train": 4.197868084907531, "val": 4.197463512420654} +{"kind": "train", "iteration": 0, "loss": 4.212620735168457, "lr": 9.900990099009901e-06, "seconds": 1.3801348749548197, "tokens_per_second": 2225.8694101187507, "verified": true, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 5456, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 1, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 4, "loss": 3.5160000324249268, "lr": 4.950495049504951e-05, "seconds": 0.14052720798645169, "tokens_per_second": 21860.535365480067, "verified": true, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"checkpoint": "benchmarks/apple/shakespeare-resident-verified-gpu/last.npz", "next_iteration": 5, "measurements": {"dense_cache_evictions": 0, "dense_peak_cached_bytes": 11111962625, "dense_programs": 374, "dense_executions": 1739, "training_steps": 5, "training_exports": 282, "dense_call_seconds": 7.864916921011172, "dense_lowering_seconds": 1.7172620004275814, "key_cache_bytes": 0, "gpu_input_arrays": 2801, "gpu_input_bytes": 609340168, "gpu_status_readbacks": 98, "gpu_fused_programs": 91, "gpu_fused_executions": 8738, "gpu_contractions": 6760, "gpu_readbacks": 388, "gpu_readback_bytes": 1861487776, "gpu_training_compilations": 1, "gpu_scatter_reductions": 156, "gpu_union_reductions": 146, "gpu_training_executions": 5, "gpu_loss_readbacks": 5, "dense_last_decline": null, "training_version": 5}} diff --git a/benchmarks/apple/shakespeare-resident-verified-gpu/metrics.jsonl b/benchmarks/apple/shakespeare-resident-verified-gpu/metrics.jsonl new file mode 100644 index 00000000..4ac4633a --- /dev/null +++ b/benchmarks/apple/shakespeare-resident-verified-gpu/metrics.jsonl @@ -0,0 +1,6 @@ +{"kind": "eval", "iteration": 0, "train": 4.197868084907531, "val": 4.197463512420654} +{"kind": "train", "iteration": 0, "loss": 4.212620735168457, "lr": 9.900990099009901e-06, "seconds": 1.3801348749548197, "tokens_per_second": 2225.8694101187507, "verified": true, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 5456, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 1, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 1, "loss": 4.043956279754639, "lr": 1.9801980198019803e-05, "seconds": 0.15056524996180087, "tokens_per_second": 20403.11426958997, "verified": true, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 2, "loss": 3.8534018993377686, "lr": 2.9702970297029702e-05, "seconds": 0.15196337504312396, "tokens_per_second": 20215.397289828765, "verified": true, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 3, "loss": 3.6280620098114014, "lr": 3.9603960396039605e-05, "seconds": 0.14072529203258455, "tokens_per_second": 21829.764611813254, "verified": true, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} +{"kind": "train", "iteration": 4, "loss": 3.5160000324249268, "lr": 4.950495049504951e-05, "seconds": 0.14052720798645169, "tokens_per_second": 21860.535365480067, "verified": true, "device_step": {"gpu_readback_bytes": 0, "gpu_input_bytes": 0, "gpu_status_readbacks": 1, "training_exports": 0, "gpu_training_compilations": 0, "gpu_loss_readbacks": 1}} diff --git a/benchmarks/apple/shakespeare-summary.json b/benchmarks/apple/shakespeare-summary.json new file mode 100644 index 00000000..d9a6fbbb --- /dev/null +++ b/benchmarks/apple/shakespeare-summary.json @@ -0,0 +1,290 @@ +{ + "date": "2026-11-01", + "device": "Apple M5 Ultra, 30 CPU cores, 64 GPU cores, 256 GiB unified memory", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "python": "3.12.14", + "packages": { + "xarray-sql": "0.6.0", + "ddxdb": "0.4.6+xqlmath.2", + "einfold": "0.2.0+xqlmlx.5", + "mlx": "0.32.4", + "datafusion": "55.0.0", + "numpy": "2.5.4", + "pyarrow": "26.0.0", + "xarray": "2026.9.0" + }, + "ddx_revision": "af71683f6678bb093e600f948c212490a4d8b74b", + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "code_sha256": { + "benchmarks/ml/02_nanogpt.py": "a8a45f5daad78e64cd8a6d0957cc4092cb28b2116d6136797b3695be7a6573a9", + "benchmarks/ml/03_shakespeare.py": "c1407a2f4564cb47607ef46e3a215e5e2003775b27022a52be0b9c5394d5429e", + "benchmarks/ml/_shakespeare_torch.py": "f341fc1ff6c3c70c368b78fca2efa21cbbe9521b00e14a3996aed602b7d8d9e0", + "accelerators/einfold_mlx/dense_program.py": "5151e331bfdd1ba88a7dca0e553d27660cd8a1ebfe93684ef1f7f5725507ea84", + "accelerators/ddx_math/erf.patch": "66e35a7330d47f9944e8ab025f092fd627c5715cc07efd06845298431403c24e", + "xarray_sql/_dense_mlx.py": "caf4fb2c2412ec5bad8dd284297c94d26e7941b904759076191f5bb8aecd13d9", + "xarray_sql/math.py": "cbb68d604168da39d7ab2b3e1068cb1d8534f3cbe3845215626e4df57370d499", + "xarray_sql/dense.py": "c5c0e431f6c872022eff1089000e5cd2c67038910a13d23e248d62ad525ebc86" + }, + "environment": { + "MLX_ENABLE_TF32": "0 (GPU run)", + "PYTHONPATH": "/Users/cromwellian/xql/xarray-sql:/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages" + }, + "scope": "SQL loss forward + ddx backward + global clipping + SQL AdamW + result registration. Batch/mask sampling, evaluation, reference checks and checkpoint I/O excluded. Warm mean is steps 2-5. Runs are serial. Five updates do not establish convergence.", + "runs": [ + { + "backend": "dense_gpu", + "precision": "float32", + "cache_gib": 16, + "config": { + "backend": "dense_gpu", + "data_dir": "/Users/cromwellian/nanoGPT/data/shakespeare_char", + "out_dir": "benchmarks/apple/shakespeare-matched-release-gpu-stable", + "resume": null, + "steps": 5, + "verify_nanogpt": "/Users/cromwellian/nanoGPT", + "batch_size": 12, + "block_size": 256, + "n_layer": 6, + "n_head": 6, + "n_embd": 384, + "gradient_accumulation_steps": 1, + "max_iters": 2000, + "lr_decay_iters": 2000, + "warmup_iters": 100, + "eval_interval": 200, + "eval_iters": 20, + "log_interval": 10, + "seed": 1337, + "dropout": 0.2, + "learning_rate": 0.001, + "min_lr": 0.0001, + "beta1": 0.9, + "beta2": 0.99, + "weight_decay": 0.1, + "grad_clip": 1.0, + "cache_gib": 16 + }, + "steps": 5, + "verified_every_gradient_weight_and_moment": true, + "warm_mean_seconds": 2.02414754149504, + "warm_tokens_per_second": 1517.6759287670372, + "cold_seconds": 2.52290066704154, + "training": [ + { + "kind": "train", + "iteration": 0, + "loss": 4.221166610717773, + "lr": 9.900990099009901e-06, + "seconds": 2.52290066704154, + "tokens_per_second": 1217.646037409137, + "verified": true + }, + { + "kind": "train", + "iteration": 1, + "loss": 4.05787992477417, + "lr": 1.9801980198019803e-05, + "seconds": 2.5857169170631096, + "tokens_per_second": 1188.0650893096283, + "verified": true + }, + { + "kind": "train", + "iteration": 2, + "loss": 3.844763994216919, + "lr": 2.9702970297029702e-05, + "seconds": 1.8406671659322456, + "tokens_per_second": 1668.9600688585758, + "verified": true + }, + { + "kind": "train", + "iteration": 3, + "loss": 3.62751841545105, + "lr": 3.9603960396039605e-05, + "seconds": 1.8549383329227567, + "tokens_per_second": 1656.1197455872104, + "verified": true + }, + { + "kind": "train", + "iteration": 4, + "loss": 3.511723756790161, + "lr": 4.950495049504951e-05, + "seconds": 1.8152677500620484, + "tokens_per_second": 1692.3123323790635, + "verified": true + } + ], + "evaluation": [ + { + "kind": "eval", + "iteration": 0, + "train": 4.197868084907531, + "val": 4.197463512420654 + } + ], + "measurements": { + "table_registers": 55, + "inplace_updates": 302, + "dense_programs": 770, + "dense_executions": 3265, + "dense_cache_evictions": 0, + "dense_peak_cached_bytes": 10523102578, + "dense_lowering_seconds": 7.730973405065015, + "dense_call_seconds": 12.962130957050249, + "key_cache_bytes": 0, + "gpu_input_arrays": 6905, + "gpu_input_bytes": 9938858000, + "gpu_fused_programs": 73, + "gpu_fused_executions": 9695, + "gpu_contractions": 12560, + "gpu_readbacks": 325, + "gpu_readback_bytes": 859607295, + "gpu_scatter_reductions": 260, + "gpu_union_reductions": 245, + "dense_last_decline": null + }, + "log": "benchmarks/apple/shakespeare-matched-release-gpu-stable.log", + "checkpoint": "benchmarks/apple/shakespeare-matched-release-gpu-stable/last.npz", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "benchmarks/ml/03_shakespeare.py", + "--backend", + "dense_gpu", + "--steps", + "5", + "--cache-gib", + "16", + "--verify-nanogpt", + "/Users/cromwellian/nanoGPT", + "--out-dir", + "benchmarks/apple/shakespeare-matched-release-gpu-stable" + ] + }, + { + "backend": "dense_cpu", + "precision": "float64", + "cache_gib": 32, + "config": { + "backend": "dense_cpu", + "data_dir": "/Users/cromwellian/nanoGPT/data/shakespeare_char", + "out_dir": "benchmarks/apple/shakespeare-matched-release-cpu-stable", + "resume": null, + "steps": 5, + "verify_nanogpt": "/Users/cromwellian/nanoGPT", + "batch_size": 12, + "block_size": 256, + "n_layer": 6, + "n_head": 6, + "n_embd": 384, + "gradient_accumulation_steps": 1, + "max_iters": 2000, + "lr_decay_iters": 2000, + "warmup_iters": 100, + "eval_interval": 200, + "eval_iters": 20, + "log_interval": 10, + "seed": 1337, + "dropout": 0.2, + "learning_rate": 0.001, + "min_lr": 0.0001, + "beta1": 0.9, + "beta2": 0.99, + "weight_decay": 0.1, + "grad_clip": 1.0, + "cache_gib": 32.0 + }, + "steps": 5, + "verified_every_gradient_weight_and_moment": true, + "warm_mean_seconds": 9.746508520736825, + "warm_tokens_per_second": 315.18979267949794, + "cold_seconds": 10.362358084064908, + "training": [ + { + "kind": "train", + "iteration": 0, + "loss": 4.221166694066472, + "lr": 9.900990099009901e-06, + "seconds": 10.362358084064908, + "tokens_per_second": 296.45761853415195, + "verified": true + }, + { + "kind": "train", + "iteration": 1, + "loss": 4.057880311620805, + "lr": 1.9801980198019803e-05, + "seconds": 10.331209582975134, + "tokens_per_second": 297.3514355049353, + "verified": true + }, + { + "kind": "train", + "iteration": 2, + "loss": 3.844764134425487, + "lr": 2.9702970297029702e-05, + "seconds": 9.599913167068735, + "tokens_per_second": 320.0028944572228, + "verified": true + }, + { + "kind": "train", + "iteration": 3, + "loss": 3.6275185921665165, + "lr": 3.9603960396039605e-05, + "seconds": 9.483316457946785, + "tokens_per_second": 323.93730754663784, + "verified": true + }, + { + "kind": "train", + "iteration": 4, + "loss": 3.511723839465803, + "lr": 4.950495049504951e-05, + "seconds": 9.571594874956645, + "tokens_per_second": 320.94964738192755, + "verified": true + } + ], + "evaluation": [ + { + "kind": "eval", + "iteration": 0, + "train": 4.1978681571745575, + "val": 4.197463574385281 + } + ], + "measurements": { + "table_registers": 55, + "inplace_updates": 302, + "dense_programs": 770, + "dense_executions": 3265, + "dense_cache_evictions": 3802, + "dense_peak_cached_bytes": 34359736986, + "dense_lowering_seconds": 61.90397226775531, + "dense_call_seconds": 90.60476937738713, + "key_cache_bytes": 0, + "dense_last_decline": null + }, + "log": "benchmarks/apple/shakespeare-matched-release-cpu-stable.log", + "checkpoint": "benchmarks/apple/shakespeare-matched-release-cpu-stable/last.npz", + "command": [ + "/private/tmp/xql-framework-venv/bin/python", + "benchmarks/ml/03_shakespeare.py", + "--backend", + "dense_cpu", + "--steps", + "5", + "--cache-gib", + "32", + "--verify-nanogpt", + "/Users/cromwellian/nanoGPT", + "--out-dir", + "benchmarks/apple/shakespeare-matched-release-cpu-stable" + ] + } + ], + "gpu_speedup_vs_cpu": 4.815117633933952, + "earlier_trials": "Other shakespeare-matched logs are diagnostics on a newer unreleased ddx snapshot or the intermediate window/CASE implementations. Only the release-*-stable runs describe the final implementation." +} diff --git a/benchmarks/apple/shakespeare-torch-mps.json b/benchmarks/apple/shakespeare-torch-mps.json new file mode 100644 index 00000000..7013e377 --- /dev/null +++ b/benchmarks/apple/shakespeare-torch-mps.json @@ -0,0 +1,127 @@ +{ + "backend": "torch_mps", + "date": "2026-11-01", + "python": "3.12.14", + "platform": "macOS-27.0.1-arm64-arm-64bit", + "torch_version": "2.15.0", + "steps": 5, + "parameters": 10745088, + "batch_tokens": 3072, + "config": { + "out_dir": "/Users/cromwellian/xql/xarray-sql/benchmarks/apple/shakespeare-torch-mps-out", + "eval_interval": 200, + "log_interval": 1, + "eval_iters": 20, + "eval_only": false, + "always_save_checkpoint": false, + "init_from": "scratch", + "wandb_log": false, + "wandb_project": "shakespeare-char", + "wandb_run_name": "mini-gpt", + "dataset": "shakespeare_char", + "gradient_accumulation_steps": 1, + "batch_size": 12, + "block_size": 256, + "n_layer": 6, + "n_head": 6, + "n_embd": 384, + "dropout": 0.2, + "bias": false, + "learning_rate": 0.001, + "max_iters": 4, + "weight_decay": 0.1, + "beta1": 0.9, + "beta2": 0.99, + "grad_clip": 1.0, + "decay_lr": true, + "warmup_iters": 100, + "lr_decay_iters": 2000, + "min_lr": 0.0001, + "backend": "nccl", + "device": "mps", + "dtype": "float32", + "compile": false + }, + "nanogpt_revision": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291", + "source_sha256": { + "train.py": "413bd97b40bb400a2a0d01230e2946f056c263ebac1b7281c2367d964a123085", + "model.py": "7c01703240dbec5d554527dc666e35b3df8391d0b117fddc07afcf325a21d11c", + "config/train_shakespeare_char.py": "9b41cdfb2c917259d796a8184d835221a349c23cd18a0a63b46da431d95d9c05" + }, + "instrumentation_sha256": "85cfe28aeaf7d105149a4bee073825bb80c61262aa82c0a7999f95146c265df1", + "native_argv": [ + "train.py", + "config/train_shakespeare_char.py", + "--device=mps", + "--dtype=float32", + "--compile=False", + "--batch_size=12", + "--block_size=256", + "--n_layer=6", + "--n_head=6", + "--n_embd=384", + "--gradient_accumulation_steps=1", + "--max_iters=4", + "--lr_decay_iters=2000", + "--warmup_iters=100", + "--eval_interval=200", + "--eval_iters=20", + "--log_interval=1", + "--wandb_log=False", + "--out_dir=/Users/cromwellian/xql/xarray-sql/benchmarks/apple/shakespeare-torch-mps-out" + ], + "rows": [ + { + "iteration": 0, + "lr": 9.900990099009901e-06, + "loss": 4.258163928985596, + "seconds": 0.5524522500345483, + "tokens_per_second": 5560.661577191311 + }, + { + "iteration": 1, + "lr": 1.9801980198019803e-05, + "loss": 4.132935047149658, + "seconds": 0.03179487492889166, + "tokens_per_second": 96619.34531494278 + }, + { + "iteration": 2, + "lr": 2.9702970297029702e-05, + "loss": 3.9018430709838867, + "seconds": 0.029624666087329388, + "tokens_per_second": 103697.37133725565 + }, + { + "iteration": 3, + "lr": 3.9603960396039605e-05, + "loss": 3.6646385192871094, + "seconds": 0.0316031250404194, + "tokens_per_second": 97205.57685580173 + }, + { + "iteration": 4, + "lr": 4.950495049504951e-05, + "loss": 3.5102603435516357, + "seconds": 0.02439495804719627, + "tokens_per_second": 125927.66070991736 + } + ], + "warm_mean_seconds": 0.02935440602595918, + "warm_tokens_per_second": 104652.09199884057, + "attention": "original model.py scaled_dot_product_attention selection", + "scope": "Synchronized native forward, backward, gradient clipping, AdamW, zero_grad and next-batch prefetch. Evaluation, initialization, logging and checkpoint I/O excluded. Weights and optimizer state remain on MPS.", + "matching": "Same model, data split, optimizer, dropout and LR schedule as the SQL recipe; stock PyTorch RNGs, rather than SQL NumPy RNGs. max_iters=steps-1 stops native's inclusive loop after exactly steps updates. compile=False.", + "sql_comparison": [ + { + "backend": "dense_gpu", + "seconds": 2.02414754149504, + "sql_over_torch_ratio": 68.9554930767467 + }, + { + "backend": "dense_cpu", + "seconds": 9.746508520736825, + "sql_over_torch_ratio": 332.02881067045365 + } + ] +} diff --git a/benchmarks/apple/shakespeare-training.md b/benchmarks/apple/shakespeare-training.md new file mode 100644 index 00000000..6e8180ad --- /dev/null +++ b/benchmarks/apple/shakespeare-training.md @@ -0,0 +1,193 @@ +# SQL Shakespeare training matching nanoGPT + +This report records the earlier Arrow-output implementation. The later +GPU-resident implementation measures 101.3 ms per warm step over 100 +updates (19.98× faster), with a separate fully checked five-step run. +See [the complete performance report](../../NANOGPT_PERFORMANCE.md) and +[resident measurements](resident-summary.json) for its timing scope and +the updated native PyTorch comparison. + +[nanoGPT](https://github.com/karpathy/nanoGPT) is a small implementation +of GPT training in PyTorch. The new +[`03_shakespeare.py`](../ml/03_shakespeare.py) trains its Shakespeare +character model through SQL, with gradients generated by +[ddx](https://github.com/xqlsystems/ddx), compact array programs from +[EinFold](https://github.com/xqlsystems/einfold), and optional +[MLX](https://github.com/ml-explore/mlx) GPU execution. It initializes once +per run, then keeps updating the parameters; these are training timings. + +The recipe matches the local nanoGPT command: batch 12, context 256, +6 layers, 6 heads, 384 channels, gradient accumulation 1, max iteration +2000, cosine decay through iteration 2000, warmup 100, evaluation every +200 iterations and 20 batches per train/validation split. It also matches +inherited settings: exact GELU, dropout 0.2 at all four types of dropout +site, tied embeddings, no biases, global gradient clipping at 1, and +AdamW with betas 0.9/0.99, epsilon 1e-8, learning rate 1e-3 to 1e-4 and +weight decay 0.1 excluding layer norm gains. The vocabulary has 65 +characters and the model has **10,745,088 total parameters**. It is not a +123M-parameter GPT-2 vocabulary model. + +## Measured results + +Measured November 1, 2026, on Apple M5 Ultra, 30 CPU cores, 64 GPU cores +and 256 GiB unified memory. Both backends completed exactly **five +updates**, using `--steps 5` without shortening the learning-rate schedule. +Each also evaluated 20 train and 20 validation batches at iteration 0. +Runs were serial. PyTorch reference checks were outside each timer. + +| Backend | Arithmetic | Computed cache limit | First step | Mean steps 2–5 | Tokens/s, warm | +|---|---|---:|---:|---:|---:| +| Compact CPU | float64 | 32 GiB | 10.362 s | **9.747 s** | 315 | +| Compact MLX GPU | float32 | 16 GiB | 2.523 s | **2.024 s** | 1,518 | + +The GPU path was **4.82× faster** than the measured CPU configuration. +The precision and cache budgets differ as shown. This compares these +SQL execution configurations, not native PyTorch or DuckDB performance. +First-step measurements include plan/JIT preparation in the final +process; driver/kernel caches may already have been warmed by earlier +trials. Five steps do not establish model convergence. + +Training time includes a loss forward call, ddx backward gradients, +global clipping, all SQL AdamW updates, Arrow conversion and registration +of the new weights/moments. Batch and dropout-mask sampling, evaluation, +reference checks and checkpoint I/O are outside the timer. There are +3,072 training tokens per update. These timings therefore retain SQL's +public Arrow boundaries and the separate loss-forward call. + +Every evaluation/training loss, every raw parameter gradient, every +updated weight and both AdamW moments were checked against the original +nanoGPT `model.py` plus PyTorch AdamW after every update. The oracle uses +identical initial arrays, batches and dropout masks, exact GELU and +explicit causal attention. CPU checks use 1e-8 absolute/relative tolerance; +GPU checks use the existing 3e-5 tolerance. Stock nanoGPT samples with +PyTorch RNGs; the SQL script uses NumPy RNGs, so an integer seed alone +does not produce identical independent runs. + +Both final runs used strict compact execution with no SQL fallback. The +GPU's peak computed cache was approximately 9.80 GiB. The CPU reached its +32 GiB computed-cache limit and evicted intermediates 3,802 times. These +cache counters are not total process memory. A diagnostic CPU trial with +a 16 GiB cache evicted more often and was slower; it is excluded from the +final comparison because it used a newer ddx source snapshot. + +### Native PyTorch MPS comparison + +The original nanoGPT `train.py` was also measured with the same model, +batch, context, data split, dropout, exact GELU, clipping, AdamW and +learning-rate schedule, using PyTorch 2.15.0 on MPS, float32 and +`compile=False`. It completed five actual updates; the warm mean again +uses updates 2–5: + +| Training implementation | Warm time/update | Tokens/s | Relative to native MPS | +|---|---:|---:|---:| +| Native nanoGPT / PyTorch MPS | **29.35 ms** | **104,652** | 1× | +| SQL / MLX GPU | 2,024.15 ms | 1,518 | **68.96× slower** | +| SQL / float64 CPU, 32 GiB cache | 9,746.51 ms | 315 | 332.03× slower | + +The [native results](shakespeare-torch-mps.json) include every synchronized +duration, source hashes, package version and exact configuration. +[`compare_shakespeare_mps.py`](compare_shakespeare_mps.py) executes the +original local trainer with timing instrumentation; it does not modify +the nanoGPT source files. It synchronizes MPS immediately before the +forward/backward/update interval and after gradient clearing. Timing +includes native next-batch prefetch. Evaluation, initialization, logging +and checkpoint I/O are outside that interval. + +This is a comparison of the practical training paths. Native PyTorch +keeps weights and optimizer state resident on MPS and performs one +training forward/backward pass. SQL returns Arrow parameter/moment tables +and computes its reported loss in a separate forward call. Consequently +the paths have different execution and output overhead. Matching recipe +settings does not remove those differences. Native RNGs also produce +different initial weights and dropout masks from the independent SQL +run. These measurements are five-step samples, not long-run averages. + +The native loop stops when `iter_num > max_iters`; setting `max_iters=4` +therefore runs exactly five updates. `lr_decay_iters=2000` and +`warmup_iters=100` remain unchanged. To repeat the instrumented comparison +in an environment with PyTorch installed: + +```sh +PYTHONPATH="$PWD:$PWD/.venv/lib/python3.12/site-packages" \ + /private/tmp/xql-framework-venv/bin/python benchmarks/apple/compare_shakespeare_mps.py \ + --nanogpt-source /Users/cromwellian/nanoGPT --steps 5 \ + --output benchmarks/apple/shakespeare-torch-mps.json +``` + +## Changes needed for this recipe + +- Dropout masks are exogenous SQL tables, freshly sampled for each + training update and absent from the evaluation graph. ddx differentiates + the actual masked SQL model. +- An optional [ddx math patch](../../accelerators/ddx_math/README.md) adds + erf's analytic scalar derivative and preserves erf in derivative plans. + `xarray_sql.math.register_erf` supplies its SQL implementation and + equivalent array primitive. This enables nanoGPT's exact GELU. +- Softmax and cross-entropy stabilization use `GROUP BY` and joins. + Profiling an earlier window-based trial showed repeated traversal of + shared subqueries during ddx window lowering. The stabilizing maximum + is held constant: the shared shift cancels from softmax and cross + entropy, including their parameter gradients. +- Conditional products retain their factors through aliases and + zero-valued CASE branches. This avoids expanding an attention backward + temporary of shape 256 × 256 × 12 × 6 × 64, which exceeded the existing + conservative 1 GiB array guard. The guard remains unchanged. +- AdamW moments occupy separate tables from differentiated weights; + clipping is computed once in SQL, and prepared update programs are + reused across iterations. +- Safe NPZ checkpoints contain weights, moments, next iteration, RNG + states, vocabulary and recipe settings. A correctness test checks that + resuming produces exactly the uninterrupted SQL trajectory. + +The optional builds are `ddxdb==0.4.6+xqlmath.2` and +`einfold==0.2.0+xqlmlx.5`. The ddx patch is based on the exact 0.4.6 release +commit `af71683f6678bb093e600f948c212490a4d8b74b`, excluding newer upstream +changes. The nanoGPT reference is commit +`3adf61e154c3fe3fca428ad6bc3818b27a3b8291`. Other dependencies, commands, +per-step records, counters and source hashes are recorded in +[`shakespeare-summary.json`](shakespeare-summary.json). Intermediate +`shakespeare-matched-*` logs retain failed/superseded trials; only the +`release-*-stable` runs describe the final comparison. + +## Run the training script + +Use the existing checkout environment after the optional builders have +been installed. From the xarray-sql directory: + +```sh +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/python benchmarks/ml/03_shakespeare.py \ + --backend dense_gpu \ + --data-dir /Users/cromwellian/nanoGPT/data/shakespeare_char \ + --out-dir out-shakespeare-sql +``` + +Defaults match the recipe above. As in native `train.py`, `max_iters=2000` +includes iterations 0–2000, or 2001 updates. Add `--steps 5` for an exact +five-update smoke run. Use `--backend dense_cpu --cache-gib 32` for the +measured CPU configuration. For optional independent checks, add +`--verify-nanogpt /Users/cromwellian/nanoGPT` in an environment with +PyTorch installed. See the [training documentation](../ml/README.md) for +resume instructions and timing scope. + +The final verified five-step checkpoints are: + +- [`GPU last.npz`](shakespeare-matched-release-gpu-stable/last.npz) +- [`CPU last.npz`](shakespeare-matched-release-cpu-stable/last.npz) + +Checkpoint binaries are ignored by git; text metrics and logs are retained. + +## Validation + +The full local suite passed with GPU checks enabled: + +```sh +MLX_ENABLE_TF32=0 PYTHONPATH="$PWD" .venv/bin/pytest -q -m 'not integration' +# 704 passed, 277 skipped, 102 network/integration tests deselected +``` + +The [test log](shakespeare-correctness.log) records the result and existing +warnings. New checks cover erf values/NULLs, UDF replacement, exact GELU +gradients in SQL/CPU/GPU, bounded conditional contractions, and complete +SQL training checkpoint/resume. ddx-core's scalar-rule suite also passed, +including the new erf chain-rule and second-derivative check. Ruff and +`git diff --check` passed. diff --git a/benchmarks/apple/shared-correctness.log b/benchmarks/apple/shared-correctness.log new file mode 100644 index 00000000..96ee50c5 --- /dev/null +++ b/benchmarks/apple/shared-correctness.log @@ -0,0 +1,87 @@ +.............sssssssss...sssssssss...ssssssssss..ssssssssss..ssssssssss. [ 7%] +.ssssssssss..sssssssss.....ssssssssssssssssss....sssssssss...sssssssss.. [ 15%] +...ssssssssssssssssss....sssssssss...sssssssss...sssssssss...sssssssss.. [ 23%] +.sssssssss...sssssssss...sssssssss...sssssssss...sssssssss..ssssssssssss [ 31%] +sssssssssssssssssssssssssssssssssssssssssssssssssssssssssss............. [ 39%] +........................................................................ [ 47%] +........................................................................ [ 55%] +........................................................................ [ 63%] +....sss.....s........................................................... [ 70%] +........................................................................ [ 78%] +........................................................................ [ 86%] +........................................................................ [ 94%] +.................................................. [100%] +=============================== warnings summary =============================== +tests/test_df.py:688 + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:688: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT"), np.datetime64("2000-01-01")], + +tests/test_adbc_backend.py: 28 warnings +tests/test_flight_sql.py: 10 warnings + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py:371: Warning: Cannot disable autocommit; conn will not be DB-API 2.0 compliant + warnings.warn( + +tests/test_arrow_dataset.py::test_unchunked_zarr_scans_by_storage_chunks + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/zarr/api/asynchronous.py:260: ZarrUserWarning: Consolidated metadata is currently not part in the Zarr format 3 specification. It may not be supported by other zarr implementations and may change in the future. + warnings.warn( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/tests/test_coord_lookup.py:111: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + template = xr.Dataset( + +tests/test_coord_lookup.py::test_float16_axis_roundtrip + /Users/cromwellian/xql/xarray-sql/xarray_sql/ds.py:707: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + return xr.Dataset(data_vars=data_vars, coords=coords_arg) + +tests/test_df.py::test_partition_metadata_preserves_coordinate_dtype[float16-values9-expected9] + /Users/cromwellian/xql/xarray-sql/tests/test_df.py:639: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"v": (["x"], np.arange(len(coord)))}, coords={"x": coord}) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'z' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'u' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_ds.py::test_round_trip_identity[eraint_uvz] + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/xarray/conventions.py:210: SerializationWarning: variable 'v' has non-conforming '_FillValue' np.float64(nan) defined, dropping '_FillValue' entirely. + var = coder.decode(var, name=name) + +tests/test_flight_sql.py::test_shutdown_does_not_wait_forever_on_an_unread_result + /Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:63: PytestUnraisableExceptionWarning: Exception ignored in: + + Traceback (most recent call last): + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 1053, in __del__ + self.close() + File "/Users/cromwellian/xql/xarray-sql/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 851, in close + self._stmt.close() + File "adbc_driver_manager/_lib.pyx", line 1555, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 1561, in adbc_driver_manager._lib.AdbcStatement.close + File "adbc_driver_manager/_lib.pyx", line 262, in adbc_driver_manager._lib.check_error + adbc_driver_manager.OperationalError: IO: [FlightSQL] connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:59206: connect: connection refused" (Unavailable; ClosePreparedStatement). Vendor code: 14 + + Enable tracemalloc to get traceback where the object was allocated. + See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info. + warnings.warn(pytest.PytestUnraisableExceptionWarning(msg)) + +tests/test_lazy_roundtrip.py: 32 warnings + /Users/cromwellian/xql/xarray-sql/xarray_sql/lazyscan.py:63: DeprecationWarning: `is_in` with a collection of the same datatype is ambiguous and deprecated. + Please use `implode` to return to previous behavior. + + See https://github.com/pola-rs/polars/issues/22149 for more information. + return lf.collect(engine="streaming") + +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place +tests/test_prepare.py::test_a_write_must_keep_nat_in_place + /Users/cromwellian/xql/xarray-sql/tests/test_prepare.py:169: DeprecationWarning: The 'generic' unit for NumPy timedelta is deprecated, and will raise an error in the future. This includes implicit conversion of bare integers (e.g. `+ 1`). Please use a specific unit instead. + [np.datetime64("NaT") if d is None else np.datetime64(d) for d in days], + +tests/test_reader.py::TestFilterPushdown::test_float_coordinate_types_prune_with_natural_literal[float16] + /Users/cromwellian/xql/xarray-sql/tests/test_reader.py:1067: FutureWarning: `pandas.Index` does not support the `float16` dtype. Casting to `float64` for you, but in the future please manually cast to either `float32` and `float64`. + ds = xr.Dataset({"value": (["x"], np.arange(8))}, coords={"x": coord}) + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +637 passed, 277 skipped, 102 deselected, 83 warnings in 35.80s diff --git a/benchmarks/apple/shared-native-kernel-tests.log b/benchmarks/apple/shared-native-kernel-tests.log new file mode 100644 index 00000000..0120c1ba --- /dev/null +++ b/benchmarks/apple/shared-native-kernel-tests.log @@ -0,0 +1,307 @@ + Compiling proc-macro2 v1.0.107 + Compiling quote v1.0.47 + Compiling unicode-ident v1.0.26 + Compiling libc v0.2.190 + Compiling cfg-if v1.0.5 + Compiling autocfg v1.5.1 + Compiling shlex v2.0.1 + Compiling find-msvc-tools v0.1.14 + Compiling libm v0.2.16 + Compiling memchr v2.8.3 + Compiling version_check v0.9.5 + Compiling zerocopy v0.8.60 + Compiling bytes v1.12.1 + Compiling serde_core v1.0.229 + Compiling pkg-config v0.3.34 + Compiling once_cell v1.21.4 + Compiling itoa v1.0.18 + Compiling getrandom v0.3.4 + Compiling zmij v1.0.23 + Compiling core-foundation-sys v0.8.7 + Compiling serde_json v1.0.151 + Compiling equivalent v1.0.2 + Compiling allocator-api2 v0.2.21 + Compiling siphasher v1.0.4 + Compiling foldhash v0.2.0 + Compiling stable_deref_trait v1.2.1 + Compiling chrono-tz v0.10.4 + Compiling pin-project-lite v0.2.17 + Compiling lexical-util v1.0.7 + Compiling futures-sink v0.3.34 + Compiling futures-core v0.3.34 + Compiling litemap v0.8.3 + Compiling object v0.39.1 + Compiling smallvec v1.16.2 + Compiling writeable v0.6.4 + Compiling utf8_iter v1.0.4 + Compiling icu_normalizer_data v2.3.0 + Compiling semver v1.0.28 + Compiling icu_properties_data v2.3.0 + Compiling zstd-safe v7.3.0 + Compiling bitflags v2.13.2 + Compiling phf_shared v0.12.1 + Compiling futures-channel v0.3.34 + Compiling ryu v1.0.23 + Compiling unicode-width v0.2.2 + Compiling iana-time-zone v0.1.65 + Compiling slab v0.4.12 + Compiling crc32fast v1.5.2 + Compiling parking_lot_core v0.9.12 + Compiling unicode-segmentation v1.13.3 + Compiling futures-io v0.3.34 + Compiling base64 v0.23.1 + Compiling ahash v0.8.12 + Compiling futures-task v0.3.34 + Compiling regex-syntax v0.8.11 + Compiling num-traits v0.2.19 + Compiling phf v0.12.1 + Compiling rustc_version v0.4.1 + Compiling scopeguard v1.2.0 + Compiling simd-adler32 v0.3.10 + Compiling twox-hash v2.1.5 + Compiling adler2 v2.0.1 + Compiling lock_api v0.4.14 + Compiling either v1.19.0 + Compiling thiserror v2.0.21 + Compiling simdutf8 v0.1.5 + Compiling getrandom v0.4.3 + Compiling percent-encoding v2.3.2 + Compiling alloc-no-stdlib v2.0.4 + Compiling zlib-rs v0.6.8 + Compiling lz4_flex v0.14.0 + Compiling tracing-core v0.1.36 + Compiling snap v1.1.2 + Compiling hashbrown v0.17.1 + Compiling miniz_oxide v0.9.1 + Compiling alloc-stdlib v0.2.4 + Compiling same-file v1.0.6 + Compiling lexical-parse-integer v1.0.6 + Compiling lexical-write-integer v1.0.6 + Compiling flatbuffers v25.12.19 + Compiling aho-corasick v1.1.5 + Compiling csv-core v0.1.13 + Compiling form_urlencoded v1.2.2 + Compiling itertools v0.14.0 + Compiling brotli-decompressor v5.0.3 + Compiling walkdir v2.5.0 + Compiling humantime v2.4.0 + Compiling comfy-table v7.2.2 + Compiling log v0.4.34 + Compiling lexical-parse-float v1.0.6 + Compiling seq-macro v0.3.6 + Compiling http v1.5.0 + Compiling lexical-write-float v1.0.6 + Compiling syn v3.0.6 + Compiling syn v2.0.119 + Compiling itertools v0.15.0 + Compiling typenum v1.20.1 + Compiling serde v1.0.229 + Compiling generic-array v0.14.7 + Compiling datafusion-doc v55.2.0 + Compiling foldhash v0.1.5 + Compiling crossbeam-utils v0.8.23 + Compiling jobserver v0.1.35 + Compiling rustix v1.1.5 + Compiling fixedbitset v0.5.7 + Compiling const-oid v0.10.2 + Compiling hashbrown v0.14.5 + Compiling lexical-core v1.0.6 + Compiling rand_core v0.9.5 + Compiling hashbrown v0.15.5 + Compiling uuid v1.26.1 + Compiling cc v1.6.0 + Compiling errno v0.3.14 + Compiling indexmap v2.14.2 + Compiling parking_lot v0.12.5 + Compiling brotli v8.0.4 + Compiling fastrand v2.5.0 + Compiling subtle v2.6.1 + Compiling cpufeatures v0.3.1 + Compiling constant_time_eq v0.4.2 + Compiling arrayvec v0.7.8 + Compiling hex v0.4.3 + Compiling zstd-safe v8.0.0 + Compiling libbz2-rs-sys v0.2.5 + Compiling compression-core v0.4.33 + Compiling glob v0.3.4 + Compiling bigdecimal v0.4.11 + Compiling matrixmultiply v0.3.11 + Compiling hybrid-array v0.4.15 + Compiling rawpointer v0.2.1 + Compiling dashmap v6.2.1 + Compiling einfold-ir v0.1.0 (/private/tmp/xql-einfold/crates/einfold-ir) + Compiling einfold-testkit v0.1.0 (/private/tmp/xql-einfold/crates/einfold-testkit) + Compiling num-integer v0.1.47 + Compiling chrono v0.4.45 + Compiling num-complex v0.4.6 + Compiling atoi v2.0.0 + Compiling num-bigint v0.5.1 + Compiling petgraph v0.8.3 + Compiling bzip2 v0.6.1 + Compiling regex-automata v0.4.18 + Compiling crypto-common v0.2.2 + Compiling block-buffer v0.12.1 + Compiling num-bigint v0.4.8 + Compiling crypto-common v0.1.7 + Compiling block-buffer v0.10.4 + Compiling ndarray v0.16.1 + Compiling digest v0.10.7 + Compiling tempfile v3.27.0 + Compiling zstd-sys v2.1.0+zstd.1.5.7 + Compiling stacker v0.1.25 + Compiling csv v1.4.0 + Compiling blake3 v1.8.7 + Compiling blake2 v0.10.6 + Compiling flate2 v1.1.10 + Compiling liblzma-sys v0.4.9 + Compiling digest v0.11.3 + Compiling md-5 v0.11.0 + Compiling sha2 v0.11.0 + Compiling arrow-schema v59.3.1 + Compiling ar_archive_writer v0.5.3 + Compiling synstructure v0.14.0 + Compiling psm v0.1.32 + Compiling zerocopy-derive v0.8.60 + Compiling tracing-attributes v0.1.31 + Compiling recursive-proc-macro-impl v0.1.1 + Compiling regex v1.13.1 + Compiling sqlparser_derive v0.5.0 + Compiling recursive v0.1.1 + Compiling pin-project-internal v1.1.13 + Compiling zerofrom-derive v0.1.8 + Compiling yoke-derive v0.8.4 + Compiling zerovec-derive v0.11.6 + Compiling displaydoc v0.2.7 + Compiling tokio-macros v2.7.2 + Compiling futures-macro v0.3.34 + Compiling thiserror-impl v2.0.21 + Compiling tracing v0.1.44 + Compiling zerofrom v0.1.8 + Compiling async-trait v0.1.92 + Compiling yoke v0.8.3 + Compiling sqlparser v0.62.0 + Compiling tokio v1.53.2 + Compiling zerotrie v0.2.5 + Compiling futures-util v0.3.34 + Compiling zerovec v0.11.8 + Compiling pin-project v1.1.13 + Compiling serde_derive v1.0.229 + Compiling datafusion-macros v55.2.0 + Compiling tinystr v0.8.4 + Compiling potential_utf v0.1.6 + Compiling icu_collections v2.3.0 + Compiling icu_locale_core v2.3.0 + Compiling icu_provider v2.3.1 + Compiling icu_properties v2.3.0 + Compiling icu_normalizer v2.3.0 + Compiling zstd v0.13.3 + Compiling zstd v0.14.0 + Compiling liblzma v0.4.8 + Compiling compression-codecs v0.4.45 + Compiling futures-executor v0.3.34 + Compiling futures v0.3.34 + Compiling idna_adapter v1.2.2 + Compiling idna v1.1.0 + Compiling url v2.5.8 + Compiling tokio-util v0.7.19 + Compiling datafusion-common-runtime v55.2.0 + Compiling async-compression v0.4.50 + Compiling half v2.7.1 + Compiling ppv-lite86 v0.2.21 + Compiling tokio-stream v0.1.19 + Compiling object_store v0.13.2 + Compiling rand_chacha v0.9.0 + Compiling rand v0.9.5 + Compiling arrow-buffer v59.3.1 + Compiling arrow-data v59.3.1 + Compiling arrow-array v59.3.1 + Compiling arrow-select v59.3.1 + Compiling arrow-arith v59.3.1 + Compiling arrow-row v59.3.1 + Compiling arrow-ord v59.3.1 + Compiling arrow-ipc v59.3.1 + Compiling arrow-string v59.3.1 + Compiling parquet v59.3.1 + Compiling arrow-cast v59.3.1 + Compiling arrow-csv v59.3.1 + Compiling arrow-json v59.3.1 + Compiling arrow v59.3.1 + Compiling einfold-kernels v0.1.0 (/private/tmp/xql-einfold/crates/einfold-kernels) + Compiling datafusion-common v55.2.0 + Compiling datafusion-expr-common v55.2.0 + Compiling datafusion-physical-expr-common v55.2.0 + Compiling datafusion-functions-aggregate-common v55.2.0 + Compiling datafusion-functions-window-common v55.2.0 + Compiling datafusion-expr v55.2.0 + Compiling datafusion-physical-expr v55.2.0 + Compiling datafusion-execution v55.2.0 + Compiling datafusion-functions v55.2.0 + Compiling datafusion-functions-aggregate v55.2.0 + Compiling datafusion-functions-window v55.2.0 + Compiling datafusion-optimizer v55.2.0 + Compiling datafusion-physical-plan v55.2.0 + Compiling datafusion-physical-expr-adapter v55.2.0 + Compiling datafusion-functions-nested v55.2.0 + Compiling datafusion-sql v55.2.0 + Compiling datafusion-session v55.2.0 + Compiling datafusion-datasource v55.2.0 + Compiling datafusion-catalog v55.2.0 + Compiling datafusion-pruning v55.2.0 + Compiling datafusion-datasource-csv v55.2.0 + Compiling datafusion-datasource-parquet v55.2.0 + Compiling datafusion-physical-optimizer v55.2.0 + Compiling datafusion-datasource-arrow v55.2.0 + Compiling datafusion-datasource-json v55.2.0 + Compiling datafusion-functions-table v55.2.0 + Compiling datafusion-catalog-listing v55.2.0 + Compiling datafusion v55.2.0 + Finished `release` profile [optimized] target(s) in 22m 28s + Running unittests src/lib.rs (target/release/deps/einfold_kernels-414290ee973494ef) + +running 4 tests +test expression::tests::a_tile_has_each_pairs_value ... ok +test gemm::tests::products_within_the_summation_bound_and_repeatable ... ok +test keys::tests::signed_zeros_are_one_key_only_when_the_host_says_so ... ok +test discriminate::tests::sorts_keys_with_long_shared_prefixes_in_a_small_stack ... ok + +test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.09s + + Running tests/dense_metamorphic.rs (target/release/deps/dense_metamorphic-5486d4bc9691e0e6) + +running 1 test +test dense_metamorphic ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.87s + + Running tests/expression_vs_sql.rs (target/release/deps/expression_vs_sql-ef9c95a84435c7ae) + +running 2 tests +test power_of_special_values_is_datafusions ... ok +test row_expressions_compute_what_datafusion_does ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.25s + + Running tests/kernels_vs_sql.rs (target/release/deps/kernels_vs_sql-f3791a5db4e69653) + +running 1 test +test fold_matches_sql ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 37.61s + + Running tests/pipeline_vs_sql.rs (target/release/deps/pipeline_vs_sql-ada73a867d6b89ab) + +running 6 tests +test declines_a_pipeline_that_sums_its_own_dimension_away ... ok +test declines_where_a_rescale_meets_an_overflowed_sum ... ok +test gives_no_rows_without_queries_or_keys ... ok +test sliding_windows_visit_a_band_of_tiles ... ok +test causal_masks_skip_half_the_tiles ... ok +test pipeline_matches_sql ... ok + +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.84s + + Doc-tests einfold_kernels + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s diff --git a/benchmarks/apple/shared-native-planner-tests.log b/benchmarks/apple/shared-native-planner-tests.log new file mode 100644 index 00000000..0d8c242e --- /dev/null +++ b/benchmarks/apple/shared-native-planner-tests.log @@ -0,0 +1,105 @@ + Compiling einfold-datafusion v0.1.0 (/private/tmp/xql-einfold/crates/einfold-datafusion) + Finished `release` profile [optimized] target(s) in 30.45s + Running unittests src/lib.rs (target/release/deps/einfold_datafusion-61d1a80d1273af6e) + +running 1 test +test tests::supported_datafusion_version ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s + + Running tests/chains_vs_host.rs (target/release/deps/chains_vs_host-06069eff898e69c4) + +running 1 test +test chains_answer_as_the_host_does ... ok + +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.62s + + Running tests/datafusion_behaviors.rs (target/release/deps/datafusion_behaviors-763cd73013d1e14b) + +running 9 tests +test sort_merge_join_filter_reads_stale_columns ... ok +test where_equality_on_a_not_distinct_join_matches_null_with_equals ... ok +test not_distinct_join_then_cross_join_drops_null_matches ... ok +test grouping_a_dictionary_encoded_double_keeps_signed_zeros_apart ... ok +test mixed_null_equality_in_a_join_chain_matches_null_with_equals ... ok +test sort_merge_join_over_signed_zeros_drops_a_match ... ok +test joining_doubles_matches_signed_zeros_and_equal_nans ... ok +test grouping_a_double_merges_signed_zeros_into_positive_zero ... ok +test top_k_over_signed_zeros_depends_on_timing ... ok + +test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.03s + + Running tests/detect_pipelines.rs (target/release/deps/detect_pipelines-bb20a5cf3478fdc7) + +running 2 tests +test the_nadaraya_watson_control_fuses ... ok +test einfold_finds_pipelines_and_changes_no_answer ... ok + +test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.55s + + Running tests/detect_vs_host.rs (target/release/deps/detect_vs_host-d109c3a5636d37e5) + +running 22 tests +test movable_functions_never_fail ... ok +test fallbacks_hold_only_the_hosts_plan ... ok +test joined_back_fallbacks_hold_only_the_hosts_plan ... ok +test the_optimizer_sees_a_fallback_once ... ok +test folds_estimate_their_rows ... ok +test reset_plans_whose_fallback_ran_rerun ... ok +test volatile_factors_are_left_alone ... ok +test host_plan_runs_where_there_is_no_tokio_runtime ... ok +test a_sessions_own_coalesce_is_no_fill ... ok +test fallbacks_plan_with_the_sessions_variables ... ok +test masked_folds_over_heads_run_dense ... ok +test products_over_saved_scores_run_dense ... ok +test folds_whose_fallback_runs_recur ... ok +test masked_gradients_from_their_domain_run_dense ... ok +test fallible_factors_stay_above_the_join ... ok +test chains_of_three_products_contract_in_order ... ok +test infallible_factors_are_found ... ok +test joined_back_tests_apply_once ... ok +test masked_folds_keep_their_mask_under_a_domain ... ok +test einfold_changes_no_answer ... ok +test einfold_runs_short_of_memory_only_where_the_host_does ... ok +test folds_joined_back_check_their_domain ... ok + +test result: ok. 22 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.25s + + Running tests/operator.rs (target/release/deps/operator-d8640b69eb8a148c) + +running 6 tests +test fails_cleanly_when_the_memory_pool_is_too_small ... ok +test holds_only_its_output_in_the_memory_pool ... ok +test runs_where_there_is_no_tokio_runtime ... ok +test reports_which_algorithm_ran ... ok +test inputs_that_spawn_tasks_run_where_there_is_no_tokio_runtime ... ok +test matches_sql_across_partitions_threads_and_batch_sizes ... ok + +test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s + + Running tests/pipeline_operator.rs (target/release/deps/pipeline_operator-e63ed555cabe3858) + +running 3 tests +test gives_no_rows_where_there_are_no_values ... ok +test runs_the_fused_algorithm_on_complete_inputs ... ok +test runs_the_host_plan_where_an_input_has_a_hole ... ok + +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s + + Running tests/simulation.rs (target/release/deps/simulation-64b0b23043012d32) + +running 6 tests +test export ... ignored +test host_hang ... ignored +test host_rerun_hang ... ignored +test soak ... ignored +test panics_fail_alike ... ok +test simulation ... ok + +test result: ok. 2 passed; 0 failed; 4 ignored; 0 measured; 0 filtered out; finished in 0.93s + + Doc-tests einfold_datafusion + +running 0 tests + +test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s diff --git a/benchmarks/apple/shared-profile-demo.csv b/benchmarks/apple/shared-profile-demo.csv new file mode 100644 index 00000000..87f54771 --- /dev/null +++ b/benchmarks/apple/shared-profile-demo.csv @@ -0,0 +1,11 @@ +demo,step,host,batch,seconds,check,loss,operators,rtol,atol,layers,embed,heads,context,vocab,parameters,optimizer,sgd_learning_rate,forward_only,table_registers,query_preparations,cpu_matmul,gpu_matmul,gpu_attention,declined,physical_plans,plan_executions,stage_buffer_allocations,key_cache_hits,key_copy_bytes_avoided,forward_stages,planning_seconds,execution_seconds,stage_storage_seconds,key_cache_bytes,load_seconds,model_seconds,gradient_preparations,inplace_updates,stage_buffer_writes,stage_replacements_keys_or_nulls +nanogpt,"loss, step 0",shared_gpu,1,0.08210029196925461,ok (3e-09),4.185923,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,7,1,0,10,10,9,16,691712,10,0.03745054092723876,0.03704787383321673,0.0015233327867463231,297472,,,,,, +nanogpt,"train (sgd), step 0",shared_gpu,1,0.7857961249537766,ok (7e-09),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,32,1,0,86,86,63,208,13300224,10,0.35036358842626214,0.36936655023600906,0.02808791457209736,5620224,0.013338916236534715,0.0012503330362960696,1,11,, +nanogpt,"loss, step 1",shared_gpu,1,0.02988116699270904,ok (3e-10),4.111482,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,39,2,0,86,96,63,208,13300224,10,0.35036358842626214,0.3976505504688248,0.029075414640828967,5620224,,,1,12,9, +nanogpt,"train (sgd), step 1",shared_gpu,1,0.48307370801921934,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,64,2,0,99,172,73,307,19788288,10,0.4361427564872429,0.7467816746793687,0.054693212383426726,5816832,0.020218208199366927,0.00129725004080683,1,23,53,10 +nanogpt,"loss, step 2",shared_gpu,1,0.029160457896068692,ok (3e-09),4.147195,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,71,3,0,99,182,73,307,19788288,10,0.4361427564872429,0.7743723407620564,0.055730002583004534,5816832,,,1,24,62,10 +nanogpt,"train (sgd), step 2",shared_gpu,1,0.4922497500665486,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,96,3,0,112,258,83,406,26276352,10,0.5249446337111294,1.1282220911234617,0.0812208813149482,6013440,0.02633079409133643,0.0013464170042425394,1,35,106,20 +nanogpt,"loss, step 3",shared_gpu,1,0.028076375019736588,ok (2e-09),4.065422,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,103,4,0,112,268,83,406,26276352,10,0.5249446337111294,1.1544697562931105,0.08239159046206623,6013440,,,1,36,115,20 +nanogpt,"train (sgd), step 3",shared_gpu,1,0.521025542053394,ok (1e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,128,4,0,125,344,93,505,32764416,10,0.6160765929380432,1.5361583440098912,0.10795496974606067,6210048,0.0323859608033672,0.0013863339554518461,1,47,159,30 +nanogpt,"loss, step 4",shared_gpu,1,0.029403500026091933,ok (2e-09),4.03082,,3e-05,3e-05,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,135,5,0,125,354,93,505,32764416,10,0.6160765929380432,1.5639026338467374,0.10901172051671892,6210048,,,1,48,168,30 +nanogpt,"train (sgd), step 4",shared_gpu,1,0.48507858405355364,ok (2e-08),,,,,1,128,4,64,65,213504,sgd,0.01,False,13,1,0,160,5,0,138,430,103,604,39252480,10,0.701797676156275,1.9156180099816993,0.13413060072343796,6406656,0.038999419775791466,0.0014298750320449471,1,59,212,40 diff --git a/benchmarks/apple/shared-profile-demo.log b/benchmarks/apple/shared-profile-demo.log new file mode 100644 index 00000000..f67a7b1c --- /dev/null +++ b/benchmarks/apple/shared-profile-demo.log @@ -0,0 +1,27 @@ + 1-layer GPT, vocab 65, 213,504 parameters: the loss is 123 lines of SQL + steps=5, optimizer=sgd, context=64, heads=4, embed=128, forward_only=False + loss, step 0 shared_gpu batch=1 0.082s + train (sgd), step 0 shared_gpu batch=1 0.786s + loss, step 1 shared_gpu batch=1 0.030s + train (sgd), step 1 shared_gpu batch=1 0.483s + loss, step 2 shared_gpu batch=1 0.029s + train (sgd), step 2 shared_gpu batch=1 0.492s + loss, step 3 shared_gpu batch=1 0.028s + train (sgd), step 3 shared_gpu batch=1 0.521s + loss, step 4 shared_gpu batch=1 0.029s + train (sgd), step 4 shared_gpu batch=1 0.485s + +wrote benchmarks/apple/shared-profile-demo.csv + +| demo | step | host | batch | seconds | check | loss | operators | rtol | atol | layers | embed | heads | context | vocab | parameters | optimizer | sgd_learning_rate | forward_only | table_registers | query_preparations | cpu_matmul | gpu_matmul | gpu_attention | declined | physical_plans | plan_executions | stage_buffer_allocations | key_cache_hits | key_copy_bytes_avoided | forward_stages | planning_seconds | execution_seconds | stage_storage_seconds | key_cache_bytes | load_seconds | model_seconds | gradient_preparations | inplace_updates | stage_buffer_writes | stage_replacements_keys_or_nulls | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| nanogpt | loss, step 0 | shared_gpu | 1 | 0.082 | ok (3e-09) | 4.186 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 7 | 1 | 0 | 10 | 10 | 9 | 16 | 691712 | 10 | 0.037 | 0.037 | 0.002 | 297472 | n/a | n/a | n/a | n/a | n/a | n/a | +| nanogpt | train (sgd), step 0 | shared_gpu | 1 | 0.786 | ok (7e-09) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 32 | 1 | 0 | 86 | 86 | 63 | 208 | 13300224 | 10 | 0.350 | 0.369 | 0.028 | 5620224 | 0.013 | 0.001 | 1 | 11 | n/a | n/a | +| nanogpt | loss, step 1 | shared_gpu | 1 | 0.030 | ok (3e-10) | 4.111 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 39 | 2 | 0 | 86 | 96 | 63 | 208 | 13300224 | 10 | 0.350 | 0.398 | 0.029 | 5620224 | n/a | n/a | 1 | 12 | 9 | n/a | +| nanogpt | train (sgd), step 1 | shared_gpu | 1 | 0.483 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 64 | 2 | 0 | 99 | 172 | 73 | 307 | 19788288 | 10 | 0.436 | 0.747 | 0.055 | 5816832 | 0.020 | 0.001 | 1 | 23 | 53 | 10 | +| nanogpt | loss, step 2 | shared_gpu | 1 | 0.029 | ok (3e-09) | 4.147 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 71 | 3 | 0 | 99 | 182 | 73 | 307 | 19788288 | 10 | 0.436 | 0.774 | 0.056 | 5816832 | n/a | n/a | 1 | 24 | 62 | 10 | +| nanogpt | train (sgd), step 2 | shared_gpu | 1 | 0.492 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 96 | 3 | 0 | 112 | 258 | 83 | 406 | 26276352 | 10 | 0.525 | 1.128 | 0.081 | 6013440 | 0.026 | 0.001 | 1 | 35 | 106 | 20 | +| nanogpt | loss, step 3 | shared_gpu | 1 | 0.028 | ok (2e-09) | 4.065 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 103 | 4 | 0 | 112 | 268 | 83 | 406 | 26276352 | 10 | 0.525 | 1.154 | 0.082 | 6013440 | n/a | n/a | 1 | 36 | 115 | 20 | +| nanogpt | train (sgd), step 3 | shared_gpu | 1 | 0.521 | ok (1e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 128 | 4 | 0 | 125 | 344 | 93 | 505 | 32764416 | 10 | 0.616 | 1.536 | 0.108 | 6210048 | 0.032 | 0.001 | 1 | 47 | 159 | 30 | +| nanogpt | loss, step 4 | shared_gpu | 1 | 0.029 | ok (2e-09) | 4.031 | n/a | 3e-05 | 3e-05 | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 135 | 5 | 0 | 125 | 354 | 93 | 505 | 32764416 | 10 | 0.616 | 1.564 | 0.109 | 6210048 | n/a | n/a | 1 | 48 | 168 | 30 | +| nanogpt | train (sgd), step 4 | shared_gpu | 1 | 0.485 | ok (2e-08) | n/a | n/a | n/a | n/a | 1 | 128 | 4 | 64 | 65 | 213504 | sgd | 0.010 | False | 13 | 1 | 0 | 160 | 5 | 0 | 138 | 430 | 103 | 604 | 39252480 | 10 | 0.702 | 1.916 | 0.134 | 6406656 | 0.039 | 0.001 | 1 | 59 | 212 | 40 | diff --git a/benchmarks/apple/shared-profile.txt b/benchmarks/apple/shared-profile.txt new file mode 100644 index 00000000..d2096f61 --- /dev/null +++ b/benchmarks/apple/shared-profile.txt @@ -0,0 +1,103 @@ +Separate profiled run, after all final timing comparisons. +MLX_ENABLE_TF32=0 PYTHONPATH=$PWD .venv/bin/python -m cProfile -o /private/tmp/xql-shared-demo.prof benchmarks/ml/02_nanogpt.py --preset demo --steps 5 --sizes 1 --learning-rate 0.01 --hosts shared_gpu --csv benchmarks/apple/shared-profile-demo.csv + +Sun Nov 1 03:24:50 2026 /private/tmp/xql-shared-demo.prof + + 5075972 function calls (4993988 primitive calls) in 6.114 seconds + + Ordered by: cumulative time + List reduced from 9155 to 45 due to restriction <45> + + ncalls tottime percall cumtime percall filename:lineno(function) + 1833/1 0.018 0.000 6.144 6.144 {built-in method builtins.exec} + 1 0.000 0.000 6.144 6.144 02_nanogpt.py:1() + 1 0.008 0.008 5.557 5.557 02_nanogpt.py:530(main) + 10 0.000 0.000 2.966 0.297 _bench.py:85(time) + 5 0.000 0.000 2.768 0.554 _bench.py:110(attempt) + 5 0.000 0.000 2.767 0.553 _hosts.py:336(query_all) + 5 0.002 0.000 2.767 0.553 shared.py:684(execute) + 5 0.000 0.000 2.636 0.527 __init__.py:270(run) + 182 0.002 0.000 2.357 0.013 __init__.py:1() + 270 0.037 0.000 1.971 0.007 shared.py:411(materialize_compiled) + 430 0.028 0.000 1.917 0.004 shared.py:166(run) + 4 0.012 0.003 1.893 0.473 {method 'run' of 'ddxdb._Prepared' objects} + 971/529 0.001 0.000 1.843 0.003 traceback_util.py:190(reraise_with_filtered_traceback) + 200 0.001 0.000 1.590 0.008 pjit.py:1119(_pjit_call_impl_python) + 953/753 0.002 0.000 1.522 0.002 profiler.py:585(wrapper) + 200 0.001 0.000 1.433 0.007 pxla.py:1229(compile) + 174 0.003 0.000 1.432 0.008 pxla.py:1706(from_hlo) + 174 0.001 0.000 1.417 0.008 pxla.py:1519(_cached_compilation) + 174 0.001 0.000 1.402 0.008 compiler.py:436(compile_or_get_cached) + 174 0.001 0.000 1.376 0.008 compiler.py:771(_compile_and_write_cache) + 174 1.374 0.008 1.374 0.008 compiler.py:341(backend_compile_and_load) + 436/431 0.001 0.000 1.367 0.003 pjit.py:263(cache_miss) + 436 0.001 0.000 1.311 0.003 pjit.py:133(_run_python_pjit) +2784/1311 0.006 0.000 1.281 0.001 core.py:687(bind) +2825/1310 0.003 0.000 1.274 0.001 core.py:736(bind_with_trace) + 5 0.000 0.000 1.193 0.239 api.py:480(grad_f) + 5 0.000 0.000 1.193 0.239 api.py:542(value_and_grad_f) +1252/1162 0.002 0.000 1.159 0.001 core.py:1312(process_primitive) + 10 0.001 0.000 1.096 0.110 02_nanogpt.py:379(jax_loss) + 8135 0.003 0.000 0.936 0.000 record_batch.py:90(__next__) + 17878 0.758 0.000 0.936 0.000 {built-in method builtins.next} + 2234/14 0.003 0.000 0.900 0.064 :1349(_find_and_load) + 1723/14 0.002 0.000 0.900 0.064 :1304(_find_and_load_unlocked) + 1513/10 0.002 0.000 0.898 0.090 :911(_load_unlocked) + 1304/10 0.001 0.000 0.898 0.090 :993(exec_module) + 3824/24 0.021 0.000 0.897 0.037 :480(_call_with_frames_removed) + 5070 0.004 0.000 0.810 0.000 context.py:2415(execute) + 5070 0.805 0.000 0.806 0.000 {method 'execute' of 'datafusion.SessionContext' objects} + 839/95 0.000 0.000 0.786 0.008 {built-in method builtins.__import__} + 3065/886 0.001 0.000 0.778 0.001 :1390(_handle_fromlist) + 1 0.016 0.016 0.743 0.743 {built-in method ddxdb._ddxdb._prepare} + 570 0.008 0.000 0.711 0.001 pjit.py:1189(_pjit_call_impl) + 138 0.001 0.000 0.709 0.005 shared.py:140(__init__) + 138 0.000 0.000 0.702 0.005 shared.py:154(_physical) + 138 0.000 0.000 0.701 0.005 dataframe.py:1317(execution_plan) + + + +Sorted by self time: +Sun Nov 1 03:24:50 2026 /private/tmp/xql-shared-demo.prof + + 5075972 function calls (4993988 primitive calls) in 6.114 seconds + + Ordered by: internal time + List reduced from 9155 to 35 due to restriction <35> + + ncalls tottime percall cumtime percall filename:lineno(function) + 174 1.374 0.008 1.374 0.008 compiler.py:341(backend_compile_and_load) + 5070 0.805 0.000 0.806 0.000 {method 'execute' of 'datafusion.SessionContext' objects} + 17878 0.758 0.000 0.936 0.000 {built-in method builtins.next} + 138 0.701 0.005 0.701 0.005 {method 'execution_plan' of 'datafusion.DataFrame' objects} + 1115859 0.236 0.000 0.236 0.000 {method 'index' of 'list' objects} + 139 0.193 0.001 0.193 0.001 {built-in method _imp.create_dynamic} + 165/155 0.130 0.001 0.150 0.001 mlx.py:148(run) + 430 0.119 0.000 0.119 0.000 {built-in method from_bytes} + 125 0.071 0.001 0.071 0.001 compute.py:257(wrapper) + 1304 0.062 0.000 0.062 0.000 {built-in method marshal.loads} + 6 0.044 0.007 0.298 0.050 02_nanogpt.py:445(batches) + 38 0.043 0.001 0.043 0.001 {method 'to_arrow_table' of 'datafusion.DataFrame' objects} + 139/80 0.042 0.000 0.200 0.003 {built-in method _imp.exec_dynamic} + 270 0.037 0.000 1.971 0.007 shared.py:411(materialize_compiled) + 126 0.032 0.000 0.032 0.000 {built-in method from_substrait_plan} + 430 0.028 0.000 1.917 0.004 shared.py:166(run) + 1304 0.027 0.000 0.027 0.000 {built-in method _io.open_code} + 1304 0.024 0.000 0.086 0.000 :755(_compile_bytecode) +386824/385413 0.023 0.000 0.026 0.000 {built-in method builtins.isinstance} + 3824/24 0.021 0.000 0.897 0.037 :480(_call_with_frames_removed) + 3065 0.021 0.000 0.021 0.000 {method 'to_pyarrow' of 'datafusion.RecordBatch' objects} + 315 0.020 0.000 0.134 0.000 shared.py:191(store) + 1833/1 0.018 0.000 6.144 6.144 {built-in method builtins.exec} +3671/3652 0.017 0.000 0.069 0.000 {built-in method builtins.__build_class__} + 16 0.016 0.001 0.032 0.002 mlir.py:628(_create_ir_context) + 412 0.016 0.000 0.016 0.000 {built-in method numpy.array} + 1 0.016 0.016 0.743 0.743 {built-in method ddxdb._ddxdb._prepare} + 61 0.016 0.000 0.016 0.000 {built-in method serialize_bytes} + 6273 0.015 0.000 0.015 0.000 {built-in method posix.stat} + 1002/999 0.014 0.000 0.016 0.000 {method 'astype' of 'numpy.ndarray' objects} + 300 0.013 0.000 0.097 0.000 shared.py:95(_canonical) +244816/224578 0.013 0.000 0.016 0.000 {built-in method builtins.len} + 4 0.012 0.003 1.893 0.473 {method 'run' of 'ddxdb._Prepared' objects} + 125 0.010 0.000 0.010 0.000 compute.py:468(take) + 310 0.010 0.000 0.014 0.000 algorithms.py:473(unique_with_mask) diff --git a/benchmarks/apple/summarize_dense.py b/benchmarks/apple/summarize_dense.py new file mode 100644 index 00000000..99806f12 --- /dev/null +++ b/benchmarks/apple/summarize_dense.py @@ -0,0 +1,112 @@ +"""Summarize checked serial SQL benchmarks without including failed runs. + +Warm time is the mean paired loss/update API time at steps 1–4. Setup and +first-use costs are reported separately. Per-parameter validation and +conversion of returned Arrow rows to Datasets are outside the API timers. +""" + +from __future__ import annotations + +import argparse +import csv +import json +import statistics +from pathlib import Path + + +def _summarize(manifest): + results = [] + for record in json.loads(manifest.read_text()): + result = dict(record) | {"manifest": manifest.name} + if record["status"] != "passed": + results.append(result) + continue + rows = list(csv.DictReader((manifest.parent / record["csv"]).open())) + command = record["command"] + steps = int(command[command.index("--steps") + 1]) + forward = "--forward-only" in command + if len(rows) != steps * (1 if forward else 2) or any( + not r["check"].startswith("ok (") for r in rows + ): + raise ValueError(f"incomplete correctness records: {record['csv']}") + loss = { + int(r["step"].rsplit(" ", 1)[1]): float(r["seconds"]) + for r in rows + if r["step"].startswith("loss") + } + train = { + int(r["step"].rsplit(" ", 1)[1]): float(r["seconds"]) + for r in rows + if r["step"].startswith("train") + } + pairs = [loss[i] + train.get(i, 0) for i in range(steps)] + first, last = rows[0], rows[-1] + setup = sum( + float(last.get(k) or 0) for k in ("load_seconds", "model_seconds") + ) + result |= { + "steps": steps, + "batch": int(first["batch"]), + "parameters": int(first["parameters"]), + "context": int(first["context"]), + "optimizer": first["optimizer"], + "forward_only": forward, + "rtol": float(first["rtol"]), + "atol": float(first["atol"]), + "paired_api_seconds": pairs, + "warm_loss_seconds": statistics.mean( + loss[i] for i in range(1, steps) + ), + "warm_training_seconds": statistics.mean( + train[i] for i in range(1, steps) + ) + if train + else 0, + "warm_combined_seconds": statistics.mean(pairs[1:]), + "five_api_seconds": sum(pairs), + "loading_and_model_seconds": setup, + "five_api_with_loading_model_seconds": sum(pairs) + setup, + "final_statistics": { + k: v + for k, v in last.items() + if k.startswith( + ( + "dense_", + "gpu_", + "physical_", + "planning_", + "stage_", + "key_", + ) + ) + }, + } + results.append(result) + return results + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("manifests", nargs="+", type=Path) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args() + runs = [r for path in args.manifests for r in _summarize(path)] + args.output.write_text( + json.dumps({"method": __doc__, "runs": runs}, indent=2) + "\n" + ) + for run in runs: + if run["status"] == "passed": + print( + run["manifest"], + run["host"], + f"{run['parameters']:,}", + f"ctx={run['context']}", + f"warm={run['warm_combined_seconds']:.6f}s", + f"rss={run['peak_rss_gib']:.2f}GiB", + ) + else: + print(run["manifest"], run["host"], run["status"]) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/apple/summarize_resident.py b/benchmarks/apple/summarize_resident.py new file mode 100644 index 00000000..8ee046f1 --- /dev/null +++ b/benchmarks/apple/summarize_resident.py @@ -0,0 +1,64 @@ +"""Rebuild the resident-training summary from recorded benchmark artifacts.""" + +import hashlib +import json +import statistics +from pathlib import Path + + +def summarize(root): + baseline = json.loads((root / "shakespeare-summary.json").read_text()) + before = next(r for r in baseline["runs"] if r["backend"] == "dense_gpu") + native = json.loads((root / "shakespeare-resident-torch-mps-100.json").read_text()) + runs = [] + for name in ("shakespeare-resident-verified-gpu", "shakespeare-resident-steady-gpu"): + path = root / name / "metrics.jsonl" + rows = [json.loads(line) for line in path.read_text().splitlines()] + train = [r for r in rows if r["kind"] == "train"] + seconds = statistics.mean(r["seconds"] for r in train[1:]) + device = [r["device_step"] for r in train] + run = { + "metrics": str(path.relative_to(root.parent.parent)), + "metrics_sha256": hashlib.sha256(path.read_bytes()).hexdigest(), + "updates": len(train), + "verified_all_updates": all(r["verified"] for r in train), + "first_step_seconds": train[0]["seconds"], + "warm_mean_seconds": seconds, + "warm_tokens_per_second": 3072 / seconds, + "speedup_over_arrow_gpu": before["warm_mean_seconds"] / seconds, + "ratio_to_native_mps_100": seconds / native["warm_mean_seconds"], + "total_training_compilations": sum(r["gpu_training_compilations"] for r in device), + "timed_array_readback_bytes": sum(r["gpu_readback_bytes"] for r in device), + "timed_training_exports": sum(r["training_exports"] for r in device), + "status_reads_per_step": sorted({r["gpu_status_readbacks"] for r in device}), + "loss_reads_per_step": sorted({r["gpu_loss_readbacks"] for r in device}), + } + if "rss_bytes" in train[0]: + run.update({ + "first_rss_bytes": train[0]["rss_bytes"], + "last_rss_bytes": train[-1]["rss_bytes"], + "last_mlx_memory": train[-1]["mlx_memory"], + }) + runs.append(run) + return { + "date": "2026-11-01", + "device": baseline["device"], + "platform": baseline["platform"], + "parameters": 10745088, + "batch_tokens": 3072, + "arithmetic": "float32; MLX_ENABLE_TF32=0; compile_training and gpu_dropout enabled", + "packages": {**baseline["packages"], "ddxdb": "0.4.6+xqlmath.3", "einfold": "0.2.0+xqlmlx.6"}, + "ddx_base": baseline["ddx_revision"], + "einfold_base": "39b25210d3b39027ef91584383fb114dfa8a4a48", + "nanogpt_revision": native["nanogpt_revision"], + "baseline_gpu_seconds": before["warm_mean_seconds"], + "native_mps_100_warm_seconds": native["warm_mean_seconds"], + "native_evidence": "benchmarks/apple/shakespeare-resident-torch-mps-100.json", + "scope": "Synchronized SQL/ddx forward, backward, clipping and AdamW, resident state publication and loss scalar. Evaluation, batch sampling/upload, oracle and checkpoint I/O excluded; GPU dropout is inside. First-step excludes prior evaluation and process startup. Caches may already be warm. Warm mean excludes first update. Native includes next-batch prefetch and uses its own RNG trajectory.", + "runs": runs, + } + + +if __name__ == "__main__": + directory = Path(__file__).resolve().parent + (directory / "resident-summary.json").write_text(json.dumps(summarize(directory), indent=2) + "\n") diff --git a/benchmarks/apple/thread-confirmation.json b/benchmarks/apple/thread-confirmation.json new file mode 100644 index 00000000..f583c061 --- /dev/null +++ b/benchmarks/apple/thread-confirmation.json @@ -0,0 +1,82 @@ +{ + "environment": { + "cpu_count": 30, + "platform": "macOS-27.0.1-arm64-arm-64bit", + "reps": 9, + "label": "confirmation" + }, + "results": [ + { + "task": "thread_sweep", + "host": "datafusion", + "threads": 30, + "tokens": 2048, + "seconds": 0.7901488339994103, + "samples": [ + 0.7901488339994103, + 0.7951917499303818, + 0.7973077079514042, + 0.7708741249516606, + 0.7871014999691397, + 0.7713399579515681, + 0.7890463749645278, + 0.7966605830006301, + 0.8032787080155686 + ] + }, + { + "task": "thread_sweep", + "host": "einfold", + "threads": 30, + "tokens": 2048, + "seconds": 0.008736749994568527, + "samples": [ + 0.009245082968845963, + 0.010036124964244664, + 0.009555666940286756, + 0.00918883306439966, + 0.008736749994568527, + 0.008482916979119182, + 0.008702125051058829, + 0.008304540999233723, + 0.008283833041787148 + ] + }, + { + "task": "thread_sweep", + "host": "datafusion", + "threads": 60, + "tokens": 2048, + "seconds": 0.7470870000543073, + "samples": [ + 0.7470870000543073, + 0.7418220840627328, + 0.7904250409919769, + 0.7422788330586627, + 0.7533866659505293, + 0.7288017499959096, + 0.7582202500198036, + 0.7344983340008184, + 0.7588597090216354 + ] + }, + { + "task": "thread_sweep", + "host": "einfold", + "threads": 60, + "tokens": 2048, + "seconds": 0.009075542096979916, + "samples": [ + 0.009495667065493762, + 0.009194542071782053, + 0.009275374934077263, + 0.009075542096979916, + 0.009464459028095007, + 0.008870750083588064, + 0.008511792053468525, + 0.008415707969106734, + 0.008375791949220002 + ] + } + ] +} diff --git a/benchmarks/apple/training-overheads-before.json b/benchmarks/apple/training-overheads-before.json new file mode 100644 index 00000000..bd31bcf5 --- /dev/null +++ b/benchmarks/apple/training-overheads-before.json @@ -0,0 +1,358 @@ +{ + "environment": { + "cpu_count": 30, + "platform": "macOS-27.0.1-arm64-arm-64bit", + "reps": 9, + "label": "before" + }, + "results": [ + { + "task": "register_dataset", + "rows": 32, + "seconds": 0.00012712494935840368, + "samples": [ + 0.00020295800641179085, + 0.00015791598707437515, + 0.00014529202599078417, + 0.00012979202438145876, + 0.0001224579755216837, + 0.00011991697829216719, + 0.00012712494935840368, + 0.0001180840190500021, + 0.00011499994434416294 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 32, + "seconds": 0.00020170898642390966, + "samples": [ + 0.0002983750309795141, + 0.000242958078160882, + 0.000249749980866909, + 0.00020404194947332144, + 0.0001917920308187604, + 0.0001847910461947322, + 0.0001812080154195428, + 0.00018745800480246544, + 0.00020170898642390966 + ] + }, + { + "task": "register_and_sum", + "rows": 32, + "seconds": 0.0002515410305932164, + "samples": [ + 0.0003529160749167204, + 0.0002911670599132776, + 0.0002703750506043434, + 0.00025812501553446054, + 0.0002515410305932164, + 0.0002489999169483781, + 0.0002446670550853014, + 0.00024083408061414957, + 0.00024108297657221556 + ] + }, + { + "task": "register_dataset", + "rows": 4096, + "seconds": 0.00024754193145781755, + "samples": [ + 0.0002756669418886304, + 0.00025566702242940664, + 0.00024754193145781755, + 0.0002470000181347132, + 0.00024562503676861525, + 0.0002459579845890403, + 0.0002443330595269799, + 0.00025337503757327795, + 0.0002672090195119381 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 4096, + "seconds": 0.0005107499891892076, + "samples": [ + 0.0005197919672355056, + 0.0005107499891892076, + 0.000492165912874043, + 0.00044791598338633776, + 0.0005475840298458934, + 0.0005687919910997152, + 0.000521583016961813, + 0.0004645830485969782, + 0.00044179195538163185 + ] + }, + { + "task": "register_and_sum", + "rows": 4096, + "seconds": 0.0006732919719070196, + "samples": [ + 0.0008729170076549053, + 0.0007517080521211028, + 0.0006227080011740327, + 0.0006732919719070196, + 0.000650875037536025, + 0.0006044999463483691, + 0.0006545829819515347, + 0.0007172090699896216, + 0.0007172080222517252 + ] + }, + { + "task": "register_dataset", + "rows": 65536, + "seconds": 0.002363709034398198, + "samples": [ + 0.0023943750420585275, + 0.00231870892457664, + 0.002363709034398198, + 0.0023380829952657223, + 0.00241950002964586, + 0.002272374927997589, + 0.0023080840473994613, + 0.0023818330373615026, + 0.0024416670203208923 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 65536, + "seconds": 0.0009260409278795123, + "samples": [ + 0.000890707946382463, + 0.0009960000170394778, + 0.0010424579959362745, + 0.0009741670219227672, + 0.0009213339071720839, + 0.0009260409278795123, + 0.0008223330369219184, + 0.000977999996393919, + 0.0009202500805258751 + ] + }, + { + "task": "register_and_sum", + "rows": 65536, + "seconds": 0.003002749988809228, + "samples": [ + 0.0033863340504467487, + 0.003018291899934411, + 0.00313220894895494, + 0.003002749988809228, + 0.002753000007942319, + 0.0028048750245943666, + 0.0028373750392347574, + 0.0030807500006631017, + 0.002943583996966481 + ] + }, + { + "task": "sql", + "host": "datafusion", + "query": "product", + "seconds": 0.0008686250075697899, + "samples": [ + 0.0009977500885725021, + 0.0009561249753460288, + 0.0009101670002564788, + 0.0008686250075697899, + 0.000837208004668355, + 0.0008367910049855709, + 0.0008273749845102429, + 0.0008634999394416809, + 0.000921166967600584 + ] + }, + { + "task": "prepared", + "host": "datafusion", + "query": "product", + "seconds": 0.0007229580078274012, + "samples": [ + 0.0008207919308915734, + 0.0007840830367058516, + 0.0007229580078274012, + 0.0006890411023050547, + 0.0006823340663686395, + 0.0006787499878555536, + 0.0007661250419914722, + 0.0007251660572364926, + 0.0006953750271350145 + ] + }, + { + "task": "sql", + "host": "datafusion", + "query": "repeated_cte", + "seconds": 0.001690582954324782, + "samples": [ + 0.0017165420576930046, + 0.001690582954324782, + 0.0017382910009473562, + 0.0017208750359714031, + 0.001790915965102613, + 0.0016832080436870456, + 0.001659124973230064, + 0.0016511250287294388, + 0.0016588750295341015 + ] + }, + { + "task": "prepared", + "host": "datafusion", + "query": "repeated_cte", + "seconds": 0.001312374952249229, + "samples": [ + 0.0013360000448301435, + 0.0013662090059369802, + 0.001312374952249229, + 0.0013496249448508024, + 0.001371084013953805, + 0.001287874998524785, + 0.0013010830152779818, + 0.00131145806517452, + 0.0013072500005364418 + ] + }, + { + "task": "gradient_prepared", + "host": "datafusion", + "seconds": 0.0036672919522970915, + "samples": [ + 0.0036672919522970915, + 0.0038730000378564, + 0.003969416953623295, + 0.003425707924179733, + 0.003713875077664852, + 0.0037400840083137155, + 0.003490207949653268, + 0.0035422920482233167, + 0.0033359579974785447 + ] + }, + { + "task": "sql", + "host": "einfold", + "query": "product", + "seconds": 0.0012989579699933529, + "samples": [ + 0.0015357499942183495, + 0.0014398329658433795, + 0.0012989579699933529, + 0.0013017909368500113, + 0.001301709096878767, + 0.0012537920847535133, + 0.0012414590455591679, + 0.0012590840924531221, + 0.0012578750029206276 + ] + }, + { + "task": "prepared", + "host": "einfold", + "query": "product", + "seconds": 0.001173041993752122, + "samples": [ + 0.0013259591069072485, + 0.001173041993752122, + 0.0011105829617008567, + 0.0012064171023666859, + 0.0011468339944258332, + 0.0012579580070450902, + 0.0012408330803737044, + 0.0011690830579027534, + 0.0011508339084684849 + ] + }, + { + "task": "sql", + "host": "einfold", + "query": "repeated_cte", + "seconds": 0.001877542003057897, + "samples": [ + 0.0020875410409644246, + 0.0019669580506160855, + 0.0018942920723930001, + 0.001877542003057897, + 0.0018610829720273614, + 0.0018292919266968966, + 0.0017963750287890434, + 0.001785542001016438, + 0.0021478330017998815 + ] + }, + { + "task": "prepared", + "host": "einfold", + "query": "repeated_cte", + "seconds": 0.001981208915822208, + "samples": [ + 0.0018939999863505363, + 0.002255333005450666, + 0.002011333010159433, + 0.001981208915822208, + 0.0018877920228987932, + 0.0019936669850721955, + 0.0018754580523818731, + 0.002010042080655694, + 0.0018965830095112324 + ] + }, + { + "task": "gradient_prepared", + "host": "einfold", + "seconds": 0.009082957985810935, + "samples": [ + 0.008911916986107826, + 0.010163999977521598, + 0.009411791921593249, + 0.009082957985810935, + 0.009333624970167875, + 0.008591708028689027, + 0.009091541985981166, + 0.008726166095584631, + 0.008915040991269052 + ] + }, + { + "task": "thread_sweep", + "host": "datafusion", + "threads": 30, + "tokens": 256, + "seconds": 0.045458749984391034, + "samples": [ + 0.05048225005157292, + 0.045819457969628274, + 0.045458749984391034, + 0.04530299990437925, + 0.044829958002083004, + 0.04552145802881569, + 0.04515074996743351, + 0.04511454200837761, + 0.04585320805199444 + ] + }, + { + "task": "thread_sweep", + "host": "einfold", + "threads": 30, + "tokens": 256, + "seconds": 0.0034217090578749776, + "samples": [ + 0.0037245419807732105, + 0.003661583992652595, + 0.0035677499836310744, + 0.0035481250379234552, + 0.00338487490080297, + 0.0034217090578749776, + 0.0032776249572634697, + 0.0033539170399308205, + 0.003177290898747742 + ] + } + ] +} diff --git a/benchmarks/apple/training-overheads-final.json b/benchmarks/apple/training-overheads-final.json new file mode 100644 index 00000000..b005410c --- /dev/null +++ b/benchmarks/apple/training-overheads-final.json @@ -0,0 +1,371 @@ +{ + "environment": { + "cpu_count": 30, + "platform": "macOS-27.0.1-arm64-arm-64bit", + "reps": 9, + "label": "final" + }, + "results": [ + { + "task": "register_dataset", + "rows": 32, + "seconds": 0.00010645901784300804, + "samples": [ + 0.0001489589922130108, + 0.0001261249417439103, + 0.00010645901784300804, + 0.00010391592513769865, + 0.00010362500324845314, + 0.00010341696906834841, + 0.00010433397255837917, + 0.0001663749571889639, + 0.0001443750225007534 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 32, + "seconds": 0.00024341605603694916, + "samples": [ + 0.00041733402758836746, + 0.0003140839980915189, + 0.0005200420273467898, + 0.0002979579148814082, + 0.00024341605603694916, + 0.00019754096865653992, + 0.00018879096023738384, + 0.00017937493976205587, + 0.00017658399883657694 + ] + }, + { + "task": "register_and_sum", + "rows": 32, + "seconds": 0.00023758294992148876, + "samples": [ + 0.0003131669946014881, + 0.00026412506122142076, + 0.00024695799220353365, + 0.0002448749728500843, + 0.00023120793048292398, + 0.00022624991834163666, + 0.00022333301603794098, + 0.00023758294992148876, + 0.00022195791825652122 + ] + }, + { + "task": "register_dataset", + "rows": 4096, + "seconds": 0.0002240829635411501, + "samples": [ + 0.00025795900728553534, + 0.00023508304730057716, + 0.000225124997086823, + 0.0002230409299954772, + 0.0002227500081062317, + 0.0002240829635411501, + 0.00022149994038045406, + 0.0002246249932795763, + 0.00022391590755432844 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 4096, + "seconds": 0.0004723339807242155, + "samples": [ + 0.0005690830294042826, + 0.0006104169879108667, + 0.0004588329466059804, + 0.000470208004117012, + 0.0004927919944748282, + 0.0004289579810574651, + 0.0004133750917389989, + 0.00048562500160187483, + 0.0004723339807242155 + ] + }, + { + "task": "register_and_sum", + "rows": 4096, + "seconds": 0.0006517079891636968, + "samples": [ + 0.0007588750449940562, + 0.0006517079891636968, + 0.0006984169594943523, + 0.0007941670482978225, + 0.0006428749766200781, + 0.0006678329082205892, + 0.0006379160331562161, + 0.0006192500004544854, + 0.0006311660399660468 + ] + }, + { + "task": "register_dataset", + "rows": 65536, + "seconds": 0.002375875017605722, + "samples": [ + 0.002375875017605722, + 0.00228433299344033, + 0.002462125034071505, + 0.002346666995435953, + 0.0024745409609749913, + 0.0023931250907480717, + 0.002377042081207037, + 0.0022560410434380174, + 0.0023356250021606684 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 65536, + "seconds": 0.001027416903525591, + "samples": [ + 0.0010781249729916453, + 0.0010483339428901672, + 0.001027416903525591, + 0.00090954196639359, + 0.0008824999677017331, + 0.0008387499256059527, + 0.0010642920387908816, + 0.001043915981426835, + 0.0008438749937340617 + ] + }, + { + "task": "register_and_sum", + "rows": 65536, + "seconds": 0.0027691249269992113, + "samples": [ + 0.002858957974240184, + 0.0028876670403406024, + 0.002827499993145466, + 0.002896999940276146, + 0.0027397499652579427, + 0.0026956249494105577, + 0.002665917039848864, + 0.0027691249269992113, + 0.0027113750111311674 + ] + }, + { + "task": "register_fine_chunks", + "rows": 4096, + "columns": 16, + "seconds": 0.0004051249707117677, + "samples": [ + 0.000469875056296587, + 0.00041379197500646114, + 0.0004051249707117677, + 0.00039858289528638124, + 0.0003944169729948044, + 0.0003918340662494302, + 0.0003934999695047736, + 0.0004116250202059746, + 0.00041279103606939316 + ] + }, + { + "task": "sql", + "host": "datafusion", + "query": "product", + "seconds": 0.0008812920423224568, + "samples": [ + 0.000993583002127707, + 0.0008934999350458384, + 0.0008812920423224568, + 0.0008645419729873538, + 0.0008415419142693281, + 0.0008394999895244837, + 0.0008381250081583858, + 0.00091104197781533, + 0.0009936249116435647 + ] + }, + { + "task": "prepared", + "host": "datafusion", + "query": "product", + "seconds": 0.0006823749281466007, + "samples": [ + 0.0007148749427869916, + 0.0006863339804112911, + 0.0006726249121129513, + 0.0006692500319331884, + 0.000668209046125412, + 0.0007860419573262334, + 0.0007349159568548203, + 0.0006823749281466007, + 0.000670416047796607 + ] + }, + { + "task": "sql", + "host": "datafusion", + "query": "repeated_cte", + "seconds": 0.0017094999784603715, + "samples": [ + 0.0017254999838769436, + 0.0017460829112678766, + 0.0017094999784603715, + 0.0017092919442802668, + 0.0020056249340996146, + 0.0017505409196019173, + 0.0017039589583873749, + 0.0016880419570952654, + 0.0016672080382704735 + ] + }, + { + "task": "prepared", + "host": "datafusion", + "query": "repeated_cte", + "seconds": 0.0013071250868961215, + "samples": [ + 0.0013127080164849758, + 0.0013365410268306732, + 0.0013071250868961215, + 0.0013364589540287852, + 0.0012950419913977385, + 0.0012892918894067407, + 0.0013047500979155302, + 0.0012981250183656812, + 0.0013085000682622194 + ] + }, + { + "task": "gradient_context_prepared", + "host": "datafusion", + "seconds": 0.003434707992710173, + "samples": [ + 0.003739292034879327, + 0.0038018750492483377, + 0.00346791697666049, + 0.003314291941933334, + 0.004152124980464578, + 0.003434707992710173, + 0.0033455000957474113, + 0.0034343749284744263, + 0.0033617500448599458 + ] + }, + { + "task": "gradient_prepared", + "host": "datafusion", + "seconds": 0.003718332969583571, + "samples": [ + 0.0033787089632824063, + 0.003396832966245711, + 0.0033159590093418956, + 0.006050167023204267, + 0.003846750012598932, + 0.003718332969583571, + 0.003558000084012747, + 0.00409337505698204, + 0.003755416953936219 + ] + }, + { + "task": "sql", + "host": "einfold", + "query": "product", + "seconds": 0.001436999998986721, + "samples": [ + 0.0015172080602496862, + 0.00133200001437217, + 0.0012446669861674309, + 0.001874250010587275, + 0.0017677079886198044, + 0.001460542087443173, + 0.001436999998986721, + 0.0012977499281987548, + 0.001252374961040914 + ] + }, + { + "task": "prepared", + "host": "einfold", + "query": "product", + "seconds": 0.0011767089599743485, + "samples": [ + 0.001198875019326806, + 0.0011767089599743485, + 0.001114542013965547, + 0.0011638749856501818, + 0.001364082912914455, + 0.001235750038176775, + 0.0012203749502077699, + 0.0011497079394757748, + 0.0011304590152576566 + ] + }, + { + "task": "sql", + "host": "einfold", + "query": "repeated_cte", + "seconds": 0.0018477500416338444, + "samples": [ + 0.0018477500416338444, + 0.001814875053241849, + 0.001812709029763937, + 0.0017883339896798134, + 0.002385542029514909, + 0.0020317500457167625, + 0.001892582979053259, + 0.0018425409216433764, + 0.0018761670216917992 + ] + }, + { + "task": "prepared", + "host": "einfold", + "query": "repeated_cte", + "seconds": 0.0017464590491726995, + "samples": [ + 0.0018430419731885195, + 0.002097583026625216, + 0.001705583999864757, + 0.0016526670660823584, + 0.0016428750241175294, + 0.001751875039190054, + 0.0016900839982554317, + 0.0017464590491726995, + 0.0018924169708043337 + ] + }, + { + "task": "gradient_context_prepared", + "host": "einfold", + "seconds": 0.009094916982576251, + "samples": [ + 0.009345291997306049, + 0.009007250075228512, + 0.009130332968197763, + 0.009412707993760705, + 0.009209917043335736, + 0.009094916982576251, + 0.008615250000730157, + 0.009059040923602879, + 0.008686457993462682 + ] + }, + { + "task": "gradient_prepared", + "host": "einfold", + "seconds": 0.00904766598250717, + "samples": [ + 0.00922454206738621, + 0.008593791048042476, + 0.009089207975193858, + 0.008508125087246299, + 0.009798625018447638, + 0.00874391698744148, + 0.009182333014905453, + 0.00904766598250717, + 0.008820499991998076 + ] + } + ] +} diff --git a/benchmarks/apple/training-overheads.json b/benchmarks/apple/training-overheads.json new file mode 100644 index 00000000..f06b5d85 --- /dev/null +++ b/benchmarks/apple/training-overheads.json @@ -0,0 +1,502 @@ +{ + "environment": { + "cpu_count": 30, + "platform": "macOS-27.0.1-arm64-arm-64bit", + "reps": 9, + "label": "after" + }, + "results": [ + { + "task": "register_dataset", + "rows": 32, + "seconds": 0.00010954204481095076, + "samples": [ + 0.0001598750241100788, + 0.0001264170277863741, + 0.00011212506797164679, + 0.0001079590292647481, + 0.00010954204481095076, + 0.00010675005614757538, + 0.00010325002949684858, + 0.0001041659852489829, + 0.00011012505274266005 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 32, + "seconds": 0.00018745800480246544, + "samples": [ + 0.0006457919953390956, + 0.0005402080714702606, + 0.000482583069242537, + 0.00023875001352280378, + 0.00018745800480246544, + 0.00018416706006973982, + 0.0001700409920886159, + 0.0001667090691626072, + 0.00016533304005861282 + ] + }, + { + "task": "register_and_sum", + "rows": 32, + "seconds": 0.00024687498807907104, + "samples": [ + 0.0003261670935899019, + 0.00026662496384233236, + 0.0002508750185370445, + 0.00024687498807907104, + 0.00024800002574920654, + 0.00024579197634011507, + 0.00023308303207159042, + 0.0002292919671162963, + 0.00022787495981901884 + ] + }, + { + "task": "register_dataset", + "rows": 4096, + "seconds": 0.00025158305652439594, + "samples": [ + 0.0002663329942151904, + 0.0002519169356673956, + 0.00023808295372873545, + 0.00023304205387830734, + 0.00024079193826764822, + 0.0002532500075176358, + 0.0002509999321773648, + 0.00025829195510596037, + 0.00025158305652439594 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 4096, + "seconds": 0.0004678330151364207, + "samples": [ + 0.0005803329404443502, + 0.0004688749322667718, + 0.0005575420800596476, + 0.0004545829724520445, + 0.0004678330151364207, + 0.00045720895286649466, + 0.0006029170472174883, + 0.00044670794159173965, + 0.00046445801854133606 + ] + }, + { + "task": "register_and_sum", + "rows": 4096, + "seconds": 0.0006899170111864805, + "samples": [ + 0.0007077499758452177, + 0.0006899170111864805, + 0.0007315829861909151, + 0.0006989999674260616, + 0.0005919580580666661, + 0.0006543330382555723, + 0.000601666048169136, + 0.0006380409467965364, + 0.0007614579517394304 + ] + }, + { + "task": "register_dataset", + "rows": 65536, + "seconds": 0.00234412495046854, + "samples": [ + 0.00234412495046854, + 0.0024157920852303505, + 0.0024086249759420753, + 0.0023014580365270376, + 0.0023864590330049396, + 0.0022683750139549375, + 0.0022810830269008875, + 0.0022876670118421316, + 0.0024922500597313046 + ] + }, + { + "task": "prepared_write_and_sum", + "rows": 65536, + "seconds": 0.0009534170385450125, + "samples": [ + 0.0009922910248860717, + 0.0010079999919980764, + 0.0008682080078870058, + 0.0009868339402601123, + 0.0008707499364390969, + 0.0008653340628370643, + 0.0009348340099677444, + 0.0009534170385450125, + 0.001017875038087368 + ] + }, + { + "task": "register_and_sum", + "rows": 65536, + "seconds": 0.0033994170371443033, + "samples": [ + 0.0029995000222697854, + 0.003025667043402791, + 0.0033994170371443033, + 0.0037623329553753138, + 0.003365917014889419, + 0.003273208043538034, + 0.0035405829548835754, + 0.004040250089019537, + 0.00351208308711648 + ] + }, + { + "task": "sql", + "host": "datafusion", + "query": "product", + "seconds": 0.0011444169795140624, + "samples": [ + 0.001288625062443316, + 0.0011444169795140624, + 0.0014107079477980733, + 0.0012428329791873693, + 0.0009879160206764936, + 0.0009857499971985817, + 0.0014517080271616578, + 0.001098083914257586, + 0.0008762090001255274 + ] + }, + { + "task": "prepared", + "host": "datafusion", + "query": "product", + "seconds": 0.0006741250399500132, + "samples": [ + 0.0007239578990265727, + 0.0006817498942837119, + 0.0007114580366760492, + 0.0006665409309789538, + 0.0006759170209988952, + 0.0006741250399500132, + 0.0006637920159846544, + 0.0006720829987898469, + 0.0006557919550687075 + ] + }, + { + "task": "sql", + "host": "datafusion", + "query": "repeated_cte", + "seconds": 0.0023224999895319343, + "samples": [ + 0.001735959085635841, + 0.0023224999895319343, + 0.0022287500323727727, + 0.0018957500578835607, + 0.0017628750065341592, + 0.0030990419909358025, + 0.0028174580074846745, + 0.0024897500406950712, + 0.0024012909270823 + ] + }, + { + "task": "prepared", + "host": "datafusion", + "query": "repeated_cte", + "seconds": 0.001995875034481287, + "samples": [ + 0.0021742500830441713, + 0.002208749996498227, + 0.0018810420297086239, + 0.0018634169828146696, + 0.001995875034481287, + 0.0022771249059587717, + 0.0021090409718453884, + 0.0019037090241909027, + 0.0019660829566419125 + ] + }, + { + "task": "gradient_prepared", + "host": "datafusion", + "seconds": 0.004188833059743047, + "samples": [ + 0.005623165983706713, + 0.0066480840323492885, + 0.0055510420352220535, + 0.0057469160528853536, + 0.004012292018160224, + 0.004137082956731319, + 0.0037805839674547315, + 0.0036733750021085143, + 0.004188833059743047 + ] + }, + { + "task": "sql", + "host": "einfold", + "query": "product", + "seconds": 0.0014984169974923134, + "samples": [ + 0.001577167073264718, + 0.0015121659962460399, + 0.0014984169974923134, + 0.0014285000506788492, + 0.0012985829962417483, + 0.0014405419351533055, + 0.001416041050106287, + 0.001851332955993712, + 0.0017582080326974392 + ] + }, + { + "task": "prepared", + "host": "einfold", + "query": "product", + "seconds": 0.0011858749203383923, + "samples": [ + 0.0011949160834774375, + 0.0011969159822911024, + 0.0011425840202718973, + 0.0011740420013666153, + 0.001121666980907321, + 0.0012039580615237355, + 0.0011858749203383923, + 0.0011671249521896243, + 0.001561500015668571 + ] + }, + { + "task": "sql", + "host": "einfold", + "query": "repeated_cte", + "seconds": 0.001958582899533212, + "samples": [ + 0.001958582899533212, + 0.0019546670373529196, + 0.0019297089893370867, + 0.0018382919952273369, + 0.0018907909980043769, + 0.001973208039999008, + 0.0023236670531332493, + 0.002298167091794312, + 0.0020657910499721766 + ] + }, + { + "task": "prepared", + "host": "einfold", + "query": "repeated_cte", + "seconds": 0.0018105839844793081, + "samples": [ + 0.0016767920460551977, + 0.0018105839844793081, + 0.0020094580249860883, + 0.0023384999949485064, + 0.0023963330313563347, + 0.001867417013272643, + 0.001697832951322198, + 0.0017190419603139162, + 0.0016771659720689058 + ] + }, + { + "task": "gradient_prepared", + "host": "einfold", + "seconds": 0.00952566706109792, + "samples": [ + 0.01042745797894895, + 0.015069916960783303, + 0.009525208035483956, + 0.010767375002615154, + 0.009642166900448501, + 0.009454458020627499, + 0.00949904101435095, + 0.008906667004339397, + 0.00952566706109792 + ] + }, + { + "task": "thread_sweep", + "host": "datafusion", + "threads": 1, + "tokens": 2048, + "seconds": 6.797253291937523, + "samples": [ + 6.966559167020023, + 6.866939791943878, + 6.773954042000696, + 6.797253291937523, + 6.908656374900602, + 6.794572124956176, + 6.77834212500602, + 6.944329541991465, + 6.764209542074241 + ] + }, + { + "task": "thread_sweep", + "host": "einfold", + "threads": 1, + "tokens": 2048, + "seconds": 0.025110292015597224, + "samples": [ + 0.025809749960899353, + 0.025502000004053116, + 0.02475137508008629, + 0.02484495798125863, + 0.025110292015597224, + 0.025271792081184685, + 0.025262875016778708, + 0.024914125096984208, + 0.0250011250609532 + ] + }, + { + "task": "thread_sweep", + "host": "datafusion", + "threads": 10, + "tokens": 2048, + "seconds": 1.4564081249991432, + "samples": [ + 1.454833082971163, + 1.4737039579777047, + 1.4681051670340821, + 1.4636198750231415, + 1.44507508398965, + 1.4543672499712557, + 1.4733307500137016, + 1.4564081249991432, + 1.4505268749780953 + ] + }, + { + "task": "thread_sweep", + "host": "einfold", + "threads": 10, + "tokens": 2048, + "seconds": 0.010010165977291763, + "samples": [ + 0.010010165977291763, + 0.010035041021183133, + 0.009709417005069554, + 0.010043749934993684, + 0.012923042057082057, + 0.010610709083266556, + 0.00949716696050018, + 0.009967792080715299, + 0.009332500048913062 + ] + }, + { + "task": "thread_sweep", + "host": "datafusion", + "threads": 20, + "tokens": 2048, + "seconds": 0.9636518330080435, + "samples": [ + 0.9981592909898609, + 0.9892778329085559, + 0.9781492919428274, + 0.9904568750644103, + 0.9627878749743104, + 0.9593620840460062, + 0.9513139999471605, + 0.9629402080317959, + 0.9636518330080435 + ] + }, + { + "task": "thread_sweep", + "host": "einfold", + "threads": 20, + "tokens": 2048, + "seconds": 0.009234709083102643, + "samples": [ + 0.009800750063732266, + 0.009873583097942173, + 0.009299624944105744, + 0.009733624989166856, + 0.009233000106178224, + 0.009083583950996399, + 0.009092833963222802, + 0.008824583957903087, + 0.009234709083102643 + ] + }, + { + "task": "thread_sweep", + "host": "datafusion", + "threads": 30, + "tokens": 2048, + "seconds": 0.7947687909472734, + "samples": [ + 0.809984833933413, + 0.8019449169514701, + 0.8249652090016752, + 0.7947687909472734, + 0.7744551659561694, + 0.7864025419112295, + 0.7744688329985365, + 0.7976371670374647, + 0.7918349580140784 + ] + }, + { + "task": "thread_sweep", + "host": "einfold", + "threads": 30, + "tokens": 2048, + "seconds": 0.00929850002285093, + "samples": [ + 0.009170583100058138, + 0.00871741701848805, + 0.008498999988660216, + 0.009191917022690177, + 0.010007916018366814, + 0.00929850002285093, + 0.009511707932688296, + 0.009643833036534488, + 0.009299541008658707 + ] + }, + { + "task": "thread_sweep", + "host": "datafusion", + "threads": 60, + "tokens": 2048, + "seconds": 0.7485410000663251, + "samples": [ + 0.7676040000515059, + 0.7385192079236731, + 0.7446305840276182, + 0.7601484169717878, + 0.7227030420908704, + 0.7730295408982784, + 0.7440219999989495, + 0.7485410000663251, + 0.750917625031434 + ] + }, + { + "task": "thread_sweep", + "host": "einfold", + "threads": 60, + "tokens": 2048, + "seconds": 0.008907875046133995, + "samples": [ + 0.015425999998115003, + 0.009728500037454069, + 0.008934915997087955, + 0.008639874984510243, + 0.008907875046133995, + 0.008516790927387774, + 0.008795166038908064, + 0.008549582911655307, + 0.008925708010792732 + ] + } + ] +}