Skip to content

Add CLIP Contrastive Loss challenge (hard #162) - #387

Open
claude[bot] wants to merge 1 commit into
mainfrom
add-challenge-162-clip-contrastive-loss
Open

claude[bot] wants to merge 1 commit into
mainfrom
add-challenge-162-clip-contrastive-loss

Conversation

@claude

@claude claude Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds CLIP Contrastive Loss (hard #162): the symmetric InfoNCE loss used to train CLIP/ALIGN-style image–text models (and dual-encoder retrievers). Inputs are image_embeds and text_embeds, each N × D with arbitrary row norms, plus logit_scale. Each row is L2-normalized, S = logit_scale · x̂ ŷᵀ is formed, and the output is the mean of the row-softmax (image→text) and column-softmax (text→image) cross-entropies, with the diagonal as targets.

What a solver learns

  • Fusing a GEMM with a streaming log-sum-exp reduction in two directions at once. Each S tile updates the running (max, sum-exp) state for its rows and its columns.
  • Merging online-softmax state across thread blocks. The row reduction stays inside one block-row, but the column reduction for a given column is spread across every block-row of the grid. Solvers must emit per-tile (max, sum) partials and merge them in a separate pass, or use another cross-CTA scheme.
  • Folding a normalization prologue into the GEMM epilogue. You don't need to write normalized copies of the inputs: compute 1/‖x_i‖ and 1/‖y_j‖ once and apply them as rank-1 scales to each accumulator tile.

Novelty vs. the closest existing challenges

Naive vs. optimized

  • Naive (correct): normalize into two new buffers, run a GEMM that writes the N×N S (1 GB at the benchmark size), then run a row-LSE kernel and a column-LSE kernel. A naive column-LSE kernel reads S with stride N, which is uncoalesced. Alternatively, a "fused" version that recomputes the GEMM once per direction doubles the FLOPs.
  • Optimized (same result): one tiled pass over S that never writes it. Each block computes a 64×64 (or larger) tile from shared-memory-staged x and y, applies scale · inv_x[i] · inv_y[j], and reduces the tile into row partials (warp shuffles) and column partials (shared memory). A small merge kernel combines the N/BM partials per row and per column, then a deterministic final reduction computes the mean.

How the benchmark exposes it

Performance test: N = 16,384, D = 512, logit_scale = 100. The GEMM is about 275 GFLOP, so a good solution is compute-bound on the T4's FP32 units.

  • Materializing S adds a 1 GB write plus two 1 GB reads (the column read is strided unless transposed), and memory use grows as O(N²). The fused version only touches the 64 MB of inputs plus O(N²/BM) partials (about 2 × 256 × 16,384 float2 ≈ 67 MB at BM = 64).
  • Recomputing the GEMM per direction doubles the dominant cost, so sharing one tile between both reductions removes ~50% of the FLOPs compared with the naive fused alternative.
  • The data is paired (text = image + 5·noise), so the loss is O(1) nats (~3.2) and off-diagonal logits really contribute (CLIP-like rather than degenerate). logit_scale = 100 is CLIP's clamp value, so logits reach tens of nats and need a correct running max.

Targeted tests (mistakes caught)

I checked locally that each mistake below differs from the reference by far more than the tolerance in at least one test:

  • Single softmax direction (row-only or column-only), or assuming S is symmetric: an asymmetric case (near-identical image rows, diverse text rows; 1.68 nat gap), plus the example (0.21) and N=255 (3.56). Also confirmed on a T4: a row-only CUDA variant fails the example with 1.9451 vs 1.7325.
  • Missing L2 normalization / normalizing the wrong axis: rows scaled ×0.01…×100 with N≠D.
  • No max subtraction: all embeddings clustered around one direction, so every logit is in [98, 100] and exp overflows float32 while the loss stays non-trivial (3.48).
  • Wrong target index / sum instead of mean / dividing by logit_scale: the large-diagonal cases (N=512, 1000, 4096).
  • Uninitialized accumulators: N=1, D=1 (loss must be exactly 0).
  • Wrong constant handling: logit_scale = 0 is the zero-input case (loss must be exactly log N).
  • Tail tiles: N=255 with D=3, and N=1000 with D=100.
  • Ties: duplicate text rows (repeated captions).
  • Independent check: the example was hand-computed (row losses 0.2633 / 3.6270, column losses 0.9130 / 2.1269, loss 1.7325) and matches the reference to 1e-7.

Tolerance: atol = rtol = 1e-3. The loss is a mean of LSE values of magnitude ≤ ~10, built from unit-vector dot products with |logit| ≤ 100. FP32 accumulation-order error is ~1e-5, and every mistake above is off by ≥ 0.2 nats.

Validation

  • scripts/run_challenge.py --accelerator T4 with a fused tiled CUDA solution: --action run passed and --action submit passed all functional tests plus the performance test. The solution file is not committed.
  • A deliberately row-only variant was rejected (see above).
  • pre-commit run --all-files and scripts/validate_challenges.py pass.
  • CLAUDE.md checklist verified:
    • The HTML starts with <p>, uses <h2> sections, includes an SVG (#222 background) and LaTeX matrix examples, and has the performance bullet.
    • There is no __init__, and all allocations use device=self.device.
    • The reference uses only standard PyTorch ops (XLA-safe).
    • The 6 starters each have exactly one parameter comment (medium/hard style, no parenthetical).

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Review bot: low risk, safe to merge. Lint and structure checks passed; no sensitive paths or grader edits; 141 lines changed.

  • Numeric ID 162 is unused by any existing challenge. The challenge is distinct from #129, #128, #141, #25 and #150 because it needs a two-direction (row and column) LSE merge from one GEMM.
  • challenge.py defines the 5 required methods with no init override and sets name, atol, rtol, num_gpus and access_tier. The helper methods _pack, _make and _randn are harmless.
  • Imports are limited to ctypes, typing, torch and the challenge base. There is no network, file or subprocess access and no import-time code.
  • reference_impl asserts shapes and dtypes and uses only standard PyTorch ops. All allocations in generate_* use device=self.device.
  • Functional tests cover N=1, logit_scale=0, wildly varying row norms, asymmetric logits, overflow-range logits, tail tiles, duplicate columns, and power-of-two and non-power-of-two sizes. The 1e-3 tolerances are sensible for a loss of about 3.
  • Performance inputs and outputs total about 64 MB, well under the memory limit.
  • challenge.html is consistent with the signature, and I re-checked the Example 1 numbers (1.7325); they are correct.
  • All six starters are empty stubs with only the parameter comment. The Mojo starter uses max.gpu imports and MutAnyOrigin.
  • The PR body is cut off mid-sentence, but the diff is complete and fully judgeable.

@shxjames shxjames left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed f4bfa52. Spec and reference agree; no correctness blocker.

Spec / reference. Every row is L2-normalized with no epsilon (constraints already require a nonzero L2). S = logit_scale * x̂ ŷᵀ: logit_scale is the already-scaled multiplier in [0, 100], not exp(log_scale) and not a temperature divisor. The loss is ½ (mean of row CE + mean of column CE) with diagonal targets. The only output is a scalar loss of shape (1,); no gradients. On a 1-element tensor the max-scaled grader is just |\u0394| ≤ 1e-3 · (1 + |loss|), which is a meaningful check (budget ≈ 2e-3–1.5e-2 on the non-trivial cases).

An independent fp64 CPU reimplementation matches the reference. The HTML example (row losses 0.2633 / 3.6270, column 0.9130 / 2.1269, loss 1.7325) matches both (fp64 1.73254570, ref 1.73254561).

Headroom (share of atol + rtol · |ref|). Correct fp32, F.cross_entropy, and tiled online-LSE (256 / 128 / 64-reversed) stay at ≤ 2.7e-4 of budget on every functional case and ≤ 4.9e-5 on perf (≥ 3700×). Reduction order is not a problem.

TF32 / FP16 / BF16 all pass, though: worst shares are 0.055 / 0.056 / 0.177 on the N=48 asymmetric case, and 0.00034 / 0.0017 / 0.016 on perf. Because the output is one averaged scalar, 1e-3 is loose enough that reduced-precision shortcuts succeed everywhere. Optional: tighten to ~5e-5 (correct fp32 would still have ~190× room, and that same case would reject all three). Not a correctness blocker if that is acceptable.

Injected bugs. Missing L2, L2 along dim 0, divide by ‖x‖², divide by / omit / exp of logit_scale, off-by-one or zero targets, no /2, sum instead of mean, and uniform CE are all caught, usually by a large margin (N=1 and logit_scale=0 miss several by construction). One-direction-only is missed on the nearly-symmetric cases, including perf (0.63 of budget; i2t/t2i gap 0.003 nats), but is caught hard on the designed asymmetric tests (N=48: gap 3.37 nats, 257×; N=255: gap 7.11 nats, 234×) and the example (78×). A missing max-subtraction in fp32 overflows only on the clustered scale=100 case (logits ≈ 100); every other case, including perf (diag in [1.8, 36]), stays finite and matches — so only that one test is doing that job. Adding 1e-6 (or 1e-12) to the L2 denom is never caught (worst 0.0015 of budget), which is fine given the spec has no eps.

Perf. N=16384, D=512, scale=100: loss is 1.4535 (the PR text says ~3.2; same order, just stale). I/O 67 MB; materializing S is 1.07 GB, still fine on a T4. GEMM is ~275 GFLOP, theoretical T4 fp32 floor ~34 ms, compute-bound as claimed. Every test dict includes the loss buffer (zeros, not a 123 prefill; an unwritten 0 fails every case except N=1, where the expected loss is 0).

All six starters match the signature; JAX returns the output. CI is green (Lint Python, Lint C++/CUDA, Check Mojo, check-index, checks, review). I have not approved or triggered a T4 run; the author reports a fused CUDA solution already passed --action submit on a T4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant