Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions challenges/hard/162_clip_contrastive_loss/challenge.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
<p>
Compute the symmetric InfoNCE loss used to train CLIP-style image&ndash;text models. Given a
batch of <code>N</code> paired embeddings <code>image_embeds</code> and <code>text_embeds</code>,
each of shape <code>N &times; D</code> (row <code>i</code> of both is a matching pair), L2-normalize
every row, form the scaled cosine-similarity matrix, and average the cross-entropy of
classifying each image's matching text (softmax over each row) and each text's matching image
(softmax over each column). All tensors are 32-bit floats; write the scalar loss to
<code>loss[0]</code>.
</p>
<p>
\[
\hat{x}_i = \frac{x_i}{\lVert x_i \rVert_2}, \quad
\hat{y}_j = \frac{y_j}{\lVert y_j \rVert_2}, \quad
S_{i,j} = \text{logit_scale} \cdot \hat{x}_i \cdot \hat{y}_j
\]
\[
\text{loss} = \frac{1}{2}\left(
\frac{1}{N}\sum_{i=0}^{N-1}\Big(\log\sum_{j=0}^{N-1} e^{S_{i,j}} - S_{i,i}\Big) +
\frac{1}{N}\sum_{j=0}^{N-1}\Big(\log\sum_{i=0}^{N-1} e^{S_{i,j}} - S_{j,j}\Big)
\right)
\]
</p>
<p>
where <code>x</code> = <code>image_embeds</code> and <code>y</code> = <code>text_embeds</code>.
The <code>N &times; N</code> logit matrix is much larger than the inputs, so a fast solution
computes each tile of <code>S</code> once and folds it into running log-sum-exp statistics for
both its rows and its columns instead of materializing <code>S</code>.
</p>

<svg width="560" height="250" viewBox="0 0 560 250" xmlns="http://www.w3.org/2000/svg"
style="display:block; margin:20px auto;">
<defs>
<marker id="arr" markerWidth="8" markerHeight="6" refX="7" refY="3" orient="auto">
<polygon points="0 0,8 3,0 6" fill="#bbb"/>
</marker>
</defs>
<rect width="560" height="250" fill="#222" rx="8"/>
<text x="20" y="28" fill="#ddd" font-family="monospace" font-size="13">S = logit_scale &#183; x&#770; y&#770;&#7488; (N &#215; N)</text>
<g stroke="#555" fill="#2a2a2a">
<rect x="120" y="50" width="40" height="40"/><rect x="160" y="50" width="40" height="40"/>
<rect x="200" y="50" width="40" height="40"/><rect x="240" y="50" width="40" height="40"/>
<rect x="120" y="90" width="40" height="40"/><rect x="160" y="90" width="40" height="40"/>
<rect x="200" y="90" width="40" height="40"/><rect x="240" y="90" width="40" height="40"/>
<rect x="120" y="130" width="40" height="40"/><rect x="160" y="130" width="40" height="40"/>
<rect x="200" y="130" width="40" height="40"/><rect x="240" y="130" width="40" height="40"/>
<rect x="120" y="170" width="40" height="40"/><rect x="160" y="170" width="40" height="40"/>
<rect x="200" y="170" width="40" height="40"/><rect x="240" y="170" width="40" height="40"/>
</g>
<g fill="#2d5a27" stroke="#5c9c3a">
<rect x="120" y="50" width="40" height="40"/><rect x="160" y="90" width="40" height="40"/>
<rect x="200" y="130" width="40" height="40"/><rect x="240" y="170" width="40" height="40"/>
</g>
<text x="20" y="74" fill="#8ecae6" font-family="monospace" font-size="12">image 0</text>
<text x="20" y="114" fill="#8ecae6" font-family="monospace" font-size="12">image 1</text>
<text x="20" y="154" fill="#8ecae6" font-family="monospace" font-size="12">image 2</text>
<text x="20" y="194" fill="#8ecae6" font-family="monospace" font-size="12">image 3</text>
<text x="122" y="228" fill="#e3b07e" font-family="monospace" font-size="11">txt0 txt1 txt2 txt3</text>
<line x1="290" y1="110" x2="350" y2="110" stroke="#bbb" marker-end="url(#arr)"/>
<text x="358" y="106" fill="#8ecae6" font-family="monospace" font-size="12">row logsumexp:</text>
<text x="358" y="122" fill="#8ecae6" font-family="monospace" font-size="12">image &#8594; text loss</text>
<line x1="220" y1="215" x2="300" y2="236" stroke="#bbb" marker-end="url(#arr)"/>
<text x="308" y="236" fill="#e3b07e" font-family="monospace" font-size="12">column logsumexp: text &#8594; image</text>
<text x="358" y="170" fill="#5c9c3a" font-family="monospace" font-size="12">diagonal = targets</text>
</svg>

<h2>Implementation Requirements</h2>
<ul>
<li>Do not change the function signature.</li>
<li>Do not use external libraries beyond what is available in the starter.</li>
<li>Write the scalar loss to <code>loss[0]</code>.</li>
<li>The computation must be numerically stable for logits up to <code>logit_scale</code>
(subtract the running maximum).</li>
</ul>

<h2>Example 1:</h2>
<p>
Input (<code>N</code> = 2, <code>D</code> = 2, <code>logit_scale</code> = 2.0):
\[
\text{image_embeds} = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}, \quad
\text{text_embeds} = \begin{bmatrix} 3 & 4 \\ 0 & -1 \end{bmatrix}
\]
Normalized rows and logits:
\[
\hat{x} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}, \quad
\hat{y} = \begin{bmatrix} 0.6 & 0.8 \\ 0 & -1 \end{bmatrix}, \quad
S = \begin{bmatrix} 1.2 & 0 \\ 1.6 & -2 \end{bmatrix}
\]
Row (image &rarr; text) losses: \(\log(e^{1.2} + e^{0}) - 1.2 \approx 0.2633\) and
\(\log(e^{1.6} + e^{-2}) + 2 \approx 3.6270\), mean 1.9451.<br>
Column (text &rarr; image) losses: \(\log(e^{1.2} + e^{1.6}) - 1.2 \approx 0.9130\) and
\(\log(e^{0} + e^{-2}) + 2 \approx 2.1269\), mean 1.5200.<br>
Output: <code>loss</code> = 0.5 &times; (1.9451 + 1.5200) = 1.7325
</p>

<h2>Constraints</h2>
<ul>
<li>1 &le; <code>N</code> &le; 32,768</li>
<li>1 &le; <code>D</code> &le; 1,024</li>
<li>0 &le; <code>logit_scale</code> &le; 100</li>
<li>Every row of <code>image_embeds</code> and <code>text_embeds</code> has a nonzero L2 norm</li>
<li>Rows may have arbitrary magnitudes; only their directions affect the loss</li>
<li>Performance is measured with <code>N</code> = 16,384, <code>D</code> = 512, <code>logit_scale</code> = 100.0</li>
</ul>
141 changes: 141 additions & 0 deletions challenges/hard/162_clip_contrastive_loss/challenge.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
import ctypes
from typing import Any, Dict, List

import torch
from core.challenge_base import ChallengeBase


class Challenge(ChallengeBase):
name = "CLIP Contrastive Loss"
# The loss is the average of two means of per-row / per-column log-sum-exp values
# (magnitude ~log N <= ~10). Logits are bounded by |logit_scale| <= 100 and come from
# dot products of unit vectors, so float32 accumulation-order differences stay ~1e-5
# in the final mean. Every targeted mistake below (missing L2 normalization, a single
# softmax direction, an off-diagonal target, no max subtraction) is off by >= 0.05.
atol = 1e-03
rtol = 1e-03
num_gpus = 1
access_tier = "free"

def reference_impl(
self,
image_embeds: torch.Tensor,
text_embeds: torch.Tensor,
loss: torch.Tensor,
N: int,
D: int,
logit_scale: float,
):
assert image_embeds.shape == (N, D)
assert text_embeds.shape == (N, D)
assert loss.shape == (1,)
assert image_embeds.dtype == text_embeds.dtype == loss.dtype == torch.float32

img = image_embeds / image_embeds.norm(dim=1, keepdim=True)
txt = text_embeds / text_embeds.norm(dim=1, keepdim=True)
logits = logit_scale * (img @ txt.T)
diag = torch.diagonal(logits)
loss_i2t = (torch.logsumexp(logits, dim=1) - diag).mean()
loss_t2i = (torch.logsumexp(logits, dim=0) - diag).mean()
loss.copy_((0.5 * (loss_i2t + loss_t2i)).reshape(1))

def get_solve_signature(self) -> Dict[str, tuple]:
return {
"image_embeds": (ctypes.POINTER(ctypes.c_float), "in"),
"text_embeds": (ctypes.POINTER(ctypes.c_float), "in"),
"loss": (ctypes.POINTER(ctypes.c_float), "out"),
"N": (ctypes.c_int, "in"),
"D": (ctypes.c_int, "in"),
"logit_scale": (ctypes.c_float, "in"),
}

def _pack(self, img, txt, logit_scale):
N, D = img.shape
return {
"image_embeds": img.contiguous(),
"text_embeds": txt.contiguous(),
"loss": torch.zeros(1, device=self.device, dtype=torch.float32),
"N": N,
"D": D,
"logit_scale": logit_scale,
}

def _make(self, N, D, logit_scale, seed, noise=1.0):
# Paired embeddings: text = image + noise. Matching pairs have the largest expected
# similarity (as in a trained CLIP model); `noise` is chosen per shape so the loss is
# O(1) nats and off-diagonal logits contribute meaningfully (not a degenerate ~0 loss).
torch.manual_seed(seed)
img = self._randn(N, D)
txt = img + noise * self._randn(N, D)
return self._pack(img, txt, logit_scale)

def _randn(self, *shape):
return torch.randn(*shape, device=self.device, dtype=torch.float32)

def generate_example_test(self) -> Dict[str, Any]:
img = torch.tensor([[1.0, 0.0], [0.0, 2.0]], device=self.device, dtype=torch.float32)
txt = torch.tensor([[3.0, 4.0], [0.0, -1.0]], device=self.device, dtype=torch.float32)
return self._pack(img, txt, 2.0)

def generate_functional_test(self) -> List[Dict[str, Any]]:
tests = [self.generate_example_test()]

# N = 1, D = 1: the only logit is the diagonal, so the loss must be exactly 0.
# Catches uninitialized accumulators and empty-reduction bugs.
tests.append(self._make(1, 1, 10.0, seed=1))

# logit_scale = 0: every logit is 0, so the loss must be exactly log(N).
# This is the "zero input" case (zero embeddings are excluded since rows are normalized).
tests.append(self._make(3, 4, 0.0, seed=2))

# Rows with wildly different magnitudes (x0.01 .. x100) and N != D. Catches a
# missing L2 normalization and normalizing along the wrong axis.
torch.manual_seed(3)
img = self._randn(30, 100) * torch.logspace(-2, 2, 30, device=self.device).unsqueeze(1)
txt = self._randn(30, 100) * torch.logspace(2, -2, 30, device=self.device).unsqueeze(1)
tests.append(self._pack(img, txt, 20.0))

# Asymmetric logits: all image rows are nearly the same direction while text rows
# are diverse, so the image->text and text->image losses differ by several nats.
# Catches computing only the row (or only the column) softmax, or assuming the
# logit matrix is symmetric.
torch.manual_seed(4)
base = self._randn(1, 64)
img = base + 0.05 * self._randn(48, 64)
txt = self._randn(48, 64) + 0.5 * base
tests.append(self._pack(img, txt, 30.0))

# Large logits: every embedding points close to one shared direction, so with
# logit_scale = 100 (CLIP's clamp value) all logits lie in ~[98, 100]. exp(>88.7)
# overflows float32, so this catches a missing max subtraction; the loss stays
# non-trivial because the logits are tightly packed.
torch.manual_seed(5)
base = self._randn(1, 64)
img = base + 0.1 * self._randn(64, 64)
txt = img + 0.05 * self._randn(64, 64)
tests.append(self._pack(img, txt, 100.0))

# All-negative image entries, tiny D and N = 255 (tail tiles in both dimensions).
neg = self._make(255, 3, 15.0, seed=6)
neg["image_embeds"].abs_().neg_()
tests.append(neg)

# Duplicate text rows (repeated captions): tied columns in the logit matrix.
torch.manual_seed(7)
img = self._randn(16, 32)
txt = self._randn(4, 32).repeat(4, 1)
tests.append(self._pack(img, txt, 10.0))

# Power-of-two shapes.
tests.append(self._make(512, 256, 50.0, seed=8, noise=5.0))

# Non-power-of-two shapes: N and D are not multiples of common tile sizes.
tests.append(self._make(1000, 100, 100.0, seed=9, noise=2.5))

# Realistic CLIP batch shard.
tests.append(self._make(4096, 512, 100.0, seed=10, noise=5.0))

return tests

def generate_performance_test(self) -> Dict[str, Any]:
return self._make(16384, 512, 100.0, seed=0, noise=5.0)
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
#include <cuda_runtime.h>

// image_embeds, text_embeds, loss are device pointers
extern "C" void solve(const float* image_embeds, const float* text_embeds, float* loss, int N,
int D, float logit_scale) {}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import cutlass
import cutlass.cute as cute


# image_embeds, text_embeds, loss are tensors on the GPU
@cute.jit
def solve(
image_embeds: cute.Tensor,
text_embeds: cute.Tensor,
loss: cute.Tensor,
N: cute.Int32,
D: cute.Int32,
logit_scale: cute.Float32,
):
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import jax
import jax.numpy as jnp


# image_embeds, text_embeds are tensors on device
@jax.jit
def solve(
image_embeds: jax.Array, text_embeds: jax.Array, N: int, D: int, logit_scale: float
) -> jax.Array:
# return output tensor directly
pass
17 changes: 17 additions & 0 deletions challenges/hard/162_clip_contrastive_loss/starter/starter.mojo
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
from max.gpu.host import DeviceContext
from max.gpu import block_dim, block_idx, thread_idx
from std.memory import UnsafePointer
from std.math import ceildiv


# image_embeds, text_embeds, loss are device pointers
@export
def solve(
image_embeds: UnsafePointer[Float32, MutAnyOrigin],
text_embeds: UnsafePointer[Float32, MutAnyOrigin],
loss: UnsafePointer[Float32, MutAnyOrigin],
N: Int32,
D: Int32,
logit_scale: Float32,
) raises:
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import torch


# image_embeds, text_embeds, loss are tensors on the GPU
def solve(
image_embeds: torch.Tensor,
text_embeds: torch.Tensor,
loss: torch.Tensor,
N: int,
D: int,
logit_scale: float,
):
pass
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
import torch
import triton
import triton.language as tl


# image_embeds, text_embeds, loss are tensors on the GPU
def solve(
image_embeds: torch.Tensor,
text_embeds: torch.Tensor,
loss: torch.Tensor,
N: int,
D: int,
logit_scale: float,
):
pass
Loading