[mnapoli]

Summary

The Growth page only had cumulative retention: "came back on day 1 or later". It keeps rising, can't be read per signup cohort, and can't be compared with benchmarks, which use day-N retention (active on day N). With user_active_days (#1324) recording each day a person is seen, this adds two panels next to the funnel's rate:

• Retention by signup day: the share of each day's sign-ups active exactly 1 day later, or 7 with the switch, with the range's average. Built like "Activation by signup day".
• Retention by cohort: weekly cohorts × days 1, 3, 7, 14, 30, darker when higher, and the curve of all cohorts pooled from day 0 to day 30.

Days before 2026-10-30 were backfilled from what people did, which misses the days someone only read (the backfill found 72–82% of each day's active people), so they're marked "approximate": striped bars, "≥" cells, hollow points on the curve.

Changes

• RetentionByDayQuery and RetentionCohortsQuery: a signup counts on day N only once day N is over (signup date + N < today, UTC). Cohort cells only count the members who reached that day. Bots are left out. RetentionByDayQuery::RECORDED_SINCE is the first recorded day.
• The cohort query cross-joins the range's sign-ups with days 1–30 and looks up user_active_days by its primary key: about 150 ms for 12k sign-ups, measured by the reviewer.
• Heat colors stop where regular text on them would drop under 4.5:1 contrast (88% of the primary in light mode, 56% in dark), so cells always use the regular text color.
• PRODUCT_METRICS.md documents day-N retention, what "approximate" means, and that it's now comparable with the a16z benchmarks.

Validation

All local, on devnull-pg:

• composer test:lint and bun run test:lint: pass
• PHPStan: pass
• composer test:type-coverage: 100%
• pest --parallel --coverage --exactly=100.0: 5,145 passed, 100% coverage
• bun run test:unit: 212 passed

Visual evidence

Same fake data on both sides: the repo's OperationsSeeder (843 sign-ups over 60 days) plus synthetic active days. Every day is before Oct 30, so almost everything shows as approximate, as production will for the next days. The region goes from "Acquisition by day" to "Conversion by device".

	Before (production)	After
Light, 1440
Dark, 1440
Light, 390

On 390px with 4- and 5-digit cohorts (two counts set by hand in the page), no horizontal overflow:

Device verification

Chromium viewport emulation at 390px, not a device. The page is an admin dashboard with no text input. Checked: the table and curve fit with no horizontal scroll, the day 1 / day 7 switch, cell and point tooltips (left and right edges, the first row flipping below), fresh loads and range changes.

Independent review

Independent agent review of base fbc9ac8a, head 774da4bc: CODE PASS, VISUAL PASS, on the screenshots above. The first pass raised three minor findings, all fixed in 774da4bc and re-reviewed: the note shifting the chart when switching days, a screen-reader label for cohort sizes, and large cohorts on phones.

Limitations

• The first cohort row is labelled with its Monday even when the range starts later in that week: it then holds only the in-range part.
• Weekly cohorts of a few hundred people are still noisy. Read a column over several weeks.

Summary by CodeRabbit

• New Features
  • Added day 1 and day 7 retention reporting by signup date, alongside weekly cohort tables for days 1, 3, 7, 14, and 30 and an aggregate retention curve.
  • Marked estimates that include backfilled activity data with visual indicators and explanatory notes.
  • Added documentation describing retention metrics and their comparability with external benchmarks.
• Bug Fixes
  • Clarified that funnel retention is not comparable to bounded retention benchmarks.

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: a4900b96-d62c-4132-852f-d70f8e50fc26

Commits

Reviewing files that changed from the base of the PR and between fbc9ac8 and 774da4b.

Files selected for processing: 10

• PRODUCT_METRICS.md
• app/Http/Controllers/ProductMetricsController.php
• app/Queries/RetentionByDayQuery.php
• app/Queries/RetentionCohortsQuery.php
• resources/js/layouts/operations/layout.tsx
• resources/js/pages/operations/metrics.tsx
• resources/js/types/operations.ts
• tests/Feature/Controllers/ProductMetricsControllerTest.php
• tests/Unit/Queries/RetentionByDayQueryTest.php
• tests/Unit/Queries/RetentionCohortsQueryTest.php

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 7 remain after this review.

Walkthrough

The metrics page now reports bounded retention from user_active_days through daily day-1 and day-7 charts, weekly cohorts, and a pooled curve through day 30. Queries exclude configured bots and mark pre-2026-10-30 activity data as backfilled. The documentation distinguishes bounded Day-N retention from the existing last_seen_at rate.

Changes: Bounded retention reporting

Layer / File(s)	Summary
Retention queries and controller data app/Queries/RetentionByDayQuery.php, app/Queries/RetentionCohortsQuery.php, app/Http/Controllers/ProductMetricsController.php, tests/Unit/Queries/*Retention*Test.php, tests/Feature/Controllers/ProductMetricsControllerTest.php, PRODUCT_METRICS.md	The queries calculate daily and weekly cohort retention, eligibility, and backfilled counts. The controller provides the results and recording start date to the page. Tests cover query and page data. The documentation describes the bounded metrics and their backfill limits.
Daily retention view resources/js/types/operations.ts, resources/js/layouts/operations/layout.tsx, resources/js/pages/operations/metrics.tsx	The page adds day-1 and day-7 selection, signup-day retention charts, and visual indicators and tooltips for backfilled values.
Cohort table and retention curve resources/js/types/operations.ts, resources/js/pages/operations/metrics.tsx	The page adds weekly cohort cells for days 1, 3, 7, 14, and 30, plus a pooled retention curve through day 30. Backfilled rates are marked in the table and curve.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant ProductMetricsController
  participant RetentionByDayQuery
  participant RetentionCohortsQuery
  participant OperationsMetricsPage
  ProductMetricsController->>RetentionByDayQuery: get day 1 and day 7 retention
  RetentionByDayQuery-->>ProductMetricsController: daily retention results
  ProductMetricsController->>RetentionCohortsQuery: get cohorts for requested date range
  RetentionCohortsQuery-->>ProductMetricsController: cohort and curve results
  ProductMetricsController->>OperationsMetricsPage: provide retention props

Loading

Possibly related PRs

• dev-null-sh/dev-null.sh#1324: Adds and records user_active_days, which the new bounded retention queries read.

Suggested reviewers: xheaven, nunomaduro

Merge Risk: Minimal · up to 774da

The retention panels have no identified issue that needs correction before merge.

Security architecture risk: Low · up to 774da

The new reporting remains behind existing administrator authorization and uses read-only, parameterized queries. No introduced security concern was identified in the reviewed retention flow. The assessment does not establish deployment-wide security or production capacity.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• observed — The added responses expose aggregate activity across non-bot signups in an administrator-selected range, including the existing all-time option. Their scope is application-wide aggregate reporting, not a caller-selected individual activity history.

Trust Boundaries and Controls

• observed — The new data remains behind the existing administrator boundary. Date ranges, day offsets, and bot exclusions are passed as SQL bindings rather than interpolated into executable SQL, preserving the input-to-database boundary in the inspected queries.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: adding day-N retention views by signup day and weekly cohort to the Growth page.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 71.43% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 7 functions across 9 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}