Skip to content

Return a data.frame from loo_compare() for subsampled loo objects - #393

Draft
florence-bockting wants to merge 2 commits into
fix-subsamplingfrom
fix-subsampling-df
Draft

Return a data.frame from loo_compare() for subsampled loo objects#393
florence-bockting wants to merge 2 commits into
fix-subsamplingfrom
fix-subsampling-df

Conversation

@florence-bockting

@florence-bockting florence-bockting commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Closes #392

What

loo_compare() returns a matrix for subsampled loo objects, but a data.frame
for all other loo objects. This PR aligns the two.

loo_compare.psis_loo_ss_list() now builds the same structure as
loo_compare() in R/loo_compare.R.

Output

Before:

       elpd_diff se_diff subsampling_se_diff
model2  0.0       0.0     0.0
model1 -16.5     22.5     0.4

After:

  model elpd_diff se_diff subsampling_se_diff
 model2       0.0     0.0                 0.0
 model1     -16.5    22.5                 0.4

Breaking change

Code that indexes the result by position breaks. comp[, 1] now returns the
model names. Use comp$elpd_diff. Code that reads rownames(comp) also
breaks. Use comp$model.

Checks

  • devtools::test(filter = "loo_subsampling") passes.
  • Snapshots in tests/testthat/_snaps/loo_subsampling_cases.md updated.
  • vignettes/loo2-large-data.Rmd output blocks updated.
  • reverse dependency checks

AI assistance

I used AI assistance while working on this PR. I have reviewed all changes.

@florence-bockting
florence-bockting changed the base branch from master to fix-subsampling September 3, 2026 06:39
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.87%. Comparing base (8635a88) to head (8a8f767).

Additional details and impacted files
@@                 Coverage Diff                 @@
##           fix-subsampling     #393      +/-   ##
===================================================
+ Coverage            92.86%   92.87%   +0.01%     
===================================================
  Files                   31       31              
  Lines                 3041     3046       +5     
===================================================
+ Hits                  2824     2829       +5     
  Misses                 217      217              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 8a8f767 is merged into master:

  • ✔️loo_function: 1.94s -> 1.94s [-0.77%, +0.47%]
  • ✔️loo_matrix: 1.76s -> 1.75s [-1.53%, +0.87%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update output structure of loo_compare for subsampling case

2 participants