Skip to content

Fix loo_compare() for subsampling results - #391

Open
florence-bockting wants to merge 5 commits into
masterfrom
fix-subsampling
Open

Fix loo_compare() for subsampling results#391
florence-bockting wants to merge 5 commits into
masterfrom
fix-subsampling

Conversation

@florence-bockting

@florence-bockting florence-bockting commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fixes #390

Description

loo_compare for subsampling results provided a difference based on reference model - comparison model, while loo_compare for other loo objects returned the reverse difference leading to a flip in the sign of elpd_diff.

This PR fixes the sign flip, adds a test for the comparison direction, and updates the related outputs of the tutorial vignettes/loo2-large-data.Rmd.

Tests

test_loo_subsampling.R: [ FAIL 0 | WARN 0 | SKIP 0 | PASS 408 ]

Documentation

Updated the three printed tables in loo2-large-data.Rmd.

TODO

  • devtools::check()
  • Fill the PR number into NEWS.md

AI assistance

I used AI assistance while working on this PR. I have reviewed all changes.

@florence-bockting florence-bockting changed the title fix: correct elpd_diff sign and return a data.frame in loo_compare() … Fix loo_compare() for subsampling results Sep 2, 2026
@codecov-commenter

codecov-commenter commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.86%. Comparing base (a59519c) to head (8635a88).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #391   +/-   ##
=======================================
  Coverage   92.86%   92.86%           
=======================================
  Files          31       31           
  Lines        3041     3041           
=======================================
  Hits         2824     2824           
  Misses        217      217           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

This is how benchmark results would change (along with a 95% confidence interval in relative change) if 8635a88 is merged into master:

  • ✔️loo_function: 1.51s -> 1.5s [-1.61%, +0.23%]
  • ✔️loo_matrix: 1.5s -> 1.49s [-1.79%, +0.23%]
    Further explanation regarding interpretation and methodology can be found in the documentation.

@florence-bockting
florence-bockting marked this pull request as ready for review September 3, 2026 06:48

@jgabry jgabry left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks. I made one testing suggestion and I noticed one other small bug that doesn't have to be fixed here, but is one line so maybe we can just include it (see review comment)?


comp <- loo_compare_matrix.psis_loo_ss_list(x)
ord <- loo_compare_order(x)
names(x) <- rownames(comp)[ord]

@jgabry jgabry Sep 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to this PR, but I think this line should be

names(x) <- rownames(comp)[order(ord)] 

that is, inside the brackets it should be order(ord) instead of just ord.

I noticed this while reviewing this PR and I thought about opening a separate PR, but then I decided to just suggest including it here.

for (m in list(lcss, lcss2, lcssohh)) {
expect_lt(m[2, "elpd_diff"], 0)
expect_lt(m[3, "elpd_diff"], 0)
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not strictly necessary, but maybe we should also add a test like this, since I don't think we have anything like it? It tests both the sign and the magnitude at the same time:

# subsampling should agree with non-subsampling when nothing
# is actually subsampled (pins the direction *and* the magnitude)
l1ss <- loo:::as.psis_loo_ss.psis_loo(l1)
l2ss <- loo:::as.psis_loo_ss.psis_loo(l2)
l3ss <- loo:::as.psis_loo_ss.psis_loo(l3)
expect_equal(
  unname(loo_compare(l1ss, l2ss, l3ss)[, "elpd_diff"]),
  loo_compare(l1, l2, l3)$elpd_diff
)

The first argument to expect_equal() could then be simplified with the data.frame conversion in the other PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

loo_compare() returns elpd_diff with the opposite sign for psis_loo_ss objects

3 participants