Fix loo_compare() for subsampling results - #391
Conversation
…for subsampled loo objects
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #391 +/- ##
=======================================
Coverage 92.86% 92.86%
=======================================
Files 31 31
Lines 3041 3041
=======================================
Hits 2824 2824
Misses 217 217 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
This is how benchmark results would change (along with a 95% confidence interval in relative change) if 8635a88 is merged into master:
|
jgabry
left a comment
There was a problem hiding this comment.
Looks good, thanks. I made one testing suggestion and I noticed one other small bug that doesn't have to be fixed here, but is one line so maybe we can just include it (see review comment)?
|
|
||
| comp <- loo_compare_matrix.psis_loo_ss_list(x) | ||
| ord <- loo_compare_order(x) | ||
| names(x) <- rownames(comp)[ord] |
There was a problem hiding this comment.
Unrelated to this PR, but I think this line should be
names(x) <- rownames(comp)[order(ord)]
that is, inside the brackets it should be order(ord) instead of just ord.
I noticed this while reviewing this PR and I thought about opening a separate PR, but then I decided to just suggest including it here.
| for (m in list(lcss, lcss2, lcssohh)) { | ||
| expect_lt(m[2, "elpd_diff"], 0) | ||
| expect_lt(m[3, "elpd_diff"], 0) | ||
| } |
There was a problem hiding this comment.
Not strictly necessary, but maybe we should also add a test like this, since I don't think we have anything like it? It tests both the sign and the magnitude at the same time:
# subsampling should agree with non-subsampling when nothing
# is actually subsampled (pins the direction *and* the magnitude)
l1ss <- loo:::as.psis_loo_ss.psis_loo(l1)
l2ss <- loo:::as.psis_loo_ss.psis_loo(l2)
l3ss <- loo:::as.psis_loo_ss.psis_loo(l3)
expect_equal(
unname(loo_compare(l1ss, l2ss, l3ss)[, "elpd_diff"]),
loo_compare(l1, l2, l3)$elpd_diff
)
The first argument to expect_equal() could then be simplified with the data.frame conversion in the other PR.
Fixes #390
Description
loo_comparefor subsampling results provided a difference based on reference model - comparison model, whileloo_comparefor other loo objects returned the reverse difference leading to a flip in the sign ofelpd_diff.This PR fixes the sign flip, adds a test for the comparison direction, and updates the related outputs of the tutorial
vignettes/loo2-large-data.Rmd.Tests
test_loo_subsampling.R: [ FAIL 0 | WARN 0 | SKIP 0 | PASS 408 ]Documentation
Updated the three printed tables in
loo2-large-data.Rmd.TODO
devtools::check()NEWS.mdAI assistance
I used AI assistance while working on this PR. I have reviewed all changes.