Dependence aware tests for `ppc*ecdf` by florence-bockting · Pull Request #428 · stan-dev/bayesplot

florence-bockting · 2026-03-04T08:43:59Z

Description

The current approach in ppc_loo_pit_ecdf and ppc_pit_ecdf assumes independence of LOO-PIT values which is not valid (Marhunenda et al., 2005). The corresponding graphical test yields an envelope that is too wide, reducing the test's ability to reveal model miscalibration.
Tesso & Vehtari (2026, see preprint) propose three testing procedures that can handle any dependent uniform values and provide an updated graphical representation that uses color coding to indicate influential regions or most influential points of the ECDF. This PR implements the new development, by adding the updated approach (method = "correlated") additionally to the previous approach (method = "independent").

TODOs

updated ppc_loo_pit_ecdf() function in ppc-loo.R
updated ppc_pit_ecdf() and ppc_pit_ecdf_grouped() function in ppc-distributions.R
add unittests and visual regression tests
update documentation
deprecation suggestion of old method
- P1: old method (default) and new method available via method argument
- P2: new method (default) and old method available via method argument
- P3: old method is removed

…it_ecdf

…s-ppc.R

…improve input validation checks

…dation

jgabry

Here are a few comments and questions. So far I've only looked at some of the code, so I may have more, but I thought I would give you these now so we could start discussing.

…_pit'

… corresponding unittests

…e-bockting/bayesplot into dependence-aware-LOO-PIT

jgabry · 2026-04-13T18:25:06Z

@florence-bockting The changes look good, but I think there are the same issues (with the gamma doc and the pareto_pit issue) in the ppc-distributions.R file too, right? Also, how much of the doc in the two R files now overlaps? Could we write some of it once and share it? (If it's only a small amount of overlap then maybe not worth it).

…uments into single docs file

…f plots

florence-bockting · 2026-04-15T06:31:03Z

@jgabry
Thanks for the comments. I was planning to do the refactoring for reducing duplicate code between the ppc-ecdf functions at a later stage as this was already in the original version the case. But given your comment I realized that I should probably do it just now. Thus:

I have refactored the code such that it removes duplicate code and documentation by creating new helper functions or template files.
Furthermore, I increased the helper-text size but also added now the factor for controlling size as an additional argument. When we improve the theme, we can probably get rid of it again. @avehtari

…gests

jgabry · 2026-04-16T19:54:24Z

@florence-bockting please also officially add yourself to the DESCRIPTION file as part of this PR!

florence-bockting · 2026-05-08T10:49:02Z

@jgabry just as a little reminder that this PR is still waiting for final review. Thank you

jgabry

Here are some review comments. Sorry for the delay! Overall I think this is in good shape, just a few relatively minor comments.

jgabry · 2026-06-02T13:52:11Z

+) {
+  # pareto-pit values
+  if (isTRUE(pareto_pit) && is.null(pit)) {
+    suggested_package("rstantools")


I don't think rstantools is actually needed for this branch here, right?

jgabry · 2026-06-02T13:56:23Z

+    }
+  }
+
+  alpha <- 1 - prob


Should we validate that prob is a valid probability? Either here or before it's passed in? I guess whatever is consistent with how we've been checking other arguments.

jgabry · 2026-06-02T14:04:37Z

+#' @param color When `method = "correlated"`, a vector with base color and
+#'   highlight color for the ECDF plot. Defaults to
+#'   `c(ecdf = "grey60", highlight = "red")`. The first element is used for
+#'   the main ECDF line, the second for highlighted suspicious regions.


The code that processes the color vector requires these exact names to be used in order to work properly (I think) but it doesn't check for the names. It indexes color["ecdf"] and color["highlight"]. The doc also mentions "first element" and "second element", but it's the names, not the position, that are used in the code. The user can pass incorrect names or unnamed vectors and I think the colors won't render properly. We should probably either check for the names or just use the position (first element and second element) and forget about the names.

jgabry · 2026-06-02T14:13:40Z

+#'   For `ppc_loo_pit_ecdf()` when `method = 'independent'`.
+#'   The default is to use interpolation if `K` is greater than 200.
+#' @param pit An optional vector of probability integral transformed values for
+#'   which the ECDF is to be drawn. For `ppc_loo_pit_ecdf()`. If `NULL`, PIT


Should we also mention the other ppc_loo_* functions that take the pit argument in addition to ppc_loo_pit_ecdf()?

jgabry · 2026-06-02T14:21:16Z

+#' @note
+#' Note that the default "independent" method is **superseded** by
+#' the "correlated" method (Tesso & Vehtari, 2026) which accounts for dependent
+#' PIT values.


Because this is documented on the same page as the other function that the same @note it shows up twice in the .Rd file. However, it also doesn't show up at all (at least for me) when rendering the doc in RStudio. Maybe the Note field isn't rendered? Do you see that too?

Another option would be to remove both uses of @note and instead use the Plot Descriptions section. And I guess same thing for the loo versions, if applicable.

florence-bockting added 30 commits February 20, 2026 22:39

add helper functions for correlation-aware uniformity test

3e0e568

incorporate in ppc_loo_pit_ecdf correlation-aware uniformity test

c02215c

add tests for updated ppc_loo_pit_ecdf function

0d58176

formatting and efficiency improvements

ee2a3fc

improve description and computation of uniformity tests for pcc_loo_p…

583b3e2

…it_ecdf

add unit-tests for dependence-aware uniformity tests

317a753

do not remove NA when using sort() in pot_test

3afa4ac

move unittest for uniformity tests from test-ppc-loo.R to test-helper…

c8e63fd

…s-ppc.R

improve descripting and naming of function to compute shapley values

609f48f

improve descripting and naming of function to compute shapley values

bbfedc5

add unittest for compute_shapley_values

18f6539

improve computation of Cauchy combination test and std. Cauchy values

2ca91a3

make variable namings more descriptive

d6a18ce

add unittests for Cauchy combination test

8b55c2f

improve efficiency of influential_points_idx

da23cdf

add unittests for influential_points_idx

32a4087

update unittests for ppc_loo_pit_ecdf

7ca5b9b

remove influential_points_idx function

a66358f

remove unittests for influential_points_idx

d62cd06

remove infl_points_only argument, add linewith, color arguments, and …

753053f

…improve input validation checks

minor fixes in plotting behavior (fontsize, linestyle) and input vali…

1ae2595

…dation

adjust tests to new ppc_loo_pit_ecdf implementation

270f67e

add compute_cauchy function

7489701

transition from -qcauchy(x) to compute_cauchy(x)

22048bd

add unittests for compute_cauchy

8180cb7

minor adjustments for regression-tests

703b9f0

change svg snapshots for regression tests

7f6fb12

improve documentation for arguments linewidth, color in ppc_loo_pit_ecdf

11086f9

update PPC-loo.Rd file with new arguments

5321ac0

add tests for plot_diff=TRUE and method=correlated

478e96e

Merge branch 'master' into dependence-aware-LOO-PIT

3b73f35

florence-bockting marked this pull request as ready for review April 7, 2026 05:46

florence-bockting requested review from avehtari and jgabry April 7, 2026 05:47

florence-bockting mentioned this pull request Apr 7, 2026

Vignette for use of ppc*pit_ecdf() with new method argument #527

Draft

1 task

jgabry reviewed Apr 8, 2026

View reviewed changes

Comment thread R/ppc-loo.R Outdated

Comment thread R/ppc-loo.R Outdated

Comment thread R/ppc-loo.R Outdated

Florence Bockting and others added 5 commits April 10, 2026 12:06

docs: fix description of 'gamma' and add special condition to 'pareto…

e91bde2

…_pit'

fix: update condition for 'pareto_pit', improve input checks, and add…

0afafbe

… corresponding unittests

Merge branch 'dependence-aware-LOO-PIT' of https://github.com/florenc…

ed7d6fa

…e-bockting/bayesplot into dependence-aware-LOO-PIT

Merge branch 'master' into dependence-aware-LOO-PIT

2cd5577

fix: merge description

2eac7c2

avehtari reviewed Apr 13, 2026

View reviewed changes

Comment thread tests/testthat/_snaps/ppc-distributions/ppc-pit-ecdf-color-change.svg

Florence Bockting added 3 commits April 15, 2026 09:21

refactor: create helpers to remove duplicate code between ecdf-functions

8c2a6bd

docs,refactor: update function documentation and outsource common arg…

ee34125

…uments into single docs file

tests: update snapshots wrt to increased size of help-text in ppc-ecd…

ebd74a7

…f plots

Florence Bockting and others added 6 commits April 15, 2026 10:04

fix: fix duplicate linewidth, prob arguments

41c142a

chore: label size compatible with var. ggplot2 versions

76b2e27

Merge branch 'master' into dependence-aware-LOO-PIT

b953e94

chore: check for S4 class using base R

8693d43

chore: remove tests that require brms and also brms dependency in Sug…

f0ce52e

…gests

fix: handle 0-1 PIT values for POT and PIET

a5e8236

florence-bockting and others added 4 commits April 21, 2026 12:47

Merge branch 'master' into dependence-aware-LOO-PIT

30ee389

fix: ensure warning/error messages are below 80 characters

a164a38

chore: add florence-bockting to Description file

e6c6220

merge: update from master

3b5b513

jgabry reviewed Jun 2, 2026

View reviewed changes

Uh oh!

Conversation

florence-bockting commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

TODOs

Uh oh!

jgabry left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgabry commented Apr 13, 2026

Uh oh!

Uh oh!

florence-bockting commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jgabry commented Apr 16, 2026

Uh oh!

florence-bockting commented May 8, 2026

Uh oh!

jgabry left a comment

Choose a reason for hiding this comment

Uh oh!

jgabry Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

jgabry Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

jgabry Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

jgabry Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

jgabry Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

florence-bockting commented Mar 4, 2026 •

edited

Loading

florence-bockting commented Apr 15, 2026 •

edited

Loading