question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

nan uncertainty estimates

See original GitHub issue

For certain types of overlapping distributions, the uncertainty estimate returned by BAR can benan

%matplotlib inline
import numpy as np
import pymbar
from matplotlib import pyplot as plt
for _ in range(10):
    fwd = np.random.normal(0, 10.0, size=50000)
    rev = np.random.normal(0, 175.0, size=50000)
    print(pymbar.BAR(fwd, rev))

Returns

(0.03461197183239051, nan)
(0.01789748600329588, nan)
(-0.041729050167845116, 0.006057615238226341)
(-0.0008426040270812507, nan)
/home/yutong/venv37/lib/python3.7/site-packages/numpy/core/_methods.py:48: RuntimeWarning: overflow encountered in reduce
  return umr_sum(a, axis, dtype, out, keepdims, initial, where)
(0.011851492025385824, nan)
(-0.011125827676117694, nan)
(-0.0011024556901020333, 0.006058115351639581)
(-0.14948909238990282, 0.006102804127491066)
(0.03077049043731961, 0.006088478465760845)
(-0.10509980343390524, nan)

The distributions look like: image

Edit: G0 and G1 below is wrong, confused the distributions, there is probably no issue with the bias!

Futhermore, since the analytical partition functions are Z(sig)=1/(sig*sqrt(2pi)), so using G=-ln(Z) the analytical free energies are G0=3.221 and G1=6.083, The estimator also appears to be biased.

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:10 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
mrshirtscommented, Feb 1, 2022

@jchodera - see the discussion above - the example was using U_1 and U_2, not U_1-U2 and U_2-U1, so the results were nonsensical because the data was not physical. Good question about what should be returned in those cases, though, and if they should be trapped (if possible? Might be hard to distinguish in many cases from numerically bad data).

1reaction
jchoderacommented, Feb 1, 2022

I’ve just suggested we also add a method to compute the number of effective samples: #427

I’m guessing these cases are seeing a collapse in the number of effective samples, meaning the estimates will be unreliable.

We should add some protection to make sure we don’t return nans, thought. And we may want to warn when there is a sample size collapse.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Day 6, Keynote Lecture 3. Uncertainty estimation in ... - YouTube
Slides: https://github.com/bayesgroup/deepbayes-2019/blob/master/lectures/day6/2.
Read more >
A Review of Uncertainty Quantification in Deep Learning - arXiv
This study reviews recent advances in UQ methods used in deep learning, investigates the application of these methods in reinforcement learning, ...
Read more >
Robust uncertainty estimates with out-of-distribution pseudo ...
This paper considers the idea of generating pseudo-inputs (i.e. extra augmented samples) to improve OOD uncertainty estimates by training with an ...
Read more >
A review of uncertainty quantification in deep learning
Uncertainty estimation and quantification have been studied extensively in DL and ... Nan and Ji [655], 2020, BSDS500 and three other standard benchmark ......
Read more >
Parameter estimation in uncertain differential equations based ...
The parameter estimation of uncertain differential equation is a significant subject in uncertainty theory. This paper introduces three ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found