Additional documentation needed on difference between calculator type test statistic return type

Summary

There is a large discrepancy in the value of the discovery test statistic if it is generated via an Asymptotic based or toy based calculator.

Related Issues:

• https://github.com/scikit-hep/pyhf/issues/1587
• https://github.com/scikit-hep/pyhf/issues/1712

OS / Environment

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Steps to Reproduce

# issue_1792.py
import pyhf


def discovery_from_test_stat(model, data):
    init_pars = model.config.suggested_init()
    par_bounds = model.config.suggested_bounds()
    fixed_params = model.config.suggested_fixed()
    return pyhf.infer.test_statistics.q0(
        0.0,
        data,
        model,
        init_pars,
        par_bounds,
        fixed_params,
    )


def discovery_from_calculator(model, data, calc_type="asymptotics", **kwargs):
    calculator = pyhf.infer.utils.create_calculator(
        calc_type, data, model, test_stat="q0", **kwargs
    )
    return calculator.teststatistic(0.0)


def discovery_from_asymptotic_calculator(model, data):
    calculator = pyhf.infer.calculators.AsymptoticCalculator(
        data, model, test_stat="q0"
    )
    return calculator.teststatistic(0.0)


if __name__ == "__main__":
    model = pyhf.simplemodels.uncorrelated_background([25], [2500], [2.5])
    data = [2600] + model.config.auxdata

    discovery_test_stat = discovery_from_test_stat(model, data)
    print(f"Discovery test stat from infer.test_statistics.q0: {discovery_test_stat}")

    discovery_test_stat = discovery_from_calculator(model, data, calc_type="toybased")
    print(
        f"Discovery test stat from toy based infer.utils.create_calculator: {discovery_test_stat}"
    )

    discovery_test_stat = discovery_from_calculator(model, data)
    print(
        f"Discovery test stat from asymptotics infer.utils.create_calculator: {discovery_test_stat}"
    )

    discovery_test_stat = discovery_from_asymptotic_calculator(model, data)
    print(
        f"Discovery test stat from infer.calculators.AsymptoticCalculator: {discovery_test_stat}"
    )

File Upload (optional)

No response

Expected Results

The discovery test statistic would be the same regardless of calculator type.

Actual Results

$ python issue_1792.py 
Discovery test stat from infer.test_statistics.q0: 3.9377335959507036
Discovery test stat from toy based infer.utils.create_calculator: 3.9377335959507036
Discovery test stat from asymptotics infer.utils.create_calculator: 1.485845489706193
Discovery test stat from infer.calculators.AsymptoticCalculator: 1.485845489706193

pyhf Version

9fd99be886349a90e927672e950cc233fad0916c on master

Code of Conduct

• I agree to follow the Code of Conduct

[kratsg]

My main complaint at the moment is that while we make it clear that the asymptotic test stats are in -^mu/sigma space

this part I think is what confuses me, so if you have a better grasp, it would be great to clarify this for the upcoming release.

[matthewfeickert]

At the moment this doesn’t run any toys as far as I can tell, so I’m assuming this is a question about the calculators, and not about asymptotic vs toy agreement in general?

Yes, I didn’t phrase the original text clearly as I was dumping things in for myself to clarify later.