Iteration limit reached when using histosys modifier.

Summary

Hello everyone,

First of all let me thank you for the wonderful package. As a PhD student playing with LHC data for the first time, it is really helpful.

I would like to take into account a systematic uncertainty in my signal, to do that I tried to add the histosys modifier to my signal in addition to the normfactor for signal strength and the uncorr_bkguncrt for my background.

For some reason this caused pyhf.infer.hypotest to be stuck in a loop and raise an iteration limit exception (it also prints a lot of “nan” in the error report for the function to minimize, its jacobian and minimum).

I tried replacing histosys by shapesys and it seems to work normally.

Am I doing something wrong ? Is this a bug? Can you suggest a workaround for what I am trying to do?

Many thanks

OS / Environment

Ubuntu 20.04

Steps to Reproduce

mod = {
        "channels": [
            { "name": "singlechannel",
              "samples": [
                { "name": "signal",
                  "data": list(signal),
                  "modifiers": [{ "name": "mu", "type": "normfactor", "data": None},
                                { "name": "mod_name", "type": "histosys", "data": {"hi_data": list(hi), "lo_data": list(lo)} }]

                },
                { "name": "background",
                  "data": list(backgr),
                  "modifiers": [ {"name": "uncorr_bkguncrt", "type": "shapesys", "data": list(b_err)} ]
                }
              ]
            }
        ],
        "observations": [
            { "name": "singlechannel", "data": list(obs) }
        ],
        "measurements": [
            { "name": "Measurement", "config": {"poi": "mu", "parameters": [{"bounds": [[0, 1e+6]],"inits": [1.0], "name": "mu"}]} }
        ],
        "version": "1.0.0"
        }

File Upload (optional)

No response

Expected Results

A p-value.

Actual Results

fun: nan
     jac: array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan])
 message: 'Iteration limit reached'
    nfev: 2999989
     nit: 100000
    njev: 100000
  status: 9
 success: False
       x: array([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan])
Traceback (most recent call last):
   File "/mnt/c/pyhf/optimize/mixins.py", line 49
      assert result.success
AssertionError

pyhf Version

0.62

Code of Conduct

• I agree to follow the Code of Conduct

[FlorentinJ]

Thank you so much! That wes exactly it. A simple pip install scipy --upgrade later (from version 1.5.4), and it now works fine.

[matthewfeickert]

Thank you so much! That wes exactly it. A simple pip install scipy --upgrade later (from version 1.5.4), and it now works fine.

Glad to hear it! 🙂 The reason I suspected scipy was because the error was coming from the minimizer failing to converge, and there isn’t much in terms of pyhf code there.

Though maybe it was actually a bug in an older version of NumPy (SciPy imposes both lower and upper limits on NumPy) or an interaction between NumPy and SciPy, as if I use scipy==1.5.4 I can still get it to pass:

$ docker run --rm -ti python:3.9 /bin/bash
root@177469c2b9f1:/# python -m venv venv && . venv/bin/activate
(venv) root@177469c2b9f1:/# python -m pip install --upgrade pip setuptools wheel
(venv) root@177469c2b9f1:/# python -m pip install 'scipy==1.5.4' 'pyhf==0.6.3'
(venv) root@177469c2b9f1:/# python -m pip list
Package     Version
----------- -------
attrs       21.2.0
click       8.0.3
jsonpatch   1.32
jsonpointer 2.2
jsonschema  4.2.1
numpy       1.21.4
pip         21.3.1
pyhf        0.6.3
pyrsistent  0.18.0
PyYAML      6.0
scipy       1.5.4
setuptools  59.5.0
tqdm        4.62.3
wheel       0.37.0
(venv) root@177469c2b9f1:/# tail -n 5 issue_1722.py  # copying the example in Issue #1722

# this line takes a long time then fail
cls_obs, cls_expected = pyhf.infer.hypotest(1, wsdata, ws.model(), calctype='asymptotics', return_expected=True, test_stat='qtilde')
print(f"CL_s observed: {cls_obs}")
print(f"CL_s expected: {cls_expected}")
(venv) root@177469c2b9f1:/# python issue_1722.py 
/venv/lib/python3.9/site-packages/pyhf/tensor/numpy_backend.py:352: RuntimeWarning: invalid value encountered in log
  return n * np.log(lam) - lam - gammaln(n + 1.0)
CL_s observed: 0.26319911317665245
CL_s expected: 0.2012008685590555