Toys struggling with signal uncertainties for low-background region

Hey folks,

I’m using the toycalc branch to assess a limit on this model using toys and the signal, background, and uncertainty values shown:

nSig = 4.166929245
errSig = 4.166929245
nBkg = 0.11
errBkgUp = 0.20
errBkgDown = 0.11

model_json = {
    "channels": [
        { "name": "SR_combined",
          "samples": [
            { "name": "signal",
              "data": [nSig],
              "modifiers": [ { "name": "mu", "type": "normfactor", "data": None},
                              {"name": "uncorr_siguncrt", "type": "shapesys", "data": [errSig]} 
              ]
            },
            { "name": "background",
              "data": [nBkg],
              "modifiers": [ {"name": "uncorr_bkguncrt", "type": "normsys", "data": {"hi": errBkgUp, "lo": errBkgDown}} ]
            }
          ]
        }
    ],
    "observations": [
        { "name": "SR_combined", "data": [0.0] }
    ],
    "measurements": [
        { "name": "Measurement", "config": {"poi": "mu", "parameters": []} }
    ],
    "version": "1.0.0"
}

There are zero observed events, and we’re using an asymmetric background error to avoid further confusing the system with potential results outside reasonable bounds.

When I try to assess a CLs for this setup I get the following error:

"""
Traceback (most recent call last):
  File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.7.6-7a85e/x86_64-centos7-gcc8-opt/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/cvmfs/sft.cern.ch/lcg/releases/Python/3.7.6-7a85e/x86_64-centos7-gcc8-opt/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "my_mods.py", line 62, in getCLS
    qmu_sig, qmu_bkg = toy_calculator.distributions(mu_seed[0])
  File "/afs/cern.ch/work/k/kpachal/DisplacedLeptons/pyhf_tests/pythonenv/lib/python3.7/site-packages/pyhf/infer/calculators.py", line 445, in distributions
    qmu(poi_test, sample, self.pdf, bkg_pars, self.par_bounds)
  File "/afs/cern.ch/work/k/kpachal/DisplacedLeptons/pyhf_tests/pythonenv/lib/python3.7/site-packages/pyhf/infer/test_statistics.py", line 56, in qmu
    data, pdf, init_pars, par_bounds, return_fitted_val=True
  File "/afs/cern.ch/work/k/kpachal/DisplacedLeptons/pyhf_tests/pythonenv/lib/python3.7/site-packages/pyhf/infer/mle.py", line 61, in fit
    return opt.minimize(twice_nll, data, pdf, init_pars, par_bounds, **kwargs)
  File "/afs/cern.ch/work/k/kpachal/DisplacedLeptons/pyhf_tests/pythonenv/lib/python3.7/site-packages/pyhf/optimize/mixins.py", line 136, in minimize
    result = self._internal_minimize(**minimizer_kwargs, options=kwargs)
  File "/afs/cern.ch/work/k/kpachal/DisplacedLeptons/pyhf_tests/pythonenv/lib/python3.7/site-packages/pyhf/optimize/mixins.py", line 45, in _internal_minimize
    options=options,
  File "/afs/cern.ch/work/k/kpachal/DisplacedLeptons/pyhf_tests/pythonenv/lib/python3.7/site-packages/pyhf/optimize/opt_scipy.py", line 73, in _minimize
    options=dict(maxiter=maxiter, disp=bool(verbose)),
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/scipy/optimize/_minimize.py", line 626, in minimize
    constraints, callback=callback, **options)
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/scipy/optimize/slsqp.py", line 426, in _minimize_slsqp
    g = append(sf.grad(x), 0.0)
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 188, in grad
    self._update_grad()
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 171, in _update_grad
    self._update_grad_impl()
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/scipy/optimize/_differentiable_functions.py", line 92, in update_grad
    **finite_diff_options)
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages/scipy/optimize/_numdiff.py", line 391, in approx_derivative
    raise ValueError("`x0` violates bound constraints.")
ValueError: `x0` violates bound constraints.
"""

If I set the signal uncertainty to zero, however, it runs successfully.

Any idea what might be going on?

Thanks a lot for any and all help - I’m also happy to assist however I can, and can do any tests etc that might be useful.

[lukasheinrich]

hi @kpachal:

thanks for raising this. There is a small bug in the way we run the fit. I’ll fix this right now. I tested your example on the fix and it runs through. I’ll post here again.

EDIT:

sorry wrong alarm, it’s not a bug on our side, but rather in scipy. @kratsg will give more details

[kpachal]

Thanks very much to all of you! I’ve subscribed to the list as you suggest and I’ll keep an eye out for branch deletion notices. I also tested the fix suggested above (switching to Minuit) and it works 😁

This is a great tool, many thanks to you all for developing it!