Bug with extended CrystalBall + other PDF, with `result.hesse()`

Hello,

Bugs

Current Behaviour

I’ve built a PDF in the following fashion

1. Create a CrystalBall PDF (so-called signal)
2. Extend it (create_extended)
3. Create another PDF (so-called background), extend it
4. Sum up the extended signal and background PDFs

Then, I’ve performed a fit using the Minuit minimizer and the Unbinned nll (nll).

EDIT: the sample that is fitted is weighted. If it isn’t weighted, there is no problem whatsoever!

From the minimisation, I’ve got the variable result = minimizer.minimize(nll). In order to get the errors with Hesse, I’ve run : result.hesse()

and by doing so, I’ve got errors, that I’ve uploaded in this message: assertionerror.txt

Reproduce the error in a notebook

I’ve tried to reproduce the errors in a colab notebook. By doing so, I noticed that:

• the errors only show up with the Crystal Ball
• the errors only show up when we add up two extended PDFs, one of the PDFs being a Crystal Ball

The minimalist example with a notebook is available here: https://colab.research.google.com/drive/18UWMr7keOBDt4PjW1uk6r88X62oMvlqT?usp=sharing

EDIT: I’ve used weights of 1 in this notebook

Context (Environment)

• zfit version: 0.7.0 but I’ve also tried previous versions, 0.6.6 and 0.5.6 (and their associated Tensorflow versions)
• Python version: 3.7
• Operating System: Linux
• Tensorflow version: for zfit=0.7.0, tensorflow=2.5.0

Have a nice day, Anthony

[jonas-eschle]

Good to know, another solution is to use:

with zfit.run.set_autograd_mode(False):
    param_hesse = result.hesse()

This uses the graph mode but does the derivatives numerically. The difference between using weights and not is that with weights, an additional correction factor comes into play. And for that, the gradients are calculated differently (Forward Mode instead of Backward Mode). Seemingly, this fails in compiled mode…

Thanks again for the report!

[jonas-eschle]

Hi Anthony, many thanks for this clean report and the reproducible example!

This seems to be something nasty, if TensorFlow throws an assertion error, something is somewhere deeply wrong, clearly! I’ll investigate that further.

Meanwhile, what works is to run the relevant piece of code in eager (=Python-like) mode. This can be done with

with zfit.run.set_graph_mode(False):
    param_hesse = result.hesse()

The fact that this runs clearly shows that it is something about the graph in TF and not about derivatives or similar.

Does this work? Btw, if it works, let’s keep it open to track the upstream TF bug