question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Splitting data for a model into main and aux data

See original GitHub issue

Question

Is there a recommended way to split data (main + aux) into the “main” part (yields per bin) and auxiliary?

Given a model: pyhf.pdf.Model and data: List[float] (e.g. from workspace.data(model)), the following works as long as there is at least a single aux parameter (I do not remember from where I picked this up, but I believe it was a suggestion):

main_data, aux_data = model.fullpdf_tv.split(pyhf.tensorlib.astensor(data))

With no aux parameters, this fails since a list of length 1 is returned. I could instead split like this

main_data = data[:model.config.nmaindata]
aux_data = data[model.config.nmaindata:]

which would return [] when model.config.nauxdata is zero.

Do you see any possible issue with this second approach? Looking at https://github.com/scikit-hep/pyhf/blob/f380fddf331c89030384e7949c76099e9277693a/src/pyhf/probability.py#L281-L283 I doubt that changing the .split() behavior would make sense for the API.

Relevant Issues and Pull Requests

none I am aware of

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:6 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
lukasheinrichcommented, Mar 23, 2021

ok, i’ll close this then for now, but feel free to reopen if it becomes annoying (I do think this will be rare in real-life scenarios)

0reactions
alexander-heldcommented, Mar 23, 2021

This is fine, happy to use an approach like this. If this kind of split is commonly needed it might be useful to have something in the API for it, but I suspect it is a rare thing to use. model.fullpdf_tv.split(data) indeed always works, just need to handle the length-1 case accordingly.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Data splitting technique to fit any Machine Learning Model
The main aim of deciding the splitting ratio is that all three sets should have the general trend of our original dataset.
Read more >
What is data splitting and why is it important? - TechTarget
Data splitting is when data is divided into two or more subsets. Typically, with a two-part split, one part is used to evaluate...
Read more >
How to Split The Data - Model Building and Validation
This video is part of an online course, Model Building and Validation. Check out the course here: https://www.udacity.com/course/ud919.
Read more >
Splitting your data to fit any machine learning model
Today I want to share with you a few very simple lines of code that will divide any data set into variables that...
Read more >
Splitting your data: growing beyond train_test_split - Medium
Typically, you divide your data into three categories: ~80 % training, 10 % validation & 10 % testing. Then, you fit your model...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found