question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Performance of compute residuals

See original GitHub issue

@ybagdasa wrote

@CamDavidsonPilon I’m trying to use compute_residuals on a dataframe with 5M observations and after tens of minutes it is unclear whether it will ever finish computing. I suspect the dataframe is probably too large to do the computation as is in a reasonable amount of time. I’d like to avoid significantly scaling down as events constitute a small fraction of the observations and I need the statistics. Is there an existing solution for this?

@ybagdasa, to confirm, you were computing the schoenfeld residuals?

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
ybagdasacommented, Jul 8, 2020

@CamDavidsonPilon Yes that’s fitting.

I ended up doing a little workaround where I divied up the data into 50 k samples and then ran those in parallel and combined the coefficients and covariance matrix afterwards using some normal approximation assumptions. Took about 6 hours to run. Not the most ideal, but it worked.

1reaction
ybagdasacommented, Jun 26, 2020

like 6 covariates to fit

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to Find Residuals in Regression Analysis | Built In
Calculating the residual provides a valuable clue into how well your model fits the data set. To calculate residuals we need to find...
Read more >
Residual Calculator | Analyse Linear Regression
The residual calculator helps you to calculate the residuals of a linear regression analysis.
Read more >
Interpreting Residual Plots to Improve Your Regression
When you run a regression, Stats iQ automatically calculates and plots residuals to help you understand and improve your regression model.
Read more >
Residual analysis - statistics - Encyclopedia Britannica
The analysis of residuals plays an important role in validating the regression ... These residuals, computed from the available data, are treated as...
Read more >
AP Stats Unit 2 Notes: Residuals Study Guide - Fiveable
Calculating Residuals. In order to calculate a residual for a given data point, we need the LSRL for that data set and the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found