Generalizing Parametric Regression models

Design goals:

1. Easy to add custom regression models, similar to how one can create custom univariate models.

2. Arbitrary number of parameters to fit to. Example: an exponential AFT model has a single parameter, whereas Weibull AFT model has two, and a custom one could have N.

3. Fixing parameters. I should be able to fix n < N parameters to set parameters.

4. custom penalizers (this enables point 5 below) for users to allow some interesting parameter/information sharing.

5. Stretch goal: make PiecewiseExponentialRegressionFitter less of a snowflake.

Point 2. and 3. allow the Exponential AFT fitter to be a special case of the the Weibull AFT fitter.

Prior workarounds or issues

See also #720 and this answer

Some thoughts:

• I think I need to expel the idea of “primary” and “ancillary” for the general case, and make Known models reimplement those
• PiecewiseExponentialRegressionFitter is currently a snowflake. One of the reasons was because it has an arbitrary number of parameters and a customer penalizer function. I might be able to fold that into this class.

Custom AFT API

class ShiftedWeibull(ParametericRegressionFitter):

    _fitted_parameter_names = ['lambda_', 'rho_', 'delta_']
    # no point in _bounds: assume they range over all floats, the user can change to positive only 
    # using the cumulative hazard and exp()

    def _cumulative_hazard(self, params, T, Xs):
        lambda_ = np.exp(np.dot(Xs["lambda_"], params["lambda_"])) # > 0
        rho_    = np.exp(np.dot(Xs["rho"],     params["rho_"]))    # > 0
        delta_  =        np.dot(Xs["delta_"],  params["delta_"])

        return ((T - delta_) / lambda_) ** rho_


    def _add_penalty(self, neg_ll, params):
        """ 
        only penalize the delta parameter, for whatever reason
        """
        penalty = params['delta_'] ** 2
        return neg_ll + self.penalizer * penalty



swf = ShiftedWeibull(penalizer=1.0)

covariates = {
  'lambda_': [variable names], # need a shortcut for all columns?
  'rho_':    [variable names], 
  'delta_':  [variable names] # (or fixed constant),
}

swf.fit_right_censoring(df, "T", "E", regressors=covariates) # TODO: name 

[CamDavidsonPilon]

@helloannietran yes, you’re correct, that makes more sense for a parameter called “cure” - the interpretation is all relative though!

[helloannietran]

In the cumulative hazard func for CureModel, should it return -np.log(c + (1 - c) * sf) instead of -np.log((1 - c) + c * sf) if c is the cure rate? I’m new to survival analysis and currently experimenting with lifelines. Let me know if I should post this somewhere else…