CoxTimeVaryingFitter is actually faster than CoxPHFitter...

CoxPHFitter test

    import pandas as pd
    import time

    from lifelines import CoxPHFitter
    from lifelines.datasets import load_rossi

    df = load_rossi()
    df = pd.concat([df] * 20)
    cp = CoxPHFitter()
    start_time = time.time()
    cp.fit(df, duration_col="week", event_col="arrest")
    print("--- %s seconds ---" % (time.time() - start_time))
    cp.print_summary()

takes about 2.3 seconds.

CoxTimeVaryingFitter test

    import time
    import pandas as pd
    from lifelines import CoxTimeVaryingFitter
    from lifelines.datasets import load_rossi
    from lifelines.utils import to_long_format

    df = load_rossi()
    df = pd.concat([df] * 20)
    df = df.reset_index()
    df = to_long_format(df, duration_col='week')
    ctv = CoxTimeVaryingFitter()
    start_time = time.time()
    ctv.fit(df, id_col="index", event_col="arrest", start_col="start", stop_col="stop")
    time_took = time.time() - start_time
    print("--- %s seconds ---" % time_took)
    ctv.print_summary()

takes about 1.65 seconds.

Note that the datasets between the two are identical. Even the results are identical (as expected). The internal differences are that CoxPHFitter looks at each row individually, while the CoxTimeVaryingFitter looks at all rows grouped by duration. The latter is much more efficient when there are lots of ties (i.e. when cardinality / row count is low).

This is kinda shocking to me. It means I can improve CoxPHFitter performance by like 30%.

[CamDavidsonPilon]

Down to ~0.17 with #609

[CamDavidsonPilon]

Down to 0.67 with #595