1.8 seconds for ... from prometheus_api_client import PrometheusConnect

Describe the bug It seems to take a long time to import from the prometheus_api_client library. So I tested it. Note: This is purely import related and nothing to do with actual data query/transfer.

To Reproduce The following trivial code in file time-import.py:

#!/usr/bin/env python3

import sys
from datetime import datetime, timedelta

n = int(sys.argv[1])

if n == 0:
    now = datetime.now()
    import os
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'os'))

if n == 1:
    now = datetime.now()
    import numpy
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'numpy'))

if n == 2:
    now = datetime.now()
    import pandas
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'pandas'))

if n == 3:
    now = datetime.now()
    import prometheus_api_client
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'import prometheus_api_client'))

if n == 4:
    now = datetime.now()
    from prometheus_api_client import PrometheusConnect
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'from prometheus_api_client import PrometheusConnect'))

if n == 5:
    now = datetime.now()
    from prometheus_api_client import PrometheusConnect, MetricsList, Metric
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'from prometheus_api_client import PrometheusConnect, MetricsList, Metric'))

if n == 6:
    now = datetime.now()
    from prometheus_api_client import PrometheusConnect,  MetricSnapshotDataFrame, MetricRangeDataFrame
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'from prometheus_api_client import PrometheusConnect,  MetricSnapshotDataFrame, MetricRangeDataFrame'))

if n == 7:
    now = datetime.now()
    from prometheus_api_client.utils import parse_datetime
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'from prometheus_api_client.utils import parse_datetime'))

When run, I get the following results (R.Pi4 Linux pi.local 5.15.72-v7l+ #1591 SMP Wed Oct 5 12:05:33 BST 2022 armv7l GNU/Linux).

$ for t in 0 1 2 3 4 5 6 7 ; do ./time-import.py $t ; done
delay: 0.046 os
delay: 225.053 numpy
delay: 872.548 pandas
delay: 1848.370 import prometheus_api_client
delay: 1863.928 from prometheus_api_client import PrometheusConnect
delay: 1867.375 from prometheus_api_client import PrometheusConnect, MetricsList, Metric
delay: 1860.085 from prometheus_api_client import PrometheusConnect,  MetricSnapshotDataFrame, MetricRangeDataFrame
delay: 2879.714 from prometheus_api_client.utils import parse_datetime
$

Times are in mSec’s

This loop of tests was run a few times in order to make sure files were memory cached.

Clearly both numpy and pandas aren’t exactly the best at importing themselves (and this issue can’t address that fact); however, I wonder where the rest of the time is being spent and why so long? Remember, this is just import and no actual query or class was created. (BTW: class creation is 1.3 msec and a local LAN call to Prometheus server for all_metrics() is 38.2 msecs).

tuna Here’s what tuna has to say about all this.

$ python3 -X importtime -c "import prometheus_api_client" 2> prometheus_api_client.log
$ tuna prometheus_api_client.log 

Well heck, maybe this isn’t fixable within this package?

Expected behavior I guess this issue is about opening the discussion about speed vs. requesting a fix. Can this be sped up? If so how? Would fixing imports simply make later code calls slow?

For example, the from prometheus_api_client import PrometheusConnect (i.e. no Metric) also imports the metrics code; however, that code is yet to be used/called.

Additional context The same was run on a MacBook Air (M1 Darwin MacBook-Air.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 29 04:31:12 PDT 2022; root:xnu-7195.141.39~2/RELEASE_ARM64_T8101 arm64) and to be honest, I thought it would be a lot quicker.

$ for t in 0 1 2 3 4 5 6 7 ; do ./time-import.py $t ; done
delay: 0.006 os
delay: 56.143 numpy
delay: 189.823 pandas
delay: 407.442 import prometheus_api_client
delay: 427.759 from prometheus_api_client import PrometheusConnect
delay: 399.677 from prometheus_api_client import PrometheusConnect, MetricsList, Metric
delay: 389.309 from prometheus_api_client import PrometheusConnect,  MetricSnapshotDataFrame, MetricRangeDataFrame
delay: 627.775 from prometheus_api_client.utils import parse_datetime
$ 

[mahtin]

Give me some time and I’ll upload what I think works cleanly. I still need to run all tests. I deferred the import to the plot() call so that only folks doing plots see the slower start time. It’s something like this:

    def plot(self):
        if not Metric._bar:
            from good import bar
            Metric._bar = bar
        Metric.bar.call()
        ...

This works quite well. The import stays referenced and is usable later.

More later

[4n4nd]

@chauhankaranraj @harshad16 wdyt of only importing specific items from a package?

For example, here instead of

import os

we do

from os import makedirs 

Do you think that would make the imports faster?