question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

1.8 seconds for ... from prometheus_api_client import PrometheusConnect

See original GitHub issue

Issue Description

Describe the bug It seems to take a long time to import from the prometheus_api_client library. So I tested it. Note: This is purely import related and nothing to do with actual data query/transfer.

To Reproduce The following trivial code in file time-import.py:

#!/usr/bin/env python3

import sys
from datetime import datetime, timedelta

n = int(sys.argv[1])

if n == 0:
    now = datetime.now()
    import os
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'os'))

if n == 1:
    now = datetime.now()
    import numpy
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'numpy'))

if n == 2:
    now = datetime.now()
    import pandas
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'pandas'))

if n == 3:
    now = datetime.now()
    import prometheus_api_client
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'import prometheus_api_client'))

if n == 4:
    now = datetime.now()
    from prometheus_api_client import PrometheusConnect
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'from prometheus_api_client import PrometheusConnect'))

if n == 5:
    now = datetime.now()
    from prometheus_api_client import PrometheusConnect, MetricsList, Metric
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'from prometheus_api_client import PrometheusConnect, MetricsList, Metric'))

if n == 6:
    now = datetime.now()
    from prometheus_api_client import PrometheusConnect,  MetricSnapshotDataFrame, MetricRangeDataFrame
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'from prometheus_api_client import PrometheusConnect,  MetricSnapshotDataFrame, MetricRangeDataFrame'))

if n == 7:
    now = datetime.now()
    from prometheus_api_client.utils import parse_datetime
    print('delay: %.3f %s' % (float((datetime.now() - now) / timedelta(milliseconds=1)), 'from prometheus_api_client.utils import parse_datetime'))

When run, I get the following results (R.Pi4 Linux pi.local 5.15.72-v7l+ #1591 SMP Wed Oct 5 12:05:33 BST 2022 armv7l GNU/Linux).

$ for t in 0 1 2 3 4 5 6 7 ; do ./time-import.py $t ; done
delay: 0.046 os
delay: 225.053 numpy
delay: 872.548 pandas
delay: 1848.370 import prometheus_api_client
delay: 1863.928 from prometheus_api_client import PrometheusConnect
delay: 1867.375 from prometheus_api_client import PrometheusConnect, MetricsList, Metric
delay: 1860.085 from prometheus_api_client import PrometheusConnect,  MetricSnapshotDataFrame, MetricRangeDataFrame
delay: 2879.714 from prometheus_api_client.utils import parse_datetime
$

Times are in mSec’s

This loop of tests was run a few times in order to make sure files were memory cached.

Clearly both numpy and pandas aren’t exactly the best at importing themselves (and this issue can’t address that fact); however, I wonder where the rest of the time is being spent and why so long? Remember, this is just import and no actual query or class was created. (BTW: class creation is 1.3 msec and a local LAN call to Prometheus server for all_metrics() is 38.2 msecs).

tuna Here’s what tuna has to say about all this.

$ python3 -X importtime -c "import prometheus_api_client" 2> prometheus_api_client.log
$ tuna prometheus_api_client.log 
tuna_prometheus_api_client

Well heck, maybe this isn’t fixable within this package?

Expected behavior I guess this issue is about opening the discussion about speed vs. requesting a fix. Can this be sped up? If so how? Would fixing imports simply make later code calls slow?

For example, the from prometheus_api_client import PrometheusConnect (i.e. no Metric) also imports the metrics code; however, that code is yet to be used/called.

Additional context The same was run on a MacBook Air (M1 Darwin MacBook-Air.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 29 04:31:12 PDT 2022; root:xnu-7195.141.39~2/RELEASE_ARM64_T8101 arm64) and to be honest, I thought it would be a lot quicker.

$ for t in 0 1 2 3 4 5 6 7 ; do ./time-import.py $t ; done
delay: 0.006 os
delay: 56.143 numpy
delay: 189.823 pandas
delay: 407.442 import prometheus_api_client
delay: 427.759 from prometheus_api_client import PrometheusConnect
delay: 399.677 from prometheus_api_client import PrometheusConnect, MetricsList, Metric
delay: 389.309 from prometheus_api_client import PrometheusConnect,  MetricSnapshotDataFrame, MetricRangeDataFrame
delay: 627.775 from prometheus_api_client.utils import parse_datetime
$ 

Issue Analytics

  • State:open
  • Created 3 months ago
  • Reactions:1
  • Comments:5 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
mahtincommented, Nov 1, 2022

Give me some time and I’ll upload what I think works cleanly. I still need to run all tests. I deferred the import to the plot() call so that only folks doing plots see the slower start time. It’s something like this:

    def plot(self):
        if not Metric._bar:
            from good import bar
            Metric._bar = bar
        Metric.bar.call()
        ...

This works quite well. The import stays referenced and is usable later.

More later

1reaction
4n4ndcommented, Oct 31, 2022

@chauhankaranraj @harshad16 wdyt of only importing specific items from a package?

For example, here instead of

import os

we do

from os import makedirs 

Do you think that would make the imports faster?

Read more comments on GitHub >

github_iconTop Results From Across the Web

prometheus-api-client - PyPI
This module is essentially a class created for the collection of metrics from a Prometheus host. It stores the following connection parameters:
Read more >
Source code for prometheus_api_client.prometheus_connect
Source code for prometheus_api_client.prometheus_connect. """A Class for collection of metrics from a Prometheus Host.""" from urllib.parse import ...
Read more >
HTTP API - Prometheus.io
The following example evaluates the expression up over a 30-second range with a query resolution of 15 seconds. $ curl 'http://localhost:9090/api/v1/query_range ...
Read more >
Prometheus API Client in Python - Donald Le - Medium
yml Example prometheus. yml file # my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds.
Read more >
Metrics API Code Samples | Embrace Docs
This example uses the prometheus-api-client package for python. from datetime import datetime from prometheus_api_client import PrometheusConnect, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found