question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Memory usage grows when using StackdriverExporter for Flask app

See original GitHub issue

I am using opencensus-python library to trace requests to my Python Flask application.

I have noticed that when using a StackdriverExporter instance, the memory usage of my app grows steadily for each traced request. When observed using memory_profiler, the memory usage grows roughly 700KB to 1MB per request. I have also noticed similar usage growth on GKE metrics dashboard when running the app on GKE.

All other factors being the same, if I use a LoggingExporter instance (instead of StackdriverExporter), the memory usage stays constant (as expected).

I have put together a sample app along with commands to repro this issue: https://gist.github.com/Amey-D/ebbf7acb77446e81998cf325c7737e73

Any ideas?

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:2
  • Comments:6 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
c24tcommented, Nov 26, 2018

Reports of this bug’s death may have been exaggerated. After adding OC instrumentation to the GCP microservices demo app, I’m seeing GKE containers leaking memory. This only happens with the stackdriver exporter, and seems to happen regardless of the transport and sampler.

screen shot 2018-11-21 at 1 04 02 pm-1

I don’t know if it’s worth reopening yet, but I’m looking into it now and will let you know what I find.

0reactions
Amey-Dcommented, Nov 26, 2018

Thanks @c24t for investigating this. Per your suggestion I upgraded the version of opencensus-python package to 0.1.8, and I no longer see the memory leaks.
flask_app_with_stackdriver_exporter_0 1 8

Side notes:

  1. I had to add opencensus-python[stackdriver]==0.1.8 to my requirements.txt file because the newer version treats stackdriver exporter as an extra functionality.
  2. I also had to install grpcio separately because apparently the transitive dependency is not handled correctly by the package dependencies.
Read more comments on GitHub >

github_iconTop Results From Across the Web

Troubleshoot Dataflow out of memory errors - Google Cloud
Memory usage from SDK shared objects and data grows linearly with the number of Apache Beam SDK processes. In Java and Go pipelines,...
Read more >
Flask App Memory Leak caused by each API call
Suppose you have a simple stateless Flask app with only one endpoint named 'foo'. Note that the other endpoints 'memory' and 'snapshot' ...
Read more >
Detecting Memory Leak in Python - Superuser
Confirm there's a leak:​​ To give bit more context on application which was leaking memory, it was a flask app with traffic mostly...
Read more >
Google Associate Cloud Engineer Practice Exam Part 1
Use the Stackdriver Logging export feature to create a sink to Cloud Storage. Create a Cloud Dataflow job that imports log files from...
Read more >
Blog | Prometheus
Furthermore, it enables application instrumentation and exporters to be ... Many of us already use Prometheus to scrape all required metrics ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found