question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Optimize text_classifer model memory usage

See original GitHub issue

Related issue: https://github.com/brave/brave-browser/issues/21602 Parsed text classifier model (component id : emgmepnebbddgnkhfmhdhmjifkglkamo) takes ~24mb of memory in Ads Service process.

It’s looks excessive and we should find a way to make it smaller. Another problem is that peak usage on start is more than 200mb.

Here is the code that produce 24mb memory allocation: https://github.com/brave/brave-core/blob/master/vendor/bat-native-ads/src/bat/ads/internal/ml/pipeline/text_processing/text_processing.cc#L63

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
LaurenWagscommented, May 16, 2022

Makes sense, thanks for the clarification @atuchin-m 👍🏻

1reaction
atuchin-mcommented, May 13, 2022

@LaurenWags @LaurenWags @btlechowski Big thanks for verification!

Just want to clarify one thing: Memory footprint on Linux/Mac is counted in a different way that on Windows. That’s why it’s ok that absolute numbers are not match across OSes, so please don’t be confused by this.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Text Classification: All Tips and Tricks from 5 Kaggle ...
Optimize the memory by reducing the size of some attributes · Use open-source libraries such as Dask to read and manipulate the data,...
Read more >
Optimize PyTorch Performance for Speed and Memory ...
Overall, you can optimize the time and memory usage by 3 key points. First, reduce the i/o (input/output) as much as possible so...
Read more >
Text Mining - Computation time and memory usage
Dear all, I am working on a text mining use case with a data set of around 80.000 data sets and 33 Attributes...
Read more >
Expanded fastText library now fits on smaller-memory devices
fastText's models now fit on smartphones and small computers like Raspberry Pi devices thanks to a new functionality that reduces memory usage.
Read more >
An Improved Text Classification Model Based on Memory ...
This paper proposes a text classification model, called improved memory neural network model, which is used to process large-scale training ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found