question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Heavy disk write in long run

See original GitHub issue

I set up a local instance (iBot) of Smokey in a Docker container, and recently I noticed unintelligibly high disk write amount (read from Block I/O column of docker stats). At present, the container that’s been up for 7 days has accrued a total block write amount of more than 150 GiB, which divides into roughly 20 GiB per day.

Upon closer inspection, I noticed that the file bodyfetcherQueueTimings.p was written for once every few seconds, and at the time it had a size of 2.1 MiB. For every update its size grew by a few bytes. Given that it’s written once for every API call to fetch posts from SE and that we’re using around 10k of our daily quota, it’s most likely the culprit for such a huge write amount.

Per my knowledge, that pickle file doesn’t have any use for Smokey’s primary functionalities (it’s for data analysis by human), and this file alone is wearing out SSDs very quickly (one Samsung 970 EVO 500 GB model in less than 3 years if unattended, calculating from raw bytes). Should we take some measures to reduce unnecessary hardware workloads?

Issue Analytics

  • State:open
  • Created 4 years ago
  • Comments:8 (7 by maintainers)

github_iconTop GitHub Comments

5reactions
angussidneycommented, Jul 22, 2019

I have two proposals for how we can go about fixing this issue:

  1. Disable queue timing logging by default. This data isn’t necessary for Smokey to correctly function, and isn’t accessed during normal operation. We only use this data on an infrequent basis; thus, it would make sense to disable this logging by default, and only enable it on the rare occasion that we actually need to collect the data (e.g. by passing some form of option to the startup command, like we have for standby etc)

  2. Change the data to a format which can be easily be appended to the end of a file, so that it doesn’t require an overwrite. Pickle, as a serialised format, is difficult to modify without loading the binary data into Python objects, changing it, and re-writing the data to file. This downside is worth it if you’re dealing with complex Python objects that are hard to store by other means; however, we’re only using it to store relatively simple data (a series of lists of integers (or floats?), with each number representing the time the post waited for an API request):

    queue_timings = {
        "site": [int, int, int...],
        ...
    }
    

    (side note: wouldn’t our current implementation benefit from the use of collections.defaultdict?)

    Given that all the analysis is done by a totally separate script, I don’t think we’re benefiting greatly from pickle’s ability to store python objects directly. Instead, I propose that we take a simpler approach, and convert the data to one timing per line, in the format site timing (e.g. stackoverflow.com 231). Then, at a later time the data analysis script can aggregate the data into a dictionary, or whatever other format is most convenient for analysis. This method has the advantage that new data can easily be appended to the end of the file without having to overwrite it, thus saving disk writes.

Better ideas for a new data format are welcomed.

0reactions
stale[bot]commented, Oct 30, 2019

This issue has been closed because it has had no recent activity. If this is still important, please add another comment and find someone with write permissions to reopen the issue. Thank you for your contributions.

Read more comments on GitHub >

github_iconTop Results From Across the Web

[SOLVED] Windows 10 100% disk usage in Task Manager
If your Windows 10 Task Manager shows 100% disk usage, something is causing your hard drive to over-work. The good news is, it's...
Read more >
How to Fix 100% Disk Usage in Windows 10 - AVG
Running lots of applications at once may cause 100% disk usage in Windows 10. Uninstalling these programs often isn't possible, because you may ......
Read more >
Explained: What 100% Disk Usage Mean and How to Fix it
100% disk usage means that your disk has reached its maximum capacity i.e. it is fully occupied by some or the other task....
Read more >
How to Fix Hard Drive Very Slow in Windows 10/8/7 - EaseUS
1. Hard drive very slow to read ... Check for programs making frequent read/write requests to the hard disk in Task Manager. Make...
Read more >
Reducing disk writes | Apple Developer Documentation
Overview · Optimize SSD access · Eliminate excessive write operations · Gather metrics about your app's disk usage · Identify the code causing...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found