question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. ItΒ collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Race condition in accept loop when stats are enabled

See original GitHub issue

❓ I’m submitting a …

  • 🐞 bug report
  • 🐣 feature request
  • ❓ question about the decisions made in the repository

🐞 Describe the bug. What is the current behavior?

When starting a server with monitoring enabled, there seems to be an intermittent race condition where the start time is accessed before it has been initialised, resulting in an exception.

❓ What is the motivation / use case for changing the behavior?

Fewer exceptions πŸ˜ƒ

πŸ’‘ To Reproduce

Toggle stats reporting on while a request is in progress, e.g. starting the server while an uptime monitoring query is being performed.

πŸ’‘ Expected behavior

Stats are recorded.

πŸ“‹ Details

Example stack trace:

[03/May/2019:14:29:50] ENGINE Bus STARTING
[03/May/2019:14:29:50] ENGINE Started monitor thread 'Autoreloader'.
[03/May/2019:14:29:50] ENGINE Serving on http://0.0.0.0:50001
[03/May/2019:14:29:50] ENGINE Bus STARTED
Exception in thread CP Server Thread-3:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/path/to/project/venv/lib/python3.7/site-packages/cheroot/workers/threadpool.py", line 121, in run
    self.work_time += time.time() - self.start_time
TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'

πŸ“‹ Environment

  • Cheroot version: 6.5.5
  • CherryPy version: 18.1.1
  • Python version: 3.7.3
  • OS: OS X 10.14.4
  • Browser: all

πŸ“‹ Additional context

I believe this is probably occurring as threadpool.py does two independent checks on self.server.stats['Enabled'] - one to initialise the start time, one to record the results. If the value of self.server.stats['Enabled'] changes while a request is in progress then we can assume there’s a chance that we’ll try the result block without the initialising the start time.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
webknjazcommented, Sep 4, 2019

Released the patch in v6.5.7.

1reaction
jshiellcommented, Sep 2, 2019

For the record; no - and no, we don’t toggle. It just seemed to be a timing issue on startup. It looks like we fixed it by toggling thread order to give the stats reporting thread a little longer to startup.

That commit looks perfect - thanks very much @webknjaz, much obliged!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Go race condition trivia - by Jack Lindamood - Medium
I recently ran into a -race error message that stumped me for a while, so I'll retell it here in phases to see...
Read more >
Possible race condition in OpenMP parallel for loop
Thus I suspect there is a race condition in my loop, but I haven't been able to find it. Does anyone know what...
Read more >
Race Condition (Software) - Devopedia
Race condition in software is an undesirable event that can happen when multiple entities access or modify shared resources in a system.
Read more >
Ignore stale read errors from the race detector - Google Groups
I've been trying to find a way to ignore errors from the golang race detector. In particular I have a stats loop that...
Read more >
Invisible race conditions: The cache has poisoned us
As you can see, we are looking at a loop that may be executed several times, as such the updateValueFactory can be called...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found