Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
KeyError: 'services'
See original GitHub issueI was playing wtih long-running tasks with Distributed Dask scaling a number of workers up and down, so I’m not exactly sure what actions led to the hanging Distributed master with the following log:
7/22/2016 12:39:37 PM distributed.utils - ERROR - 'services'
7/22/2016 12:39:37 PM Traceback (most recent call last):
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/utils.py", line 206, in log_errors
7/22/2016 12:39:37 PM yield
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in restart
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in <dictcomp>
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM KeyError: 'services'
Full log:
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Starting worker compute stream, 10.42.145.247:42383
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Register 10.42.145.247:40939
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Starting worker compute stream, 10.42.145.247:46179
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Starting worker compute stream, 10.42.145.247:40939
7/22/2016 12:39:20 PM distributed.core - INFO - Connection from 10.42.145.247:53876 to Scheduler
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Register 10.42.145.247:43829
7/22/2016 12:39:20 PM distributed.core - INFO - Connection from 10.42.145.247:53878 to Scheduler
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Starting worker compute stream, 10.42.145.247:43829
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Register 10.42.145.247:42268
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Starting worker compute stream, 10.42.145.247:42268
7/22/2016 12:39:20 PM distributed.core - INFO - Connection from 10.42.145.247:53880 to Scheduler
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Register 10.42.145.247:46043
7/22/2016 12:39:20 PM distributed.scheduler - INFO - Starting worker compute stream, 10.42.145.247:46043
7/22/2016 12:39:37 PM distributed.utils - ERROR - 'services'
7/22/2016 12:39:37 PM Traceback (most recent call last):
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/utils.py", line 206, in log_errors
7/22/2016 12:39:37 PM yield
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in restart
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in <dictcomp>
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM KeyError: 'services'
7/22/2016 12:39:37 PM distributed.scheduler - ERROR - 'services'
7/22/2016 12:39:37 PM Traceback (most recent call last):
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1426, in handle_messages
7/22/2016 12:39:37 PM yield result
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
7/22/2016 12:39:37 PM value = future.result()
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
7/22/2016 12:39:37 PM raise_exc_info(self._exc_info)
7/22/2016 12:39:37 PM File "<string>", line 3, in raise_exc_info
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 282, in wrapper
7/22/2016 12:39:37 PM yielded = next(result)
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in restart
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in <dictcomp>
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM KeyError: 'services'
7/22/2016 12:39:37 PM distributed.utils - ERROR - 'services'
7/22/2016 12:39:37 PM Traceback (most recent call last):
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/utils.py", line 206, in log_errors
7/22/2016 12:39:37 PM yield
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1426, in handle_messages
7/22/2016 12:39:37 PM yield result
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
7/22/2016 12:39:37 PM value = future.result()
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
7/22/2016 12:39:37 PM raise_exc_info(self._exc_info)
7/22/2016 12:39:37 PM File "<string>", line 3, in raise_exc_info
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 282, in wrapper
7/22/2016 12:39:37 PM yielded = next(result)
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in restart
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in <dictcomp>
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM KeyError: 'services'
7/22/2016 12:39:37 PM distributed.scheduler - INFO - Close connection to Scheduler, 996a9048-4fda-11e6-90b1-0211f410711f
7/22/2016 12:39:37 PM distributed.core - ERROR - 'services'
7/22/2016 12:39:37 PM Traceback (most recent call last):
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/core.py", line 206, in handle_stream
7/22/2016 12:39:37 PM result = yield gen.maybe_future(handler(stream, **msg))
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
7/22/2016 12:39:37 PM value = future.result()
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
7/22/2016 12:39:37 PM raise_exc_info(self._exc_info)
7/22/2016 12:39:37 PM File "<string>", line 3, in raise_exc_info
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1017, in run
7/22/2016 12:39:37 PM yielded = self.gen.send(value)
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1353, in add_client
7/22/2016 12:39:37 PM yield self.handle_messages(stream, bstream, client=client)
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
7/22/2016 12:39:37 PM value = future.result()
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
7/22/2016 12:39:37 PM raise_exc_info(self._exc_info)
7/22/2016 12:39:37 PM File "<string>", line 3, in raise_exc_info
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1014, in run
7/22/2016 12:39:37 PM yielded = self.gen.throw(*exc_info)
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1426, in handle_messages
7/22/2016 12:39:37 PM yield result
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
7/22/2016 12:39:37 PM value = future.result()
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
7/22/2016 12:39:37 PM raise_exc_info(self._exc_info)
7/22/2016 12:39:37 PM File "<string>", line 3, in raise_exc_info
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 282, in wrapper
7/22/2016 12:39:37 PM yielded = next(result)
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in restart
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM File "/opt/conda/lib/python3.5/site-packages/distributed/scheduler.py", line 1616, in <dictcomp>
7/22/2016 12:39:37 PM for addr, d in self.worker_info.items()}
7/22/2016 12:39:37 PM KeyError: 'services'
7/22/2016 12:39:37 PM distributed.core - INFO - Lost connection: ('10.42.138.167', 38072)
7/22/2016 12:39:37 PM distributed.core - INFO - Close connection from 10.42.138.167:38072 to Scheduler
7/22/2016 12:39:43 PM /opt/conda/lib/python3.5/site-packages/distributed/bokeh/status_monitor.py:132: BokehDeprecationWarning: bokeh.io.vplot was deprecated in Bokeh 0.12.0; please use bokeh.models.layouts.Column instead
7/22/2016 12:39:43 PM table = vplot(slow, fast)
7/22/2016 12:39:43 PM bokeh.server.views.ws - INFO - WebSocket connection closed: code=1001, reason=None
7/22/2016 12:39:45 PM bokeh.server.views.ws - INFO - WebSocket connection opened
7/22/2016 12:39:45 PM /opt/conda/lib/python3.5/site-packages/distributed/bokeh/status_monitor.py:132: BokehDeprecationWarning: bokeh.io.vplot was deprecated in Bokeh 0.12.0; please use bokeh.models.layouts.Column instead
7/22/2016 12:39:45 PM table = vplot(slow, fast)
7/22/2016 12:39:45 PM bokeh.server.views.ws - INFO - ServerConnection created
Issue Analytics
- State:
- Created 7 years ago
- Comments:8 (8 by maintainers)
Top Results From Across the Web
Python KeyError Exceptions and How to Handle Them
The Python KeyError is a type of LookupError exception and denotes that there was an issue retrieving the key you were looking for....
Read more >I'm getting Key error in python - Stack Overflow
A KeyError generally means the key doesn't exist. So, are you sure the path key exists? From the official python docs: exception KeyError....
Read more >How to Fix KeyError Exceptions in Python - Rollbar
The Python KeyError is an exception that occurs when an attempt is made to access an item in a dictionary that does not...
Read more >How to fix Python KeyError Exceptions in simple steps?
A Python KeyError is raised when you try to access an invalid key in a dictionary. In simple terms, when you see a...
Read more >keyerror in Python – How to Fix Dictionary Error
When working with dictionaries in Python, a KeyError gets raised when you try to access an item that doesn't exist in a Python...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Closing for now. Please reopen if the problem persists.
@mrocklin I will sure be able to run the fixed version. Let me know once you have it. These types of bugs are really nasty, so it will be nice to have detailed logging there, so we can see whether we could have had the state in which it failed previously.