Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Frequent CommClosed issues w/ inproc comms on Prefect
See original GitHub issueIn Prefect we’re seeing issues with distributed >= 2.23.0 when using inproc comms. Everything works fine in 2.22.0, but in 2.23.0 we see comm closed errors and workers marked as dead.
I have a reproducer with this, but unfortunately haven’t been able to reproduce without prefect yet (will keep trying):
Reproducer (requires prefect installed):
from prefect import Flow, task
from prefect.engine.executors import DaskExecutor
from time import sleep
@task
def get_data():
sleep(0.5)
return list(range(180))
@task
def inc(user):
sleep(0.5)
return user + 1
@task
def accum(data):
return sum(data)
executor = DaskExecutor(cluster_kwargs={"nanny": False, "memory_limit": 0, "processes": False, "n_workers": 20, "threads_per_worker": 1, "silence_logs": 0})
with Flow("test_flow") as flow:
accum(inc.map(get_data()))
if __name__ == "__main__":
flow.run(executor=executor)
Logs on dask & distributed `2.30.0`:
[2020-10-12 22:12:30] INFO - prefect.FlowRunner | Beginning Flow run for 'test_flow'
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: inproc://192.168.1.36/40918/1
distributed.scheduler - INFO - dashboard at: localhost:8787
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/3
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65097
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-84m1pdwc
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/4
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65098
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-tghwjywi
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/5
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65099
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-aa13mh4n
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/6
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65100
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-tgp7cb_b
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/7
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65101
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-4ncd59jb
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/8
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65102
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-50ukew9q
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/9
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65103
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-0unm12uq
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/10
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65104
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-xvxakipk
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/11
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65105
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-88ryax75
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/12
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65106
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-ojz616ac
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/13
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65107
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-l97cqvxc
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/14
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65108
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-e22w6nsn
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/15
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65109
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-lgan9zf1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/16
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65110
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-t3r_s91p
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/17
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65111
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-vk8f4abf
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/18
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65112
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-vip55j4t
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/19
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65113
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-antub0bv
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/20
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65114
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-t4x30du5
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/21
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65115
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-h9kq54de
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: inproc://192.168.1.36/40918/22
distributed.worker - INFO - Listening to: inproc192.168.1.36
distributed.worker - INFO - dashboard at: 192.168.1.36:65116
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-u5ijheja
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/3', name: 12, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/3
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/4', name: 7, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/4
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/5', name: 11, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/5
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/6', name: 6, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/6
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/7', name: 10, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/7
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/8', name: 5, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/8
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/9', name: 9, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/9
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/10', name: 4, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/10
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/11', name: 0, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/11
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/12', name: 3, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/12
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/13', name: 2, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/13
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/14', name: 1, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/14
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/15', name: 19, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/15
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/16', name: 18, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/16
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/17', name: 8, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/17
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/18', name: 17, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/18
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/19', name: 16, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/19
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/20', name: 15, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/20
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/21', name: 14, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/21
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/22', name: 13, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/22
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Receive client connection: Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.core - INFO - Starting established connection
[2020-10-12 22:12:31] INFO - prefect.TaskRunner | Task 'get_data': Starting task run...
[2020-10-12 22:12:31] INFO - prefect.TaskRunner | Task 'get_data': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc': finished task run for task with final state: 'Mapped'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[0]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[3]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[1]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[11]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[12]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[2]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[4]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[5]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[6]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[0]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[7]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[20]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[8]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[3]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[1]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[12]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[11]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[9]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[2]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[4]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[10]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[5]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[38]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[6]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[50]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[21]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[49]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[39]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[7]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[20]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[8]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[22]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[9]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[23]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[44]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[10]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[45]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[40]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[46]': Starting task run...
distributed.comm.inproc - WARNING - Closing dangling queue in <InProc local=inproc://192.168.1.36/40918/64 remote=inproc://192.168.1.36/40918/1>
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[38]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[47]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[48]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[50]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[49]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[21]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[39]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[98]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[22]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[23]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[44]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[29]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[99]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[41]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[30]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[40]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[45]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[46]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[47]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[42]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[48]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[43]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[24]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[98]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[27]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[25]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[26]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[100]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[28]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[29]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[99]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[41]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[30]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[78]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[42]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[89]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[43]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[24]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[79]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[101]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[90]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[27]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[25]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[100]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[26]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[28]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[102]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[103]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[84]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[78]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[87]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[89]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[85]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[80]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[101]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[88]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[58]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[86]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[79]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[90]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[69]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[81]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[102]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[59]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[84]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[103]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[70]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[87]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[85]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[82]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[67]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[80]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[64]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[83]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[86]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[58]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[88]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[81]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[69]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[65]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[59]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[60]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[70]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[66]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[118]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[61]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[82]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[64]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[83]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[68]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[129]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[67]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[130]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[65]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[119]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[62]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[63]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[127]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[60]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[118]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[124]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[66]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[61]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[125]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[68]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[129]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[120]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[126]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[121]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[158]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[130]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[119]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[128]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[62]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[63]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[109]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[127]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[124]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[110]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[125]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[159]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[122]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[123]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[126]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[107]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[104]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[120]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[121]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[158]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[105]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[128]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[109]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[106]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[110]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[161]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[160]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[138]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[159]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[108]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[122]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[123]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[139]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[104]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[150]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[149]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[107]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[105]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[106]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[162]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[138]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[161]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[147]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[108]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[160]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[164]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[163]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[165]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[139]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[149]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[150]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[146]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[148]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[141]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[178]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[140]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[179]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[162]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[169]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[170]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[147]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[142]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[164]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[163]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[165]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[145]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[167]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[144]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[146]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[143]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[148]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[141]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[178]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[140]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[179]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[169]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[170]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[168]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[57]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[142]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[166]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[77]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[117]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[145]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[167]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[144]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[131]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[143]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[37]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[75]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[157]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[53]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[97]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[168]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[94]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[57]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[166]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[77]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[117]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[36]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[92]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[131]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[37]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[72]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[111]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[51]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[91]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[75]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[157]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[97]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[53]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[94]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[113]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[153]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[36]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[92]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[74]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[174]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[34]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[134]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[72]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[111]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[51]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[54]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[91]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[114]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[136]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[113]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[153]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[132]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[172]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[52]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[74]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[31]': Starting task run...
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/10
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[174]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[34]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[134]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/11
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/12
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/13
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/14
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/15
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/16
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/17
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/18
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/19
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/11', name: 0, memory: 11, processing: 0>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/11
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/10', name: 4, memory: 11, processing: 1>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/10
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/12', name: 3, memory: 11, processing: 2>
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[54]': finished task run for task with final state: 'Success'
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/12
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/13', name: 2, memory: 11, processing: 3>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/13
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/14', name: 1, memory: 11, processing: 4>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/14
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/15', name: 19, memory: 10, processing: 5>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/15
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/16', name: 18, memory: 10, processing: 6>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/16
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/17', name: 8, memory: 11, processing: 7>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/17
distributed.batched - INFO - Batched Comm Closed:
distributed.batched - INFO - Batched Comm Closed:
distributed.batched - INFO - Batched Comm Closed:
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/18', name: 17, memory: 10, processing: 9>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/18
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/19', name: 16, memory: 10, processing: 10>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/19
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/20
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[152]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[114]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[136]': finished task run for task with final state: 'Success'
distributed.batched - INFO - Batched Comm Closed:
distributed.batched - INFO - Batched Comm Closed:
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/21
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/22
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[132]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[172]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[52]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[31]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/20', name: 15, memory: 10, processing: 13>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/20
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/21', name: 14, memory: 11, processing: 17>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/21
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/22', name: 13, memory: 11, processing: 21>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/22
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[152]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Comm closed
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[95]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[71]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[55]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[96]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[137]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[32]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[33]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[95]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[71]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[55]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[96]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[33]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[137]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[32]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[23]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[7]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[9]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'get_data': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[3]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[10]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[0]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[23]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[9]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[51]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[7]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'get_data': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[8]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[0]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[10]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[3]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[6]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[5]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[1]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[53]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[22]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[51]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[45]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[8]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[4]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[6]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[5]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[12]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[11]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[1]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[22]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[53]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[30]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[2]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[29]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[45]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[40]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[4]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[20]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[12]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[35]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[11]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[41]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[30]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[2]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[29]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[47]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[21]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[40]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[46]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[20]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[31]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[35]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[50]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[37]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[41]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[27]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[47]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[46]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[21]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[54]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[38]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[26]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[50]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[31]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[37]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[27]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[44]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[70]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[28]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[42]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[54]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[38]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[26]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[48]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[49]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[67]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[44]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[28]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[70]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[42]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[90]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[34]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[63]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[25]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[48]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[49]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[36]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[67]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[24]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[69]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[90]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[34]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[63]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[25]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[103]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[52]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[130]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[39]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[36]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[102]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[24]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[69]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[43]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[62]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[103]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[52]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[130]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[39]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[80]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[124]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[66]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[77]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[102]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[97]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[43]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[62]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[64]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[65]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[80]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[66]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[124]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[77]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[57]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[128]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[83]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[97]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[94]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[100]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[64]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[88]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[65]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[84]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[57]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[92]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[128]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[83]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[94]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[100]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[88]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[76]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[109]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[84]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[87]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[79]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[82]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[72]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[92]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[74]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[76]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[109]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[56]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[79]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[87]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[82]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[136]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[72]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[89]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[58]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[86]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[74]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[98]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[68]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[56]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[112]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[136]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[106]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[89]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[58]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[86]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[98]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[68]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[81]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[117]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[101]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[99]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[85]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[112]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[129]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[106]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[118]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[81]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[61]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[101]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[117]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[85]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[99]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[114]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[113]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[129]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[93]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[118]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[91]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[120]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[135]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[61]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[132]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[113]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[114]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[105]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[121]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[93]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[91]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[73]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[120]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[75]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[135]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[116]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[115]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[132]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[107]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[105]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[121]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[125]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[127]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[73]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[123]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[75]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[116]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[115]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[59]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[107]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[138]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[160]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[125]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[127]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[123]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[126]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[134]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[131]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[111]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[59]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[60]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[138]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[165]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[160]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[147]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[126]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[148]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[111]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[134]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[131]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[60]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[104]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[119]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[152]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[78]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[165]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[153]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[147]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[154]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[148]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[163]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[104]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[152]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[119]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[78]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[122]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[153]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[141]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[110]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[133]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[154]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[161]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[163]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[144]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[164]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[122]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[133]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[157]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[141]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[110]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[150]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[161]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[145]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[108]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[164]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[159]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[144]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[142]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[162]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[157]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[169]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[150]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[158]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[145]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[108]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[159]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[140]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[149]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[156]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[142]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[162]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[139]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[151]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[169]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[174]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[158]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[143]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[140]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[149]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[156]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[146]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[168]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[139]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[170]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[151]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[155]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[171]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[174]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[177]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[143]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[172]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[146]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[167]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[168]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[170]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[176]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[166]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[171]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[155]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[177]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[178]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[172]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[179]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[173]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[175]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[167]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[176]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[166]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[178]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[179]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[173]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[175]': finished task run for task with final state: 'Success'
distributed.worker - WARNING - Compute Failed
Function: _maybe_run
args: ('prefect-ea630327465142ab9810ea3f6966882e', <function run_task at 0x7fdaad74c310>)
kwargs: {'task': <Task: accum>, 'state': None, 'upstream_states': {<Edge(key=data, mapped=False, flattened=False): inc to accum>: <Mapped: "Ready to proceed with mapping.">}, 'context': {'config': <Box: {'debug': False, 'home_dir': '/Users/jcristharif/.prefect', 'backend': 'cloud', 'server': {'host': 'http://localhost', 'port': 4200, 'host_port': 4200, 'endpoint': 'http://localhost:4200', 'database': {'host': 'localhost', 'port': 5432, 'host_port': 5432, 'name': 'prefect_server', 'username': 'prefect', 'password': 'test-password', 'connection_url': 'postgresql://prefect:test-password@localhost:5432/prefect_server', 'volume_path': '/Users/jcristharif/.prefect/pg_data'}, 'graphql': {'host': '0.0.0.0', 'port': 4201, 'host_port': 4201, 'debug': False, 'path': '/graphql/'}, 'hasura': {'host': 'localhost', 'port': 3000, 'host_port': 3000, 'admin_secret': '', 'claims_namespace': 'hasura-claims', 'graphql_url': 'http://localhost:3000/v1alpha1/graphql', 'ws_url': 'ws://localhost:3000/v1alpha1/graphql',
Exception: AttributeError("'NoneType' object has no attribute 'result'")
distributed.scheduler - INFO - Remove client variable-prefect-ea630327465142ab9810ea3f6966882e
distributed.scheduler - INFO - Remove client Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.scheduler - INFO - Remove client Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.scheduler - INFO - Close client connection: Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
Issue Analytics
- State:
- Created 3 years ago
- Comments:8 (8 by maintainers)
Top Results From Across the Web
GitHub Discussions: An exploratory study of early adoption
Discussions is a new feature of GitHub for asking questions or discussing topics outside of specific Issues or Pull Requests.
Read more >Physical Layer Security for Beyond 5G - IEEE Xplore
Latency increases accrue because data must propagate through the lower layers before communication can begin. However, several systems such as ...
Read more >A Secure and Lightweight D2D Communication System With ...
This proposed scheme can be used for determining if these two devices are physically close or not according to the time consumption of...
Read more >Distributed Neural Precoding for Hybrid mmWave MIMO ...
MIMO Communications with Limited Feedback ... address this issue, hybrid analog-and-digital precoding was proposed for mmWave MIMO system ...
Read more >Dask for Machine Learning — Dask Examples documentation
Scheduler ; Comm: inproc://10.1.1.64/7078/1, Workers: 1 ; Dashboard: http://10.1.1.64:8787/status, Total threads: 4 ; Started: Just now, Total memory: 1.86 GiB ...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Ok, I’ve figured this out, it’s a bug in the connect timeout logic in distributed. PR incoming.
Fixed by #4176, closing.