Why is the walltime is set by default?

I was wondering why the default walltime for workers is set to 30 minutes instead of None?

For example, see https://github.com/dask/dask-jobqueue/blob/1f9ae1ecc79a5b76930e56c8801f3b8ace659877/dask_jobqueue/jobqueue.yaml#L71

I kept having failed jobs for 2 days until I realized that it’s actually this library is setting a hard limit for my jobs, why? This was very hard to debug among all the other errors that I was trying to fix.

If you want your users to be aware of walltime, isn’t there a better solution than setting it to a random number? Can’t you make it a required variable? Can’t the default value be something like "not-set-by-user" and complain if it’s not set?

[jacobtomlinson]

I’m definitely in favour of documenting it.

I am also in favour of logging output to the user when we create cluster managers for them. In dask-cloudprovider we log some information about the cluster object when it gets instantiated and that information is also available at cluster.get_logs(). We log things like VM type, docker image, region, etc. I think other cluster managers should start implementing this too.

[guillaumeeb]

OK, what you say makes complete sense!

So to fix this issues, we should both :

• update documentation to inform about the default setting (and explaining it a bit),
• log some output with the jobqueue cluster main settings by implementing the _log method of the Cluster object.