Tracking adoption of AzureMLClass

Hello,

We’d like to add a minimal logging capability to the AzureMLCluster. This would only report the following name-value pair: {"AzureMLCluster-Dask": "0.1"} (or whatever the current version is) with the AzureML Service and nothing else – once the cluster is instantiated NO information of what the users do on the Dask Cluster or anything else would be logged.

The only use of this signal would be to account for the adoption of the AzureMLClass: how many jobs were submitted to AzureML using the AzureMLCluster, how many users used it, how many core-hours it generated. All these information would only be used in an aggregated form.

We have a solution ready to go just looking for your agreement that the above is OK.

[drabastomek]

@martindurant that is correct – this is part of the #67 PR. @TomAugspurger @jrbourbeau Totally fine to make this opt-out. Will update the documentation and provide an opt-out mechanism.

Thank you all!

[jrbourbeau]

Thanks for raising this issue @drabastomek. In general, logging the version of dask-cloudprovider used seems reasonable and would be a valuable piece of information to have.

IIUC we don’t log this type of information with other cluster managers (e.g. FargateCluster doesn’t log this information with AWS). Would you be okay adding some documentation so users are aware of what is being logged and some way for users to opt out if they don’t want to send this information (e.g. adding a log_version, or some other name, which can be set to False)?

EDIT: Ah, I missed @TomAugspurger’s comment right before mine. Looks like we’re all in favor of an opt-out option and some documentation describing what’s logged