bug: `Task.set_initial_iteration` doesn't affect the server

I want to report two problems:

1. The docs for the parameter iteration of any Logger.report_* method are not matching what happens.
2. Setting the initial iteration doesn’t affect the server.

For instance, if I have a clearml.Task instance which is being resumed from iteration 100, the initial iteration (as returned by Task.get_initial_iteration) is 100. Lets say I want to report a scalar for iteration 105 by calling Logger.report_scalar

clearml_logger.report_scalar(value, iteration=105)

However, it gets plotted for iteration 205 on the server (i.e. 100 + 105). Hence the docs should say here that iteration really is an offset to the initial iteration. There would be two ways to do this properly:

1. manually calculate the offset

   clearml_logger.report_scalar(value, iteration=105 - clearml_task.get_initial_iteration())
   

   This works for all manual summaries, however not for the automatically done summaries (GPU / CPU / memory utilization; see #440)
2. Initially set the initial iteration statically to 0, relying on an external recovery of the current iteration. This is broken however. When I do this, my local instance of clearml_task gets updated properly. However,

   clearml_logger.report_scalar(value, iteration=105)
   

   will still cause the server to show that summary value for iteration 205, not 105.

[patzm]

I think this comment is a good summary of the idea already: https://github.com/allegroai/clearml/issues/440#issuecomment-908284422

[bmartinn]

I like the idea. I would see (2) as an extension to (1) though. I would still implement set_last_checkpoint (and thus expose), and then simply call it when detecting that a checkpoint was written

Agreed! Let’s call it a new feature request 😃 @patzm Could you close this issue and update this feature request issue with the discussed design ?