create_lr_scheduler_with_warmup does not change init_lr to proper value

Hi,

In order to get expected sequence of lrs from create_lr_scheduler_with_warmup’s scheduler, one must not attach it to engine on event EPOCH_COMPLETED because it produces the lr passed to optimizer’s constructor at the beginning and then warmup lrs. This hurts warmup procedure. As a workaround, one could use event EPOCH_STARTED but it might not be a good solution.

It seems there should be something like below in line 1017 of param_scheduler.py within the for loop .

param_group['lr'] = warmup_start_value

To reproduce current behaviour:

param = nn.Parameter(torch.Tensor([3.]))
optimizer = torch.optim.SGD([param], lr=1e-3)
scheduler = StepLR(optimizer, 3)
with_warmup_scheduler = create_lr_scheduler_with_warmup(scheduler, warmup_start_value=1e-5, warmup_duration=3)

def process_func(e,b):
  param.grad = torch.Tensor([1.])
  optimizer.step()
trainer = Engine(process_func)
@trainer.on(Events.EPOCH_COMPLETED)
def _():
  print(op.param_groups[0]['lr'])
trainer.add_event_handler(Events.EPOCH_COMPLETED, with_warmup_scheduler)

output:

0.001
1e-05
0.000505
0.001
0.001
0.001
0.0001
0.0001
0.0001
1e-05

[vfdev-5]

@sadra-barikbin you can simply schedule your post-warm-up epoch-wise scheduling using iterations if possible. Otherwise, you can try to combine events as below:

import torch
import torch.optim as optim
from torch.optim.lr_scheduler import ExponentialLR

from ignite.engine import Engine, Events
from ignite.handlers import create_lr_scheduler_with_warmup


def train_step(e, b):
    print(trainer.state.epoch, trainer.state.iteration, " | ", optimizer.param_groups[0]["lr"])

    
trainer = Engine(train_step)
optimizer = optim.SGD([torch.tensor([0.1])], lr=0.1234)


torch_lr_scheduler = ExponentialLR(optimizer=optimizer, gamma=0.5)

data = [0] * 8
epoch_length = len(data)
warmup_duration = 5
scheduler = create_lr_scheduler_with_warmup(torch_lr_scheduler,
                                            warmup_start_value=0.0,
                                            warmup_duration=warmup_duration)

# Trigger scheduler on iteration_started events before reaching warmup_duration
combined_events = Events.ITERATION_STARTED(event_filter=lambda _, __: trainer.state.iteration <= warmup_duration)
# Trigger scheduler on epoch_started events after the warm-up. Epochs are 1-based, thus we do 1 + 
combined_events |= Events.EPOCH_STARTED(event_filter=lambda _, __: trainer.state.epoch > 1 + warmup_duration / epoch_length)
trainer.add_event_handler(combined_events, scheduler)
   
trainer.run(data, max_epochs=10)
> 
1 1  |  0.0
1 2  |  0.03085
1 3  |  0.0617
1 4  |  0.09255
1 5  |  0.1234
1 6  |  0.1234
1 7  |  0.1234
1 8  |  0.1234
2 9  |  0.0617
2 10  |  0.0617
2 11  |  0.0617
2 12  |  0.0617
2 13  |  0.0617
2 14  |  0.0617
2 15  |  0.0617
2 16  |  0.0617
3 17  |  0.03085
3 18  |  0.03085
3 19  |  0.03085
3 20  |  0.03085
3 21  |  0.03085
3 22  |  0.03085
3 23  |  0.03085
3 24  |  0.03085
4 25  |  0.015425
4 26  |  0.015425
4 27  |  0.015425
4 28  |  0.015425
4 29  |  0.015425
4 30  |  0.015425

By the way, our docs on create_lr_scheduler_with_warmup is incorrect, cc @sdesrozis . We should trigger scheduler on ITERATION_STARTED instead of ITERATION_COMPLETED if we would like to avoid the first iteration to take optimizer’s default value.

[trsvchn]

@sadra-barikbin so basically you add example to this examples and it will rendered on the main website (the new one) and we can reference it then.

please check the contributing guide for the examples and do not hesitate to ask for the help!