"lr_schedule" option ignored using torch framework and PPO algorithm

Ray version and other system information (Python version, TensorFlow version, OS):

• Ray: 0.9.0.dev0 (2c599dbf05e41e338920ee2fbe692658bcbec4dd)
• CUDA: 10.1
• Pytorch: 1.4.0 with GPU support
• Ubuntu 18.04
• Python 3.6

What is the problem?

Setting the hyperparameter “lr_schedule” as no effect when using PyTorch as backend framework and PPO learning algorithm.

Reproduction (REQUIRED)

import ray 
from ray.rllib.agents.ppo import PPOTrainer, DEFAULT_CONFIG 
 
config = DEFAULT_CONFIG.copy() 
for key, val in { 
    "env": "CartPole-v0", 
    "num_workers": 0, 
    "use_pytorch": False, 
    "lr": 1.0e-5, 
    "lr_schedule": [ 
        [0, 1.0e-6], 
        [1, 1.0e-7], 
    ] 
}.items(): config[key] = val 
 
ray.init() 

for use_pytorch in [False, True]: 
    config["use_pytorch"] = use_pytorch 
    agent = PPOTrainer(config, "CartPole-v0") 
    for _ in range(2): 
        result = agent.train() 
        print(f"use_pytorch: {use_pytorch} - Current learning rate: "\
              f"{result['info']['learner']['default_policy']['cur_lr']}")

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[duburcqa]

Nice, thank you ! The learning rate using tensorflow comes from a conversion from float32 to float64 that must be done somewhere. If you want to check:

import numpy as np
print(np.float64(np.float32(1.0e-5)))
print(np.float64(np.float32(1.0e-7)))

[janblumenkamp]

Hmmm I made some more experiments and I am not convinced that the lr is actually properly updated… Is it possible that the learning rate in cur_lr is different from the actual learning rate?