[tune] get_checkpoint_paths fails due to glob command for .tune_metadata file

TrainableUtil.get_checkpoints_paths does not recognize the .tune_metadata files due to the fact that glob.glob does not match files starting with ., see the end of the glob documentation.

https://github.com/ray-project/ray/blob/e0573df337d9d6ec4298e4d3dce056e071abe9e4/python/ray/tune/utils/trainable.py#L151-L156

Possible solution: It seems that only .tune_metadata files are used throughout the library, so one could just omit glob.glob, e.g.

metadata_file = os.path.join(chkpt_dir, ".tune_metadata")
if not os.path.isfile(metadata_file):
     raise ValueError( 
         "{} has no tune_metadata.".format(chkpt_dir))

Ray version: 1.0.1.post1 Python version: 3.7.3 (issue should be independent of the python version)

[vuoristo]

Hi @krfricke, it leads to problems when using the experiment analysis tools with a trial directory instead of a trial instance.

I have created a colab notebook to demonstrate the problem: https://colab.research.google.com/drive/12yc4xH4yYKlwbnVZEwkX44n7EUnj7PWp?usp=sharing

For anyone who runs into this issue: I don’t think this is explicitly documented anywhere but you can get around it by making sure you return the full path to your checkpoint, not just the checkpoint dir in save_checkpoint. E.g.

def save_checkpoint(self, tmp_checkpoint_dir):
    return os.path.join(tmp_checkpoint_dir, 'checkpoint')

Not

def save_checkpoint(self, tmp_checkpoint_dir):
    return tmp_checkpoint_dir

[karstenddwx]

Hi, I’ve run into similar issues with trial_path having special characters. No checkpoint paths returned as glob doesn’t match special characters.

If hyper parameters are set by tune/grid_search, the parameters get part of the trial_path. If in that case the hyperparameter is a list, … e.g. “model”: { “fcnet_hiddens”: grid_search([[256, 256]]), }

…the resulting trial_path looks like that and is not working with TrainableUtil.get_checkpoints_paths(trial_path). DQN_simple_env_0_fcnet_hiddens=[256, 256]_2021-01-06_11-52-03

glob doesn’t match ‘[’ and ‘]’.