Checkpointing fails to save config data because it is a `dict` not a `str`

Describe the bug Saving model checkpoints fails with stack trace:

Traceback (most recent call last):^M
  File "/mnt/nvme/home/dashiell/gpt-neox/train.py", line 27, in <module>^M
    pretrain(neox_args=neox_args)^M
  File "/mnt/nvme/home/dashiell/gpt-neox/megatron/training.py", line 106, in pretrain^M
    iteration = train(^M
  File "/mnt/nvme/home/dashiell/gpt-neox/megatron/training.py", line 613, in train^M
    save_checkpoint(^M
  File "/mnt/nvme/home/dashiell/gpt-neox/megatron/checkpointing.py", line 208, in save_checkpoint^M
    save_ds_checkpoint(iteration, model, neox_args)^M
  File "/mnt/nvme/home/dashiell/gpt-neox/megatron/checkpointing.py", line 201, in save_ds_checkpoint^M
    f.write(config_data)^M
TypeError: write() argument must be str, not dict^M

To Reproduce python3 deepy.py --conf_dir configs 1-3B.yml local_setup.yml

Expected behavior The checkpoint should save without failing.

[dashstander]

I’m on DeeperSpeed and GPT-NeoX main. I actually think this is an(other) SlurmRunner issue. It’s failing specifically with the version of the config files that are saved as strings in the arguments. Those specifically were giving me trouble when implementing the SlurmRunner and there’s Deep(er)Speed logic to basically clean them up. They get written back to strings to get passed in to srun as a command line arg, but it’s too much of a coincidence for me not to suspect it.

Maybe make another issue about loading the checkpoints? Or comment on #673 ? That seems like something we should all be aware of

[haileyschoelkopf]

(Made an issue at #732 for the above error I described)!