Aim doesn't launch

Hi!

As the title says, Aim doesn’t launch for some reason. My tf.keras code looks like this:

from aim import Session

# some code

history = model.fit(training_generator,
              validation_data=valid_generator,
              epochs=100,
              callbacks=[
                  AimCallback(session=Session(experiment='test_tf_keras_cb')),
                  save_best_model_callback,
                  tf_board_callback,
                  cp_callback
              ],
              verbose=1)

The training process seems to go as expected, the epochs begin and end, and there’s no message/error/warning from Aim. But when I run Aim commands in the terminal (aim up, aim version, aim upgrade), it kind of freezes: there’s no output for more than an hour. Pressing Ctrl+C aborts the command as usual. Other callbacks such as that of wandb work.

I have tried:

• Running aim up before and during the code execution
• Uninstalling and reinstalling Aim with pip install aim
• Replacing aim.tensorflow with aim.keras, and removing other callbacks

The code is run on Python 3.6.6, in PyCharm 2020.3.3 Professional, Windows 10, on CPU, with Docker Desktop installed, and here is a screenshot from Task Manager.

Is there something I’ve missed? 🙂

[gorarakelyan]

Thanks for the details @aprikyan!

The repo structure is exactly like this. It seems artifacts(metrics, params) are logged properly. I wonder why Aim gets stuck in windows then. Kinda weird behavior. Will troubleshoot it soon. Thanks for your help 😊

[aprikyan]

This could be helpful too. It looks like Aim registers logs for each run in the .aim directory like this:

.aim
│
└───test_tf_keras_cb
    │
    └───d07a118-92ad-11eb-9a6d-1831bfba6804
        │  config.json
        │
        └───objects
            │  meta.json
            │
            └───map
            │   │  dictionary.txt
            │
            └───aimrecords_storage
                │
                └───acc
                    │  bucket_offsets.bin
                    │  current_bucket.bin
                    │  data_chunk_00000.bin
                    │  metadata.json
                    │  record_offsets.bin
                    │
                    └───indices
                        │
                        └───3BNVQjBZ68x9kvpey
                        │   │  index_offsets.bin
                        │
                        └───7VEAAo43RxBXvs
                            │  index_offsets.bin

There are dozens of folders (for each run I think) like this in .aim/test_tf_keras_cb with fancy names, and also an empty folder named “index”.