How to scale aim repo to support simultaneous writes from multiple training runs

❓Question

in order to compare multiple training runs side by side, I tried make them write to the same aim repo located on a shared EFS. However when I do this, I see a huge number of error logs like below printed in some of those runs:

Traceback (most recent call last):
  File "src/aimrocks/lib_rocksdb.pyx", line 89, in aimrocks.lib_rocksdb.check_status
aimrocks.errors.RocksIOError: b'IO error: While appending to file: /mnt/model_factory_pipeline_data/experiment_tracking/aim/.aim/seqs/chunks/fa74e34e388248a68339f974/000012.log: Bad file descriptor'
Exception ignored in: 'aimrocks.lib_rocksdb.DB.write'
Traceback (most recent call last):
  File "src/aimrocks/lib_rocksdb.pyx", line 89, in aimrocks.lib_rocksdb.check_status
aimrocks.errors.RocksIOError: b'IO error: While appending to file: /mnt/model_factory_pipeline_data/experiment_tracking/aim/.aim/seqs/chunks/fa74e34e388248a68339f974/000012.log: Bad file descriptor'

My questions are:

• Does aim repo current support simultaneous writes from multiple training runs?
• If yes, how scalable is that? In my case I encountered these errors when just running 3 simultaneous jobs
• If not, is there a plan to support that?
• Could this be related to that fact that I’m using EFS to store the repo? If so, are there better alternatives?

[jiyuanq]

To share more information:

1. The two jobs started encountering the issue at around the same time
2. While training logs stopped updating at minibatch 24500-ish, it looks like aim metrics were updated to 40k minibatches, with a weird pattern (see below), so i guess training actually continued for a while it’s just no more logs where printed?
3. The EFS I’m using for aim is also used by other use cases, and I found that it’s network IO usage can sometimes reach almost 100%, but i’m not sure how that may impact aim writes

I think what I can try on my end is to see if switching to another EFS with more network bandwidth would help. At the same time I also hope aim can provide a robust way of writing experiment logs

[jiyuanq]

Update: I switched to a new EFS with no other users, and tried 7 runs today. 2 of those runs still hit the same issue… I’ll try remote tracker as well

@jiyuanq, are those two runs started from scratch or they are the same which were failing before?

I’m running experiments for hyperparameter tuning, so all the 7 runs are similar to the failed ones except some hyperparameter values. (eg. activation function)