Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Run cannot be ended or deleted if process crashed

See original GitHub issue

🐛 Bug

When a training script is canceled before completion or when it crashes, the run remains in In Progress state and cannot be deleted, even when clicking Delete in the run’s settings tab.

Screenshot 2022-10-11 at 16 14 41

To reproduce

Expected behavior

Environment

  • Aim Version 3.14.0
  • Python version
  • pip version
  • OS (e.g., Linux)
  • Any other relevant information

Additional context

Issue Analytics

  • State:open
  • Created a year ago
  • Reactions:2
  • Comments:13 (6 by maintainers)

github_iconTop GitHub Comments

2reactions
Pyrestonecommented, Nov 4, 2022

minimal version of the workaround seems to be something like:

# make sure aim server is not running while you do this (not sure if necessary but I did it for safety)

RUN_HASH=83d793ef7b4b4d12924a2202 # put your stalled hash here
touch .aim/meta/locks/$RUN_HASH
rm .aim/meta/locks/$RUN_HASH.softlock

aim storage reindex --finalize-only

# re-start aim server (e.g. aim up ...)
0reactions
gorarakelyancommented, Nov 4, 2022

There might be need to remove respective locks in .aim/seqs/locks/ as well

Read more comments on GitHub >

github_iconTop Results From Across the Web

Unable to delete EXE after it crashes even though process not ...
Your process is not being terminated all the way. Since it is not list in the task manager, you can use PSKILL to...
Read more >
Attempt to delete the physical volume store while geometry ...
When I increase the energy of primary protons, that these pass throughout the first volume (box), these WARNINGS happen and the run crashed....
Read more >
4 Ways to Solve the Cannot Delete Files Error
To see all the files and running processes, open Start Task Manager and close from there the files by clicking End Task. 2....
Read more >
Elixir, A Little Beyond The Basics - Part 7: supervisors
Crashing Supervisor​​ Except... By design, Supervisors will terminate themselves if a supervised process is being restarted too fast and too often. ...
Read more >
How To Fix explorer.exe Crashing In Windows 10 - YouTube
The “ Process ” tab of the Task Manager window shows you apps and background processes currently running on your PC. Scroll down...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

`.repo_lock.softlock` could not be acquired when multiple slurm jobs are started

Next page

`aim up` is not user-friendly when training on remote machine