Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

database disk image is malformed

See original GitHub issue

When launching multiple instances of optuna study optimize ~in rapid succession (but by hand)~, I get the following error:

[W 2019-12-26 08:12:45,772] Setting status of trial#13 as TrialState.FAIL because of the following error: DatabaseError('(sqlite3.DatabaseError) database disk image is malformed')

Expected behavior

No error.

Environment

Optuna version: 0.19.0
Python version: Python 3.7.4
OS: CentOS Linux release 7.5.1804 (Core)

Error messages, stack traces, or logs

https://pastebin.com/UPaiAm5t

Steps to reproduce

Launch multiple instances of optuna study optimize ~in rapid succession~.

Additional context (optional)

I’m launching the tasks over LSF on a cluster with a shared file system.

EDIT: the tasks keep failing (even after executing some trials) so it probably isn’t related to launching them in rapid succession.

Issue Analytics

State:
Created 4 years ago
Comments:6 (2 by maintainers)

Top GitHub Comments

1reaction

g-vottecommented, Dec 27, 2019

Thank you for trying your best for this problem.

As you mention, it’s sometimes hard to deploy a server-based RDB on the user’s environment. So, if there’s any workaround to enable distributed optimization on SQLite & NFS, that’s should be documented.

0reactions

louisabrahamcommented, Dec 26, 2019

So, according to the sqlalchemy documentation, I was doing it wrong.

I think the correct way to use VFS is the following: --storage 'sqlite:///file:trials.db?vfs=unix-dotfile&uri=true'

Now I can sometimes see a trials.db.lock file appearing, so I don’t know if it is using unix-dotfile. I’m pretty sure the trials.db.lock file doesn’t appear without the vfs option, but at the same time it doesn’t look like a dotfile.

However, I still have various DB errors, but I think they appear even more frequently with vfs=unix-dotfile.