[RFC] Support easy-to-use Storage in Multi-process on the single server.

Motivation

In the current implementation, there is no option to run Optuna in multi process without setup such as RDB. There is a need for a function that can be executed in a multi process without any special setup on the single server.

Note

For the case of Multi-process on the multiple severs, some users want to run distributed optimization without RDB setting. That issue will be addressed in another issue if the demand is high.

Description

We will select a solution from the follows:

1. We will not do anything for it
2. Check if the problem in SQLite ( https://github.com/optuna/optuna/issues/820 ) can be solved, and implement and provide it if possible.
3. Implemented and provided within the Optuna module, such as https://github.com/optuna/optuna/pull/2759
4. Implemented and provided using 3rd party module such as Dask like https://github.com/optuna/optuna/pull/2023

The solution will be selected from the following viewpoints, and the scope of change (e.g. whether a major release is required or not) will also be investigated.

• Usability
  • Demand for the functionality
  • Quality of the functionality
  • Environment / conditions that can provide the functionality
• Implementation / maintenance cost

TODOs

• Select a solution for multiprocess in-memory storage
• Implement it
  • At first, we try to resolve this issue by 2: Check if the problem in SQLite can be solved, and implement and provide it if possible.
  • If not, we will try to resolve this issue by others.

[keisuke-umezawa]

@MaksimEkin Thank you for the comment! We have just published Optuna-V3-Roadmap and this issue is on it. We will soon resolve it.

[keisuke-umezawa]

Yes!

For the case of Multi-process on the multiple severs, some users want to run distributed optimization without RDB setting. That issue will be addressed in another issue if the demand is high.

If we want to do something for it, we will create another issue.