Close connection after optimize

Expected behavior

I need to build many, fast-training models. For this i’m using multiprocessing.Pool:

with Pool(maxtasksperchild=10) as pool:
    res=pool.map(task,iter)

In each task i’m creating stydy with unique name and run optimize. As storage is used Postgres 12.

After a few cycle, i get error from psycopg2: “too many client” How I can manualy close connection for finished task and study. Or exist anotherway? P.S. I know about the possibility to шncrease quantity of clients on Postgres, but It only test data. Real task is much more.

Environment

• Optuna version: 1.5.0
• Python version: 3.7.7
• OS: Ubuntu 20.04
• psycopg2-binary: 2.8.5
• Postgres version: 12

Error messages, stack traces, or logs

# error messages, stack traces, or logs
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ilya/anaconda3/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/ilya/anaconda3/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "<ipython-input-11-68b9f5f269e0>", line 3, in task
    df:pd.DataFrame=pd.read_sql_query(select_query.format(pan=pan),connect_string)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 332, in read_sql_query
    chunksize=chunksize,
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1218, in read_query
    result = self.execute(*args)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1087, in execute
    return self.connectable.execute(*args, **kwargs)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2243, in execute
    connection = self._contextual_connect(close_with_result=True)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2311, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2349, in _wrap_pool_connect
    e, dialect, self
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1591, in _handle_dbapi_exception_noconnection
    sqlalchemy_exception, with_traceback=exc_info[2], from_=e
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2345, in _wrap_pool_connect
    return fn()
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 364, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
    rec = pool._do_get()
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 140, in _do_get
    self._dec_overflow()
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 137, in _do_get
    return self._create_connection()
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
    return _ConnectionRecord(self)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
    self.__connect(first_connect_check=True)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
    exc_value, with_traceback=exc_tb,
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
    connection = pool._invoke_creator(self)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 490, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/home/ilya/anaconda3/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  sorry, too many clients already

(Background on this error at: http://sqlalche.me/e/e3q8)

[keisuke-umezawa]

@hvy Thank you for addressing to me! I will work on it. At first, I will create multiple plans and designs for fixing the bug.

[keisuke-umezawa]

I will close this issue, because the issue 1 and 2 were resolved.

The issue 1 was resolved by this workaround. https://github.com/optuna/optuna/issues/1347#issuecomment-732042329

And, the issue 2 was also resolved by #2079 .