MOTPESampler StorageInternalError after ~300 iterations

Running a multiobjective optimization using the MOTPESampler, an exception is raised after ~300 iterations when writing to a postgres DB. Apparently, the length of the insert into trial_system_attributes is too long.

• The optuna exceptions is: “optuna.exceptions.StorageInternalError: An exception is raised during the commit. This typically happens due to invalid data in the commit, e.g. exceeding max length.”
• It occurs when inserting into trial_system_attributes and the key: ‘motpe:splitcache’.

Expected behavior

(i) Is there a chance to increase the limit (maybe no limit) of the column, such that this exception is not raised? (ii) If not, can this be handled in another way ? I actually do not need this information myself, so maybe there is an option to not write it to the RDB

Environment

• Optuna version: 2.5.0
• Python version: 3.8.5
• OS: Windows10

Error messages, stack traces, or logs

• This is the error from the example provided below. The one I get is basically the same. Is use three objective functions but this should not matter here!

[I 2021-02-09 10:51:19,607] Trial 298 finished with values: [4.007762489974416, 38.003975116711004] and parameters: {‘x’: 0.3695368246169375, ‘y’: 0.9302597259613226}. [I 2021-02-09 10:51:20,211] Trial 299 finished with values: [49.20740296652893, 12.833723674946304] and parameters: {‘x’: 2.291235054547539, ‘y’: 2.655577652121054}. [I 2021-02-09 10:51:21,086] Trial 300 finished with values: [90.85183690981745, 12.874199880388066] and parameters: {‘x’: 4.5426754735860415, ‘y’: 1.4412004611205884}. [I 2021-02-09 10:51:21,778] Trial 301 finished with values: [24.243756546890843, 23.964240426067477] and parameters: {‘x’: 2.279352836003099, ‘y’: 0.9303170350624244}. [I 2021-02-09 10:51:22,231] Trial 303 finished with values: [35.489315997021954, 24.6078917994922] and parameters: {‘x’: 0.48805437834067955, ‘y’: 2.938389341635649}. [I 2021-02-09 10:51:22,303] Trial 302 finished with values: [29.709922859084305, 19.10484199146676] and parameters: {‘x’: 2.121506619593528, ‘y’: 1.7107572527369035}. [I 2021-02-09 10:51:23,111] Trial 304 finished with values: [37.020124677459904, 24.090399224150516] and parameters: {‘x’: 0.5188215621728609, ‘y’: 2.997641632348585}. [I 2021-02-09 10:51:24,065] Trial 305 finished with values: [35.64562542691311, 24.521412656755626] and parameters: {‘x’: 0.4951533240150495, ‘y’: 2.9438460459822156}. [W 2021-02-09 10:51:24,414] Trial 306 failed because of the following error: StorageInternalError('An exception is raised during the commit. This typically happens due to invalid data in the commit, e.g. exceeding max length. ') Traceback (most recent call last): File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1276, in _execute_context self.dialect.do_execute( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\default.py”, line 593, in do_execute cursor.execute(statement, parameters) psycopg2.errors.StringDataRightTruncation: value too long for type character varying(2048)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_rdb\storage.py”, line 53, in _create_scoped_session session.commit() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 1042, in commit self.transaction.commit() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 504, in commit self._prepare_impl() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 483, in _prepare_impl self.session.flush() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 2536, in flush self._flush(objects) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 2678, in flush transaction.rollback(capture_exception=True) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\util\langhelpers.py”, line 68, in exit compat.raise( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\util\compat.py”, line 182, in raise raise exception File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 2638, in _flush flush_context.execute() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\unitofwork.py”, line 422, in execute rec.execute(self) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\unitofwork.py”, line 586, in execute persistence.save_obj( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\persistence.py”, line 239, in save_obj _emit_insert_statements( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\persistence.py”, line 1135, in _emit_insert_statements result = cached_connections[connection].execute( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1011, in execute return meth(self, multiparams, params) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\sql\elements.py”, line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1124, in _execute_clauseelement ret = self._execute_context( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1316, in _execute_context self.handle_dbapi_exception( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1510, in handle_dbapi_exception util.raise( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\util\compat.py”, line 182, in raise raise exception File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1276, in _execute_context self.dialect.do_execute( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\default.py”, line 593, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.DataError: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(2048)

[SQL: INSERT INTO trial_system_attributes (trial_id, key, value_json) VALUES (%(trial_id)s, %(key)s, %(value_json)s) RETURNING trial_system_attributes.trial_system_attribute_id] [parameters: {‘trial_id’: 307, ‘key’: ‘motpe:splitcache’, ‘value_json’: ‘{“indices_below”: [193, 213, 105, 262, 187, 271, 123, 31, 153, 168, 97, 159, 8, 274, 246, 44, 203, 270, 149, 42, 110, 226, 59, 264, 229, 52, 241, 268 … (1753 characters truncated) … 65, 0.2898335894719627, 0.2834317038837007, 0.25384174600837245, 0.22567013790028984, 0.22467341592524187, 0.17310754919901758, 0.17209539166212695]}’}] (Background on this error at: http://sqlalche.me/e/13/9h9h)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna_optimize.py”, line 211, in _run_trial value_or_values = func(trial) File “optuna_git.py”, line 15, in objective x = trial.suggest_float(“x”, 0, 5) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\trial_trial.py”, line 178, in suggest_float return self.suggest_uniform(name, low, high) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\trial_trial.py”, line 237, in suggest_uniform return self._suggest(name, distribution) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\trial_trial.py”, line 716, in _suggest param_value = self.study.sampler.sample_independent( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\samplers_tpe\multi_objective_sampler.py”, line 182, in sample_independent below_param_values, above_param_values = self._split_mo_observation_pairs( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\samplers_tpe\multi_objective_sampler.py”, line 291, in _split_mo_observation_pairs study._storage.set_trial_system_attr(trial._trial_id, _SPLITCACHE_KEY, attrs) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_cached_storage.py”, line 340, in set_trial_system_attr self._flush_trial(trial_id) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_cached_storage.py”, line 418, in _flush_trial return self._backend._update_trial( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_rdb\storage.py”, line 739, in _update_trial session.add(trial_model) File “C:\Users\sebas\anaconda3\envs\tvp\lib\contextlib.py”, line 120, in exit next(self.gen) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_rdb\storage.py”, line 70, in _create_scoped_session raise optuna.exceptions.StorageInternalError(message) from e optuna.exceptions.StorageInternalError: An exception is raised during the commit. This typically happens due to invalid data in the commit, e.g. exceeding max length. Traceback (most recent call last): File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1276, in _execute_context self.dialect.do_execute( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\default.py”, line 593, in do_execute cursor.execute(statement, parameters) psycopg2.errors.StringDataRightTruncation: value too long for type character varying(2048)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_rdb\storage.py”, line 53, in _create_scoped_session session.commit() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 1042, in commit self.transaction.commit() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 504, in commit self._prepare_impl() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 483, in _prepare_impl self.session.flush() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 2536, in flush self._flush(objects) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 2678, in flush transaction.rollback(capture_exception=True) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\util\langhelpers.py”, line 68, in exit compat.raise( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\util\compat.py”, line 182, in raise raise exception File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\session.py”, line 2638, in _flush flush_context.execute() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\unitofwork.py”, line 422, in execute rec.execute(self) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\unitofwork.py”, line 586, in execute persistence.save_obj( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\persistence.py”, line 239, in save_obj _emit_insert_statements( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\orm\persistence.py”, line 1135, in _emit_insert_statements result = cached_connections[connection].execute( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1011, in execute return meth(self, multiparams, params) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\sql\elements.py”, line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1124, in _execute_clauseelement ret = self._execute_context( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1316, in _execute_context self.handle_dbapi_exception( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1510, in handle_dbapi_exception util.raise( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\util\compat.py”, line 182, in raise raise exception File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\base.py”, line 1276, in _execute_context self.dialect.do_execute( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\sqlalchemy\engine\default.py”, line 593, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.DataError: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(2048)

[SQL: INSERT INTO trial_system_attributes (trial_id, key, value_json) VALUES (%(trial_id)s, %(key)s, %(value_json)s) RETURNING trial_system_attributes.trial_system_attribute_id] [parameters: {‘trial_id’: 307, ‘key’: ‘motpe:splitcache’, ‘value_json’: ‘{“indices_below”: [193, 213, 105, 262, 187, 271, 123, 31, 153, 168, 97, 159, 8, 274, 246, 44, 203, 270, 149, 42, 110, 226, 59, 264, 229, 52, 241, 268 … (1753 characters truncated) … 65, 0.2898335894719627, 0.2834317038837007, 0.25384174600837245, 0.22567013790028984, 0.22467341592524187, 0.17310754919901758, 0.17209539166212695]}’}] (Background on this error at: http://sqlalche.me/e/13/9h9h)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File “optuna_git.py”, line 25, in <module> study.optimize(objective, n_trials=1000, n_jobs=5) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\study.py”, line 376, in optimize _optimize( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna_optimize.py”, line 109, in _optimize parallel( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\joblib\parallel.py”, line 1061, in call self.retrieve() File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\joblib\parallel.py”, line 940, in retrieve self._output.extend(job.get(timeout=self.timeout)) File “C:\Users\sebas\anaconda3\envs\tvp\lib\multiprocessing\pool.py”, line 771, in get raise self._value File “C:\Users\sebas\anaconda3\envs\tvp\lib\multiprocessing\pool.py”, line 125, in worker result = (True, func(*args, **kwds)) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\joblib_parallel_backends.py”, line 595, in call return self.func(*args, **kwargs) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\joblib\parallel.py”, line 262, in call return [func(*args, **kwargs) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\joblib\parallel.py”, line 262, in <listcomp> return [func(*args, **kwargs) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna_optimize.py”, line 164, in _optimize_sequential trial = _run_trial(study, func, catch) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna_optimize.py”, line 262, in _run_trial raise func_err File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna_optimize.py”, line 211, in _run_trial value_or_values = func(trial) File “optuna_git.py”, line 15, in objective x = trial.suggest_float(“x”, 0, 5) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\trial_trial.py”, line 178, in suggest_float return self.suggest_uniform(name, low, high) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\trial_trial.py”, line 237, in suggest_uniform return self._suggest(name, distribution) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\trial_trial.py”, line 716, in _suggest param_value = self.study.sampler.sample_independent( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\samplers_tpe\multi_objective_sampler.py”, line 182, in sample_independent below_param_values, above_param_values = self._split_mo_observation_pairs( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\samplers_tpe\multi_objective_sampler.py”, line 291, in _split_mo_observation_pairs study._storage.set_trial_system_attr(trial._trial_id, _SPLITCACHE_KEY, attrs) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_cached_storage.py”, line 340, in set_trial_system_attr self._flush_trial(trial_id) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_cached_storage.py”, line 418, in _flush_trial return self._backend._update_trial( File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_rdb\storage.py”, line 739, in _update_trial session.add(trial_model) File “C:\Users\sebas\anaconda3\envs\tvp\lib\contextlib.py”, line 120, in exit next(self.gen) File “C:\Users\sebas\anaconda3\envs\tvp\lib\site-packages\optuna\storages_rdb\storage.py”, line 70, in _create_scoped_session raise optuna.exceptions.StorageInternalError(message) from e optuna.exceptions.StorageInternalError: An exception is raised during the commit. This typically happens due to invalid data in the commit, e.g. exceeding max length.

Reproducible examples (optional)

The example is taken from Release Note v2.4.0.

import optuna
def objective(trial):
    # The Binh and Korn function. It has two objectives to minimize.
    x = trial.suggest_float("x", 0, 5)
    y = trial.suggest_float("y", 0, 3)

    v0 = 4 * x ** 2 + 4 * y ** 2
    v1 = (x - 5) ** 2 + (y - 5) ** 2
    return v0, v1
    
storage='postgresql://postgres:xxx@localhost:xxx/GitExample' 
sampler = optuna.samplers.MOTPESampler(n_startup_trials=100, n_ehvi_candidates=50, seed=1)
study = optuna.create_study(directions=["minimize", "minimize"],storage=storage, sampler=sampler)
study.optimize(objective, n_trials=1000, n_jobs=5)

[SammyRamone]

I had the same issue (also using postgresql). I managed to fix this by changing the type of the value_json column in the trial_system_attributes from character varying(2048) to text. This removes the limit and I did not encounter the bug afterwards again (yet). Obviously, this is not a real solution to this issue, as I think there should not be such long entries there in the first place. Still, it provides a workaround until this is fixed. For anyone who wants to perform the same change on the database, do the following:

1. Connect to database: psql DATABASENAME -U USERNAME -h SERVER
2. Change column type: ALTER TABLE trial_system_attributes ALTER COLUMN value_json TYPE text;

[Crissman]

Was unable to replicate this issue using the code above. It appears this issue was fixed by PR #2395 . Please confirm, and close this issue. 🙇‍♂️