sqlite3.OperationalErrors in hyperparameter tuning

Description

I am trying to tune ComplEx with ampligraph.evaluation.select_best_model_ranking, but I keep getting various sqlite3.OperationalError exceptions at seemingly random iterations of the grid search saying “no such table: triples_table”, “no such table: integrity_check”, etc. I am not modifying my directory structure while the code is running. If I understand correctly this error has something to do with the sqlite DB built for the filter triples.

Actual Behavior

Example:

16it [11:49, 41.87s/it]2020-04-13 06:53:32.031940: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: OperationalError: no such index: triples_table_sp_idx
Traceback (most recent call last):

  File "/x/tsafavi/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
    ret = func(*args)

  File "/x/tsafavi/.local/lib/python3.6/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 594, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "/x/tsafavi/.local/lib/python3.6/site-packages/ampligraph/latent_features/models/EmbeddingModel.py", line 1060, in _test_generator
    out, indices_obj, indices_sub = next(batch_iterator)

  File "/x/tsafavi/.local/lib/python3.6/site-packages/ampligraph/datasets/numpy_adapter.py", line 111, in get_next_batch
    participating_objects, participating_subjects = self.filter_adapter.get_participating_entities(out)

  File "/x/tsafavi/.local/lib/python3.6/site-packages/ampligraph/datasets/sqlite_adapter.py", line 405, in get_participating_entities
    cur1.execute(query1)

sqlite3.OperationalError: no such index: triples_table_sp_idx

Another example for a different hyperparameter configuration:

42it [35:50, 68.89s/it]2020-04-13 08:00:56.142500: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: OperationalError: no such table: integrity_check
Traceback (most recent call last):

  File "/x/tsafavi/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in __call__
    ret = func(*args)

  File "/x/tsafavi/.local/lib/python3.6/site-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 594, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "/x/tsafavi/.local/lib/python3.6/site-packages/ampligraph/latent_features/models/EmbeddingModel.py", line 1060, in _test_generator
    out, indices_obj, indices_sub = next(batch_iterator)

  File "/x/tsafavi/.local/lib/python3.6/site-packages/ampligraph/datasets/numpy_adapter.py", line 111, in get_next_batch
    participating_objects, participating_subjects = self.filter_adapter.get_participating_entities(out)

  File "/x/tsafavi/.local/lib/python3.6/site-packages/ampligraph/datasets/sqlite_adapter.py", line 395, in get_participating_entities
    cur_integrity.execute("SELECT * FROM integrity_check")

sqlite3.OperationalError: no such table: integrity_check

Expected Behavior

Full grid search without errors

Steps to Reproduce

...

X_train, X_valid, X_test = [  # small custom dataset
        load_from_csv(data_dir, fname, sep=sep, header=header)
        for fname in (train, valid, test)]

X_dict = {'train': X_train, 'valid': X_valid, 'test': X_test}

params = {
        'seed': 0,
        'batches_count': [10, 50, 100],  
        'epochs': [500],
        'k': [50, 100, 200],  
        'eta': [1, 5, 10],  
        'loss': ['pairwise', 'nll', 'self_adversarial'],
        'loss_params': {
            'margin': [1, 5],  
            'alpha': [0.5]  
        },
        'embedding_model_params': {
            'negative_corruption_entities': 'all'
        },
        'regularizer': [None, 'LP'],
        'regularizer_params': {
            'p': [3],
            'lambda': [1e-4, 1e-5]
        },
        'optimizer': ['adagrad', 'adam'],
        'optimizer_params': {
            'lr': [0.1, 0.01, 0.0001]
        },
        'verbose': False
}

X_filter = np.concatenate(
        (X_dict['train'], X_dict['valid'], X_dict['test'])
)

return select_best_model_ranking(
        ComplEx,
        X_dict['train'],
        X_dict['valid'],
        X_dict['test'],
        params,
        early_stopping=True,
        early_stopping_params={
            'x_valid': X_dict['valid'],
            'criteria': 'mrr',
            'burn_in': 100,
            'check_interval': 10,
            'stop_interval': 3,
            'x_filter': X_filter,
            'corruption_entities': 'all',
        },
        use_filter=True,
        corrupt_side='s,o',
        verbose=False
)

[tsafavi]

Ubuntu 18.04.2.

[sumitpai]

Sorry. I am unable to reproduce this issue.