Worker stops indexing new blocks/transactions, gets stuck

Describe the bug I’ve been running the safe-transaction-service (v3.1.24) standalone for ETH mainnet which worked fine in the first instance. However, a second and third instance is experiencing issues which lead to repeated failure of finding transactions (as can be seen in the worker logs. Also the last block which is shown in the database is from a couple of days ago (e.g. block number 13063935).

Is there some specific task I need to restart to get the indexer back into a coherent working mode? Restarting the worker process didn’t help. I tried the task check_index_problems which did show that safe nonces are out of sync, which is expected at that point.

To Reproduce

1. Start a transaction service from scratch against Ethereum mainnet

Expected behavior

• all gnosis safes are indexed, indexer keeps on going

Additional context

Logs:

2021-08-26 07:03:56,674 [INFO] [181d3752/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099516 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:03:56,886 [INFO] [181d3752/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[181d3752-df66-4780-9b78-e4dbb17d1700] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:03:58,391 [INFO] [97407077/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:03:58,456 [INFO] [97407077/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses
2021-08-26 07:04:00,589 [INFO] [97407077/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099516 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:04:00,795 [INFO] [97407077/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[97407077-3392-4bb8-b9f7-76c8f09ef9af] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:04:04,558 [INFO] [6842e41f/index_internal_txs_task] Task safe_transaction_service.history.tasks.index_internal_txs_task[6842e41f-3ae9-4fe0-b854-c89a67258b0e] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0x24c0a32d259fca363b718a2077c8bedc30c4983d12a8ac4d6b666ab51c61ae10')
2021-08-26 07:04:07,321 [INFO] [240083d9/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:04:07,365 [INFO] [240083d9/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses
2021-08-26 07:04:10,322 [INFO] [ccf57d16/index_internal_txs_task] Start indexing of internal txs
2021-08-26 07:04:10,363 [INFO] [ccf57d16/index_internal_txs_task] InternalTxIndexer: Processing 8 not updated addresses
2021-08-26 07:04:11,464 [INFO] [240083d9/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099516 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:04:11,703 [INFO] [240083d9/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[240083d9-362e-42fc-a182-fa7c9db66cf7] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:04:11,886 [INFO] [181d3752/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:04:11,950 [INFO] [181d3752/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses
2021-08-26 07:04:14,261 [INFO] [181d3752/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099516 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:04:14,509 [INFO] [181d3752/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[181d3752-df66-4780-9b78-e4dbb17d1700] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:04:15,794 [INFO] [97407077/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:04:18,699 [INFO] [97407077/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses
2021-08-26 07:04:22,146 [INFO] [97407077/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099516 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:04:22,376 [INFO] [97407077/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[97407077-3392-4bb8-b9f7-76c8f09ef9af] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:04:26,701 [INFO] [240083d9/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:04:26,753 [INFO] [240083d9/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses
2021-08-26 07:04:30,787 [INFO] [240083d9/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099517 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:04:30,998 [INFO] [240083d9/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[240083d9-362e-42fc-a182-fa7c9db66cf7] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:04:35,322 [INFO] [3589eabc/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:04:35,386 [INFO] [3589eabc/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses
2021-08-26 07:04:37,994 [INFO] [ccf57d16/index_internal_txs_task] Found 734 relevant txs with internal txs between block-number=13042668 and block-number=13045068. Addresses=['0x34CfAC646f301356fAa8B21e94227e3583Fe3F5F']
2021-08-26 07:04:37,996 [INFO] [ccf57d16/index_internal_txs_task] InternalTxIndexer: block_process_limit decreased to 2380
2021-08-26 07:04:38,992 [INFO] [3589eabc/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099518 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:04:39,219 [INFO] [3589eabc/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[3589eabc-7d40-4a2d-aed7-20280701299e] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:04:45,997 [INFO] [240083d9/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:04:46,037 [INFO] [240083d9/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses
2021-08-26 07:04:51,232 [INFO] [240083d9/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099520 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:04:51,446 [INFO] [240083d9/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[240083d9-362e-42fc-a182-fa7c9db66cf7] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:04:54,218 [INFO] [3589eabc/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:04:54,271 [INFO] [3589eabc/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses
2021-08-26 07:04:57,743 [INFO] [3589eabc/index_erc20_events_task] Erc20EventsIndexer: Found 2 events from block-number=13095006 to block-number=13099520 for 225 addresses: ['0xcd9B5f73cebF760E7113A71B5f47C4430F42AA28', '0xA8E06CABc30cC4A8f42ADDB461458047Fd9ef59F', '0x8DfdEEfbC84A0C504bb2058fc4c23900Cb0B18e4', '0xFCa9e7387159d46516f59842fb6C6806e2a18D0A', '0x35C30d31B3B9D2c684d0D4FCD4dC1fd3E554bDb8', '0xb891ad9b23826e61F010D5cd90d6a24Ea55010Bd', '0xBcb9E0951268685fd218758d2bAFBeE9113554c3', '0xaF43006604c640f04ae9a9a0169d600F2f5677b0', '0x20dcD414A3D6E57e92373871715FB9343c43B1a3', '0x954A1dBF025876cEBF1211eAF0b1062b31B3Aa21']
2021-08-26 07:04:57,999 [INFO] [3589eabc/index_erc20_events_task] Task safe_transaction_service.history.tasks.index_erc20_events_task[3589eabc-7d40-4a2d-aed7-20280701299e] retry: Retry in 15s: TransactionWithoutBlockException('Cannot find blockNumber for tx with tx-hash=0xacc1835abbeb0aff3fa41fb5346ed663fc0d53cca49bc3072b0f0da1bec71069')
2021-08-26 07:05:03,337 [INFO] [18a2bf2c/index_erc20_events_task] Start indexing of erc20/721 events
2021-08-26 07:05:03,389 [INFO] [18a2bf2c/index_erc20_events_task] Erc20EventsIndexer: Processing 225 almost updated addresses

[tolbrino]

That fixed the issue.

@Uxio0 @jpalvarezl Thanks so much for your help!

[tolbrino]

I will put the setup I’ve created in a repo and send you a link, in case you want to reference it.