Reseting DB through test support endpoint takes a lot of time sometimes

This is a…

[ ] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[X] Bug report  
[ ] Documentation issue or request

Description

I thought it was related to the openstack environment, but I managed to reproduce on minishift as well. The resetDB endpoint is called before each test and after several tests, the resetDB starts to takes more and more time, on openstack I saw the call to take 50 minutes once.

There are some idle connections in the DB that may be related to that:

 pid  |        state        |          query_start          |                                                                                                                                                                                                                     query                                                                                                                                                                                                                      
------+---------------------+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  130 | idle                | 2019-01-18 11:38:26.663967+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
  131 | idle in transaction | 2019-01-18 11:38:25.988674+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
  132 | active              | 2019-01-18 11:38:25.520318+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
  133 | idle                | 2019-01-18 11:37:38.739764+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
  134 | idle                | 2019-01-18 11:17:19.974948+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
  135 | idle                | 2019-01-18 11:33:39.611987+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
  136 | idle in transaction | 2019-01-18 10:07:44.501777+00 | SELECT p.proname,p.oid  FROM pg_catalog.pg_proc p, pg_catalog.pg_namespace n  WHERE p.pronamespace=n.oid AND n.nspname='pg_catalog' AND ( proname = 'lo_open' or proname = 'lo_close' or proname = 'lo_creat' or proname = 'lo_unlink' or proname = 'lo_lseek' or proname = 'lo_lseek64' or proname = 'lo_tell' or proname = 'lo_tell64' or proname = 'loread' or proname = 'lowrite' or proname = 'lo_truncate' or proname = 'lo_truncate64')
  137 | idle                | 2019-01-18 11:12:03.668276+00 | COMMIT
  138 | active              | 2019-01-18 11:38:23.892177+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
  139 | idle in transaction | 2019-01-18 11:19:48.943358+00 | SELECT data FROM filestore WHERE path=$1
 8061 | active              | 2019-01-18 11:38:26.982092+00 | select pid,state,query_start,query from pg_stat_activity where pg_stat_activity.datname = current_database()
(11 rows)

When I restart syndesis-server pod, all connections are cleared and resetDB is almost instant again.

This may be related to the excessive usage of reset db endpoint and customers may not hit that, but I would like to have a confirmation from ENG.

[avano]

@zregvart looks ok now, at least it doesn’t get stuck after the mentioned scenario. I just started the tests and hopefully, everything remains ok 😃

[avano]

For some reason, after creating the AMQ to REST quickstart integration, DB reset seem to be stuck on TRUNCATE TABLE filestore, causing reset db to not finish at all. May this be related to the fact that the extension is in the integration?

 pid  |        state        |          query_start          |                                                                                                                                                                                                                     query                                                                                                                                                                                                                      
------+---------------------+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 1810 | idle                | 2019-01-29 10:29:56.884961+00 | SET extra_float_digits = 3
 1811 | idle                | 2019-01-29 10:29:57.172033+00 | SET extra_float_digits = 3
 1812 | idle                | 2019-01-29 10:29:57.186167+00 | SET extra_float_digits = 3
 1813 | idle                | 2019-01-29 10:29:57.198546+00 | SET extra_float_digits = 3
 1814 | idle                | 2019-01-29 10:29:57.26837+00  | SET extra_float_digits = 3
 1815 | idle in transaction | 2019-01-29 10:34:29.265117+00 | SELECT p.proname,p.oid  FROM pg_catalog.pg_proc p, pg_catalog.pg_namespace n  WHERE p.pronamespace=n.oid AND n.nspname='pg_catalog' AND ( proname = 'lo_open' or proname = 'lo_close' or proname = 'lo_creat' or proname = 'lo_unlink' or proname = 'lo_lseek' or proname = 'lo_lseek64' or proname = 'lo_tell' or proname = 'lo_tell64' or proname = 'loread' or proname = 'lowrite' or proname = 'lo_truncate' or proname = 'lo_truncate64')
 1816 | idle in transaction | 2019-01-29 10:37:39.721377+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
 1817 | idle                | 2019-01-29 10:34:15.488159+00 | COMMIT
 1818 | active              | 2019-01-29 10:37:39.813338+00 | TRUNCATE TABLE filestore
 1819 | idle                | 2019-01-29 10:38:04.907101+00 | select path,value,ovalue from jsondb where path LIKE $1 order by path ASC
  227 | idle                | 2019-01-29 10:38:07.610391+00 |  SELECT pg_database.datname, pg_database_size(pg_database.datname) as size FROM pg_database
 2407 | active              | 2019-01-29 10:38:08.21825+00  | select pid,state,query_start,query from pg_stat_activity where pg_stat_activity.datname = current_database()
(12 rows)

steps to reproduce:

• follow these steps (it is automated official AMQ to REST tutorial): https://github.com/syndesisio/syndesis-qe/blob/master/ui-tests/src/test/resources/features/integrations/amq-to-rest.feature
• try to reset DB through reset-db endpoint