Connection timeout

I have a massive table (about 4GB) in a MySQL source db that I’m trying to sync with my PostgreSQL db. I keep getting a timeout error and unsure if this is a chameleon issue or a MySQL issue. I don’t see why MySQL would drop a client connection running a long query. Could someone take a look? I’m using version 2.0.5.

2018-04-05 01:43:24 MainProcess INFO mysql_lib.py (640): Copying the source table requestsend into _soomgo_rep_tmp.requestsend
2018-04-05 01:43:24 MainProcess DEBUG mysql_lib.py (452): estimating rows in soomgo.requestsend
2018-04-05 01:43:24 MainProcess DEBUG mysql_lib.py (482): The table soomgo.requestsend will be copied in 2  estimated slice(s) of 52806975 rows
2018-04-05 01:43:24 MainProcess DEBUG mysql_lib.py (420): locking the table `soomgo`.`requestsend`
2018-04-05 01:43:24 MainProcess DEBUG mysql_lib.py (422): collecting the master's coordinates for table `soomgo`.`requestsend`
2018-04-05 01:43:29 MainProcess DEBUG mysql_lib.py (491): Executing query for table soomgo.requestsend
2018-04-05 02:24:50 MainProcess INFO mysql_lib.py (646): Could not copy the table requestsend. Excluding it from the replica.
2018-04-05 02:24:50 MainProcess DEBUG mysql_lib.py (247): Dropping the schema _soomgo_rep_tmp.
2018-04-05 02:24:50 MainProcess DEBUG pg_lib.py (653): Changing the lock timeout for the session to 3600s.
2018-04-05 02:24:55 MainProcess DEBUG mysql_lib.py (247): Dropping the schema _sgforce_rep_tmp.
2018-04-05 02:24:56 MainProcess DEBUG pg_lib.py (653): Changing the lock timeout for the session to 3600s.
2018-04-05 02:24:57 MainProcess DEBUG mysql_lib.py (247): Dropping the schema _sgkit_rep_tmp.
2018-04-05 02:24:57 MainProcess DEBUG pg_lib.py (653): Changing the lock timeout for the session to 3600s.
2018-04-05 02:24:58 MainProcess CRITICAL mysql_lib.py (1198): init replica for source mysql failed
2018-04-05 02:24:58 MainProcess ERROR global_lib.py (55): Could not send the message to rollbar.
Traceback (most recent call last):
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1025, in _read_bytes
    data = self._rfile.read(num_bytes)
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pg_chameleon/venv/bin/chameleon", line 5, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/root/pg_chameleon/venv/bin/chameleon.py", line 56, in <module>
    getattr(replica, args.command)()
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pg_chameleon/lib/global_lib.py", line 318, in init_replica
    self.__init_mysql_replica()
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pg_chameleon/lib/global_lib.py", line 328, in __init_mysql_replica
    self.mysql_source.init_replica()
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pg_chameleon/lib/mysql_lib.py", line 1179, in init_replica
    self.__copy_tables()
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pg_chameleon/lib/mysql_lib.py", line 642, in __copy_tables
    master_status = self.copy_data(schema, table)
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pg_chameleon/lib/mysql_lib.py", line 495, in copy_data
    csv_results = self.cursor_unbuffered.fetchmany(copy_limit)
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/cursors.py", line 485, in fetchmany
    row = self.read_next()
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/cursors.py", line 446, in read_next
    return self._conv_row(self._result._read_rowdata_packet_unbuffered())
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1430, in _read_rowdata_packet_unbuffered
    packet = self.connection._read_packet()
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1008, in _read_packet
    recv_data = self._read_bytes(bytes_to_read)
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1033, in _read_bytes
    "Lost connection to MySQL server during query (%s)" % (e,))
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([Errno 104] Connection reset by peer)')
Exception ignored in: <bound method MySQLResult.__del__ of <pymysql.connections.MySQLResult object at 0x7f3fff2b5780>>
Traceback (most recent call last):
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1345, in __del__
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1447, in _finish_unbuffered_query
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/connections.py", line 991, in _read_packet
  File "/root/pg_chameleon/venv/lib/python3.5/site-packages/pymysql/connections.py", line 1022, in _read_bytes
AttributeError: 'NoneType' object has no attribute 'settimeout'

[the4thdoctor]

hi, it seems that the pymysql or mysql driver disconnects for some reasons. I suggest you two course of action.

I would first change the copy_max_memory: for your the source to a smaller value. I would try “30M”. This will result in a copy with more slices of reduced size.

Then I would increase the connect timeout to a larger value. I suggest connect_timeout: 600 (10 minutes).

[Hoya]

lowering the copy_max_memory did the trick! thanks