batch_execute_statement parameter mismatches

Describe the bug Datetime parameters for insert statements sent to Aurora MySQL Serverless by batch_execute_statement will sometimes have incorrect values.

Steps to reproduce

1. Create a new Aurora MySQL 5.7 Serverless instance
2. Execute the CREATE TABLE in the comments of badDBvalues2.py in the attached file badDBvalues2.py.zip
3. Start 2 instances badDBvalues2.py in separate terminal windows

Expected behavior The test Python code creates the same set of rows using batch_execute_statement then reads them back repeatedly. It should never report a mismatch, but it has reported an error for us every time we run it. It may take 10 iterations or a 1000, but it will eventually fail.

Debug logs The following log files from our production environment are attached. These were captured running live applications and not the test script against an Aurora MySQL 5.7 Serverless database:

1. dbBadValues2ParameterSets.txt in dbBadValues2ParameterSets.txt.zip - application logs showing the parameter sets sent to batch_execute_statement. These are the same parameter values used in the test Python code.
2. dbBadValues2.general.txt in dbBadValues2.general.txt.zip - Aurora MySQL general logs showing values actually inserted (includes the CloudWatch log entry as well as the INSERT columns and values parsed into JSON)

The problem occurred at 2021-03-29T12:07:25.561662Z according to the MySQL general log. The values the application passed to batch_execute_statement were:

                {
                    "name": "startedAt",
                    "typeHint": "TIMESTAMP",
                    "value": {
                        "stringValue": "2021-03-29 08:06:01.000000"
                    }
                },
                {
                    "name": "endedAt",
                    "typeHint": "TIMESTAMP",
                    "value": {
                        "stringValue": "2021-03-29 08:06:07.000000"
                    }
                },

And the values shown in the MySQL general logs are: “startedAt”: “2021-03-29 08:06:56.0”, “endedAt”: “2021-03-29 08:06:07.0”,

As you can see, the value for startedAt that was inserted does not match what was passed to batch_execute_statement. This incorrect value does not appear anywhere else in the parameter sets this time, which is different from the first trace, so we have no idea where it came from.

We can find no pattern with this. It occurs rarely and randomly. As far as we can tell, this has only been a problem for datetime columns, but we can’t be sure so it does call into question other data types. It appears the problem lies somewhere between boto3 and Aurora, although we can’t be certain of that.

We have a support case open with AWS on this problem which they are researching, but they suggested also opening an issue here.

badDBvalues2.parameterSets.txt.zip badDBvalues2.py.zip badDBvalues2.general.txt.zip

[jamie-burks]

Omitting the TIMESTAMP hint does seem to get around the issue with the 6 decimal places that we are using. Previously we could not get more than about a thousand rows inserted without an issue, but after omitting the hint we did not see a failure in over 8 million rows inserted.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.