question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

When running with Aurora MySQL, the application stops when the second query is issued.

See original GitHub issue

The issue

The following code works fine for local MySQL 5.7, but when run on AWS 5.7.mysql_aurora.2.07.2, it stops working when the second query is issued. Also, synchronous execution will stop when the second query is issued as well.

var hostBuilder = Host.CreateDefaultBuilder(args)
    .ConfigureServices((context, services) =>
    {
        services.AddDbContext<MyDbContext>(options =>
        {
            const string connectionString = "Server=aurora-mysql-master.enhance.xxxxx.jp;Port=3306;Uid=xxxx;Pwd=xxx;Database=xxx;";
            optionsUseMySql(connectionString, new MySqlServerVersion(new Version(5, 7))).EnableDetailedErrors();
        });
        services.AddSingleton<App>();
    });

await hostBuilder.Build().Services
    .GetRequiredService<App>()
    .RunAsync();

public class App
{
    private readonly MyDbContext _myDbContext;
    public App(MyDbContext myDbContext)
    {
        _myDbContext = myDbContext;
    }

    public async Task RunAsync()
    {
        Log.Information("count={count}", await _myDbContext.Users.AsNoTracking().CountAsync());
        Log.Information("count={count}", await _myDbContext.Users.AsNoTracking().CountAsync()); // App Stoped
        //Log.Information("count={count}", _myDbContext.Users.AsNoTracking().Count());
        //Log.Information("count={count}", _myDbContext.Users.AsNoTracking().Count()); // App Stoped
    }
}

From the log, it looks like execution is stopped for a long time at the database connection when the second query is executed.

[10:30:08 DBG] [Microsoft.EntityFrameworkCore.Infrastructure] An 'IServiceProvider' was created for internal use by Entity Framework.
[10:58:03 INF] [Microsoft.EntityFrameworkCore.Infrastructure] Entity Framework Core 6.0.1 initialized 'MyDbContext' using provider 'Pomelo.EntityFrameworkCore.MySql:6.0.1' with options: DetailedErrorsEnabled ServerVersion 5.7-mysql
[10:58:03 DBG] [Microsoft.EntityFrameworkCore.Query] Compiling query expression:
'DbSet<User>()
    .AsNoTracking()
    .Count()'
[10:58:03 DBG] [Microsoft.EntityFrameworkCore.Query] Generated query execution expression:
'queryContext => ShapedQueryCompilingExpressionVisitor.SingleAsync<int>(
    asyncEnumerable: new SingleQueryingEnumerable<int>(
        (RelationalQueryContext)queryContext,
        RelationalCommandCache.SelectExpression(
            Projection Mapping:
                EmptyProjectionMember -> 0
            SELECT COUNT(*)
            FROM USER AS i),
        Func<QueryContext, DbDataReader, ResultContext, SingleQueryResultCoordinator, int>,
        ConsoleApp24.MyDbContext,
        False,
        True,
        True
    ),
    cancellationToken: queryContext.CancellationToken)'
[10:58:03 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Creating DbCommand for 'ExecuteReader'.
[10:58:03 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Created DbCommand for 'ExecuteReader' (11ms).
[10:58:03 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Opening connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxxx.jp'.
[10:58:04 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Opened connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxxx.jp'.
[10:58:04 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Executing DbCommand [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT COUNT(*)
FROM `USER` AS `i`
[10:58:04 INF] [Microsoft.EntityFrameworkCore.Database.Command] Executed DbCommand (35ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT COUNT(*)
FROM `USER` AS `i`
[10:58:04 DBG] [Microsoft.EntityFrameworkCore.Database.Command] A data reader was disposed.
[10:58:04 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Closing connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxxx.jp'.
[10:58:04 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Closed connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxxx.jp'.
[10:58:04 INF] [] count=127797
[10:58:26 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Creating DbCommand for 'ExecuteReader'.
[10:58:26 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Created DbCommand for 'ExecuteReader' (13ms).
[10:58:26 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Opening connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxxx.jp'.

If pooling is disabled in the connection string, the second query will also succeed. We would like to consider other ways to deal with this because of the performance issues.

const string connectionString = "Server=aurora-mysql-master.enhance.xxxxx.jp;Port=3306;Uid=xxxx;Pwd=xxx;Database=xxx;Pooling=false";
[11:36:40 DBG] [Microsoft.EntityFrameworkCore.Infrastructure] An 'IServiceProvider' was created for internal use by Entity Framework.
[11:36:45 INF] [Microsoft.EntityFrameworkCore.Infrastructure] Entity Framework Core 6.0.1 initialized 'MyDbContext' using provider 'Pomelo.EntityFrameworkCore.MySql:6.0.1' with options: DetailedErrorsEnabled ServerVersion 5.7-mysql
[11:36:45 DBG] [Microsoft.EntityFrameworkCore.Query] Compiling query expression:
'DbSet<User>()
    .AsNoTracking()
    .Count()'
[11:36:45 DBG] [Microsoft.EntityFrameworkCore.Query] Generated query execution expression:
'queryContext => ShapedQueryCompilingExpressionVisitor.SingleAsync<int>(
    asyncEnumerable: new SingleQueryingEnumerable<int>(
        (RelationalQueryContext)queryContext,
        RelationalCommandCache.SelectExpression(
            Projection Mapping:
                EmptyProjectionMember -> 0
            SELECT COUNT(*)
            FROM USER AS i),
        Func<QueryContext, DbDataReader, ResultContext, SingleQueryResultCoordinator, int>,
        ConsoleApp24.MyDbContext,
        False,
        True,
        True
    ),
    cancellationToken: queryContext.CancellationToken)'
[11:36:45 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Creating DbCommand for 'ExecuteReader'.
[11:36:45 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Created DbCommand for 'ExecuteReader' (16ms).
[11:36:45 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Opening connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxx.jp'.
[11:36:46 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Opened connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxx.jp'.
[11:36:46 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Executing DbCommand [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT COUNT(*)
FROM `USER` AS `i`
[11:36:46 INF] [Microsoft.EntityFrameworkCore.Database.Command] Executed DbCommand (49ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT COUNT(*)
FROM `USER` AS `i`
[11:36:46 DBG] [Microsoft.EntityFrameworkCore.Database.Command] A data reader was disposed.
[11:36:46 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Closing connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxx.jp'.
[11:36:46 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Closed connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxx.jp'.
[11:36:46 INF] [] count=127797
[11:36:47 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Creating DbCommand for 'ExecuteReader'.
[11:36:47 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Created DbCommand for 'ExecuteReader' (1ms).
[11:36:47 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Opening connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxx.jp'.
[11:36:47 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Opened connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxx.jp'.
[11:36:47 DBG] [Microsoft.EntityFrameworkCore.Database.Command] Executing DbCommand [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT COUNT(*)
FROM `USER` AS `i`
[11:36:47 INF] [Microsoft.EntityFrameworkCore.Database.Command] Executed DbCommand (16ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT COUNT(*)
FROM `USER` AS `i`
[11:36:47 DBG] [Microsoft.EntityFrameworkCore.Database.Command] A data reader was disposed.
[11:36:47 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Closing connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxx.jp'.
[11:36:47 DBG] [Microsoft.EntityFrameworkCore.Database.Connection] Closed connection to database 'xxx' on server 'aurora-mysql-master.enhance.xxxx.jp'.
[11:36:47 INF] [] count=127797

Further technical details

Aurora MySQL Information

  • Engine version 5.7.mysql_aurora.2.07.2

Operating system: Windows Pomelo.EntityFrameworkCore.MySql version: 6.0.1 Microsoft.AspNetCore.App version:

Other details about my project setup:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
    <PackageReference Include="Pomelo.EntityFrameworkCore.MySql" Version="6.0.1" />
    <PackageReference Include="Serilog.Extensions.Hosting" Version="4.2.0" />
    <PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
    <PackageReference Include="Serilog.Settings.Configuration" Version="3.3.0" />
    <PackageReference Include="Serilog.Sinks.Console" Version="4.0.1" />
  </ItemGroup>

</Project>

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9

github_iconTop GitHub Comments

2reactions
mguinnesscommented, Feb 18, 2022

Thanks for doing the research to track down this issue, documented at MySqlConnection.Open Freezes with Amazon Aurora.

The cause of this problem is Amazon Aurora not correctly supporting pipelining in the MySQL protocol. This is known to be a problem with 2.x versions of Aurora (that implement MySQL 5.7), but not with 3.x versions (that implement MySQL 8.0).

1reaction
karuakuncommented, Feb 18, 2022

Resolved.

Added to ConnectionStrings as suggested in the next page.Pipelining = False https://mysqlconnector.net/troubleshooting/aurora-freeze/

const string connectionStrings = "Server=aurora-mysql-master.enhance.xxxx.jp;Port=3306;Uid=xxx;Pwd=xxx;Database=xxx;Pipelining=False";
Read more comments on GitHub >

github_iconTop Results From Across the Web

Working with parallel query for Amazon Aurora MySQL
Each Aurora DB instance can run only a certain number of parallel query sessions at one time. If a query has multiple parts...
Read more >
Troubleshoot problems connecting to Aurora clusters or ...
I am having issues when connecting to an Amazon Relational Database Service (Amazon RDS) cluster or instance that's running Amazon Aurora.
Read more >
Amazon RDS MySQL/Aurora query sometimes hangs ...
Just some contexts: In our old data pipeline system, we are running MySQL 5.6. or Aurora on Amazon rds. Bad thing about our...
Read more >
AWS Aurora running out of freeable memory and crashing
We have an issue with our AWS Aurora MySQL instance running out of freeable memory and crashing as a result. AWS's response was...
Read more >
Aurora Serverless: The Good, the Bad and the Scalable
The documentation specifies long-running queries, transactions, temporary tables, and table locks as reasons why it might not be able to scale.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found