ManagedDataAccess.Core - unpredictable pooling errors between operating systems

Hello,

We’re trying to use Oracle.ManagedDataAccess.Core with .NET Core 3.1 running on Linux (Debian 10) Docker container, but we’re experiencing critical pooling errors. Docker host is Ubuntu 16.04.

Environment:

Oracle.ManagedDataAccess.Core - 2.19.60 .NET Core - 3.1 Oracle database - 12.1.0.2 Standard Edition, running on Azure virtual machine Azure public IP address configuration - drop idle TCP connections after 4 minutes of inactivity (default config). Docker host operating system - Ubuntu 16.04 Container base image - Debian 10 (mcr.microsoft.com/dotnet/core/runtime:3.1-buster-slim)

Expected behaviour:

After 4 minutes of inactivity, TCP connection to Oracle database is dropped, but provider can detect that and reconnects without errors and without significant delay.

Actual behaviour:

When the connection is dropped and connection is returned from the pool for another operation the command that is about to be executed freezes for approx. 15 minutes and then throws ORA-12570 exception.

Code for reproducing error:

Waiting for 5 minutes to be sure Azure Load Balancer drops idle TCP connection to database.

Program.cs:

using System;
using System.Threading.Tasks;
using Oracle.ManagedDataAccess.Client;

namespace OracleTest
{
    class Program
    {
        static async Task Main(string[] args)
        {
            for (var i = 0; i < 100; i++)
            {
                await Select();
                Log("Waiting for 5:00m");
                await Task.Delay(TimeSpan.FromMinutes(5));
                Log("Wait ended");
            }
        }

        static async Task Select()
        {
            try
            {
                var connectionString = "Data Source=<azure-vm-ip>:1521/cdb1;User Id=<user>;Password=<pass>;";
                await using var connection = new OracleConnection(connectionString);

                await connection.OpenAsync();

                var selectCommand = new OracleCommand("select 'success' from dual", connection);
                Log("Sending command");
                var reader = (OracleDataReader)await selectCommand.ExecuteReaderAsync();
                Log("Command executed");
                while (await reader.ReadAsync())
                {
                    Log("Result " + reader.GetFieldValue<string>(0));
                }
                await reader.CloseAsync();
                Log("Reader closed");
            }
            catch (Exception e)
            {
                Log(e.Message);
            }
        }

        static void Log(string msg)
        {
            Console.WriteLine($"[{DateTime.Now.ToString("HH:mm:ss.fff")}] {msg}");
        }
    }
}

.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.9.10" />
    <PackageReference Include="Oracle.ManagedDataAccess.Core" Version="2.19.60" />
  </ItemGroup>

</Project>

And now the weirdest part. This behaviour varies between different configurations of operating system (details below).

1. Windows 10 1909 (Azure VM/local PC) - code running directly without Docker container

ORA-12570 thrown after ~15-20 seconds when provider tries to use dropped connection returned from pool. Output:

[19:08:45.968] Sending command
[19:08:46.238] Command executed
[19:08:46.293] Result success
[19:08:46.298] Reader closed
[19:08:46.315] Waiting for 5:00m
[19:13:46.345] Wait ended
[19:13:46.354] Sending command
[19:14:07.973] ORA-12570: Network Session: Unexpected packet read error

Works correctly when connection.KeepAlive = true:

[19:31:07.841] Sending command
[19:31:08.017] Command executed
[19:31:08.041] Result success
[19:31:08.054] Reader closed
[19:31:08.087] Waiting for 5:00m
[19:36:08.094] Wait ended
[19:36:08.110] Sending command
[19:36:08.221] Command executed
[19:36:08.221] Result success
[19:36:08.221] Reader closed
[19:36:08.222] Waiting for 5:00m

2. Host - Windows 10 1909 (Azure VM/local PC) . Code running on Docker container using base image specified above (mcr.microsoft.com/dotnet/core/runtime:3.1-buster-slim)

The only configuration that works correctly. Application doesn’t throw any errors. This configuration can work for all day without throwing any error.

Output:

[19:09:04.809] Sending command
[19:09:05.019] Command executed
[19:09:05.048] Result success
[19:09:05.064] Reader closed
[19:09:05.094] Waiting for 5:00m
[19:14:05.097] Wait ended
[19:14:05.113] Sending command
[19:14:05.236] Command executed
[19:14:05.236] Result success
[19:14:05.236] Reader closed
[19:14:05.237] Waiting for 5:00m

With connection.KeepAlive = true still works correctly (as expected):

[19:29:59.188] Sending command
[19:29:59.352] Command executed
[19:29:59.368] Result success
[19:29:59.373] Reader closed
[19:29:59.391] Waiting for 5:00m
[19:34:59.442] Wait ended
[19:34:59.453] Sending command
[19:34:59.559] Command executed
[19:34:59.559] Result success
[19:34:59.559] Reader closed
[19:34:59.560] Waiting for 5:00m

3. Ubuntu 16.04 (Azure VM) - code running directly without Docker container

ORA-12570 thrown after ~15-20 minutes when provider tries to use dropped connection returned from pool.

[19:12:00.083] Sending command
[19:12:00.138] Command executed
[19:12:00.283] Result success
[19:12:00.287] Reader closed
[19:12:00.299] Waiting for 5:00m
[19:17:00.300] Wait ended
[19:17:00.309] Sending command
[19:32:47.955] ORA-12570: Network Session: Unexpected packet read error

Setting connection.KeepAlive = true doesn’t change anything.

[19:40:22.521] Sending command
[19:40:22.560] Command executed
[19:40:22.570] Result success
[19:40:22.574] Reader closed
[19:40:22.587] Waiting for 5:00m
[19:45:22.592] Wait ended
[19:45:22.603] Sending command
[20:01:09.841] ORA-12570: Network Session: Unexpected packet read error

4. Host - Ubuntu 16.04 (Azure VM). Code running on Docker container using base image specified above.

ORA-12570 thrown after ~15-20 minutes when provider tries to use dropped connection returned from pool.

[18:57:07.295] Sending command
[18:57:07.315] Command executed
[18:57:07.315] Result success
[18:57:07.315] Reader closed
[18:57:07.315] Waiting for 5:00m
[19:02:07.316] Wait ended
[19:02:07.317] Sending command
[19:17:58.290] ORA-12570: Network Session: Unexpected packet read error

Setting connection.KeepAlive = true doesn’t change anything.

[19:36:07.779] Sending command
[19:36:07.821] Command executed
[19:36:07.832] Result success
[19:36:07.836] Reader closed
[19:36:07.849] Waiting for 5:00m
[19:41:07.850] Wait ended
[19:41:07.859] Sending command
[19:56:59.174] ORA-12570: Network Session: Unexpected packet read error

With pooling disabled the application doesn’t freeze but that’s not the way it should be fixed (with pooling disabled and high number of concurrent requests we’re getting another error indicating that db cannot create more connections).

Any ideas why this works inconsistently and how to make pooling work correctly regardless of Docker host operating system?

[alexkeh]

In the case of ODP.NET Core, you would have to use sqlnet.ora. Easy Connect Plus can also set EXPIRE_TIME as well.

[alexkeh]

@rajsharmabannu The 19.10 release will occur this month. We’re wrapping up some testing. It could be released as early as this week.

Without a discernible pattern (and little understanding what your app is doing when it fails), I don’t have any educated guesses.

If the problem is the firewall and you can’t change it, you can use the ODP.NET KeepAlive setting in your app to send a probe packet to ensure the connection isn’t severed by the firewall. You can ask your firewall team what is the TCP timeout for idle connections they have set and can it be increased.

To track down the problem, open up a service request with Oracle Support to work on getting diagnostics needed to lock down the error pattern and isolate the possible root causes. If the problem happens early on, collect some ODP.NET traces for Oracle Support to compare a working situation vs. the one with errors.