Expired token exception after upgrade to 11.2.0.jre11

Versions

Driver version: JDBC driver version: 11.2.0.jre11 (com.microsoft.sqlserver:mssql-jdbc:11.2.0.jre11)

SQL Server version: Microsoft SQL Azure (RTM) - 12.0.2000.8 Aug 10 2022 15:14:09 Copyright © 2022 Microsoft Corporation

Client Operating System: Linux on Azure App Service, single instance Docker base image: eclipse-temurin:11-jre-alpine

JAVA/JVM version: Java 11

Java Spring: 2.7.2

JDBC connection string: jdbc:sqlserver://<db-server-name>.database.windows.net:1433;database=<db-name>;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;Authentication=ActiveDirectoryMSI;

Problem description

Since we upgraded from 9.4.1.jre11 to 11.2.0.jre11 some days ago, we get failures periodically.

We updated 4 Microservices and all of them show the new bevhaviour: They are mostly working normally, but after some time (sometimes around a day), the one or more connection from the pool fail for around an hour (which generates around 80 exceptions in that time).

For example, we had the behaviour twice for the last 3 days (only a single service shown):

Exception

org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms. at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:83) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:330) … Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128) at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:159) at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:117) at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) … 119 more Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user ‘<token-identified principal>’. Token is expired. ClientConnectionId:<uuid> at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:265) at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:300) at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:133) …

We think the JDBC token renewal process was changed with the newer version, but we don’t know what exactly causes the exception. We read somewhere that Azure does a health check on active connections every now and then (10 hours?) but we do not see why this should now fail when it was working before.

[Jeffery-Wasty]

Hi @akleemans,

This seems to be very similar to another ongoing issue of ours, also with Hikari. Can you send us the configuration used in Hikari? I think ‘maximum-pool-size’ is what we’re most concerned with now, but all the information you can send will be helpful. Additionally, is your application taking advantage of Idle Connection Resiliency? Finally, can you please try running with these custom jars, and sending us the resulting logs. They have custom logging that should provide more insight around token expiry. If this is a token issue, this may help as well.

Custom_Logging_Jars.zip

[Jeffery-Wasty]

Hi @akleemans,

Just wanted to check in and see whether the issue persists, and whether you have tried any of the solutions described above. If there is no further reply, we’ll mark the issue as closed. Thanks!