Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

The TDS protocol stream is not valid. Unexpected token unknown token (0xA3)

See original GitHub issue

Driver version

Driver: Microsoft JDBC Driver 8.2 for SQL Server Driver version: 8.2.2.0

SQL Server version

Azure SQL DB (Application logs show version 12.00.2000)

Client Operating System

Host Operating System is Windows Server 2016, version 10.0 Java version = 1.8.0_241, Java Runtime Version = 8.0.6.5 - pwa6480sr6fp5-20200111_02(SR6 FP5), Java Compiler = j9jit29, Java VM name = IBM J9 VM

JAVA/JVM version

Java Runtime Version = 8.0.6.5 IBM J9 VM

Problem description

Customer upgraded recently his MSJDBC from 6.4 to 8.2. The Java application starts, connects to the database and then reports that an invalid token was received from Azure SQL DB. We thought it was related with #1225 but MS engineers confirmed that the version of Azure SQL DB is updated.

As soon as the connection is estabilished, it tries to open a database and receive the following error message:

There was a problem opening a database instance SYSTEM-STATUS-NODES PEGA!DEVVMSS000000!PRVSNL-CA84856F-5913-4D84-B625-F814B4347F42 0 08S01 The TDS protocol stream is not valid. Unexpected token unknown token (0xA3).

Expected behaviour: Connect and change database
Actual behaviour: Connect and reports an exception
Error message/stack trace: There was a problem opening a database instance SYSTEM-STATUS-NODES PEGA!DEVVMSS000000!PRVSNL-CA84856F-5913-4D84-B625-F814B4347F42 0 08S01 The TDS protocol stream is not valid. Unexpected token unknown token (0xA3).

2020-06-05 19:09:52,799 [       devvmss000000] [  STANDARD] [                    ] [                    ] (nternal.store.DataStoreManager) ERROR   - Error occurred performing a "read" operation in a transaction
com.pega.pegarules.pub.database.ConnectionException: Database-General	There was a problem opening a database instance SYSTEM-STATUS-NODES PEGA!DEVVMSS000000!PRVSNL-42CEFCBD-A222-499F-A191-08EA9D5365E2	0	08S01	The TDS protocol stream is not valid. Unexpected token unknown token (0xA3).
DatabaseException caused by prior exception: com.ibm.websphere.ce.cm.StaleConnectionException: The TDS protocol stream is not valid. Unexpected token unknown token (0xA3).
 | SQL Code: 0 | SQL State: 08S01
DatabaseException caused by prior exception: com.microsoft.sqlserver.jdbc.SQLServerException: The TDS protocol stream is not valid. Unexpected token unknown token (0xA3).
 | SQL Code: 0 | SQL State: 08S01
	at com.pega.pegarules.data.internal.access.ExceptionInformation.createAppropriateExceptionDueToDBFailure(ExceptionInformation.java:375) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.ExceptionInformation.createExceptionDueToDBFailure(ExceptionInformation.java:360) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.store.AutoCommitTransaction.performQuery(AutoCommitTransaction.java:387) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.Opener$1.getResultPageFromDatabase(Opener.java:761) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.Opener$1.action(Opener.java:746) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.Opener$1.action(Opener.java:743) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.store.DataStoreManager.readInTransaction(DataStoreManager.java:71) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.Opener.openResultFromDatabase(Opener.java:772) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.Opener.openData(Opener.java:716) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.Opener.open(Opener.java:2177) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.DatabaseImpl.open(DatabaseImpl.java:2604) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.DatabaseImpl.open(DatabaseImpl.java:2565) ~[prprivate-data.jar:?]
	at com.pega.pegarules.cluster.external.SystemNodesDAO.getStatusNodesPage(SystemNodesDAO.java:507) ~[prenginext.jar:?]
	at com.pega.pegarules.cluster.external.SystemNodesDAO.getProvisionalNodePage(SystemNodesDAO.java:623) ~[prenginext.jar:?]
	at com.pega.pegarules.cluster.external.SystemNodesDAO.upsertNode(SystemNodesDAO.java:612) ~[prenginext.jar:?]
	at com.pega.pegarules.cluster.external.SystemNodesDAO.insertProvisionalNode(SystemNodesDAO.java:553) ~[prenginext.jar:?]
	at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.initializeSystem(PRNodeImpl.java:937) ~[prprivate-session.jar:?]
	at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:378) ~[prprivate-session.jar:?]
	at com.pega.pegarules.session.internal.PRSessionProviderImpl.getThreadAndInitialize(PRSessionProviderImpl.java:2170) ~[prprivate-session.jar:?]
	at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineStartup.initEngine(EngineStartup.java:661) ~[prprivate-session.jar:?]
	at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._initEngine_privact(EngineImpl.java:172) ~[prprivate-session.jar:?]
	at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.doStartup(EngineImpl.java:145) ~[prprivate-session.jar:?]
	at com.pega.pegarules.web.servlet.WebAppLifeCycleListener._contextInitialized_privact(WebAppLifeCycleListener.java:216) ~[prwebj2ee.jar:?]
	at com.pega.pegarules.web.servlet.AbstractLifeCycleListener._contextInitialized_privact(AbstractLifeCycleListener.java:159) ~[prwebj2ee.jar:?]
	at com.pega.pegarules.web.servlet.AbstractLifeCycleListener.contextInitialized(AbstractLifeCycleListener.java:73) ~[prwebj2ee.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]
	at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]
	at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:380) ~[prbootstrap-8.3.1-369.jar:8.3.1-369]
	at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:422) ~[prbootstrap-8.3.1-369.jar:8.3.1-369]
	at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224) ~[prbootstrap-api-8.3.1-369.jar:8.3.1-369]
	at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273) ~[prbootstrap-api-8.3.1-369.jar:8.3.1-369]
	at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:92) ~[prbootstrap-api-8.3.1-369.jar:8.3.1-369]
	at com.ibm.ws.webcontainer.webapp.WebApp.notifyServletContextCreated(WebApp.java:1837) ~[com.ibm.ws.webcontainer.jar:?]
	at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:442) ~[com.ibm.ws.webcontainer.jar:?]
	at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88) ~[com.ibm.ws.webcontainer.jar:?]
	at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:171) ~[com.ibm.ws.webcontainer.jar:?]
	at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:904) ~[com.ibm.ws.webcontainer.jar:?]
	at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:789) ~[com.ibm.ws.webcontainer.jar:?]
	at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:427) ~[com.ibm.ws.webcontainer.jar:?]
	at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719) ~[com.ibm.ws.webcontainer.jar:?]
	at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1249) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1590) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:707) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1161) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:801) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2325) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5523) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5649) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2330) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:985) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:524) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909) ~[com.ibm.ws.runtime.jar:?]
Caused by: com.ibm.websphere.ce.cm.StaleConnectionException: The TDS protocol stream is not valid. Unexpected token unknown token (0xA3).
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:83) ~[?:1.8.0]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57) ~[?:1.8.0]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:437) ~[?:1.8.0]
	at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapExceptionHelper(GenericDataStoreHelper.java:629) ~[rsahelpers.jar:WAS90.SERV1 [f5021945.01]]
	at com.ibm.websphere.rsadapter.GenericDataStoreHelper.mapException(GenericDataStoreHelper.java:688) ~[rsahelpers.jar:WAS90.SERV1 [f5021945.01]]
	at com.ibm.ws.rsadapter.AdapterUtil.mapException(AdapterUtil.java:2273) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.rsadapter.jdbc.WSJdbcUtil.mapException(WSJdbcUtil.java:1219) ~[com.ibm.ws.runtime.jar:?]
	at com.ibm.ws.rsadapter.jdbc.WSJdbcResultSet.next(WSJdbcResultSet.java:3133) ~[com.ibm.ws.runtime.jar:?]
	at com.pega.pegarules.data.internal.store.rdbms.DatabaseResultSetForwardOnly.next(DatabaseResultSetForwardOnly.java:120) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.PageDatabaseMapperImpl.createPageFromResultSetDatabase(PageDatabaseMapperImpl.java:480) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.access.PageDatabaseMapperImpl.createPageFromResultSet(PageDatabaseMapperImpl.java:436) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.store.AutoCommitTransaction.getResultFromResultSet(AutoCommitTransaction.java:459) ~[prprivate-data.jar:?]
	at com.pega.pegarules.data.internal.store.AutoCommitTransaction.performQuery(AutoCommitTransaction.java:359) ~[prprivate-data.jar:?]
	... 56 more
[5/06/20 19:09:52:874 AEST] 0000004e TOKEN         E   ConnectionID:63 ClientConnectionId: b5a421fc-828e-46b8-8728-c546b9506c83: FetchBufferTokenHandler: Encountered unexpected unknown token (0xA3)
[5/06/20 19:09:52:874 AEST] 0000004e Reader        E   ConnectionID:63 ClientConnectionId: b5a421fc-828e-46b8-8728-c546b9506c83 got unexpected value in TDS response at offset:3
[5/06/20 19:09:52:874 AEST] 0000004e TOKEN         E   ConnectionID:63 ClientConnectionId: b5a421fc-828e-46b8-8728-c546b9506c83: FetchBufferTokenHandler: Encountered unexpected unknown token (0xA3)
[5/06/20 19:09:52:874 AEST] 0000004e Reader        E   ConnectionID:63 ClientConnectionId: b5a421fc-828e-46b8-8728-c546b9506c83 got unexpected value in TDS response at offset:
[5/06/20 19:09:52:874 AEST] 0000004e WSJdbcConnect W   DSRA8650W: Error closing a JDBC child wrapper, com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement@dc1fa2c3

Any other details that can be helpful: We have network traces and the results from the internal incident request raised with Microsoft CSS. Network traces shows TDS Version 7.3. We see that the TLS handshake goes fine and the error happens when user is already authenticated. Connection String:

jdbc:sqlserver://SERVERNAME.database.windows.net:1433;database=DATABASENAME;user=PegaAdmin;password={ };encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net