How Lightrun saved WhiteSource cycles of redeployments on AWS

The Challenge

Quickly Identifying an Error in Production

The error occurred in a method that collected results from several threads, but WhiteSource could not immediately identify which of the tasks had the problem. The stack trace showed that the last line of code running was for collecting the thread results.

The last WS line:

com.wss.util.ThreadUtils.collectResults

The last line in the stack trace:

java.util.concurrent.FutureTask.report

WhiteSource was able to identify that the overall exception was an SQL syntax error. However,
the log made it very difficult to identify which query was throwing the exception and making it fail.

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘) group by projectinv1_.projectId’ at line 1

The logs were not informative and did not enable WhiteSource to identify the solution in the current version running in production. They had to find a way to identify the root cause.

In the past, WhiteSource had resorted to adding new logs to suspected lines of codes in the next deployment. Occasionally, they had had to go through several iterations of adding logs to new versions, until the issue was detected. They would remove the logs in the following deployment, to decrease overhead and logging costs.

This process would sometimes take weeks of iterations, and many hours of developer
time -for waiting for the changes to be deployed to production, for inspecting code behavior and re-exploring the issue. The developer would also have to deal with a lot of context switches – every iteration would require the developer to re-read the relevant code, recall the assumptions and continue from there. In the meantime, the version would be running with an error.

“Using Lightrun to debug an actual issue in production enabled us to react instantly. We were able to add the right logs and identify the root-cause in a real-time session, instead of waiting for redeployments” Tom Shapira, Director of Software Engineering at WhiteSource

The Solution

Adding Logs with Lightrun On-demand

WhiteSource used Lightrun to dynamically add logs to each thread in production. They needed to identify the problematic query, among all the MySQL queries in their system.

With Lightrun, integrated into their IDE, WhiteSource was able to add these logs in real-time. The process was simple and only took them a few moments. They were then able to quickly identify where the problematic flow occurred and which lines were executed.

The Results

Exception Identification in Minutes

In just a few minutes, WhiteSource was able to identify the line of code they weren’t able to reach with the logs they had originally added, and that was generating the exception. As a result, WhiteSource was able to quickly catch and handle the bug. They discovered they had sent an empty collection of IDs to the query. In the future, they will add a check to see if the collection is empty before sending it to the query.

By using Lightrun, WhiteSource was able to quickly identify an error that would have taken them cycles of redeployments to resolve. This process would have included waiting for future deployments, which occur every two weeks, for adding logs and recreating the issue. They might have had to go through multiple iterations, before identifying the exception root cause. Then, one last iteration for removing the logs. This is a very time consuming and resource heavy process. In some cases, it prevents full deployment of code versions.

In addition, the quick identification enables WhiteSource to quickly identify a new best practice to add to their tool set. Thus, they will ensure this error will not occur again.