How Gong Enabled Secure Production Debugging Across their Entire Engineering Organization with Lightrun

Gong Engineering slashes Issue resolution time significantly and Boosts its Dev Productivity

The Challenge

Gong’s mission is to transform revenue organizations by driving business efficiency, revenue growth, and improved decision-making. The Gong Revenue Intelligence platform uses the largest customer interaction dataset, coupled with patented, in-house artificial intelligence (AI) models to power revenue teams’ most critical workflows, including deal execution, sales engagement, forecasting, coaching, and strategic initiatives.

Gong’s engineering team, which consists of hundreds of developers, grappled with debugging tasks that spanned across AWS EC2 and EKS distributed workload environments. The application is Java-based on Spring framework and also utilizes Apache Tomcat web server. The Gong product team deploys their application to production hundreds of times each day serving hundreds of thousands of users and executing millions of online and batch transactions per minute. The hurdles were primarily centered around peripheral microservices and data-storage systems tied to systems being debugged. This includes databases, messaging systems, and other services, the behavior of which could not be faithfully replicated in non-production environments. Moreover, Gong engineers strive to minimize logs due to the prohibitive costs of logging at the scale that the system operates.

Before Lightrun’s adoption, the troubleshooting procedures at Gong involved incorporating static logs and attempting to emulate production issues locally – an inefficient and resource-draining process. Each added log line required a redeployment cycle, resulting in significant time waste and financial costs while the bug still impacted customers. The Gong team was thus on a quest to streamline their troubleshooting procedures, accelerate the process, and diminish the overall expenses associated with remote debugging.

Proposed Solution

Lightrun’s Production Debugging across the SDLC as Part of Gong’s Platform Engineering Strategy 

Lightrun was able to help the entire Gong engineering organization achieve the above objectives through its dynamic observability platform. Lightrun platform enabled developers access to their production environments in a secure manner and allowed efficient debugging of complex issues directly from the developers’ workstations. Specifically, Gong developers started using the Lightrun IntelliJ IDE plugin with Java runtime support (agent) and were able to quickly ramp up and analyze production issues by placing regular as well as conditional snapshots and logs. The team was super excited by the ease of use of the Lightrun platform, which felt like a native IDE debugger but much more powerful. This resulted in employing a modified and cost-effective debugging workflow that includes dynamic logs and breakpoints in runtime within live applications.

The team at Gong embraced Lightrun with enterprise-grade security across multiple use cases.

• Solving production issues and reducing MTTR
• Validate complex production behaviors and pinpoint issues
• Asses and analyze code execution performance using Lightrun metrics
• Enhance the usage of logs while reducing static log expenses

By using Lightrun, Gong engineers were able to save daily hours to troubleshoot customer issues and connect their developers to their production environments from their IDEs without hot-fixing, redeploying, or changing the state of the app in runtime.

“I have found Lightrun to be both user-friendly and remarkably efficient for debugging complex production issues in remote environments. Utilizing Lightrun’s snapshots directly from my IDE has proven to be a straightforward yet powerful method as I worked towards identifying the issues.” – Gil Sagi, Staff Engineer, Gong R&D.

“Leveraging Lightrun, our teams unraveled complex incidents that were challenging to solve and replicate locally using conventional debugging solutions. Lightrun enables us to quickly introduce dynamic logs and snapshots surrounding the incident area, then facilitates reproduction and resolution of issues, much to the delight of our customers.” – Jacob Eckel, VP, Gong R&D

The Results

Before adopting Lightrun, Gong developers were compelled to add numerous static and expensive log lines each time a production issue arose. This necessitated debugging sessions that were long, costly, and inefficient, frequently requiring hotfixes and redeployments that could last for hours.

With the integration of Lightrun, the entire engineering team at Gong, consisting of a few hundred, transformed their troubleshooting procedures to incorporate dynamic logs and snapshots on-demand within their debugging workflows, reducing their MTTR significantly.
This adjustment frequently reduces the overall debugging time from several hours to mere minutes. The adoption of Lightrun paved the way for a more mature platform engineering infrastructure throughout the organization.