Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Looping exception
See original GitHub issueLast week, my TB server started generating extensive logs filled with exception logs. In most cases, server dies, but sometimes it manages to recover, I didn’t find a matching pattern yet. However, the exception loop stays the same all the time. It looks like this:
2018-07-02 10:42:02,312 [pool-17-thread-1] ERROR o.t.s.s.t.DefaultTelemetryWebSocketService - Failed to fetch data! java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:131) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.isDone(RateLimitedResultSetFuture.java:95) at com.google.common.util.concurrent.AggregateFuture$RunningState.handleOneInputDone(AggregateFuture.java:204) at com.google.common.util.concurrent.AggregateFuture$RunningState.access$200(AggregateFuture.java:78) at com.google.common.util.concurrent.AggregateFuture$RunningState$1.run(AggregateFuture.java:133) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902) at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:636) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:100) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.lambda$addListener$2(RateLimitedResultSetFuture.java:122) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:677) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:112) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813) at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:713) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:185) at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:162) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:138) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813) at com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:553) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.cancel(AbstractFuture.java:106) at org.thingsboard.server.dao.util.BufferedRateLimiter$LockedFuture.cancelFuture(BufferedRateLimiter.java:171) at org.thingsboard.server.dao.util.BufferedRateLimiter.printStats(BufferedRateLimiter.java:148) at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:129) ... 40 common frames omitted Caused by: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1114) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142) at com.google.common.util.concurrent.Futures.getDone(Futures.java:1174) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:110) ... 19 common frames omitted 2018-07-02 10:42:02,313 [pool-13-thread-170289] ERROR c.g.c.u.concurrent.AggregateFuture - Got more than one input Future failure. Logging failures after the first java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:131) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.isDone(RateLimitedResultSetFuture.java:95) at com.google.common.util.concurrent.Futures.getDone(Futures.java:1173) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:103) at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:952) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:129) ... 7 common frames omitted Caused by: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1114) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142) at com.google.common.util.concurrent.Futures.getDone(Futures.java:1174) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:110) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813) at com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:553) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.cancel(AbstractFuture.java:106) at org.thingsboard.server.dao.util.BufferedRateLimiter$LockedFuture.cancelFuture(BufferedRateLimiter.java:171) at org.thingsboard.server.dao.util.BufferedRateLimiter.printStats(BufferedRateLimiter.java:148) at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ... 3 common frames omitted 2018-07-02 10:42:02,323 [pool-13-thread-170310] ERROR c.g.c.u.concurrent.AggregateFuture - Got more than one input Future failure. Logging failures after the first java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:131) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.isDone(RateLimitedResultSetFuture.java:95) at com.google.common.util.concurrent.Futures.getDone(Futures.java:1173) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:103) at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:952) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:129) ... 7 common frames omitted Caused by: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1114) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142) at com.google.common.util.concurrent.Futures.getDone(Futures.java:1174) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:110) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813) at com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:553) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.cancel(AbstractFuture.java:106) at org.thingsboard.server.dao.util.BufferedRateLimiter$LockedFuture.cancelFuture(BufferedRateLimiter.java:171) at org.thingsboard.server.dao.util.BufferedRateLimiter.printStats(BufferedRateLimiter.java:148) at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ... 3 common frames omitted 2018-07-02 10:42:02,323 [pool-13-thread-170429] ERROR c.g.c.u.concurrent.AggregateFuture - Got more than one input Future failure. Logging failures after the first java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:131) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.isDone(RateLimitedResultSetFuture.java:95) at com.google.common.util.concurrent.Futures.getDone(Futures.java:1173) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:103) at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:952) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:129) ... 7 common frames omitted Caused by: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.cancellationExceptionWithCause(AbstractFuture.java:1114) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142) at com.google.common.util.concurrent.Futures.getDone(Futures.java:1174) at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:110) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:902) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:813) at com.google.common.util.concurrent.AbstractFuture.cancel(AbstractFuture.java:553) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.cancel(AbstractFuture.java:106) at org.thingsboard.server.dao.util.BufferedRateLimiter$LockedFuture.cancelFuture(BufferedRateLimiter.java:171) at org.thingsboard.server.dao.util.BufferedRateLimiter.printStats(BufferedRateLimiter.java:148) at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ... 3 common frames omitted 2018-07-02 10:42:02,324 [pool-13-thread-170355] ERROR c.g.c.u.concurrent.AggregateFuture - Got more than one input Future failure. Logging failures after the first java.lang.IllegalStateException: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:131) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.isDone(RateLimitedResultSetFuture.java:95) at com.google.common.util.concurrent.Futures.getDone(Futures.java:1173) at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:103) at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:952) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.CancellationException: Task was cancelled. at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:78) at org.thingsboard.server.dao.nosql.RateLimitedResultSetFuture.safeGet(RateLimitedResultSetFuture.java:129) ... 7 common frames omitted
Issue Analytics
- State:
- Created 5 years ago
- Comments:11 (5 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
I do not see the problem with some specific Cassandra queries. I think the root cause is the total number of ‘SELECT’ queries, that are generated during fetching data for the Dashboard widgets. In peak, you have 600 read request per second. In general, it is not a big number, but we need to take in account your instance type.
If Thingsboard and Cassandra are both executed on the same instance with 2 GB of RAM, then it is a problem. Cassandra was designed as a big data storage and it requires memory and CPU resources for normal operation. At least you need to have 2 GB RAM and 1 dedicated CPU for the Cassandra process. But in your case you share those resources between OS, Thingsboard and Cassandra - it is not enough for production systems.
What to do - you have a few options:
Again, 360 requests or even 1000 requests per second is not a problem for the Thingsboard and Cassandra, but you need to provide a meaningful number of resources for making it stable.
Those exceptions are thrown by Guava library when Future task is cancelled. It is hard to mute it.