Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Performance degradation using version 2.1.0

See original GitHub issue

Performance degradation using version 2.1.0

We have been experiencing performance degradation when using the new version of stormcrawler. We are pretty sure that the issue is being caused by this workaround

    /** Workaround for https://issues.apache.org/jira/projects/STORM/issues/STORM-3582?filter=allopenissues **/
    protected synchronized void emit(String streamId, Tuple anchor, List<Object> tuple) {
        collector.emit(streamId, anchor, tuple);
    }

it’s worth mentioning that we are using stormcrawler intensively with a high number of threads (300) so having now this method synchronized is adding extra delays and affecting the performance. Here you can see an example of how it is performing SC 2.1 vs 1.18…

image

left side SC 2.1, then we stopped the crawler and we re-started using SC 1.18. We tried removing the synchronized in the emit method, metrics started to look better but we got this kind of issue, related to this apache storm issue : https://issues.apache.org/jira/browse/STORM-3620

./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 2021-10-15 13:51:32.752 o.a.s.u.Utils Thread-16-fetcher-executor[10, 10] [ERROR] Async loop died!
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] Serialization trace:
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] md (com.digitalpebble.stormcrawler.Metadata)
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:38) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:40) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.daemon.worker.WorkerTransfer.tryTransferRemote(WorkerTransfer.java:116) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.daemon.worker.WorkerState.tryTransferRemote(WorkerState.java:524) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.ExecutorTransfer.tryTransfer(ExecutorTransfer.java:68) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.tryFlushPendingEmits(BoltExecutor.java:200) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:166) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:159) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.utils.Utils$1.run(Utils.java:394) [storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
./worker.log:2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] Caused by: java.util.ConcurrentModificationException
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) ~[?:1.8.0_292]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at java.util.HashMap$EntryIterator.next(HashMap.java:1479) ~[?:1.8.0_292]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at java.util.HashMap$EntryIterator.next(HashMap.java:1477) ~[?:1.8.0_292]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:99) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:39) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	... 15 more
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 2021-10-15 13:51:32.756 o.a.s.e.e.ReportError Thread-16-fetcher-executor[10, 10] [ERROR] Error
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] Serialization trace:
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] md (com.digitalpebble.stormcrawler.Metadata)
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at org.apache.storm.utils.Utils$1.run(Utils.java:409) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
./worker.log:2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] Caused by: com.esotericsoftware.kryo.KryoException: java.util.ConcurrentModificationException
./worker.log-2021-10-15 13:51:32.756 STDERR Thread-2 [INFO] Serialization trace:
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] md (com.digitalpebble.stormcrawler.Metadata)
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:38) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:40) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.daemon.worker.WorkerTransfer.tryTransferRemote(WorkerTransfer.java:116) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.daemon.worker.WorkerState.tryTransferRemote(WorkerState.java:524) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.ExecutorTransfer.tryTransfer(ExecutorTransfer.java:68) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.tryFlushPendingEmits(BoltExecutor.java:200) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:166) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:159) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.utils.Utils$1.run(Utils.java:394) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	... 1 more
./worker.log:2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] Caused by: java.util.ConcurrentModificationException
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1445) ~[?:1.8.0_292]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at java.util.HashMap$EntryIterator.next(HashMap.java:1479) ~[?:1.8.0_292]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at java.util.HashMap$EntryIterator.next(HashMap.java:1477) ~[?:1.8.0_292]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:99) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.MapSerializer.write(MapSerializer.java:39) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:80) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:38) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:40) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.daemon.worker.WorkerTransfer.tryTransferRemote(WorkerTransfer.java:116) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.daemon.worker.WorkerState.tryTransferRemote(WorkerState.java:524) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.ExecutorTransfer.tryTransfer(ExecutorTransfer.java:68) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.tryFlushPendingEmits(BoltExecutor.java:200) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:166) ~[storm-client-2.2.0.jar:2.2.0]
./worker.log-2021-10-15 13:51:32.757 STDERR Thread-2 [INFO] 	at org.apache.storm.executor.bolt.BoltExecutor$1.call(BoltExecutor.java:159) ~[storm-client-2.2.0.jar:2.2.0]

@jnioche we are working on a workaround to see if we can resolve this last issue with the serialization, we will keep you posted.

What kind of issue is this?

  • Question. This issue tracker is not the best place for questions. If you want to ask how to do something, or to understand why something isn’t working the way you expect it to, use StackOverflow instead with the label ‘stormcrawler’: https://stackoverflow.com/questions/tagged/stormcrawler

  • Bug report. If you’ve found a bug, please include a test if you can, it makes it a lot easier to fix things. Use the label ‘bug’ on the issue.

  • Feature request. Please use the label ‘wish’ on the issue.

Thanks!

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:2
  • Comments:9 (9 by maintainers)

github_iconTop GitHub Comments

1reaction
jniochecommented, Oct 15, 2021

The sync has been removed in #904, this will be part of the next release. Which version of Storm is your cluster on? This should have been fixed in 2.2.0. I recently upgraded the dependency to 2.3.0

0reactions
jniochecommented, Oct 22, 2021

@jcruzmartini glad it’s fixed. Thanks for checking

Read more comments on GitHub >

github_iconTop Results From Across the Web

Performance degradation with Pygame 2.0.0 and 2.1 ... - GitHub
0 - *PyGame version*: 2.0.0 same issue reproduced on 2.1.0 - *Relevant hardware* : Raspberry Pi 4 4Gb RAM *Current behavior:* My game...
Read more >
Resolving performance degradation issues with Big SQL ... - IBM
Symptom. Query response time increases considerably after you install or upgrade to InfoSphere BigInsights 2.1.0.1 or 2.1.0.2.
Read more >
1141656 – Virtio-scsi: performance degradation from 1.5.3 to 2.1.0
Bug 1141656 - Virtio-scsi: performance degradation from 1.5.3 to 2.1.0 ... Fixed In Version: qemu-kvm-rhev-2.1.2-15.el7 ... Target Upstream Version: ...
Read more >
What's New in RapidMiner Server 2.1.0?
This page describes improvements in RapidMiner Server 2.1.0. Enhancements. Uses RapidMiner Core version 6.1. Bug fixes. BUGFIX: Fixed performance degradation ...
Read more >
Spark Release 2.1.0
We have curated a list of high level changes here, grouped by major modules. Core and Spark SQL; Structured Streaming; MLlib; SparkR; GraphX;...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Log4J vulnerability - CVE-2021-44228

Next page

Issue with ConcurrentModificationException for Metadata in StatusMetricsBolt