question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

AMQP metrics report negative number of authenticated connections

See original GitHub issue

Periodically(I would say rather sporadically !) AMQP adapter report a negative metric number for authenticated connections. Unfortunately, for a while I tried to find the cause and reproduce this but without success. But anyway, when I took a look at MicrometerBasedMetrics.java, I would like to propose to have more safe incremental and detrimental operations for connections counters, something like this:

private void tryToDecrement(AtomicLong longValue) {
   longValue.getAndUpdate(oldValue -> oldValue > 0 ? oldValue - 1 : 0);
}
private void tryToIncrement(AtomicLong longValue)
 {
  longValue.getAndUpdate(oldValue -> {
   if (oldValue == Long.MAX_VALUE) {
    throw new IllegalStateException("Long value overflow!");
   }
   return oldValue + 1L;
  });
 }
}

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:7 (7 by maintainers)

github_iconTop GitHub Comments

1reaction
fkaltnercommented, Jun 23, 2020

I think I found the cause:

In the AMQP adapter the connection count was incremented when opening a new connection after checking if the adapter’s connection limit was hit. The decrement however was done in any case, since it is part of the connection’s close handler.

As you can see in e70476c I fixed this by moving the increment of the connection count to the connection’s open handler.

Nevertheless I am not sure of any side-effects. @kaniyan / @sophokles73 WDYT?

1reaction
fkaltnercommented, Jun 23, 2020

I am able to reproduce this on my machine by setting the AMQP / MQTT adapter’s connection limit to 1 and connecting 2 clients.

When I connect the second client, I see negative values in the Prometheus dashboard.

I’d like to continue to investigate this, @sophokles73 can you assign this issue to me?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Monitoring - RabbitMQ
The plugin expose a number of RabbitMQ metrics for nodes, connections, queues, message rates and so on. This plugin has low overhead and...
Read more >
Key Metrics & Best Tools for RabbitMQ Monitoring - Sematext
Discover the key metrics you should measure to improve RabbitMQ performance and what are the best monitoring tools available today.
Read more >
Configuring AMQ Broker - Red Hat Customer Portal
Configuring AMQP Security. The broker supports AMQP SASL Authentication. See Security for more information about how to configure SASL-based authentication on ...
Read more >
Key Metrics for RabbitMQ Monitoring - Datadog
With RabbitMQ monitoring, see how your messaging setup affects your system and holds up to demand.
Read more >
Monitoring - VMware Docs
The plugin expose a number of RabbitMQ metrics for nodes, connections, queues, message rates and so on. This plugin has low overhead and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found