question-mark

Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Add macro similar to $rateColumns for counter like metrics

See original GitHub issue

Issue Description

Similarly to request in #78, please consider adding support for computing $perSecond for multiple series (hosts).

Currently, the $rateColumns computes something like:

SELECT t, arrayMap(a -> (a.1, a.2/runningDifference( t/1000 )), groupArr)
FROM (
    SELECT t, groupArray((host, v)) as groupArr 
....

Which does not produce correct results for metrics which only grows over time, as one needs to compute difference between two consequitive values v as well (not just time).

Unfortunatelly, doing runningDifference(a.2)/runningDifference(t/1000) does not work, as a.2 gets materialized to different values v between different columns (host).

I have been scratching my head for a while and I am not sure how to write such query in Clickhouse at all. I hope you know better 😃

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:15

github_iconTop GitHub Comments

1reaction
hagen1778commented, Oct 18, 2018

Hi @JiriHorky Back to #78 Let’s introduce new data:

SELECT *
FROM issue_78

┌───────Date─┬────────────────Time─┬─Packets─┬─Type─┐
│ 2018-08-31 │ 2018-08-31 10:51:00 │    1000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:51:00 │    1000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:52:00 │    2000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:52:00 │    3000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:53:00 │    3000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:53:00 │    5000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:54:00 │    4000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:54:00 │    7000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:55:00 │    5000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:55:00 │    9000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:56:00 │    6000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:56:00 │   11000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:57:00 │    7000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:57:00 │   13000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:58:00 │    8000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:58:00 │   15000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:59:00 │    9000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:59:00 │   17000 │ udp  │
└────────────┴─────────────────────┴─────────┴──────┘

Here we have a Type column and counter Packets. It is obvious, that tcp grows by 1000 per minute (1000/60=16.6 per second) and udp by 2000 per minute (2000/60=33.33 per second). I think the macros should implement the following query:

SELECT
    t,
    arrayMap(a -> (a.1, a.2 / runningDifference(t)), groupArr) AS am
FROM
(
    SELECT
        t,
        groupArray((Type, runningDifference(c))) AS groupArr
    FROM
    (
        SELECT
            intDiv(toUInt32(Time), 120) * 120 AS t,
            Type,
            max(Packets) AS c
        FROM issue_78
        GROUP BY
            t,
            Type
        ORDER BY
            Type ASC,
            t ASC
    )
    GROUP BY t
    ORDER BY t ASC
)

┌──────────t─┬─am──────────────────────────────────────────────────────┐
│ 1535712600 │ [('tcp',nan),('udp',-inf)]                              │
│ 1535712720 │ [('tcp',16.666666666666668),('udp',33.333333333333336)] │
│ 1535712840 │ [('tcp',16.666666666666668),('udp',33.333333333333336)] │
│ 1535712960 │ [('tcp',16.666666666666668),('udp',33.333333333333336)] │
│ 1535713080 │ [('tcp',16.666666666666668),('udp',33.333333333333336)] │
└────────────┴─────────────────────────────────────────────────────────┘

5 rows in set. Elapsed: 0.009 sec.

Please, note some details:

  • Selecting max value for period instead of counting, since we’re dealing with counter:
max(Packets)
  • The reverted sort in inner query:
 ORDER BY
            Type ASC,
            t ASC
  • The runningDifference comparison which is possibly because of reverse sorting:
 groupArray((Type, runningDifference(c))) AS groupArr

Please, try this construction with your data. If it fits your need I’ll add a new macros into plugin. Thank you!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Add macro similar to $rateColumns for counter like metrics
1, a.2/runningDifference( t/1000 )), groupArr) FROM ( SELECT t, groupArray((host, v)) as groupArr .... Which does not produce correct results ...
Read more >
Altinity plugin for ClickHouse plugin ...
Since ad-hoc applies automatically only to outer queries the macros can be used for ... is a combination of $columns and $perSecond for...
Read more >
Alexander Bilous / clickhouse-grafana - GitLab - Nurd
Since ad-hoc applies automatically only to outer queries the macros can be used for ... is a combination of $columns and $perSecond for...
Read more >
FANUC CNC custom Macros
Macro B), sev eral re lated top ics have been added, mainly for co her ... on a lathe, count ing the parts...
Read more >
Do financial markets respond to macroeconomic surprises ...
The present paper adds to this growing corpus of knowledge by ... Macroeconomic surprises concerning retail sales, claimant count rate, GDP, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found