Add macro similar to $rateColumns for counter like metrics

Similarly to request in #78, please consider adding support for computing $perSecond for multiple series (hosts).

Currently, the $rateColumns computes something like:

SELECT t, arrayMap(a -> (a.1, a.2/runningDifference( t/1000 )), groupArr)
FROM (
    SELECT t, groupArray((host, v)) as groupArr 
....

Which does not produce correct results for metrics which only grows over time, as one needs to compute difference between two consequitive values v as well (not just time).

Unfortunatelly, doing runningDifference(a.2)/runningDifference(t/1000) does not work, as a.2 gets materialized to different values v between different columns (host).

I have been scratching my head for a while and I am not sure how to write such query in Clickhouse at all. I hope you know better 😃

[hagen1778]

Hi @JiriHorky Back to #78 Let’s introduce new data:

SELECT *
FROM issue_78

┌───────Date─┬────────────────Time─┬─Packets─┬─Type─┐
│ 2018-08-31 │ 2018-08-31 10:51:00 │    1000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:51:00 │    1000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:52:00 │    2000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:52:00 │    3000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:53:00 │    3000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:53:00 │    5000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:54:00 │    4000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:54:00 │    7000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:55:00 │    5000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:55:00 │    9000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:56:00 │    6000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:56:00 │   11000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:57:00 │    7000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:57:00 │   13000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:58:00 │    8000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:58:00 │   15000 │ udp  │
│ 2018-08-31 │ 2018-08-31 10:59:00 │    9000 │ tcp  │
│ 2018-08-31 │ 2018-08-31 10:59:00 │   17000 │ udp  │
└────────────┴─────────────────────┴─────────┴──────┘

Here we have a Type column and counter Packets. It is obvious, that tcp grows by 1000 per minute (1000/60=16.6 per second) and udp by 2000 per minute (2000/60=33.33 per second). I think the macros should implement the following query:

SELECT
    t,
    arrayMap(a -> (a.1, a.2 / runningDifference(t)), groupArr) AS am
FROM
(
    SELECT
        t,
        groupArray((Type, runningDifference(c))) AS groupArr
    FROM
    (
        SELECT
            intDiv(toUInt32(Time), 120) * 120 AS t,
            Type,
            max(Packets) AS c
        FROM issue_78
        GROUP BY
            t,
            Type
        ORDER BY
            Type ASC,
            t ASC
    )
    GROUP BY t
    ORDER BY t ASC
)

┌──────────t─┬─am──────────────────────────────────────────────────────┐
│ 1535712600 │ [('tcp',nan),('udp',-inf)]                              │
│ 1535712720 │ [('tcp',16.666666666666668),('udp',33.333333333333336)] │
│ 1535712840 │ [('tcp',16.666666666666668),('udp',33.333333333333336)] │
│ 1535712960 │ [('tcp',16.666666666666668),('udp',33.333333333333336)] │
│ 1535713080 │ [('tcp',16.666666666666668),('udp',33.333333333333336)] │
└────────────┴─────────────────────────────────────────────────────────┘

5 rows in set. Elapsed: 0.009 sec.

Please, note some details:

• Selecting max value for period instead of counting, since we’re dealing with counter:

max(Packets)

• The reverted sort in inner query:

 ORDER BY
            Type ASC,
            t ASC

• The runningDifference comparison which is possibly because of reverse sorting:

 groupArray((Type, runningDifference(c))) AS groupArr

Please, try this construction with your data. If it fits your need I’ll add a new macros into plugin. Thank you!

[hagen1778]

Please, see https://github.com/Vertamedia/clickhouse-grafana/releases/tag/1.8.0