The "sum 1 billion doubles" result on the front page can be incorrect.

I have installed QuestDB on my machine to reproduce the claimed results.

I have: AMD Ryzen 9 3950X 16-Core Processor 64 GiB RAM

QuestDB is installed as the following:

$ wget https://github.com/questdb/questdb/releases/download/5.0.0/questdb-5.0.0-bin.tar.gz
$ tar xf questdb-5.0.0-bin.tar.gz
$ sudo apt install openjdk-14-jre
$ export JAVA_HOME=/usr/lib/jvm/java-14-openjdk-amd64

./questdb.sh start

Exception in thread "main" io.questdb.network.NetworkError: [98] could not bind socket
        at io.questdb@5.0.0/io.questdb.std.ThreadLocal.initialValue(ThreadLocal.java:36)
        at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
        at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
        at io.questdb@5.0.0/io.questdb.network.NetworkError.instance(NetworkError.java:46)
        at io.questdb@5.0.0/io.questdb.network.AbstractIODispatcher.<init>(AbstractIODispatcher.java:96)
        at io.questdb@5.0.0/io.questdb.network.IODispatcherLinux.<init>(IODispatcherLinux.java:36)
        at io.questdb@5.0.0/io.questdb.network.IODispatchers.create(IODispatchers.java:41)
        at io.questdb@5.0.0/io.questdb.cutlass.http.HttpServer.<init>(HttpServer.java:72)
        at io.questdb@5.0.0/io.questdb.cutlass.http.HttpServer.create0(HttpServer.java:148)
        at io.questdb@5.0.0/io.questdb.WorkerPoolAwareConfiguration.create(WorkerPoolAwareConfiguration.java:60)
        at io.questdb@5.0.0/io.questdb.cutlass.http.HttpServer.create(HttpServer.java:110)
        at io.questdb@5.0.0/io.questdb.ServerMain.main(ServerMain.java:145)
        
The exception message is not quite handy.

$ questdb status
questdb: command not found

The example in documentation looks wrong.

$ ./questdb.sh status

  ___                  _   ____  ____
 / _ \ _   _  ___  ___| |_|  _ \| __ )
| | | | | | |/ _ \/ __| __| | | |  _ \
| |_| | |_| |  __/\__ \ |_| |_| | |_) |
 \__\_\\__,_|\___||___/\__|____/|____/
                        www.questdb.io

Not running

The link "with the HTTP API" on the web page https://questdb.io/docs/guideBinaries goes to psql.

Looks like it is using the same port as ClickHouse, so it cannot work with default configuration.

http://localhost:9000/

SELECT 1;
-- does not work

SELECT 1 FROM dual;
-- does not work

create table zz as (select rnd_double() d from long_sequence(1000000000));
select sum(d) from zz;
-- when I copy-paste both commands to the web UI, it shows me

[16:47:58] select sum(d) from zztable does not exist [name=zz]

If I execute these commands one by one, it's Ok.

Table creation is quite slow (about ten seconds), but still Ok.

select sum(d) from zz;

Ok, sustained query time after a few runs is about 282 ms.

The sustained query time is about 282 ms.

I’m trying to reproduce the same with ClickHouse (master, clang release build):

SET max_memory_usage = '20G'
CREATE TABLE zz (x Float64) ENGINE = Memory;
INSERT INTO zz SELECT number FROM numbers_mt(1000000000)
(3.8 seconds)

SELECT sum(x) FROM zz
0.241 sec.

The sustained query time is about 240 ms.

Actually there is no much difference and ClickHouse gives slightly better results. (Or I did not follow the installation steps of QuestDB correctly?)

If I reduce the number of threads to 4 to align with the results from https://questdb.io/blog/2020/04/02/using-simd-to-aggregate-billions-of-rows-per-second ClickHouse gives 232 ms (even slightly better).

The “sum int” is executed in 0.121 (ClickHouse) vs. 0.135 (QuestDB, results on AMD Ryzen 3900X from your blog article). Again, no much difference and ClickHouse is slightly better.

I suspect that the results for ClickHouse in “sum double” benchmark were improved after https://github.com/ClickHouse/ClickHouse/pull/10992

PS. Thank you for the benchmark and for the great product!

[bluestreak01]

Thank you Alexey, this is interesting! c5.metal has two 24-core CPUs. To make result more consistent in our own benchmark we set thread affinity on all 16 threads to make sure data does not have to travel between NUMA zones.

To set affinity on QuestDB:

shared.worker.affinity=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16

That said, we are working on new version of our website, which does not feaure comparison to ClickHouse anymore:

https://v2.questdb.io/

[alexey-milovidov]

I swear it wasn’t intentional but we bind the same socket by default. We only find out when we run clickhouse on the same box as questdb 😃

No problem, the 9000 port is quite popular 😃