question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Extremely slow batch insert

See original GitHub issue

I am using this schema:

        let tableCreationSQL =
            $"CREATE TABLE IF NOT EXISTS {tableName} (
                instrument varchar NOT NULL,
                ts timestamp without time zone NOT NULL,
                price decimal NOT NULL,
                volume decimal NOT NULL
            );

            CREATE INDEX IF NOT EXISTS idx_instrument ON {tableName}(instrument);
            CREATE INDEX IF NOT EXISTS idx_ts ON {tableName}(ts);"

and this insertion code:

    let insertTradesAsync (trades: TradeData list) : Async<Result<int, Exception>> =
        async {
            try
                let data =
                    trades
                    |> List.map (fun t ->
                            [
                                "@instrument", Sql.text      t.Instrument.Ticker
                                "@ts",         Sql.timestamp t.Timestamp.DateTime
                                "@price",      Sql.decimal   t.Price
                                "@volume",     Sql.decimal   (t.Quantity * t.Price)
                            ]
                        )

                let! result =
                    connectionString
                    |> Sql.connect
                    |> Sql.executeTransactionAsync [ "INSERT INTO buffer_trades (instrument, ts, price, volume) VALUES (@instrument, @ts, @price, @volume)", data]
                    |> Async.AwaitTask
                return Ok (List.sum result)

            with ex ->
                return Error ex
        }

When inserting batches of 500 to 3000 records, it will take between 20 and 80 seconds. When I do batches of 100, It takes 3-5 seconds to do the insertion.

Is there anything wrong in the way I’m using the lib?

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:11 (7 by maintainers)

github_iconTop GitHub Comments

2reactions
JesseALancastercommented, Apr 1, 2022

I confirmed by turning on logging, that this is not batch support,

    "INSERT INTO ... VALUES (@number)", [
            [ "@number", Sql.int 1 ]
            [ "@number", Sql.int 2 ]
            [ "@number", Sql.int 3 ]
        ]

Creates 3 statements as the docs indicate, so while saving on a commit until the end is great and an important feature, it’s not surprising it’s slow if the goal is batch insertion.

@Zaid-Ajaj is there a way to create a SINGLE insert statement with multiple sets of values, so the result is

INSERT INTO ... VALUES 
("@number", Sql.int 1),
("@number", Sql.int 2),
("@number", Sql.int 3)

?

0reactions
Zaid-Ajajcommented, Jun 14, 2022

@thomasd3 @Ildhesten Npgsql.FSharp v5.0.1 has been published, it now uses the batch API thanks to @Ildhesten please give it a go and let me know how it goes 🙏

Read more comments on GitHub >

github_iconTop Results From Across the Web

Batch insert takes incredible long time
We are running the project on a machine to localhost MySQL database, ensuring the performance of the server shouldn't be the problem. Is...
Read more >
Bulk Insert to SQL Server gradually slows down
It is almost certainly due to updating indexes in batches of 10,000. As your table/indexes grow, it takes longer to update. 10,000 is...
Read more >
Batch insert mysql is very slow - brettwooldridge/HikariCP
I want to use JDBC to insert a lot of data, I have set rewriteBatchedStatements to true and use jdbc batch api to...
Read more >
Extremely Slow Netezza (Database) Batch Inserts using ...
Question. Why are my Batch Inserts on Netezza extremely slow? Cause. Insert statement batches will not be performed on Netezza the overhead per ......
Read more >
How can I tell WHY an insert on a certain table is slow?
Inserting one row at a time is really a bad practice with negative performance impacts, specially on ACID DBs like SQL Server. Do...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found