Extremely slow batch insert
See original GitHub issueI am using this schema:
let tableCreationSQL =
$"CREATE TABLE IF NOT EXISTS {tableName} (
instrument varchar NOT NULL,
ts timestamp without time zone NOT NULL,
price decimal NOT NULL,
volume decimal NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_instrument ON {tableName}(instrument);
CREATE INDEX IF NOT EXISTS idx_ts ON {tableName}(ts);"
and this insertion code:
let insertTradesAsync (trades: TradeData list) : Async<Result<int, Exception>> =
async {
try
let data =
trades
|> List.map (fun t ->
[
"@instrument", Sql.text t.Instrument.Ticker
"@ts", Sql.timestamp t.Timestamp.DateTime
"@price", Sql.decimal t.Price
"@volume", Sql.decimal (t.Quantity * t.Price)
]
)
let! result =
connectionString
|> Sql.connect
|> Sql.executeTransactionAsync [ "INSERT INTO buffer_trades (instrument, ts, price, volume) VALUES (@instrument, @ts, @price, @volume)", data]
|> Async.AwaitTask
return Ok (List.sum result)
with ex ->
return Error ex
}
When inserting batches of 500 to 3000 records, it will take between 20 and 80 seconds. When I do batches of 100, It takes 3-5 seconds to do the insertion.
Is there anything wrong in the way I’m using the lib?
Issue Analytics
- State:
- Created 2 years ago
- Reactions:1
- Comments:11 (7 by maintainers)
Top Results From Across the Web
Batch insert takes incredible long time
We are running the project on a machine to localhost MySQL database, ensuring the performance of the server shouldn't be the problem. Is...
Read more >Bulk Insert to SQL Server gradually slows down
It is almost certainly due to updating indexes in batches of 10,000. As your table/indexes grow, it takes longer to update. 10,000 is...
Read more >Batch insert mysql is very slow - brettwooldridge/HikariCP
I want to use JDBC to insert a lot of data, I have set rewriteBatchedStatements to true and use jdbc batch api to...
Read more >Extremely Slow Netezza (Database) Batch Inserts using ...
Question. Why are my Batch Inserts on Netezza extremely slow? Cause. Insert statement batches will not be performed on Netezza the overhead per ......
Read more >How can I tell WHY an insert on a certain table is slow?
Inserting one row at a time is really a bad practice with negative performance impacts, specially on ACID DBs like SQL Server. Do...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found

I confirmed by turning on logging, that this is not batch support,
Creates 3 statements as the docs indicate, so while saving on a commit until the end is great and an important feature, it’s not surprising it’s slow if the goal is batch insertion.
@Zaid-Ajaj is there a way to create a SINGLE insert statement with multiple sets of values, so the result is
?
@thomasd3 @Ildhesten Npgsql.FSharp v5.0.1 has been published, it now uses the batch API thanks to @Ildhesten please give it a go and let me know how it goes 🙏