Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Improve performance of persistent write-ahead log

See original GitHub issue

See suggestions in #56 . Tasks:

Add SkipAsync to IAsyncBinaryReader
Move identifier of the log record to its metadata.
Rewrite lock mechanism of PersistentState in a way that allows to do the compaction in parallel with writes or reads
Provides buffering API for log entries
Reuse buffering API at transport-level
Split file buffer size of regular log entries and snapshot entry
fsync for every write must be disabled by default (FileOptions.WriteThrough flag)
Introduce in-memory cached log entries

Issue Analytics

State:
Created 3 years ago
Comments:78

Top GitHub Comments

1reaction

potrusil-osicommented, May 10, 2021

Hi @sakno, I also wanted to thank you for all the work. It’s been great experience to work with you on improving the WAL performance!

1reaction

afink-osicommented, May 10, 2021

@sakno We have completed our research project, unfortunately, we also did not have time to dig into the counters that you added before our time window closed. Thank you for your quick and thorough responses and improvements throughout this work! You may proceed with your release

Top Results From Across the Web

How could WAL （write ahead log） have better ...

For busy workloads, delaying a log write and then performing a single write can significantly improve throughput.

Improving software RAID with a write-ahead log

Rebuild performance. When a write-ahead log is implemented, the RAID array is created in a consistent state and the log will have consistent ......

Write-Ahead Log - Martin Fowler

Write -Ahead Log. Provide durability guarantee without the storage data structures to be flushed to disk, by persisting every state change as a...

Persistent Write Ahead Log | .NEXT

NEXT Cluster Programming Suite ships general-purpose high-performance persistent Write-Ahead Log with the following features: Log compaction based on ...

Optimizing Write Ahead Logging with Intel® Optane™ ...

In this article, we will demonstrate different approaches for implementing a WAL with Intel® Optane™ Persistent Memory (PMem) and examine the ...