[BUG] large log file,dose this normal?

Version LiteDB 5.0.11ver .net 6.0 Win10 X64

Describe the bug the size of main db file stoped increase. the size of log file keep increase

Code to Reproduce

//Full Code
class Program {
        static void Main(string[] args) {
            try {
                LiteDBTest.TestFileNameLengthEffects0();
            } catch (Exception ex) {
                Console.WriteLine(ex.Message);
            }
            Console.ReadLine();
        }
}

    public class LiteDBTest {
        static LiteDBTest() {
            try {
                db = new LiteDatabase("Filename=TestDB.ldb;");//初始化数据库connection=shared
                BsonMapper.Global.IncludeFields = true;
            } catch (Exception ex) {
                Console.WriteLine(ex.Message);
            }
        }

        public static LiteDatabase db;


        public static List<CTS1> DataGen(ref int current, int count) {
            List<CTS1> list = new(count);
            for (int i = 0; i < count; i++) {
                var val = (current + i);
                var key = $"0d3deed1234567890abcd{(100000000 - (current + i)):D12}";
                list.Add(new(key, val));
            }
            current = current + count;
            return list;
        }

        static Random r = new Random();

        public static async void SelectTest() {
            while (!IsExit()) {
                await Task.Delay(r.Next(1, 10));//随机触发查询时间
                var sw = Stopwatch.StartNew();
                var sets = db.GetCollection<CTS1>();
                var rs = r.Next(0, 10000000);
                var key = $"0d3deed1234567890abcd{(100000000 - rs):D12}";
                var rlt = sets.FindById(key);
                if (sw.Elapsed.Seconds > 1)
                    Console.WriteLine($"查询时间大于1秒-{sw.ElapsedMilliseconds}ms");
                Interlocked.Increment(ref QueryCount);
            }
        }


        public static bool IsExit() {
            return Interlocked.Read(ref exit) == 1;
        }

        static long exit = 0;

        public static long QueryCount;

        public static void TestFileNameLengthEffects0() {
            //using var db1 = new LiteDatabase("Filename=TestDB0.ldb;");
            var sw = Stopwatch.StartNew();
            var swOutput = Stopwatch.StartNew();
            for (int t = 0; t < 50; t++)
                Task.Factory.StartNew(SelectTest);

            var set = db.GetCollection<CTS1>();
            int totalIdx = 26860000;
            for (int i = 0;i < 10000; i++) {
                var list = DataGen(ref totalIdx, 20000); //每2w一个批次
                try {
                    var rlt = set.InsertBulk(list);
                } catch (Exception ex) {
                    Console.WriteLine(ex.Message);
                }
                if (swOutput.Elapsed.TotalSeconds > 8) {
                    Console.WriteLine($" 当前数量:{totalIdx} 平均插入速度:{(totalIdx / sw.Elapsed.TotalSeconds)}op/s , 查询次数{(QueryCount / sw.Elapsed.TotalSeconds)}/s");
                    swOutput.Restart();
                }
            }
            exit = 1;
        }

    /// <summary>
    /// CodeTaskSmall
    /// </summary>
    public class CTS1 {
        public CTS1(string code, int taskId) {
            Code = code;
            TaskId = taskId;
        }
        [LiteDB.BsonId]
        public string Code;
        [LiteDB.BsonField("t")]
        public int TaskId;
    }

Expected behavior A clear and concise description of what you expected to happen.

Screenshots/Stacktrace

Additional context Add any other context about the problem here.

[mbdavid]

Hi guys.

When you modify any data in LiteDB, all changes goes to log file. This file contains all changed. The CHECKPOINT command is responsable for read all this log file and persist in original datafile, and then crop/delete file. CHECKPOINT command runs when:

• Manual run: When user call Checkpoint method
• Atuomatic run: When database is closed (in Dispose)
• Atuomatic run: When log file get 1000 pages size (~8MB) ONLY if there is no other thread reading/writing. It´s because CHECKPOINT command need exclusive lock database.

So, if you want avoid large log files after long bulk inserts (in many threads) run Checkpoint command manually afer N documents. This manual run wait for exclusive lock and will remove your log file after execute.

[Paul-Williams]

Thanks @mbdavid That does clarify things.