Batch Seeding Performance

I’m currently running into issues while trying to use batch to seed my database. I have collections of around 50K records, which caused me to run into memory issues I assume are mentioned in #308.

Since I’m using batch and not sync I added my own limit of around 1000 to get around the memory issue. This however has lead to very slow initial seeding performance (taking minutes to populate fully).

#403 mentions using the new flag to achieve the 5K batch size, but I’m not sure if they’ve run into the slow seeding performance or not.

Anybody had to deal with this case, or the general case of populating watermelon with many records?

[radex]

@BruceSuperProgramer yes and no. when inserting records, most time is spent passing data between JS and native, so there isn’t a lot of opportunity for multithreading. Try out the latest v23 pre-release with jsi mode turned on - you should see batch being >2x faster. It also rearchitects batching such that moving much of the processing to a separate thread (in a future release) should be possible and practical to do.

[bzvikler]

@radex, wouldn’t using the batch functionality within swift still have the performance issues, as @esbenp noted that the bottleneck is within the Native.batch call?

What about the idea of pre-building a .db SQLite file and somehow loading that in on the native side, as is seen here.

Do you think that would work with how watermelon needs its own models and such? I’m optimistically thinking it would be possible to load in a .db file as long as the structure adheres to what watermelon would create via all the Model objects?

EDIT:

If the above would work, then we could look into being able to pass a path to db file, instead of just a db name as is currently the case.

I’m thinking of change along the lines of:

1. SQLiteAdapter (initialized from the user’s JS code) can take an optional instantiation param of path
2. SQLiteAdapter will then be able to pass path along to DatabaseBridge.initialize()
3. add the optional param to DatabaseBridge.initialize()
4. create another convenience initializer in DatabaseDriver.swift which takes in the path
5. DatabaseDriver.swift then instantiates Database.swift by passing path along, instead of assigning it to getPath(dbName: dbName)

Thoughts on this?