Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Improve SQL performance
See original GitHub issueListing and searching is relatively slow currently, see: https://github.com/bpatrik/pigallery2/blob/master/benchmark/README.md.
Did some minimal local benchmarking and found that Issuing a SELECT * FROM media_entry takes <30ms from DB Browser from SQLite, while from code (even if I list raw results, not parsed JSON) it could take 300ms.
This is without getting all the relations (directory, faces, etc) and doing any where or sorting.
A more realistic query:
SELECT *,
'[' || GROUP_CONCAT( '{"name": "' || "person"."name" || '", "box": {"top":' || "faces"."boxTop" || ', "left":' || "faces"."boxLeft" || ', "height":' || "faces"."boxHeight" ||', "width":' || "faces"."boxWidth" || '}}' ) ||']' as media_metadataFaces
FROM "media_entity" "media"
LEFT JOIN "face_region_entry" "faces" ON "faces"."mediaId"="media"."id"
LEFT JOIN "person_entry" "person"ON "person"."id"="faces"."personId"
GROUP BY "media"."id"
In the DB app: 300ms, from nodejs: 7300ms (first run), 1000ms (following runs)
(The long faces concatenation select is there as I figured that returning with less rows has 30-40% better performance.)
I did the check on a DB with 21k photos, win 10, i7-6700HQ, with SSHD.
There is clearly some significant performance loss around typerom.
There is already some harsh thread about it on reddit: TypeORM Sucks!! Something I wanted to talk about since long!
Currently I see two alternatives:
- Using sqlite3 and mysql drivers directly
- I do not prefer stepping to that path. Especially, as all queries would need to be written twice.
- Maybe at some critical path, it would be possible to speed things up, although, not sure how typeorm would behave, If it got mixed with native queries.
- Looking for a typeorm alternative
- Replacing typeorm would be a massive work as the majority of the backend code is about SQL selects and inserts.
- The Inserting is particularly complicated.
- I found mikro-orm as alternative, It has promising benchmark results: https://github.com/mikro-orm/benchmark
- Replacing typeorm would be a massive work as the majority of the backend code is about SQL selects and inserts.
Blocking:
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (5 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
The previous benchmark result was corrupted so I had to rerun it.
There is a 10-12% increase with
better-sqlite3for every scenarios.PiGallery2 v1.9.1, 23.02.2022
Version: v1.9.1, built at: Tue Feb 22 2022 23:16:30 GMT+0000 (Coordinated Universal Time), git commit:b4c64d12472bb618787168009eae103d59487ded System: Raspberry Pi 4 4G Model B, SandisK Mobile Ultra 32Gb CLass10, UHS-I, HDD: Western Digital Elements 1TB (WDBUZG0010BBK)
Gallery: directories: 31, photos: 2036, videos: 35, diskUsage : 22.08GB, persons : 1381, unique persons (faces): 25
[SQlite=better-sqlite]Scanning directory[SQlite=better-sqlite]Saving directory to DB[SQlite=better-sqlite]List directory[SQlite=better-sqlite]Listing Facesa[SQlite=better-sqlite]Searching foracaption:a[SQlite=better-sqlite]Searching forcaption:adirectory:a[SQlite=better-sqlite]Searching fordirectory:afile-name:a[SQlite=better-sqlite]Searching forfile-name:akeyword:a[SQlite=better-sqlite]Searching forkeyword:aperson:a[SQlite=better-sqlite]Searching forperson:aposition:a[SQlite=better-sqlite]Searching forposition:a.[SQlite=better-sqlite]Searching for.<Most common name>[SQlite=better-sqlite]Searching for<Most common name><Most AND second common names>[SQlite=better-sqlite]Searching for<Most AND second common names><Most OR second common names>[SQlite=better-sqlite]Searching for<Most OR second common names><Contain at least 2 out of all names>[SQlite=better-sqlite]Searching for<Contain at least 2 out of all names>a[SQlite=better-sqlite]Auto complete forarun for : 2696203.0ms
I have just tested the idea and it is simply a matter of switching out the existing sqlite3 dependency for better-sqlite3 (tested version 7.4.4). Since TypeORM is already compatible with better-sqlite3, you can then simply change the driver from
type: 'sqlite'totype: 'better-sqlite3'in SQLConnection.ts (Line 195).The positive performance impact is massive, especially on my database with ~150.000 images.
better-sqlite3 is compatible to any existing sqlite database files, so it should really be as simple as that.
On another note, I’d also update TypeORM itself to a more recent version, but I haven’t tested whether a TypeORM update breaks anything yet.