question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Massive performance issue querying entities with many relations

See original GitHub issue

Issue Description

I have a query builder I am running to query for items in a table and loading their relations

let deviceQuery = deviceRepo
    .createQueryBuilder('device')
    .leftJoinAndSelect('device.tankType', 'tankType')
    .leftJoinAndSelect('device.customer', 'customer')
    .leftJoinAndSelect('device.organization', 'organization')
    .leftJoinAndSelect('device.location', 'location')
  const devices: Device[] = await deviceQuery.getMany()

This table only has 7 items on it. When executing this query it takes upwards of 2500ms to return. If I replace .getMany() with .getRawMany() the query only takes 140ms. I can also take the generated SQL from typeorm and paste it into PGAdmin and the query will also only take 140ms there. Something is going terribly wrong with the mapping process to the entity.

Expected Behavior

For .getMany() to be performant when querying for entities that have only 4 relations on them.

Actual Behavior

.getMany() is 15x slower than .getRawMany().

Steps to Reproduce

  1. Create an entity with 4+ @ManyToOne()/@OneToMany() relations
  2. compare .getMany() to .getRawMany()

My Environment

Dependency Version
Operating System Windows
Node.js version v12.18.3
Typescript version v3.9.7
TypeORM version v0.2.29

Additional Context

Relevant Database Driver(s)

  • aurora-data-api
  • aurora-data-api-pg
  • better-sqlite3
  • cockroachdb
  • cordova
  • expo
  • mongodb
  • mysql
  • nativescript
  • oracle
  • postgres
  • react-native
  • sap
  • sqlite
  • sqlite-abstract
  • sqljs
  • sqlserver

Are you willing to resolve this issue by submitting a Pull Request?

  • Yes, I have the time, and I know how to start.
  • Yes, I have the time, but I don’t know how to start. I would need guidance.
  • No, I don’t have the time, although I believe I could do it if I had the time…
  • No, I don’t have the time and I wouldn’t even know how to start.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:5
  • Comments:9 (5 by maintainers)

github_iconTop GitHub Comments

4reactions
m4ttacommented, Nov 21, 2020

It was the @RelationIds that caused all the problems. When I would execute a .getMany() it would run about 11 queries. Picking one of these extra queries out and running it in pgadmin I noticed it was just giving me lists of Ids. This was a massive misunderstanding on my part on the cost of @RelationId. Running postgres locally was just so lightning fast it masked the “real” query time. I thought they were free, and added paired it with every relationship I had even if it wasn’t needed at the time.

Thank you so much for leading me to the right conclusion.

I think the loading of RelationIds can be improved though. When digging into all of this it seemed that when running the 11 queries instead of running in parallel they were waiting for the previous to finish. This is evident by the total query time seeming to be the sum of all 11 queries. These queries are not dependent on each other and should be able to be executed at the same time making this take as long as the slowest query.

3reactions
nebkatcommented, Nov 21, 2020

Glad you got it figured out. @RelationId is still an experimental feature so hopefully that will be improved, but there should probably be more information about how it works in the background to avoid confusion like this. Generally the ORM hides the fact that one-to-many relations are much more complex than many-to-one. I personally avoid using them except for single item queries.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Efficient Querying - EF Core - Microsoft Learn
Performance guide for efficient querying using Entity Framework Core. ... As more one-to-many relationships are loaded, the amount of ...
Read more >
Entity-framework code is slow when using Include() many times
If at least one entity does not have a relationship to some of other entities, then EF will not able to construct one...
Read more >
10 Common Hibernate Mistakes That Cripple Your Performance
Learn how to fix your performance problems by avoiding the most common Hibernate mistakes.
Read more >
3 Common Hibernate Performance Issues in Your Log
A lot of Hibernate performance issues can be found in the log file. In this article, I show you 3 commons ones and...
Read more >
Five levels of performance tuning for an EF Core query
In my book “Entity Framework Core in Action, 2 edition” I use this Book App as an example of using various EF Core...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found