Memory leak in lazy relation

Issue type:

[x] bug report [ ] feature request [ ] documentation issue

Database system/driver:

[ ] cordova [ ] mongodb [ ] mssql [ ] mysql / mariadb [ ] oracle [x] postgres [ ] cockroachdb [x] sqlite [ ] sqljs [ ] react-native [ ] expo

TypeORM version:

[x] latest [ ] @next [ ] 0.x.x (or put your version here)

Steps to reproduce or a small repository showing the problem:

import 'reflect-metadata';
import { Entity, PrimaryGeneratedColumn, Column, createConnection, Index, ManyToOne, OneToMany } from "typeorm";

type Lazy<T extends object> = Promise<T> | T;

@Entity()
class Bind {
  @PrimaryGeneratedColumn()
  id: number;

  @ManyToOne(type => Source, source => source.bind, { lazy: true })
  source: Lazy<Source>
}

@Entity()
@Index(['problemId', 'source'])
class Problem {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  problemId: string

  @ManyToOne(type => Source, source => source.problem, { lazy: true })
  source: Lazy<Source>
}

@Entity()
class Source {
  @PrimaryGeneratedColumn()
  id: number;

  @OneToMany(type => Problem, problem => problem.source, { lazy: true })
  problem: Lazy<Problem[]>

  @OneToMany(type => Bind, bind => bind.source, { lazy: true })
  bind: Lazy<Bind[]>
}

(async () => {
  const connection = await createConnection({
    type: "postgres",
    host: "localhost",
    port: 5432,
    username: "postgres",
    password: "password",
    database: "StepByStep",
    entities: [
      Bind, Source, Problem
    ],
    synchronize: true,
    logging: false
  })

  const source = new Source();
  await connection.manager.save(source);

  const bind = new Bind();
  bind.source = source;
  await connection.manager.save(bind);


  const repository = connection.getRepository(Problem);
  for (let i = 0; i < 100; i++) {
    {
      // Memory leak code
      console.time(`create ${i}`);
      const problem = repository.create({
        source,
        problemId: String(i)
      });
      await repository.save(problem);
      console.timeEnd(`create ${i}`);
    }
    {
      // pass code
      console.time(`new ${i}`);
      const problem = new Problem();
      problem.source = source;
      problem.problemId = String(i);
      await repository.save(problem);
      console.timeEnd(`new ${i}`);
    }
  }
})();

output:

create 0: 13.096ms
new 0: 4.519ms
create 1: 6.676ms
new 1: 4.373ms
create 2: 5.298ms
new 2: 4.338ms
create 3: 12.688ms
new 3: 4.261ms
create 4: 12.086ms
new 4: 4.253ms
create 5: 35.003ms
new 5: 4.682ms
create 6: 59.895ms
new 6: 4.010ms
create 7: 311.416ms
new 7: 3.859ms
create 8: 527.165ms
new 8: 3.740ms
create 9: 3783.229ms
new 9: 3.806ms
create 10: 8092.808ms
new 10: 3.579ms

<--- Last few GCs --->

[28832:0x110008000]    39560 ms: Scavenge 2042.0 (2050.6) -> 2041.4 (2051.1) MB, 4.6 / 0.0 ms  (average mu = 0.135, current mu = 0.009) allocation failure 
[28832:0x110008000]    41210 ms: Mark-sweep 2042.1 (2051.1) -> 2041.3 (2050.6) MB, 1649.4 / 0.0 ms  (average mu = 0.096, current mu = 0.066) allocation failure scavenge might not succeed
[28832:0x110008000]    41217 ms: Scavenge 2042.2 (2050.6) -> 2041.5 (2051.1) MB, 4.4 / 0.0 ms  (average mu = 0.096, current mu = 0.066) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================
......
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
......

I tried postgres and sqlite, the result is the same. There is a huge performance gap when using new and create.

English is not my native language; please excuse typing errors.

[salemhilal]

Ok, I did some more debugging and I think I found the source of this bug @pleerock. In short: When raw SQL is hydrated into an entity, it looks like the lazy relations are initially set to undefined. Since those keys get defined at all, they are visible to iterators like Object.keys and Object.values. When Object.values gets called on the entity, it inadvertently triggers the lazy loading request for that property. This appears to loop infinitely, although I’m not sure if it’s because each subsequent request hits the same logic, or because bi-directionally-related entities end up loading each other.

Let me know if you’d like me to open up a separate issue for this.

Details

It looks like, by default, defining a property doesn’t make it visible to Object.values by default (there is an enumerable property you can set to true if you’d like it to be visible).

RelationLoader.ts uses Object.defineProperty to implement lazy relations, and at first glance you’d expect those properties to be unenumerated.

However, for some reason, entities in RawSqlResultsToEntityTransformer end up with lazy relation keys and values set. On the line in transform which I mentioned above in a previous comment:

you can see this entity (Meter) having lazy relations (meteredComponents and vendor) defined as promises (I know it’s a little hard to pick through):

and this call to Object.values in turn triggers a .get() call which kicks off the lazy relation’s logic (<anonymous> here is the callback to Object.values):

I dug in a little further. I stepped into transformRawResultsGroup which calls EntityMetadata.create. As part of that, it sets up lazy relations here. Before that loop gets called, ret has all the properties of my entity, but they’re undefined since they presumably haven’t been hydrated yet.

HOWEVER: look at what Object.keys does to that value:

That made me wonder if Object.defineProperty inherits the visibility of any keys that are already on the object. It turns out that yes, it does:

In other words, by defining the keys for a lazy relationship on an entity before the lazy relationship is created, those lazy relations inadvertently get called when hydrating results. I haven’t yet figured out why those properties are instantiated to undefined or why this only seems to happen with Sql.js (and not with our production MySQL-flavored database), but this feels like enough digging to identify the issue.

If you're curious, click here to see the source for MeterEntity

My MeterEntity looks like this (S_Meter is a generated type to make sure the entity agrees with our GraphQL schema):

@Entity("meter")
export class MeterEntity implements S_Meter {
    @PrimaryColumn()
    id!: string;

    @Column({ name: "is_archived" })
    isArchived!: boolean;

    @Column("text", { nullable: true })
    description!: string | null;

    @Column("varchar", { name: "meter_type" })
    meterType!: MeterType;

    @Column({ name: "is_incremental" })
    isIncremental!: boolean;

    @Column("varchar", { name: "unit_name", nullable: true })
    unitName!: string | null;

    @Column("varchar")
    name!: string;

    @Column("varchar", { name: "display_name" })
    displayName!: string;

    @CreateDateColumn({ name: "created_at" })
    createdAt!: string;

    @ManyToOne(() => VendorEntity, (vendor) => vendor.meters)
    @JoinColumn({ name: "vendor_id" })
    vendor!: Promise<VendorEntity>;

    @OneToMany(() => MeteredComponentEntity, (mc) => mc.meter)
    meteredComponents!: Promise<MeteredComponentEntity[]>;
}

[orcwarrior]

Thanks, @imnotjames and @salemhilal for debugging the issue it really helped me to pin it down! Seems like while debugging this was a real hell, the fix was pretty easy @salemhilal sums it up in the previous post. If somebody needs to patch it locally before the fix gets merged, code like this should help:

const orgObjectDefineProperty = Object.defineProperty;

Object.defineProperty = (obj, property, attributes) => orgObjectDefineProperty(obj, property, {
    ...attributes,
    enumerable: (attributes.enumerable === true)
});

I spend quite some time, trying to replicate the issue by writing a test case for it to fail. Unfortunately wasn’t able, while having a couple of tries. The reason for the issue to happen is that setting all properties of entity instance to undefined - I’m not familiar with TypeOrm codebase, so I wasn’t able to find a reason for that happening. Those undefined fields later forces enumerable to have true value for the property which eventually leads to all relations getting queried until the memory is no more 😅