Pagination is broken when using an order by column that isn't part of the selection

Issue Description

I have two tables, posts and author. I wish to join to the author table, order by the post title, and simply return the post IDs, using pagination (take() and skip()) (I understand there’s no reason to join to the author table, but this bug is manifesting in a OData parser that I’m working on, so if the client says “join author”, we join author).

This issue looks similar to #5670, but not 100% sure it’s the same. Regardless, this is a much simpler use case and is as close to minimal reproducible as possible

Expected Behavior

I would expect TypeORM to handle this somewhat simple query easily and return my post IDs, order by the post title

Actual Behavior

The DISTINCT query that TypeORM produces fails to include the ORDER BY column in the inner query, thus throwing a SQL exception.

Here’s the query:

SELECT DISTINCT
       "distinctAlias"."Post_id" AS "ids_Post_id",
       "distinctAlias"."Post_title"
FROM
(
    SELECT "Post"."id" AS "Post_id",
           "author31"."id" AS "author31_id",
           "author31"."name" AS "author31_name"
    FROM "posts" "Post"
        LEFT JOIN "authors" "author31"
            ON "author31"."id" = "Post"."author_id"
               AND (1 = 1)
) "distinctAlias"
ORDER BY "distinctAlias"."Post_title" ASC,
         "Post_id" ASC
LIMIT 3 OFFSET 3

The entire run log from start to finish:

$ node dist/test.js
query: BEGIN TRANSACTION
query: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" IN ('authors', 'posts')
query: SELECT * FROM "sqlite_master" WHERE "type" = 'index' AND "tbl_name" IN ('authors', 'posts')
query: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = 'typeorm_metadata'
query: CREATE TABLE "authors" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, "name" varchar NOT NULL)
query: CREATE TABLE "posts" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, "title" varchar NOT NULL, "text" varchar NOT NULL, "author_id" integer)
query: CREATE TABLE "temporary_posts" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, "title" varchar NOT NULL, "text" varchar NOT NULL, "author_id" integer, CONSTRAINT "FK_312c63be865c81b922e39c2475e" FOREIGN KEY ("author_id") REFERENCES "authors" ("id") ON DELETE NO ACTION ON UPDATE NO ACTION)
query: INSERT INTO "temporary_posts"("id", "title", "text", "author_id") SELECT "id", "title", "text", "author_id" FROM "posts"
query: DROP TABLE "posts"
query: ALTER TABLE "temporary_posts" RENAME TO "posts"
query: COMMIT
query: SELECT * FROM "sqlite_master" WHERE "type" = 'table' AND "name" = '_migrations'
query: CREATE TABLE "_migrations" ("id" integer PRIMARY KEY AUTOINCREMENT NOT NULL, "timestamp" bigint NOT NULL, "name" varchar NOT NULL)
query: SELECT * FROM "_migrations" "_migrations" ORDER BY "id" DESC
query: BEGIN TRANSACTION
query: INSERT INTO "authors" (id,name)
                                      VALUES
                                        (1,'Ursula Manning'),
                                        (2,'Carly Butler'),
                                        (3,'Maggy Juarez'),
                                        (4,'Colette Murray'),
                                        (5,'Emmanuel Salinas')
query: INSERT INTO "posts" (id,title,text,author_id) 
                                      VALUES
                                        (1,'Ultricies Sem Ltd','quam. Curabitur vel lectus. Cum',1),
                                        (2,'Dolor Consulting','mauris ipsum porta elit, a',2),
                                        (3,'Feugiat Associates','tincidunt. Donec vitae erat',3),
                                        (4,'Ipsum Phasellus LLC','ut eros non enim commodo hendrerit. Donec porttitor',4),
                                        (5,'Habitant Morbi Tristique Corp.','Suspendisse',5),
                                        (6,'Nunc Interdum Feugiat LLC','elit, pellentesque a, facilisis non, bibendum sed, est.',1),
                                        (7,'Sapien Industries','dictum. Proin eget',2),
                                        (8,'In Hendrerit Consectetuer Foundation','vulputate, nisi sem semper erat,',3),
                                        (9,'Odio Aliquam Vulputate Foundation','velit eget',4),
                                        (10,'Cursus Ltd','ac mattis ornare, lectus ante dictum mi, ac mattis velit',5)
query: INSERT INTO "_migrations"("timestamp", "name") VALUES (?, ?) -- PARAMETERS: [1577087002356,"DataFilling1577087002356"]
query: COMMIT
query: SELECT DISTINCT "distinctAlias"."Post_id" as "ids_Post_id", "distinctAlias"."Post_title" FROM (SELECT "Post"."id" AS "Post_id", "author31"."id" AS "author31_id", "author31"."name" AS "author31_name" FROM "posts" "Post" LEFT JOIN "authors" "author31" ON "author31"."id"="Post"."author_id" AND (1=1)) "distinctAlias" ORDER BY "distinctAlias"."Post_title" ASC, "Post_id" ASC LIMIT 3 OFFSET 3
query failed: SELECT DISTINCT "distinctAlias"."Post_id" as "ids_Post_id", "distinctAlias"."Post_title" FROM (SELECT "Post"."id" AS "Post_id", "author31"."id" AS "author31_id", "author31"."name" AS "author31_name" FROM "posts" "Post" LEFT JOIN "authors" "author31" ON "author31"."id"="Post"."author_id" AND (1=1)) "distinctAlias" ORDER BY "distinctAlias"."Post_title" ASC, "Post_id" ASC LIMIT 3 OFFSET 3
error: Error: SQLITE_ERROR: no such column: distinctAlias.Post_title
--> in Database#all('SELECT DISTINCT "distinctAlias"."Post_id" as "ids_Post_id", "distinctAlias"."Post_title" FROM (SELECT "Post"."id" AS "Post_id", "author31"."id" AS "author31_id", "author31"."name" AS "author31_name" FROM "posts" "Post" LEFT JOIN "authors" "author31" ON "author31"."id"="Post"."author_id" AND (1=1)) "distinctAlias" ORDER BY "distinctAlias"."Post_title" ASC, "Post_id" ASC LIMIT 3 OFFSET 3', [], [Function: handler])  
    at SqliteQueryRunner.<anonymous> (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\typeorm\driver\sqlite\SqliteQueryRunner.js:57:56)  
    at step (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\tslib\tslib.js:143:27)
    at Object.next (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\tslib\tslib.js:124:57)
    at C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\tslib\tslib.js:117:75
    at new Promise (<anonymous>)
    at Object.__awaiter (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\tslib\tslib.js:113:16)
    at execute (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\typeorm\driver\sqlite\SqliteQueryRunner.js:51:64)
    at SqliteQueryRunner.<anonymous> (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\typeorm\driver\sqlite\SqliteQueryRunner.js:83:46)  
    at step (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\tslib\tslib.js:143:27) {
  errno: 1,
  code: 'SQLITE_ERROR',
  __augmented: true
}
(node:30604) UnhandledPromiseRejectionWarning: QueryFailedError: SQLITE_ERROR: no such column: distinctAlias.Post_title
    at new QueryFailedError (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\typeorm\error\QueryFailedError.js:12:28)
    at handler (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\typeorm\driver\sqlite\SqliteQueryRunner.js:77:38)
    at replacement (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\sqlite3\lib\trace.js:25:27)
    at Statement.errBack (C:\gitRoot\odata-v4-typeorm\examples\server\node_modules\sqlite3\lib\sqlite3.js:14:21)
(node:30604) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:30604) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Steps to Reproduce

I have a minimal reproduction that uses SQLite here: https://gist.github.com/blitzmann/5147a7dc018b866168af6f71fae45056

Simply build via tsc, and run via node.

This is the actual query that causes the issue:

await queryBuilder
        .select(['Post.id']) // <-- if I were to include `Post.title` here in the selection, everything works
        .leftJoinAndSelect('Post.author', 'author31', '1=1', {})
        .addOrderBy('Post.title', 'ASC')
        .skip(3)
        .take(3)
        .getMany()

My Environment

Dependency	Version
Operating System	Windows 10
Node.js version	12.20.0
Typescript version	4.3.4
TypeORM version	0.2.34

Relevant Database Driver(s)

I’ve only tested this in MSSQL and SQLite, both show similar error

DB Type	Reproducible
aurora-data-api	no
aurora-data-api-pg	no
better-sqlite3	no
cockroachdb	no
cordova	no
expo	no
mongodb	no
mysql	no
nativescript	no
oracle	no
postgres	no
react-native	no
sap	no
sqlite	yes
sqlite-abstract	no
sqljs	no
sqlserver	yes

Are you willing to resolve this issue by submitting a Pull Request?

• ✖️ Yes, I have the time, and I know how to start.
• ✅ Yes, I have the time, but I don’t know how to start. I would need guidance.
• ✖️ No, I don’t have the time, but I can support (using donations) development.
• ✖️ No, I don’t have the time and I’m okay to wait for the community / maintainers to resolve this issue.

[blitzmann]

@pleerock Having stable pagination from TypeORM is crucial to what we’re trying to achieve. if you can provide a summary of the issues with the distinct query, I would be interested in poking around the codebase and contributing if I can get something worked out.

Thanks for your time and input, I appreciate it 😃

[pleerock]

Could you provide an example when using limit / offset might break?

I don’t have a sample repository where I can should you the problem, but I’ll try to explain. If your “post” has multiple “authors” joining in RDBMS return us data in a following format:

post_id	post_title	author_id
1	bla	1
1	bla	2
2	tada	1

now, if you do “limit(2)” it will result into SQL’s LIMIT. Returned dataset for limit=2 will be:

post_id	post_title	author_id
1	bla	1
1	bla	2

and it means you’ll have only one post instead of two. It’s not what you expect.

IMO there is still a bug with take / skip as it should be able to handle distinct query properly.

we have other issues for this problem. The problem is complex and there is no silver bullet for this problem. Current solution works for the simplest and most use cases. If we change it - we can break other cases. Maybe we can improve current solution, but it increases complexity which is bad as well.

My recommendation:

• you have a simple query without selection of joins - use limit/offset
• you have a complex query with joins and skip/take solves the problem - use it
• you have a complex query with joins and skip/take is problematic, you have two options
  • instead of using joins do separate queries - this approach is faster than joins if you have bulky data selection
  • or continue to use joins, but don’t use ORM’s mapping - instead get raw results and build the object manually