modify limit offset implement #2188

Merged
lunny merged 3 commits from FlyingOnion/xorm:master into master 2023-09-20 02:07:03 +00:00
Member

Oracle and SQLServer specific: When LIMIT OFFSET function is needed, use OFFSET <offset> ROWS FETCH NEXT <limit> ROWS ONLY to replace legacy subquery.

SQLServer specific: When ORDER BY is not set and OFFSET FETCH is set, set statement.orderStr to 1 (ORDER BY 1). See here.

MySQL specific: When limit is 0 and offset > 0, use LIMIT 9223372036854775807 ($2^{63}-1$). See comments here.

Oracle and SQLServer specific: When `LIMIT OFFSET` function is needed, use `OFFSET <offset> ROWS FETCH NEXT <limit> ROWS ONLY` to replace legacy subquery. SQLServer specific: When `ORDER BY` is not set and `OFFSET FETCH` is set, set `statement.orderStr` to `1` (`ORDER BY 1`). See [here](https://learn.microsoft.com/zh-cn/sql/t-sql/queries/select-order-by-clause-transact-sql?view=sql-server-ver16). MySQL specific: When limit is 0 and offset > 0, use `LIMIT 9223372036854775807` ($2^{63}-1$). See comments [here](https://gitea.com/xorm/xorm/src/commit/15d171ea55a011eec76910353d5d8d17397d78e4/internal/statements/query.go#L314).
FlyingOnion added 1 commit 2022-10-20 06:32:47 +00:00
modify limit offset
Some checks failed
continuous-integration/drone/pr Build is failing
15d171ea55
FlyingOnion reviewed 2022-10-20 06:37:18 +00:00
FlyingOnion left a comment
Author
Member

Test result:

@Administrator ➜ xorm git(master) go test -timeout 5s -run ^TestQueryStringWithLimit$ xorm.io/xorm/integrations 
-v -count=1 -db mssql -conn_str 'sqlserver://***:********@localhost:1433?database=xorm_test'
testing mssql sqlserver://***:********@localhost:1433?database=xorm_test
[xorm] [info]  2022/10/20 14:34:06.976103 [SQL] select name from sysobjects where xtype ='U' [] - 10.0963ms
[xorm] [info]  2022/10/20 14:34:07.059214 [SQL] select a.name as name, b.name as ctype,a.max_length,a.precision,a.scale,a.is_nullable as nullable,
                  "default_is_null" = (CASE WHEN c.text is null THEN 1 ELSE 0 END),
              replace(replace(isnull(c.text,''),'(',''),')','') as vdefault,
                  ISNULL(p.is_primary_key, 0), a.is_identity as is_identity
          from sys.columns a
                  left join sys.types b on a.user_type_id=b.user_type_id
          left join sys.syscomments c on a.default_object_id=c.id
                  LEFT OUTER JOIN (SELECT i.object_id, ic.column_id, i.is_primary_key
                        FROM sys.indexes i
                  LEFT JOIN sys.index_columns ic ON ic.object_id = i.object_id AND ic.index_id = i.index_id     
                        WHERE i.is_primary_key = 1
                ) as p on p.object_id = a.object_id AND p.column_id = a.column_id
          where a.object_id=object_id('query_with_limit') [] - 51.8348ms
[xorm] [info]  2022/10/20 14:34:07.084881 [SQL] SELECT
IXS.NAME                    AS  [INDEX_NAME],
C.NAME                      AS  [COLUMN_NAME],
IXS.is_unique AS [IS_UNIQUE]
FROM SYS.INDEXES IXS
INNER JOIN SYS.INDEX_COLUMNS   IXCS
ON IXS.OBJECT_ID=IXCS.OBJECT_ID  AND IXS.INDEX_ID = IXCS.INDEX_ID
INNER   JOIN SYS.COLUMNS C  ON IXS.OBJECT_ID=C.OBJECT_ID
AND IXCS.COLUMN_ID=C.COLUMN_ID
WHERE IXS.TYPE_DESC='NONCLUSTERED' and OBJECT_NAME(IXS.OBJECT_ID) =?
 [query_with_limit] - 25.1141ms
[xorm] [info]  2022/10/20 14:34:07.087970 [SQL] BEGIN TRANSACTION [] - 3.0888ms
[xorm] [info]  2022/10/20 14:34:07.095787 [SQL] IF EXISTS (SELECT * FROM sysobjects WHERE id = object_id(N'query_with_limit') and OBJECTPROPERTY(id, N'IsUserTable') = 1) DROP TABLE "query_with_limit" [] - 7.2989ms
[xorm] [info]  2022/10/20 14:34:07.101378 [SQL] COMMIT [] - 5.5917ms
=== RUN   TestQueryStringWithLimit
[xorm] [info]  2022/10/20 14:34:07.105705 [SQL] select name from sysobjects where xtype ='U' [] - 4.327ms
[xorm] [info]  2022/10/20 14:34:07.107444 [SQL] BEGIN TRANSACTION [] - 1.7389ms
[xorm] [info]  2022/10/20 14:34:07.108522 [SQL] COMMIT [] - 1.0781ms
[xorm] [info]  2022/10/20 14:34:07.111765 [SQL] select name from sysobjects where xtype ='U' [] - 3.2431ms      
[xorm] [info]  2022/10/20 14:34:07.121200 [SQL] IF NOT EXISTS (SELECT [name] FROM sys.tables WHERE [name] = '[query_with_limit]' ) CREATE TABLE [query_with_limit] ([id] BIGINT PRIMARY KEY IDENTITY NOT NULL, [msg] VARCHAR(255) NULL, [depart_id] BIGINT NULL, [money] FLOAT NULL) [] - 8.8558ms
[xorm] [info]  2022/10/20 14:34:07.124599 [SQL] SELECT * FROM [query_with_limit] ORDER BY 1 OFFSET 20 ROWS FETCH NEXT 20 ROWS ONLY [] - 3.3988ms
--- PASS: TestQueryStringWithLimit (0.02s)
PASS
ok      xorm.io/xorm/integrations       0.553s
Test result: ```shell @Administrator ➜ xorm git(master) go test -timeout 5s -run ^TestQueryStringWithLimit$ xorm.io/xorm/integrations -v -count=1 -db mssql -conn_str 'sqlserver://***:********@localhost:1433?database=xorm_test' testing mssql sqlserver://***:********@localhost:1433?database=xorm_test [xorm] [info] 2022/10/20 14:34:06.976103 [SQL] select name from sysobjects where xtype ='U' [] - 10.0963ms [xorm] [info] 2022/10/20 14:34:07.059214 [SQL] select a.name as name, b.name as ctype,a.max_length,a.precision,a.scale,a.is_nullable as nullable, "default_is_null" = (CASE WHEN c.text is null THEN 1 ELSE 0 END), replace(replace(isnull(c.text,''),'(',''),')','') as vdefault, ISNULL(p.is_primary_key, 0), a.is_identity as is_identity from sys.columns a left join sys.types b on a.user_type_id=b.user_type_id left join sys.syscomments c on a.default_object_id=c.id LEFT OUTER JOIN (SELECT i.object_id, ic.column_id, i.is_primary_key FROM sys.indexes i LEFT JOIN sys.index_columns ic ON ic.object_id = i.object_id AND ic.index_id = i.index_id WHERE i.is_primary_key = 1 ) as p on p.object_id = a.object_id AND p.column_id = a.column_id where a.object_id=object_id('query_with_limit') [] - 51.8348ms [xorm] [info] 2022/10/20 14:34:07.084881 [SQL] SELECT IXS.NAME AS [INDEX_NAME], C.NAME AS [COLUMN_NAME], IXS.is_unique AS [IS_UNIQUE] FROM SYS.INDEXES IXS INNER JOIN SYS.INDEX_COLUMNS IXCS ON IXS.OBJECT_ID=IXCS.OBJECT_ID AND IXS.INDEX_ID = IXCS.INDEX_ID INNER JOIN SYS.COLUMNS C ON IXS.OBJECT_ID=C.OBJECT_ID AND IXCS.COLUMN_ID=C.COLUMN_ID WHERE IXS.TYPE_DESC='NONCLUSTERED' and OBJECT_NAME(IXS.OBJECT_ID) =? [query_with_limit] - 25.1141ms [xorm] [info] 2022/10/20 14:34:07.087970 [SQL] BEGIN TRANSACTION [] - 3.0888ms [xorm] [info] 2022/10/20 14:34:07.095787 [SQL] IF EXISTS (SELECT * FROM sysobjects WHERE id = object_id(N'query_with_limit') and OBJECTPROPERTY(id, N'IsUserTable') = 1) DROP TABLE "query_with_limit" [] - 7.2989ms [xorm] [info] 2022/10/20 14:34:07.101378 [SQL] COMMIT [] - 5.5917ms === RUN TestQueryStringWithLimit [xorm] [info] 2022/10/20 14:34:07.105705 [SQL] select name from sysobjects where xtype ='U' [] - 4.327ms [xorm] [info] 2022/10/20 14:34:07.107444 [SQL] BEGIN TRANSACTION [] - 1.7389ms [xorm] [info] 2022/10/20 14:34:07.108522 [SQL] COMMIT [] - 1.0781ms [xorm] [info] 2022/10/20 14:34:07.111765 [SQL] select name from sysobjects where xtype ='U' [] - 3.2431ms [xorm] [info] 2022/10/20 14:34:07.121200 [SQL] IF NOT EXISTS (SELECT [name] FROM sys.tables WHERE [name] = '[query_with_limit]' ) CREATE TABLE [query_with_limit] ([id] BIGINT PRIMARY KEY IDENTITY NOT NULL, [msg] VARCHAR(255) NULL, [depart_id] BIGINT NULL, [money] FLOAT NULL) [] - 8.8558ms [xorm] [info] 2022/10/20 14:34:07.124599 [SQL] SELECT * FROM [query_with_limit] ORDER BY 1 OFFSET 20 ROWS FETCH NEXT 20 ROWS ONLY [] - 3.3988ms --- PASS: TestQueryStringWithLimit (0.02s) PASS ok xorm.io/xorm/integrations 0.553s ```
FlyingOnion reviewed 2022-10-20 06:43:38 +00:00
@ -207,7 +207,7 @@ func (statement *Statement) writeLimitOffset(w builder.Writer) error {
_, err := fmt.Fprintf(w, " LIMIT %v OFFSET %v", *statement.LimitN, statement.Start)
return err
}
_, err := fmt.Fprintf(w, " LIMIT 0 OFFSET %v", statement.Start)
Author
Member

Should not use LIMIT 0 here, MySQL will return 0 row(s).

Should not use `LIMIT 0` here, MySQL will return 0 row(s).
FlyingOnion marked this conversation as resolved
lunny added the
kind
breaking
label 2022-10-20 12:55:19 +00:00
FlyingOnion added 1 commit 2022-10-25 03:06:38 +00:00
fix wrong statement
All checks were successful
continuous-integration/drone/pr Build is passing
630ac63125
FlyingOnion added 1 commit 2022-10-25 03:53:41 +00:00
Merge branch 'master' into master
All checks were successful
continuous-integration/drone/pr Build is passing
31f2fe5642
Author
Member

@lunny Is it able to merge?

@lunny Is it able to merge?
Owner

@lunny Is it able to merge?

Since it's a break change(some MSSQL version will not be supported), I have to do it carefully but the PR is ready.

> @lunny Is it able to merge? Since it's a break change(some MSSQL version will not be supported), I have to do it carefully but the PR is ready.
Owner

This requires SQL Server 2012, maybe we can have a global variable to switch.

This requires SQL Server 2012, maybe we can have a global variable to switch.
lunny added 1 commit 2023-07-12 09:54:14 +00:00
Merge branch 'master' of gitea.com:FlyingOnion/xorm into FlyingOnion-master
Some checks failed
test mariadb / test mariadb (pull_request) Successful in 4m25s
test cockroach / test cockroach (pull_request) Successful in 6m33s
test mssql / test mssql (pull_request) Failing after 4m40s
test mysql / test mysql (pull_request) Successful in 4m40s
test mysql8 / test mysql8 (pull_request) Successful in 4m43s
test postgres / test postgres (pull_request) Successful in 5m12s
test tidb / test tidb (pull_request) Successful in 5m11s
test sqlite / unit test & test sqlite (pull_request) Successful in 7m37s
6cb9b2ac55
Owner

CI failure is related.

CI failure is related.
Author
Member

ORDER BY is mandatory to use OFFSET and FETCH clause.

Shall we add ORDER BY 1 (order by the first column) if statement.HasOrderBy() is false?

@lunny

ORDER BY is mandatory to use OFFSET and FETCH clause. Shall we add `ORDER BY 1` (order by the first column) if `statement.HasOrderBy()` is false? @lunny
Owner

ORDER BY is mandatory to use OFFSET and FETCH clause.

Shall we add ORDER BY 1 (order by the first column) if statement.HasOrderBy() is false?

@lunny

If there is a primary key, use primary key asc, otherwise the first column asc ?

> ORDER BY is mandatory to use OFFSET and FETCH clause. > > Shall we add `ORDER BY 1` (order by the first column) if `statement.HasOrderBy()` is false? > > @lunny If there is a primary key, use primary key asc, otherwise the first column asc ?
lunny added 1 commit 2023-07-23 01:05:35 +00:00
Merge branch 'master' into master
Some checks failed
test mariadb / test mariadb (pull_request) Successful in 4m51s
test cockroach / test cockroach (pull_request) Successful in 7m31s
test mssql / test mssql (pull_request) Failing after 4m46s
test mysql / test mysql (pull_request) Successful in 5m3s
test mysql8 / test mysql8 (pull_request) Successful in 5m4s
test postgres / test postgres (pull_request) Successful in 5m43s
test tidb / test tidb (pull_request) Successful in 5m26s
test sqlite / unit test & test sqlite (pull_request) Successful in 9m4s
bff31e3db4
Owner

How about to add an option to fall back to old method.

How about to add an option to fall back to old method.
FlyingOnion changed title from modify limit offset implement to WIP: modify limit offset implement 2023-07-23 03:42:07 +00:00
Author
Member

Sorry for late response. My computer got repaired, so the PR remains WIP.


ORDER BY is mandatory to use OFFSET and FETCH clause.

Shall we add ORDER BY 1 (order by the first column) if statement.HasOrderBy() is false?

@lunny

If there is a primary key, use primary key asc, otherwise the first column asc ?

Still working on it.


How about to add an option to fall back to old method.

It's dialect-dependent (mostly used in SQLServer and Oracle) so maybe func NewEngineWithParams is a good place to set the option.

db, err := NewEngineWithParams("mssql", "your-uri", map[string]string{"USE_LEGACY_LIMIT_OFFSET", "true"}
Sorry for late response. My computer got repaired, so the PR remains WIP. --- > > ORDER BY is mandatory to use OFFSET and FETCH clause. > > > > Shall we add `ORDER BY 1` (order by the first column) if `statement.HasOrderBy()` is false? > > > > @lunny > > If there is a primary key, use primary key asc, otherwise the first column asc ? Still working on it. --- > How about to add an option to fall back to old method. It's dialect-dependent (mostly used in SQLServer and Oracle) so maybe `func NewEngineWithParams` is a good place to set the option. ```go db, err := NewEngineWithParams("mssql", "your-uri", map[string]string{"USE_LEGACY_LIMIT_OFFSET", "true"} ```
FlyingOnion force-pushed master from 39ac77bdc9 to 3726353ad0 2023-08-01 01:49:45 +00:00 Compare
FlyingOnion added 1 commit 2023-08-15 03:39:10 +00:00
更新 .gitea/workflows/test-mysql.yml
Some checks failed
test tidb / test tidb (pull_request) Failing after 4m50s
test sqlite / unit test & test sqlite (pull_request) Failing after 8m40s
test cockroach / test cockroach (pull_request) Failing after 5m23s
test mariadb / test mariadb (pull_request) Failing after 4m42s
test mysql / test mysql (pull_request) Failing after 0s
test mssql / test mssql (pull_request) Failing after 5m10s
test mysql8 / test mysql8 (pull_request) Failing after 4m52s
test postgres / test postgres (pull_request) Failing after 2m55s
db29839631
Author
Member

Hi @lunny

Could you grant me the permission to run tests? I'm not able to run it with my own runner because of poor network. I have tried several ways to deal with it but all failed.

Thank you very much.

Hi @lunny Could you grant me the permission to run tests? I'm not able to run it with my own runner because of poor network. I have tried several ways to deal with it but all failed. Thank you very much.
Owner

Hi @lunny

Could you grant me the permission to run tests? I'm not able to run it with my own runner because of poor network. I have tried several ways to deal with it but all failed.

Thank you very much.

OK now.

> Hi @lunny > > Could you grant me the permission to run tests? I'm not able to run it with my own runner because of poor network. I have tried several ways to deal with it but all failed. > > Thank you very much. OK now.
FlyingOnion force-pushed master from db29839631 to 50ed5f6bca 2023-09-07 08:41:35 +00:00 Compare
FlyingOnion added 1 commit 2023-09-07 09:48:55 +00:00
fix
Some checks failed
test mariadb / test mariadb (pull_request) Failing after 4m35s
test cockroach / test cockroach (pull_request) Failing after 5m48s
test mssql / test mssql (pull_request) Failing after 4m44s
test mysql / test mysql (pull_request) Failing after 4m37s
test mysql8 / test mysql8 (pull_request) Failing after 4m35s
test postgres / test postgres (pull_request) Failing after 4m27s
test tidb / test tidb (pull_request) Failing after 4m57s
test sqlite / unit test & test sqlite (pull_request) Failing after 7m7s
5975b057d6
FlyingOnion added 1 commit 2023-09-08 01:46:27 +00:00
fix
Some checks failed
test mariadb / test mariadb (pull_request) Failing after 4m49s
test cockroach / test cockroach (pull_request) Failing after 5m46s
test mssql / test mssql (pull_request) Failing after 4m51s
test mysql / test mysql (pull_request) Failing after 4m49s
test mysql8 / test mysql8 (pull_request) Failing after 4m44s
test postgres / test postgres (pull_request) Failing after 4m32s
test tidb / test tidb (pull_request) Failing after 4m51s
test sqlite / unit test & test sqlite (pull_request) Failing after 7m2s
f4cc1db4ba
FlyingOnion force-pushed master from f4cc1db4ba to 9ccf24b957 2023-09-08 09:45:48 +00:00 Compare
FlyingOnion added 1 commit 2023-09-11 11:10:56 +00:00
test trial 5
All checks were successful
test mariadb / test mariadb (pull_request) Successful in 4m44s
test cockroach / test cockroach (pull_request) Successful in 7m6s
test mssql / test mssql (pull_request) Successful in 5m14s
test mysql / test mysql (pull_request) Successful in 4m24s
test mysql8 / test mysql8 (pull_request) Successful in 4m47s
test postgres / test postgres (pull_request) Successful in 5m24s
test tidb / test tidb (pull_request) Successful in 5m19s
test sqlite / unit test & test sqlite (pull_request) Successful in 8m1s
bc896febff
FlyingOnion changed title from WIP: modify limit offset implement to modify limit offset implement 2023-09-11 12:11:33 +00:00
Author
Member

@lunny

I finished the task and passed all unit tests, but these tests do not contain NewEngineWithParams or engine.SetParams, so legacy cases are not tested at all.

Should an extra job be added in test-mssql ? Or open another issue?

@lunny I finished the task and passed all unit tests, but these tests do not contain `NewEngineWithParams` or `engine.SetParams`, so legacy cases are not tested at all. Should an extra job be added in `test-mssql` ? Or open another issue?
Owner

@lunny

I finished the task and passed all unit tests, but these tests do not contain NewEngineWithParams or engine.SetParams, so legacy cases are not tested at all.

Should an extra job be added in test-mssql ? Or open another issue?

There are still two files conflicted?

> @lunny > > I finished the task and passed all unit tests, but these tests do not contain `NewEngineWithParams` or `engine.SetParams`, so legacy cases are not tested at all. > > Should an extra job be added in `test-mssql` ? Or open another issue? There are still two files conflicted?
FlyingOnion force-pushed master from bc896febff to 9b0829fb30 2023-09-13 09:30:54 +00:00 Compare
Author
Member

@lunny

In this case the unit test failed.

total, err = testEngine.OrderBy("`name`").Count(CountWithTableName{})

// the orderby will be ignored by count because some databases will return errors if the orderby columns not in group by
total, err = testEngine.OrderBy("`name`").Count(CountWithTableName{})

As you commented, it happened when statement.writeSelect removed parameter needOrderBy. The ORDER BY clause is always added (if len(statement.orderStr) > 0), which is what a former commit did in my fork (it has been overwritten by merging upstream master).

If we still keep needOrderBy, an error may occur when it's false with .Limit and .Offset in sqlserver.

I suggest we remove .OrderBy("`name`") in the unit test. IMO the result of calling .OrderBy().Count() should be left to the database itself to decide. ORM should not do extra pruning. The error message is also clear enough for developers to modify their code.

Besides, it's of no use to use ORDER BY in count(*).

@lunny In this case the unit test failed. https://gitea.com/xorm/xorm/src/commit/eeacd22674314a0712f91033c91185a33c83cacb/tests/session_count_test.go#L130 ```go // the orderby will be ignored by count because some databases will return errors if the orderby columns not in group by total, err = testEngine.OrderBy("`name`").Count(CountWithTableName{}) ``` As you commented, it happened when `statement.writeSelect` removed parameter `needOrderBy`. The `ORDER BY` clause is always added (if `len(statement.orderStr) > 0`), which is what a former commit did in my fork (it has been overwritten by merging upstream master). If we still keep `needOrderBy`, an error may occur when it's false with `.Limit` and `.Offset` in sqlserver. I suggest we remove ``.OrderBy("`name`")`` in the unit test. IMO the result of calling `.OrderBy().Count()` should be left to the database itself to decide. ORM should not do extra pruning. The error message is also clear enough for developers to modify their code. Besides, it's of no use to use `ORDER BY` in `count(*)`.
FlyingOnion added 1 commit 2023-09-18 02:00:05 +00:00
remove a unit test
All checks were successful
test mariadb / test mariadb (pull_request) Successful in 4m42s
test cockroach / test cockroach (pull_request) Successful in 7m17s
test mssql / test mssql (pull_request) Successful in 5m31s
test mysql / test mysql (pull_request) Successful in 4m37s
test mysql8 / test mysql8 (pull_request) Successful in 4m58s
test postgres / test postgres (pull_request) Successful in 5m27s
test tidb / test tidb (pull_request) Successful in 5m24s
test sqlite / unit test & test sqlite (pull_request) Successful in 8m9s
5e31f7d14b
lunny reviewed 2023-09-18 04:13:15 +00:00
@ -0,0 +1,55 @@
package statements
Owner

Please add copyright header.

Please add copyright header.
lunny added this to the 1.3.4 milestone 2023-09-18 04:16:13 +00:00
FlyingOnion added 1 commit 2023-09-20 00:38:20 +00:00
add copyright header
All checks were successful
test mariadb / test mariadb (pull_request) Successful in 4m36s
test cockroach / test cockroach (pull_request) Successful in 7m17s
test mssql / test mssql (pull_request) Successful in 5m9s
test mysql / test mysql (pull_request) Successful in 4m41s
test mysql8 / test mysql8 (pull_request) Successful in 4m50s
test postgres / test postgres (pull_request) Successful in 5m24s
test tidb / test tidb (pull_request) Successful in 5m25s
test sqlite / unit test & test sqlite (pull_request) Successful in 8m26s
47cb78ba3e
lunny approved these changes 2023-09-20 02:06:56 +00:00
lunny merged commit 551de3767c into master 2023-09-20 02:07:03 +00:00
Sign in to join this conversation.
No description provided.