XORM Session.Find接口出现data race 检测报错问题 #2217

Open
opened 2023-02-08 10:05:15 +00:00 by E-c-h-o-Go · 6 comments

XORM Session.Find接口出现data race 检测报错问题

go 1.20

mysql 5.x。字段类型blob

xorm 版本 latest
代码如下

sql := mysql.Sql().NewSession()
			defer sql.Close()
			if conditions, ok := reflect.ValueOf(entity).Elem().FieldByName(target).Interface().(define.SqlConditions); ok {
				conditions.Where(func(query string, args ...any) {
					sql = sql.Where(query, args...)
				})
			}
			var result []map[string]any
			if err := sql.Table(src).Find(&result); err != nil {
				return nil, err
			}
			if target == "Hash" {
				go func() {
					for {
						fmt.Println(result[0]["cache"])
					}
				}()
			}
			return result, nil

上述代码中 go fun 是我用于测试开启的goroutine
按照基本理解sql.Table(src).Find(&result) 这段代码应该是查询完毕返回而不是异步
那么再多次fmt.Println(result[0]["cache"]) 时。会出现该字段被更改(字段类型[]byte)

更改后结果大约为该字段之前的字段的残余数据,但是之前的字段并没有数据错误

data race 结果如下

Found 1 data race(s)

WARNING: DATA RACE
Write at 0x00c000440298 by main goroutine:
runtime.racewriterange()
:1 +0x14
internal/poll.ignoringEINTRIO()
/Users//SDK/go/go1.20/src/internal/poll/fd_unix.go:794 +0x2f0
internal/poll.(FD).Read()
/Users/
/SDK/go/go1.20/src/internal/poll/fd_unix.go:163 +0x20
net.(netFD).Read()
/Users/
/SDK/go/go1.20/src/net/fd_posix.go:55 +0x44
net.(conn).Read()
/Users/
/SDK/go/go1.20/src/net/net.go:183 +0x84
net.(TCPConn).Read()
:1 +0x4c
github.com/go-sql-driver/mysql.(buffer).fill()
/Users/
/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/buffer.go:90 +0x328
github.com/go-sql-driver/mysql.(buffer).readNext()
/Users/
/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/buffer.go:119 +0x54
github.com/go-sql-driver/mysql.(mysqlConn).readPacket()
/Users/
/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/packets.go:32 +0xa4
github.com/go-sql-driver/mysql.(mysqlConn).readResultSetHeaderPacket()
/Users/
/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/packets.go:537 +0x28
github.com/go-sql-driver/mysql.(mysqlStmt).query()
/Users/
/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/statement.go:114 +0x23c
github.com/go-sql-driver/mysql.(mysqlStmt).QueryContext()
/Users/
/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/connection.go:558 +0x1e0
database/sql.ctxDriverStmtQuery()
/Users/
/SDK/go/go1.20/src/database/sql/ctxutil.go:82 +0x9c
database/sql.rowsiFromStatement()
/Users/
/SDK/go/go1.20/src/database/sql/sql.go:2801 +0x140
database/sql.(DB).queryDC()
/Users/
/SDK/go/go1.20/src/database/sql/sql.go:1778 +0x378
database/sql.(DB).query()
/Users/
/SDK/go/go1.20/src/database/sql/sql.go:1726 +0xec
database/sql.(DB).QueryContext.func1()
/Users/
/SDK/go/go1.20/src/database/sql/sql.go:1704 +0x9c
database/sql.(DB).retry()
/Users/
/SDK/go/go1.20/src/database/sql/sql.go:1538 +0x4c
database/sql.(DB).QueryContext()
/Users/
/SDK/go/go1.20/src/database/sql/sql.go:1703 +0xc8
xorm.io/xorm/core.(DB).QueryContext()
/Users/
/go/pkg/mod/xorm.io/xorm@v1.3.2/core/db.go:151 +0x224
xorm.io/xorm.(Session).queryRows()
/Users/
/go/pkg/mod/xorm.io/xorm@v1.3.2/session_raw.go:52 +0x40c
xorm.io/xorm.(Session).noCacheFind()
/Users/
/go/pkg/mod/xorm.io/xorm@v1.3.2/session_find.go:175 +0x180
xorm.io/xorm.(Session).find()
/Users/
/go/pkg/mod/xorm.io/xorm@v1.3.2/session_find.go:161 +0xcec
xorm.io/xorm.(Session).Find()
/Users/
/go/pkg/mod/xorm.io/xorm@v1.3.2/session_find.go:31 +0xa0
/data/model.(Mysql).Load.func1()
/Users/
/Server/Project/
/data/model/mysql.go:101 +0x374
/data/model.loadAndSet...
/Users/
/Server/Project/
/data/model/util.go:282 +0x334
/data/model.(Mysql).Load()
/Users/
/Server/Project/
/data/model/mysql.go:91 +0x88
main.(M).Load()
:1 +0x54
/define.Model.Load()
:1 +0x60
/data._RangeEntities()
/Users/
/Server/Project/
/data/data.go:49 +0x57c
/data.Load()
/Users/
/Server/Project/
/data/data.go:11 +0x38
main.main()
/Users/
/Server/Project/***/main.go:29 +0x170

Previous read at 0x00c00044029e by goroutine 30:
fmt.(pp).fmtBytes()
/Users/
/SDK/go/go1.20/src/fmt/print.go:528 +0x5f8
fmt.(pp).printArg()
/Users/
/SDK/go/go1.20/src/fmt/print.go:743 +0x480
fmt.(pp).doPrintln()
/Users/
/SDK/go/go1.20/src/fmt/print.go:1223 +0x40
fmt.Fprintln()
/Users/
/SDK/go/go1.20/src/fmt/print.go:304 +0x48
fmt.Println()
/Users/
/SDK/go/go1.20/src/fmt/print.go:314 +0x9c
/data/model.(Mysql).Load.func1.2()
/Users/
/Server/Project/
/data/model/mysql.go:107 +0x2c

Goroutine 30 (running) created at:
/data/model.(Mysql).Load.func1()
/Users/
/Server/Project/
/data/model/mysql.go:105 +0x3fc
/data/model.loadAndSet...
/Users/
/Server/Project/
/data/model/util.go:282 +0x334
/data/model.(Mysql).Load()
/Users/
/Server/Project/
/data/model/mysql.go:91 +0x88
main.(M).Load()
:1 +0x54
/define.Model.Load()
:1 +0x60
/data._RangeEntities()
/Users/
/Server/Project/
/data/data.go:49 +0x57c
/data.Load()
/Users/
/Server/Project/
/data/data.go:11 +0x38
main.main()
/Users/
/Server/Project/*/main.go:29 +0x170

XORM Session.Find接口出现data race 检测报错问题 go 1.20 mysql 5.x。字段类型blob xorm 版本 latest 代码如下 ``` sql := mysql.Sql().NewSession() defer sql.Close() if conditions, ok := reflect.ValueOf(entity).Elem().FieldByName(target).Interface().(define.SqlConditions); ok { conditions.Where(func(query string, args ...any) { sql = sql.Where(query, args...) }) } var result []map[string]any if err := sql.Table(src).Find(&result); err != nil { return nil, err } if target == "Hash" { go func() { for { fmt.Println(result[0]["cache"]) } }() } return result, nil ``` 上述代码中 go fun 是我用于测试开启的goroutine 按照基本理解sql.Table(src).Find(&result) 这段代码应该是查询完毕返回而不是异步 那么再多次fmt.Println(result[0]["cache"]) 时。会出现该字段被更改(字段类型[]byte) 更改后结果大约为该字段之前的字段的残余数据,但是之前的字段并没有数据错误 data race 结果如下 Found 1 data race(s) WARNING: DATA RACE Write at 0x00c000440298 by main goroutine: runtime.racewriterange() <autogenerated>:1 +0x14 internal/poll.ignoringEINTRIO() /Users/****/SDK/go/go1.20/src/internal/poll/fd_unix.go:794 +0x2f0 internal/poll.(*FD).Read() /Users/****/SDK/go/go1.20/src/internal/poll/fd_unix.go:163 +0x20 net.(*netFD).Read() /Users/****/SDK/go/go1.20/src/net/fd_posix.go:55 +0x44 net.(*conn).Read() /Users/****/SDK/go/go1.20/src/net/net.go:183 +0x84 net.(*TCPConn).Read() <autogenerated>:1 +0x4c github.com/go-sql-driver/mysql.(*buffer).fill() /Users/****/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/buffer.go:90 +0x328 github.com/go-sql-driver/mysql.(*buffer).readNext() /Users/****/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/buffer.go:119 +0x54 github.com/go-sql-driver/mysql.(*mysqlConn).readPacket() /Users/****/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/packets.go:32 +0xa4 github.com/go-sql-driver/mysql.(*mysqlConn).readResultSetHeaderPacket() /Users/****/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/packets.go:537 +0x28 github.com/go-sql-driver/mysql.(*mysqlStmt).query() /Users/****/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/statement.go:114 +0x23c github.com/go-sql-driver/mysql.(*mysqlStmt).QueryContext() /Users/****/go/pkg/mod/github.com/go-sql-driver/mysql@v1.7.0/connection.go:558 +0x1e0 database/sql.ctxDriverStmtQuery() /Users/****/SDK/go/go1.20/src/database/sql/ctxutil.go:82 +0x9c database/sql.rowsiFromStatement() /Users/****/SDK/go/go1.20/src/database/sql/sql.go:2801 +0x140 database/sql.(*DB).queryDC() /Users/****/SDK/go/go1.20/src/database/sql/sql.go:1778 +0x378 database/sql.(*DB).query() /Users/****/SDK/go/go1.20/src/database/sql/sql.go:1726 +0xec database/sql.(*DB).QueryContext.func1() /Users/****/SDK/go/go1.20/src/database/sql/sql.go:1704 +0x9c database/sql.(*DB).retry() /Users/****/SDK/go/go1.20/src/database/sql/sql.go:1538 +0x4c database/sql.(*DB).QueryContext() /Users/****/SDK/go/go1.20/src/database/sql/sql.go:1703 +0xc8 xorm.io/xorm/core.(*DB).QueryContext() /Users/****/go/pkg/mod/xorm.io/xorm@v1.3.2/core/db.go:151 +0x224 xorm.io/xorm.(*Session).queryRows() /Users/****/go/pkg/mod/xorm.io/xorm@v1.3.2/session_raw.go:52 +0x40c xorm.io/xorm.(*Session).noCacheFind() /Users/****/go/pkg/mod/xorm.io/xorm@v1.3.2/session_find.go:175 +0x180 xorm.io/xorm.(*Session).find() /Users/****/go/pkg/mod/xorm.io/xorm@v1.3.2/session_find.go:161 +0xcec xorm.io/xorm.(*Session).Find() /Users/****/go/pkg/mod/xorm.io/xorm@v1.3.2/session_find.go:31 +0xa0 ****/data/model.(*Mysql).Load.func1() /Users/****/Server/Project/****/data/model/mysql.go:101 +0x374 ****/data/model.loadAndSet[...]() /Users/****/Server/Project/****/data/model/util.go:282 +0x334 ****/data/model.(*Mysql).Load() /Users/****/Server/Project/****/data/model/mysql.go:91 +0x88 main.(*M).Load() <autogenerated>:1 +0x54 ****/define.Model.Load() <autogenerated>:1 +0x60 ****/data._RangeEntities() /Users/****/Server/Project/****/data/data.go:49 +0x57c ****/data.Load() /Users/****/Server/Project/****/data/data.go:11 +0x38 main.main() /Users/****/Server/Project/****/main.go:29 +0x170 Previous read at 0x00c00044029e by goroutine 30: fmt.(*pp).fmtBytes() /Users/****/SDK/go/go1.20/src/fmt/print.go:528 +0x5f8 fmt.(*pp).printArg() /Users/****/SDK/go/go1.20/src/fmt/print.go:743 +0x480 fmt.(*pp).doPrintln() /Users/****/SDK/go/go1.20/src/fmt/print.go:1223 +0x40 fmt.Fprintln() /Users/****/SDK/go/go1.20/src/fmt/print.go:304 +0x48 fmt.Println() /Users/****/SDK/go/go1.20/src/fmt/print.go:314 +0x9c ****/data/model.(*Mysql).Load.func1.2() /Users/****/Server/Project/****/data/model/mysql.go:107 +0x2c Goroutine 30 (running) created at: ****/data/model.(*Mysql).Load.func1() /Users/****/Server/Project/****/data/model/mysql.go:105 +0x3fc ****/data/model.loadAndSet[...]() /Users/****/Server/Project/****/data/model/util.go:282 +0x334 ****/data/model.(*Mysql).Load() /Users/****/Server/Project/****/data/model/mysql.go:91 +0x88 main.(*M).Load() <autogenerated>:1 +0x54 ****/define.Model.Load() <autogenerated>:1 +0x60 ****/data._RangeEntities() /Users/****/Server/Project/****/data/data.go:49 +0x57c ****/data.Load() /Users/****/Server/Project/****/data/data.go:11 +0x38 main.main() /Users/****/Server/Project/****/main.go:29 +0x170
Owner

How did you implement the Where function? Are there goroutines there?

conditions.Where(func(query string, args ...any) {
					sql = sql.Where(query, args...)
				})
How did you implement the `Where` function? Are there goroutines there? ```go conditions.Where(func(query string, args ...any) { sql = sql.Where(query, args...) }) ```
Author

How did you implement the Where function? Are there goroutines there?

conditions.Where(func(query string, args ...any) {
					sql = sql.Where(query, args...)
				})

this function is a interface{}

only a normal function call

code:

type Hash struct {
	xxx   string       `field:"xxxx"`
	FFF    string       `field:"fff"`
	Uint   uint         `field:"uint"`
	Int    int          `field:"int"`
	Bool   bool         `field:"bool"`
	Bool2  bool         `field:"bool2"`
	Float  *float32     `field:"float"`
	Map    Map          `field:"map"`
	Struct *ListSub     `field:"struct"`
	Array  []int        `field:"array"`
	Cache  *cache.Cache `field:"cache"`
}

func (h *Hash) Where(where func(query string, args ...any)) {
	where("xxx = ?", 2)
}

type SliceHash []*Hash

func (s SliceHash) Where(where func(query string, args ...any)) {
	where("xxx = ?", 222)
}

func (s SliceHash) PrimaryKeyField() string {
	return "xxx"
}
> How did you implement the `Where` function? Are there goroutines there? > > ```go > conditions.Where(func(query string, args ...any) { > sql = sql.Where(query, args...) > }) > ``` this function is a interface{} only a normal function call code: ``` type Hash struct { xxx string `field:"xxxx"` FFF string `field:"fff"` Uint uint `field:"uint"` Int int `field:"int"` Bool bool `field:"bool"` Bool2 bool `field:"bool2"` Float *float32 `field:"float"` Map Map `field:"map"` Struct *ListSub `field:"struct"` Array []int `field:"array"` Cache *cache.Cache `field:"cache"` } func (h *Hash) Where(where func(query string, args ...any)) { where("xxx = ?", 2) } type SliceHash []*Hash func (s SliceHash) Where(where func(query string, args ...any)) { where("xxx = ?", 222) } func (s SliceHash) PrimaryKeyField() string { return "xxx" } ```
Author

Right Bytes :
[10 49 10 5 102 117 99 107 50 18 40 10 38 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46 99 111 109 47 112 98 46 83 116 114 105 110 103 18 5 10 3 102 102 102 10 60 10 5 102 117 99 107 49 18 51 10 39 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46 99 111 109 47 112 98 46 83 116 114 105 110 103 18 6 10 4 102 102 102 49 16 184 141 198 182 161 225 134 161 23 10 80 10 4 102 117 99 107 18 72 10 60 10 28 116 121 112 101 46 103 111 111 103 108 44 34 65 50 34 58 34 50 34 44 34 65 51 34 58 51 125 16 136 195 177 235 210 200 134 161 23 10 49 10 5 102 117 99 107 50 18 40 10 38 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46]

Bad Bytes:
[52 50 51 52 125 13 49 44 50 44 51 44 52 44 53 44 54 44 55 195 10 60 10 5 102 117 99 107 49 18 51 10 39 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46 99 111 109 47 112 98 46 83 116 114 105 110 103 18 6 10 4 102 102 102 49 16 136 195 177 235 210 200 134 161 23 10 80 10 4 102 117 99 107 18 72 10 60 10 28 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46 99 111 109 47 112 98 46 66 121 116 101 115 18 28 10 26 123 34 65 49 34 58 34 49 34 44 34 65 50 34 58 34 50 34 44 34 65 51 34 58 51 125 16 136 195 177 235 210 200 134 161 23 10 49 10 5 102 117 99 107 50 18 40 10 38 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46]

This happens the second time I call this function, the bytes will be modified

The above code is an anonymous function, so it is not called continuously

Right Bytes : [10 49 10 5 102 117 99 107 50 18 40 10 38 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46 99 111 109 47 112 98 46 83 116 114 105 110 103 18 5 10 3 102 102 102 10 60 10 5 102 117 99 107 49 18 51 10 39 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46 99 111 109 47 112 98 46 83 116 114 105 110 103 18 6 10 4 102 102 102 49 16 184 141 198 182 161 225 134 161 23 10 80 10 4 102 117 99 107 18 72 10 60 10 28 116 121 112 101 46 103 111 111 103 108 44 34 65 50 34 58 34 50 34 44 34 65 51 34 58 51 125 16 136 195 177 235 210 200 134 161 23 10 49 10 5 102 117 99 107 50 18 40 10 38 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46] Bad Bytes: [52 50 51 52 125 13 49 44 50 44 51 44 52 44 53 44 54 44 55 195 10 60 10 5 102 117 99 107 49 18 51 10 39 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46 99 111 109 47 112 98 46 83 116 114 105 110 103 18 6 10 4 102 102 102 49 16 136 195 177 235 210 200 134 161 23 10 80 10 4 102 117 99 107 18 72 10 60 10 28 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46 99 111 109 47 112 98 46 66 121 116 101 115 18 28 10 26 123 34 65 49 34 58 34 49 34 44 34 65 50 34 58 34 50 34 44 34 65 51 34 58 51 125 16 136 195 177 235 210 200 134 161 23 10 49 10 5 102 117 99 107 50 18 40 10 38 10 29 116 121 112 101 46 103 111 111 103 108 101 97 112 105 115 46] This happens the second time I call this function, the bytes will be modified The above code is an anonymous function, so it is not called continuously
Author

This problem is similar to one in 2018.

https://blog.csdn.net/itfootball/article/details/84788923

But this problem has been fixed. I don't know why the bug is still there.

This problem is similar to one in 2018. > https://blog.csdn.net/itfootball/article/details/84788923 But this problem has been fixed. I don't know why the bug is still there.
lunny added the
upstream
label 2023-02-19 23:27:41 +00:00
Owner

If it's a problem. It's related with upstream library. Please report it on upstream repository.

If it's a problem. It's related with upstream library. Please report it on upstream repository.

It is because XORM uses sql.RawBytes.
Read the document for sql.RawBytes: https://pkg.go.dev/database/sql#RawBytes

After a Scan into a RawBytes, the slice is only valid until the next call to Next, Scan, or Close.

Your code used RawBytes from another goroutine. It caused race.

GenScanResult should not use sql.RawBytes. Not only for MySQL, but also other drivers.

It is because XORM uses `sql.RawBytes`. Read the document for sql.RawBytes: https://pkg.go.dev/database/sql#RawBytes > After a Scan into a RawBytes, the slice is only valid until the next call to Next, Scan, or Close. Your code used RawBytes from another goroutine. It caused race. GenScanResult should not use `sql.RawBytes`. Not only for MySQL, but also other drivers.
Sign in to join this conversation.
No Milestone
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: xorm/xorm#2217
No description provided.