cache issue #330

New Issue

lunny · 2016-01-06T09:30:20Z

hsyan2008 commented

2016-01-06 09:30:20 +00:00

(Migrated from github.com)

想请教下cache的策略是什么
主要是碰到如下问题
数据库里原先有一条记录，每隔3秒查询，语句类似
select * from table order by next_time asc
查询的时候命中缓存
然后在运行的时候插入一条数据，然后log就一直如下提示
[xorm] [debug] 2016/01/06 17:25:34.950138 [cacheFind] cache hit sql: SELECT id FROM mail_boxes ORDER BY next_time asc []
[xorm] [info] 2016/01/06 17:25:34.950222 [sql] SELECT id, is_deleted, updated_at, created_at, username, password, status, next_time FROM mail_boxes WHERE (id IN (?,?)) [args] [1 2]
[xorm] [debug] 2016/01/06 17:25:34.952121 [cacheFind] cache bean: mail_boxes [] &{{1 N 2016-01-06 17:18:06 +0800 CST 2016-01-04 21:33:17 +0800 CST} aaaaaaaa aaaaaaa S 1452072038} [0xc8205acb00 ]
[xorm] [debug] 2016/01/06 17:25:34.952158 [cacheFind] cache bean: mail_boxes [] &{{2 N 2016-01-06 17:20:43 +0800 CST 2016-01-06 17:19:21 +0800 CST} bbbbbbb bbbbbbb S 1452072343} [0xc8205acb80 ]
[xorm] [warn] 2016/01/06 17:25:34.952170 [cacheFind] cache no hit: mail_boxes [2] [0xc8205acb80 ]

想请教下cache的策略是什么主要是碰到如下问题数据库里原先有一条记录，每隔3秒查询，语句类似 select \* from table order by next_time asc 查询的时候命中缓存然后在运行的时候插入一条数据，然后log就一直如下提示 [xorm] [debug] 2016/01/06 17:25:34.950138 [cacheFind] cache hit sql: SELECT `id` FROM `mail_boxes` ORDER BY next_time asc [] [xorm] [info] 2016/01/06 17:25:34.950222 [sql] SELECT `id`, `is_deleted`, `updated_at`, `created_at`, `username`, `password`, `status`, `next_time` FROM `mail_boxes` WHERE (`id` IN (?,?)) [args] [1 2] [xorm] [debug] 2016/01/06 17:25:34.952121 [cacheFind] cache bean: mail_boxes [<nil>] &{{1 N 2016-01-06 17:18:06 +0800 CST 2016-01-04 21:33:17 +0800 CST} aaaaaaaa aaaaaaa S 1452072038} [0xc8205acb00 <nil>] [xorm] [debug] 2016/01/06 17:25:34.952158 [cacheFind] cache bean: mail_boxes [<nil>] &{{2 N 2016-01-06 17:20:43 +0800 CST 2016-01-06 17:19:21 +0800 CST} bbbbbbb bbbbbbb S 1452072343} [0xc8205acb80 <nil>] [xorm] [warn] 2016/01/06 17:25:34.952170 [cacheFind] cache no hit: mail_boxes [2] [0xc8205acb80 <nil>]

lunny commented

2016-01-06 13:23:03 +00:00

只是一些提示信息，看最后的结果是否正确。

hsyan2008 commented

2016-01-06 16:54:45 +00:00

(Migrated from github.com)

结果不太对，数据库里有两条数据，看打印的sql语句里，id也是两个，但是结果打印出来却只有一个，如果关闭缓存，一切正常，我给你贴下部分代码
1、这是查数据库的

func (mailbox *MailBoxes) Search(t *[]MailBoxes, cond map[string]interface{}) error {
    var (
            str      []string
            args     []interface{}
            orderby  string = "id desc"
            page     int    = 1
            pageSize int    = 1000
    )
    for k, v := range cond {
            if k == "orderby" {
                    orderby = v.(string)
                    continue
            }
            if k == "page" {
                    page = max(v.(int), page)
                    continue
            }
            if k == "pagesize" {
                    pageSize = max(v.(int), 1)
                    continue
            }
            str = append(str, fmt.Sprintf("%s = ?", k))
            args = append(args, v)
    }

    err := engine.Where(strings.Join(str, " "), args...).OrderBy(orderby).
            Limit(pageSize, (page-1)*pageSize).Find(t)
    if err != nil {
            logger.Warn(err)
    }

    return err
}

2、这是调用的

            var mailboxes []models.MailBoxes
            mailbox.Search(&mailboxes, map[string]interface{}{"orderby": "next_time asc"})

结果不太对，数据库里有两条数据，看打印的sql语句里，id也是两个，但是结果打印出来却只有一个，如果关闭缓存，一切正常，我给你贴下部分代码 1、这是查数据库的 ``` func (mailbox *MailBoxes) Search(t *[]MailBoxes, cond map[string]interface{}) error { var ( str []string args []interface{} orderby string = "id desc" page int = 1 pageSize int = 1000 ) for k, v := range cond { if k == "orderby" { orderby = v.(string) continue } if k == "page" { page = max(v.(int), page) continue } if k == "pagesize" { pageSize = max(v.(int), 1) continue } str = append(str, fmt.Sprintf("%s = ?", k)) args = append(args, v) } err := engine.Where(strings.Join(str, " "), args...).OrderBy(orderby). Limit(pageSize, (page-1)*pageSize).Find(t) if err != nil { logger.Warn(err) } return err } ``` 2、这是调用的 ``` var mailboxes []models.MailBoxes mailbox.Search(&mailboxes, map[string]interface{}{"orderby": "next_time asc"}) ```

hsyan2008 commented

2016-01-12 03:43:36 +00:00

(Migrated from github.com)

今天定位一下问题，找到了问题所在，限于水平有限，还是得lunny出马
首先见https://github.com/go-xorm/xorm/issues/328，我的结构体采用组合方式，这样可以复用公共的字段设置，所以结构的设置如下
type Public struct {
Id int64 xorm:"pk autoincr"
IsDeleted string
UpdatedAt time.Time xorm:"updated"
CreatedAt time.Time xorm:"created"
}
type Mails struct {
Username string xorm:"not null default '' VARCHAR(255)"
Password string xorm:"not null default '' VARCHAR(255)"
Status string xorm:"not null default 'N' ENUM('S','E','N')"
Public xorm:"extends"
}
在这种方式下，github.com/go-xorm/xorm/engine.go里的IdOfV方法，col.FieldName的结果是Public.Id，pkField是，pkField.Kind()是invalid

然后把两个结构体合并成一个，就没以上的问题了

所以现在有2个解决方法
1、我自己合并结构体
2、lunny看下能否支持

今天定位一下问题，找到了问题所在，限于水平有限，还是得lunny出马首先见https://github.com/go-xorm/xorm/issues/328，我的结构体采用组合方式，这样可以复用公共的字段设置，所以结构的设置如下 type Public struct { Id int64 xorm:"pk autoincr" IsDeleted string UpdatedAt time.Time xorm:"updated" CreatedAt time.Time xorm:"created" } type Mails struct { Username string xorm:"not null default '' VARCHAR(255)" Password string xorm:"not null default '' VARCHAR(255)" Status string xorm:"not null default 'N' ENUM('S','E','N')" Public xorm:"extends" } 在这种方式下，github.com/go-xorm/xorm/engine.go里的IdOfV方法，col.FieldName的结果是Public.Id，pkField是<invalid reflect.Value>，pkField.Kind()是invalid 然后把两个结构体合并成一个，就没以上的问题了所以现在有2个解决方法 1、我自己合并结构体 2、lunny看下能否支持

lunny commented

2016-01-12 04:39:09 +00:00

go get -u github.com/go-xorm/core
go get -u github.com/go-xorm/xorm

你两个包都更新下再试。

``` go get -u github.com/go-xorm/core go get -u github.com/go-xorm/xorm ``` 你两个包都更新下再试。

hsyan2008 commented

2016-01-12 05:19:12 +00:00

(Migrated from github.com)

执行这两个命令后，只有
github.com/go-xorm/core/db.go
github.com/go-xorm/core/db_test.go
github.com/go-xorm/core/scan.go
这三个文件有变化
执行结果还是一样，Find的时候，只拿到一条记录

执行这两个命令后，只有 github.com/go-xorm/core/db.go github.com/go-xorm/core/db_test.go github.com/go-xorm/core/scan.go 这三个文件有变化执行结果还是一样，Find的时候，只拿到一条记录

lunny commented

2016-02-16 15:20:47 +00:00

一张表有两个结构体吗？

hsyan2008 commented

2016-02-17 06:59:59 +00:00

(Migrated from github.com)

没有，只有一个结构体

polaris1119 commented

2016-02-22 09:40:03 +00:00

(Migrated from github.com)

我也遇到了同样的问题

mamoroom commented

2017-05-04 08:55:09 +00:00

(Migrated from github.com)

I could't understand Chinese but I might got same bug.
When type of id is uint64 and query is for multiple rows, It would be cached only the last row.
For the following reasons

[1] Creating Cache Key: Numeric value is always parsed as int because of generating by SQL definition.
https://github.com/go-xorm/xorm/blob/master/session_find.go#L326

[2] Getting Cache Key: Numeric value could be uint because of generating by xorm column definition.
https://github.com/go-xorm/xorm/blob/master/engine.go#L1091

Should unify which definition is used as cache key or Should be support unsigned type on SQLTYPE.
https://github.com/go-xorm/core/blob/master/type.go

This is study about a glitch of PK struct caused by using same value as creating cache and different assertion. https://play.golang.org/p/1SOgaIskTv

I could't understand Chinese but I might got same bug. When type of id is uint64 and query is for multiple rows, It would be cached only the last row. For the following reasons [1] Creating Cache Key: Numeric value is always parsed as int because of generating by SQL definition. https://github.com/go-xorm/xorm/blob/master/session_find.go#L326 [2] Getting Cache Key: Numeric value could be uint because of generating by xorm column definition. https://github.com/go-xorm/xorm/blob/master/engine.go#L1091 Should unify which definition is used as cache key or Should be support unsigned type on SQLTYPE. https://github.com/go-xorm/core/blob/master/type.go This is study about a glitch of PK struct caused by using same value as creating cache and different assertion. https://play.golang.org/p/1SOgaIskTv

lunny commented

2017-05-17 08:41:56 +00:00

@mamoroom any idea and could you send a PR to fix this?

mamoroom commented

2017-05-18 16:55:26 +00:00

(Migrated from github.com)

@lunny
the easiest way is just deleting unsigned case from bellow.
https://github.com/go-xorm/xorm/blob/master/engine.go#L1107

but if you will support unsigned type on SQLTYPE, should fix this at same time.
do you have any plan to support unsigned type? is there the branch of unsigned type supporting?

@lunny the easiest way is just deleting unsigned case from bellow. https://github.com/go-xorm/xorm/blob/master/engine.go#L1107 but if you will support unsigned type on SQLTYPE, should fix this at same time. do you have any plan to support unsigned type? is there the branch of unsigned type supporting?

lunny commented

2017-05-27 06:15:53 +00:00

@mamoroom this will result cache failed but not will return wrong error. @hsyan2008 @polaris1119, I have added a test on 5b65f21, how about these tests? are these the same as yours?

@mamoroom this will result cache failed but not will return wrong error. @hsyan2008 @polaris1119, I have added a test on 5b65f21, how about these tests? are these the same as yours?

mamoroom commented

2017-07-04 15:18:08 +00:00

(Migrated from github.com)

@lunny Your test is a bit different what I meant before. I've just sent PR which is the correct cache testing for this issue. Check this out.
https://github.com/go-xorm/xorm/pull/634

@lunny Your test is a bit different what I meant before. I've just sent PR which is the correct cache testing for this issue. Check this out. https://github.com/go-xorm/xorm/pull/634

lunny commented

2017-07-06 05:03:34 +00:00

closed by #635

lunny referenced this issue from a commit

2020-01-19 09:34:09 +00:00

add some tests for tag and cache related with #330

lunny referenced this issue from a commit

2020-01-19 09:34:10 +00:00

Topic/fix cache uint (#635) * modify cache_test for issues #330 * fix bugs cache key when pk is defined as uint (#330) * add comment

Sign in to join this conversation.