Escape string and blob results from dump more correctly #2091

Merged
lunny merged 12 commits from zeripath/xorm:escape-results-from-dump into master 2022-01-07 08:04:02 +00:00
Contributor

dumpTables currently badly handles BLOB and TEXT data containing control
characters:

  • MySQL will interpret and unescape string literals e.g.\r will become
    carriage return.

  • Postgres will not allow string literals to contain NUL nor will
    SQLite so BLOBs will not dump correctly.

  • Schemas should not be set on the destination dump

  • MSSQL needs the N prefix to correctly ensure that UTF-8 data is
    correctly transferred.

Signed-off-by: Andrew Thornton art27@cantab.net

dumpTables currently badly handles BLOB and TEXT data containing control characters: * MySQL will interpret and unescape string literals e.g.`\r` will become carriage return. * Postgres will not allow string literals to contain NUL nor will SQLite so BLOBs will not dump correctly. * Schemas should not be set on the destination dump * MSSQL needs the N prefix to correctly ensure that UTF-8 data is correctly transferred. Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-03 21:57:27 +00:00
Some checks failed
continuous-integration/drone/pr Build is failing
7dbf688ae9
Escape string and blob results from dump more correctly
dumpTables currently badly handles BLOB and TEXT data containing control
characters:

* MySQL will interpret and unescape string literals e.g.`\r` will become
carriage return.

* Postgres will not allow string literals to contain NUL nor will
SQLite so BLOBs will not dump correctly.

* Schemas should not be set on the destination dump

* MSSQL needs the N prefix to correctly ensure that UTF-8 data is
correctly transferred.

Signed-off-by: Andrew Thornton <art27@cantab.net>
Author
Contributor

I need to go through MSSQL and the other non-Gitea enabled DB types to check that blobs and control characters are being properly handled there too.

I need to go through MSSQL and the other non-Gitea enabled DB types to check that blobs and control characters are being properly handled there too.
zeripath added 1 commit 2022-01-04 18:36:29 +00:00
Some checks failed
continuous-integration/drone/pr Build is failing
b772658c0b
On postgres put the schema in if it is set
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-04 19:42:42 +00:00
Some checks failed
continuous-integration/drone/pr Build is failing
129043e50d
oops
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-04 21:37:53 +00:00
Some checks failed
continuous-integration/drone/pr Build is failing
7f471d653d
attempt to placae test
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-05 10:24:02 +00:00
Some checks failed
continuous-integration/drone/pr Build is failing
ec13261367
fix npe
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-05 11:04:06 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
47f1bd022f
attempt to fix cockroach
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-05 20:42:51 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
31415fe666
Merge remote-tracking branch 'origin/master' into escape-results-from-dump
zeripath added 1 commit 2022-01-05 23:02:44 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
d03d953a3f
handle dameng
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-05 23:25:26 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
83817e383e
handle mssql properly
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-06 04:27:01 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
9433f34337
Add some more testcases for dump
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 2022-01-06 05:02:23 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
1189a63c98
handle oracle
Signed-off-by: Andrew Thornton <art27@cantab.net>
Author
Contributor

OK, so I think I've got this working now.

However, large blobs and large text fields will not be being handled correctly.

  • MySQL - this doesn't appear to be a problem. Even \0 works in BLOB literals. I guess we need to decide if we want to use escaped strings or char() technique I've proposed here.
  • Postgres - pgdump uses COPY and an escaped format of its own design
  • Oracle - maybe something like
  dbms_lob.createtemporary(buf, FALSE);
  dbms_lob.append(buf, HEXTORAW('0EC1D7FA6B411DA58149'));
  --...lots of hex data...
  dbms_lob.append(buf, HEXTORAW('0EC1D7FA6B411DA58149'));

but I dunno

OK, so I think I've got this working now. However, large blobs and large text fields will not be being handled correctly. * MySQL - this doesn't appear to be a problem. Even \0 works in BLOB literals. I guess we need to decide if we want to use escaped strings or char() technique I've proposed here. * Postgres - pgdump uses `COPY` and an escaped format of its own design * Oracle - maybe something like ```sql dbms_lob.createtemporary(buf, FALSE); dbms_lob.append(buf, HEXTORAW('0EC1D7FA6B411DA58149')); --...lots of hex data... dbms_lob.append(buf, HEXTORAW('0EC1D7FA6B411DA58149')); ``` but I dunno
lunny approved these changes 2022-01-07 02:51:02 +00:00
lunny added 1 commit 2022-01-07 03:29:43 +00:00
All checks were successful
continuous-integration/drone/pr Build is passing
7e7ce32d82
Merge branch 'master' into escape-results-from-dump
lunny merged commit cd36b112ae into master 2022-01-07 08:04:02 +00:00
Sign in to join this conversation.
No description provided.