Escape string and blob results from dump more correctly #2091

Merged
lunny merged 12 commits from zeripath/xorm:escape-results-from-dump into master 2 weeks ago
zeripath commented 3 weeks ago

dumpTables currently badly handles BLOB and TEXT data containing control
characters:

  • MySQL will interpret and unescape string literals e.g.\r will become
    carriage return.

  • Postgres will not allow string literals to contain NUL nor will
    SQLite so BLOBs will not dump correctly.

  • Schemas should not be set on the destination dump

  • MSSQL needs the N prefix to correctly ensure that UTF-8 data is
    correctly transferred.

Signed-off-by: Andrew Thornton art27@cantab.net

dumpTables currently badly handles BLOB and TEXT data containing control characters: * MySQL will interpret and unescape string literals e.g.`\r` will become carriage return. * Postgres will not allow string literals to contain NUL nor will SQLite so BLOBs will not dump correctly. * Schemas should not be set on the destination dump * MSSQL needs the N prefix to correctly ensure that UTF-8 data is correctly transferred. Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath added 1 commit 3 weeks ago
7dbf688ae9
Escape string and blob results from dump more correctly
Poster

I need to go through MSSQL and the other non-Gitea enabled DB types to check that blobs and control characters are being properly handled there too.

I need to go through MSSQL and the other non-Gitea enabled DB types to check that blobs and control characters are being properly handled there too.
zeripath added 1 commit 3 weeks ago
b772658c0b
On postgres put the schema in if it is set
zeripath added 1 commit 3 weeks ago
129043e50d
oops
zeripath added 1 commit 2 weeks ago
7f471d653d
attempt to placae test
zeripath added 1 commit 2 weeks ago
ec13261367
fix npe
zeripath added 1 commit 2 weeks ago
47f1bd022f
attempt to fix cockroach
zeripath added 1 commit 2 weeks ago
zeripath added 1 commit 2 weeks ago
d03d953a3f
handle dameng
zeripath added 1 commit 2 weeks ago
83817e383e
handle mssql properly
zeripath added 1 commit 2 weeks ago
9433f34337
Add some more testcases for dump
zeripath added 1 commit 2 weeks ago
1189a63c98
handle oracle
Poster

OK, so I think I've got this working now.

However, large blobs and large text fields will not be being handled correctly.

  • MySQL - this doesn't appear to be a problem. Even \0 works in BLOB literals. I guess we need to decide if we want to use escaped strings or char() technique I've proposed here.
  • Postgres - pgdump uses COPY and an escaped format of its own design
  • Oracle - maybe something like
  dbms_lob.createtemporary(buf, FALSE);
  dbms_lob.append(buf, HEXTORAW('0EC1D7FA6B411DA58149'));
  --...lots of hex data...
  dbms_lob.append(buf, HEXTORAW('0EC1D7FA6B411DA58149'));

but I dunno

OK, so I think I've got this working now. However, large blobs and large text fields will not be being handled correctly. * MySQL - this doesn't appear to be a problem. Even \0 works in BLOB literals. I guess we need to decide if we want to use escaped strings or char() technique I've proposed here. * Postgres - pgdump uses `COPY` and an escaped format of its own design * Oracle - maybe something like ```sql dbms_lob.createtemporary(buf, FALSE); dbms_lob.append(buf, HEXTORAW('0EC1D7FA6B411DA58149')); --...lots of hex data... dbms_lob.append(buf, HEXTORAW('0EC1D7FA6B411DA58149')); ``` but I dunno
lunny approved these changes 2 weeks ago
lunny added 1 commit 2 weeks ago
7e7ce32d82 Merge branch 'master' into escape-results-from-dump
lunny merged commit cd36b112ae into master 2 weeks ago

Reviewers

lunny approved these changes 2 weeks ago
continuous-integration/drone/pr Build is passing
The pull request has been merged as cd36b112ae.
Sign in to join this conversation.
Loading…
There is no content yet.