同步mysql数据库
Go to file
ha666 5b775821fd 暂时先取消输出到kafka 2020-11-07 09:07:25 +08:00
config 暂时先取消输出到kafka 2020-11-07 09:07:25 +08:00
model 增加kafka配置 2020-11-01 21:18:14 +08:00
plugin/kafka 增加kafka配置 2020-11-01 21:18:14 +08:00
service 暂时先取消输出到kafka 2020-11-07 09:07:25 +08:00
.gitignore 增加编译脚本 2020-10-28 10:37:01 +08:00
build_all.sh 增加编译脚本 2020-10-28 10:37:01 +08:00
build_linux.sh 基本流程跑通了 2020-10-26 17:08:46 +08:00
build_win_to_linux64.bat 增加kafka配置 2020-11-01 21:18:14 +08:00
config.yaml 配置并发数、输出队列统计信息 2020-10-28 10:16:35 +08:00
go.mod 增加kafka配置 2020-11-01 21:18:14 +08:00
go.sum 增加kafka配置 2020-11-01 21:18:14 +08:00
init.go 暂时先取消输出到kafka 2020-11-07 09:07:25 +08:00
LICENSE Initial commit 2020-10-25 18:09:23 +08:00
main.go 从kafka读出来的消息只处理指定数据库的消息 2020-11-03 09:23:36 +08:00
README.md 暂时先取消输出到kafka 2020-11-07 09:07:25 +08:00

sync-mysql

同步mysql数据库

配置

应用配置

page_size看情况配置推荐是50~500 thread_count根据数据库服务器的配置以及网络情况来定

  page_size: 20
  thread_count: 4

源配置(数据库和kafka二选一)

  database:
    name: "abc"
    address: "127.0.0.1"
    port: 3306
    account: "root"
    password: "1234567890"
  kafka:
    version: "2.6.0"
    addresses:
     - "127.0.0.1:9092"
    topic: "maxwell"
    consumer: "maxwell-consumer"
    databaseName: "test_db"

目标配置

  databases:
    - name: "ha666db"
      address: "127.0.0.1"
      port: 3306
      account: "root"
      password: "1234567890"
  kafka:
    - addresses:
        - "127.0.0.1:9092"
      topic: "db-log-2"

表名配置

mapping:
  tables:
    "t_abcd": "t_cd"

配置示例一:从数据库到数据库

app:
  page_size: 20
  thread_count: 4
source:
  database:
    name: "abc"
    address: "127.0.0.1"
    port: 3306
    account: "root"
    password: "1234567890"
target:
  databases:
    - name: "ha666db"
      address: "127.0.0.1"
      port: 3306
      account: "root"
      password: "1234567890"

配置示例二从kafka到数据库

使用的是maxwell的方案具体的使用方法可以参考https://github.com/zendesk/maxwell

app:
  page_size: 20
  thread_count: 4
source:
  kafka:
    version: "2.6.0"
    addresses:
     - "127.0.0.1:9092"
    topic: "maxwell"
    consumer: "maxwell-consumer"
    databaseName: "test_db"
target:
  databases:
    - name: "ha666db"
      address: "127.0.0.1"
      port: 3306
      account: "root"
      password: "1234567890"

注意:

  • 如果源配置和目标配置都有数据库,那数据库配置不能完全相同
  • 如果源配置和目标配置都有kafka那kafka配置不能完全相同
  • 只同步数据,不同步结构