当前位置：网站首页>Canel Introduction & use

Canel Introduction & use

2022-07-19 07:44:00 【Top master cultivation plan】

brief introduction

canal How it works

MySQL Master slave replication process
* Master The master library will change records , Write to binary log (binary log) in
* Slave From library to mysql master send out dump agreement , take master The main library binary log
events Copy to its trunk log (relay log);
* Slave Read and redo the events in the relay log from the library , Synchronize the changed data to your own database .

canal How it works

It's simple , Namely Disguise yourself as slave, Pretend to be from master Copy the data

 

Start using  

Preparation

Turn on mysql Of binlog

sudo vi /etc/mysql/my.cnf
[mysqld]
server-id = 1
log-bin=mysql-bin
binlog_format=row
binlog-do-db=gmall

Parameter description  

  • stay [mysqld] ,log-bin=mysql-bin
this individual surface in binlog Japan Records Of front Affix yes mysql-bin , The log files generated in the future are
mysql-bin.123456 The numbers after the file are generated in order , Every time mysql Restart or reach a single file size
When the threshold is small , A new file , Number... In sequence .
  • mysql binlog There are three formats for , Namely STATEMENT,MIXED,ROW.
  • binlog-do-db Specify which database to monitor

Restart after configuration mysql

sudo systemctl restart mysqld
Grant authority 
mysql> set global validate_password_length=4;
mysql> set global validate_password_policy=0;
mysql> GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 
'canal'@'%' IDENTIFIED BY 'canal' ;

Use

canal framework

download

https://github.com/alibaba/canal/releases  

Main body configuration 

tar -zxvf canal.deployer-1.1.4.tar.gz

Modify the configuration file canal.properties

vi canal.properties
canal.serverMode = kafka
canal.mq.servers = master:9092,node1:9092,node2:9092
Through the front canal framework , We can know , One canal There can be more than one in the service instance,conf/
Under the every last example This is an example , There is a separate configuration file under each instance . There is only one real by default
example example, If you need multiple instances to handle different MySQL Data words , Directly copy out multiple example,
And rename it , The name is consistent with the name specified in the configuration file , And then modify canal.properties Medium
canal.destinations= example 1, example 2, example 3.
canal.destinations = example

The instance configuration

modify instance.properties
We only read one here MySQL data , So there's only one example , The configuration file of this instance is in
conf/example Under the table of contents 
vi instance.properties
configure connections MySQL Username and password , The default is what we authorized earlier canal
canal.instance.dbUsername=root
canal.instance.dbPassword=root
Modify output to Kafka Topics and number of partitions
Be careful ： The default is to output to the specified Kafka One of the themes kafka Partition , Because the parallelism of multiple partitions may disrupt
binlog The order of 
# mq config
canal.mq.topic=gmall_db
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*
#canal.mq.partition=0
# hash partition config, The number of partitions below is the same as that of your theme partition The same number 
canal.mq.partitionsNum=4

start-up 

bin/startup.sh

 

test 

bin/kafka-console-consumer.sh --bootstrap-server master:9092 --topic gmall_db

Then go to mysql Insert a piece of data into the monitored database to see kafka Whether there is data received

stop it 

bin/stop.sh

canal High availability

        such zookeeper The mode of monitoring for the observer , Only high availability , Not load balancing , That is, only
There is one canal-server A node can monitor a data source , As long as this node works properly , So the other monitoring
Data sources canal-server Can only do stand-by, Until the working node stops , other canal-server Node talent
Can seize . Because there is one stand-by It also takes up resources , meanwhile canal Transmission data downtime is also relatively rare , the
Many enterprises do not configure canal Of high availability .

Format example of received data

add to 

{
    "data": [
        {
            "id": "1547090631092903951",
            "user_id": "1413",
            "nick_name": null,
            "head_img": null,
            "sku_id": "16",
            "spu_id": "4",
            "order_id": "5991",
            "appraise": "1201",
            "comment_txt": " Comment content ：51811533721518964616668687831874922222995433369391",
            "create_time": "2022-06-28 13:28:44",
            "operate_time": null
        }
    ],
    "database": "gmall",
    "es": 1657690124000,
    "id": 211,
    "isDdl": false,
    "mysqlType": {
        "id": "bigint(20)",
        "user_id": "bigint(20)",
        "nick_name": "varchar(20)",
        "head_img": "varchar(200)",
        "sku_id": "bigint(20)",
        "spu_id": "bigint(20)",
        "order_id": "bigint(20)",
        "appraise": "varchar(10)",
        "comment_txt": "varchar(2000)",
        "create_time": "datetime",
        "operate_time": "datetime"
    },
    "old": null,
    "pkNames": [
        "id"
    ],
    "sql": "",
    "sqlType": {
        "id": -5,
        "user_id": -5,
        "nick_name": 12,
        "head_img": 12,
        "sku_id": -5,
        "spu_id": -5,
        "order_id": -5,
        "appraise": 12,
        "comment_txt": 12,
        "create_time": 93,
        "operate_time": 93
    },
    "table": "comment_info",
    "ts": 1657690130536,
    "type": "INSERT"
}

Delete 

{
    "data": [
        {
            "dic_code": "1102",
            "dic_name": " WeChat ",
            "parent_code": "11",
            "create_time": null,
            "operate_time": null
        }
    ],
    "database": "gmall",
    "es": 1657690329000,
    "id": 212,
    "isDdl": false,
    "mysqlType": {
        "dic_code": "varchar(10)",
        "dic_name": "varchar(100)",
        "parent_code": "varchar(10)",
        "create_time": "datetime",
        "operate_time": "datetime"
    },
    "old": null,
    "pkNames": null,
    "sql": "",
    "sqlType": {
        "dic_code": 12,
        "dic_name": 12,
        "parent_code": 12,
        "create_time": 93,
        "operate_time": 93
    },
    "table": "base_dic",
    "ts": 1657690329374,
    "type": "DELETE"
}

modify 

{
    "data": [
        {
            "id": "1",
            "login_name": "g4hpd2mp2",
            "nick_name": " modify ",
            "passwd": null,
            "name": " Shenchangcheng ",
            "phone_num": "13973123428",
            "email": "[email protected]",
            "head_img": null,
            "user_level": "1",
            "birthday": "2004-08-10",
            "gender": "M",
            "create_time": "2020-06-10 21:48:29",
            "operate_time": "2022-06-28 13:28:34",
            "status": null
        }
    ],
    "database": "gmall",
    "es": 1657690400000,
    "id": 213,
    "isDdl": false,
    "mysqlType": {
        "id": "bigint(20)",
        "login_name": "varchar(200)",
        "nick_name": "varchar(200)",
        "passwd": "varchar(200)",
        "name": "varchar(200)",
        "phone_num": "varchar(200)",
        "email": "varchar(200)",
        "head_img": "varchar(200)",
        "user_level": "varchar(200)",
        "birthday": "date",
        "gender": "varchar(1)",
        "create_time": "datetime",
        "operate_time": "datetime",
        "status": "varchar(200)"
    },
    "old": [
        {
            "nick_name": " Xiongchen "
        }
    ],
    "pkNames": [
        "id"
    ],
    "sql": "",
    "sqlType": {
        "id": -5,
        "login_name": 12,
        "nick_name": 12,
        "passwd": 12,
        "name": 12,
        "phone_num": 12,
        "email": 12,
        "head_img": 12,
        "user_level": 12,
        "birthday": 91,
        "gender": 12,
        "create_time": 93,
        "operate_time": 93,
        "status": 12
    },
    "table": "user_info",
    "ts": 1657690400556,
    "type": "UPDATE"
}

原网站

版权声明
本文为[Top master cultivation plan]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/200/202207170521048590.html

边栏推荐

猜你喜欢

随机推荐