Canal Learning notes of

1. brief introduction

Official website ：https://github.com/alibaba/canal

Canal, Waterways / The Conduit / Ditch , The main use is based on MySQL Database incremental log parsing , Provide Incremental data subscription and consumption .

canal Yes, it is java Developed incremental log parsing based on database , Provide incremental data subscription & Consumer Middleware . at present ,canal It mainly supports MySQL Of binlog analysis , Only after the parsing is completed can canal client Used to process the relevant data obtained .（ Database synchronization requires Alibaba's otter middleware , be based on canal）

Here we can simply put canal To understand as a means to A tool for synchronizing incremental data .

In the early days, Alibaba was deployed due to the dual computer rooms in Hangzhou and the United States , There is a business demand for synchronization across computer rooms , The implementation is mainly based on the business trigger Get incremental changes . from 2010 Year begins , The business gradually tries to get incremental changes from database log parsing for synchronization , Thus, a large number of incremental database subscription and consumption businesses are derived .

Incremental subscription and consumption based on logs include

• database mirroring
• Database real-time backup
• Index building and real-time maintenance （ Split heterogeneous indexes 、 Inverted index, etc ）
• Business cache Refresh
• Incremental data processing with business logic

Current canal Support source side MySQL The version includes 5.1.x,5.5.x,5.6.x,5.7.x,8.0.x

2. working principle

MySQL Principle of active / standby replication

• MySQL master Write data changes to binary log （binary log, The records are called binary log events binary log events, Can pass show binlog events To view the ）
• MySQL slave take master Of binary log events Copy to its trunk log （relay log）
• MySQL slave replay relay log Middle event , Change the data to reflect its own data

canal working principle

• cannl simulation MySQL slave Interaction protocol , Pretend to be MySQL slave, towards MySQL master send out dump agreement
• MySQL master received dump request , Start pushing binary log to slave （ namely canal）
• canal analysis binary log　 object （ Originally byte flow ）, And then send it to the storage destination , such as MySQL,Kafka,Elastic Search wait .

3. canal Download address

Download address ：https://github.com/alibaba/canal/releases