The journal is mysql An important part of the database , Record all kinds of state information during database running .mysql Logs mainly include error logs 、 Query log 、 Slow query log 、 Transaction log 、 There are several categories of binary logs .

As a development , What we need to focus on is binary logging ( binlog ) And transaction logs ( Include redo log and undo log ), This article will introduce these three kinds of logs in detail .

binlog

binlog Used to record write operations performed by the database ( Exclude queries ) Information , Stored on disk in binary form .binlog yes mysql Logical log of , And by the Server Layer to record , Using any storage engine mysql The database records binlog journal .

• Logic log ： What can be simply understood as recording is sql sentence .
• Physical log ：mysql Data is ultimately stored in the data page , Physical logging records data page changes .

binlog It is written by appending , Can pass max_binlog_size Parameter settings for each binlog File size , When the file size reaches the given value , A new file will be generated to save the log .

binlog Use scenarios

in application , binlog There are two main usage scenarios of , Namely Master slave copy and Data recovery .

1. Master slave copy ： stay Master End open binlog , And then binlog Send to each Slave End , Slave End replay binlog So as to achieve the consistency of master-slave data .
2. Data recovery ： By using mysqlbinlog Tools to recover data .

binlog When to brush the disk

about InnoDB For storage engines , Only when the transaction is committed binlog , The record is still in memory , that binlog When did it go to disk ？

mysql adopt sync_binlog Parameter control binlog When to brush the disk , The value range is 0-N：

• 0： Don't force it , It is up to the system to determine when to write to the disk ;
• 1： Every time commit All the time, we have to binlog Write to disk ;
• N： Every time N One transaction , Will be binlog Write to disk .

As can be seen from the above , sync_binlog The safest setting is 1 , This is also MySQL 5.7.7 The default value for later versions . But setting a larger value can improve database performance , Therefore, in practice, you can also increase the value appropriately , Sacrifice some consistency for better performance .

binlog Log format

binlog There are three formats for logs , Respectively STATMENT 、 ROW and MIXED.

stay MySQL 5.7.7 Before , The default format is STATEMENT , MySQL 5.7.7 after , The default value is ROW. The log format is through binlog-format Appoint .

• STATMENT： be based on SQL Copy of sentences ( statement-based replication, SBR ), Each one will modify the data sql The statement will record binlog in .

• • advantage ： You don't need to record every line change , Less binlog Log volume , Economize IO , This improves performance ;
  • shortcoming ： In some cases, the master-slave data is inconsistent , Such as execution sysdate() 、 slepp() etc. .

• ROW： Line based replication (row-based replication, RBR ), Don't record every sql The context information of the statement , Just record which data has been modified .

• • advantage ： There will be no stored procedures under certain circumstances 、 or function、 or trigger The call and trigger of cannot be copied correctly ;
  • shortcoming ： There will be a lot of logs , In especial alter table When it's time to make the journal skyrocket

• MIXED： be based on STATMENT and ROW Mixed replication of the two modes (mixed-based replication, MBR ), General replication uses STATEMENT Mode save binlog , about STATEMENT Mode can not be copied by operation using ROW Mode save binlog

Why redo log

We all know , One of the four characteristics of a transaction is persistence , To be specific As long as the transaction is committed successfully , Then the changes to the database will be permanently saved , It's impossible to return to the original state for any reason .

that mysql How to ensure consistency ？

The easiest way to do this is to do it every time a transaction is committed , Refresh all the data in the disk . But there are serious performance problems with doing this , It is mainly reflected in two aspects ：

1. because Innodb In order to page Disk interaction for units , A transaction is likely to modify only a few bytes in a data page , At this time, the complete data page will be flashed to the disk , It's a waste of resources ！
2. A transaction may involve modifying multiple data pages , And these data pages are not physically contiguous , Use random IO Poor write performance ！

therefore mysql Designed redo log , Specifically, it only records the changes made by transactions to the data page , This will solve the performance problem perfectly ( The files are relatively small and sequential IO).

redo log Basic concepts

redo log It consists of two parts ： One is the log buffer in memory ( redo log buffer ), The other is the log file on disk ( redo logfile).

mysql Every time you execute one DML sentence , First write the record to redo log buffer, At a later time point, multiple operation records will be written to redo log file. such Write the log , Write the disk again The technology is MySQL It's often said in WAL(Write-Ahead Logging) technology .

In a computer operating system , User space ( user space ) In general, the buffer data under the following conditions cannot be directly written to the disk , The middle must pass through the operating system kernel space ( kernel space ) buffer ( OS Buffer ).

therefore , redo log buffer write in redo logfile It's actually written first OS Buffer , And then through the system call fsync() Brush it to redo log file
in , The process is as follows ：

mysql Support three will redo log buffer write in redo log file The timing of , Can pass innodb_flush_log_at_trx_commit Parameter configuration , The meanings of the parameters are as follows ：

redo log Record form

As I said before , redo log Actually record changes to the data page , And this kind of change record is not necessary to keep all of them , therefore redo log The implementation adopts fixed size , How to write circularly , When it comes to the end , You'll go back to the beginning and cycle through the log . Here's the picture ：

At the same time, it's easy to know , stay innodb in , both redo log Need to brush the disk , also Data pages Also need to brush the disk , redo log The significance of existence is mainly to reduce the influence on Data pages The requirements of the brush disk .

In the diagram above , write pos Express redo log Currently recorded LSN ( Logical sequence number ) Location , check point Express Data page change record After brushing the disk, it corresponds to redo log Situated LSN( Logical sequence number ) Location .

write pos To check point The part between is redo log The empty part , Used to record new records ;check point To write pos Between redo log Change record of data page to be dropped . When write pos Catch up check point when , Will push first check point Move forward , Make room for a new log .

start-up innodb When , Whether it was a normal or abnormal shutdown last time , There will always be recovery operations . because redo log It records physical changes to the data page , So recovery is faster than logical logging ( Such as binlog ) Much faster .

restart innodb when , First of all, it will check the data page in the disk LSN , If the data page LSN Less than in the log LSN , Will follow checkpoint Start recovery .

There's another situation , Was in before the outage checkpoint The process of brushing the disk , And the disk flushing progress of the data page exceeds that of the log page , In this case, the record in the data page will appear LSN Larger than in the log LSN, At this point, the part beyond the log progress will not be redone , Because that in itself means something that has been done , No need to redo .

redo log And binlog difference

from binlog and redo log We can see the difference between ：binlog Logs are for archiving only , Rely only on binlog It's not crash-safe The ability of .

But only redo log Not good either. , because redo log yes InnoDB Peculiar , And the records on the log will be covered after the disk is dropped . Therefore need binlog and redo log Both are recorded at the same time , In order to ensure that when the database goes down and restarts , Data will not be lost .

undo log

One of the four features of database transaction is Atomicity , To be specific Atomicity refers to a series of operations on a database , All or nothing , All or nothing , Partial success is unlikely .

actually , Atomicity The bottom floor is through undo log Realized .undo log It mainly records the logical changes of data , Like one INSERT sentence , There's a corresponding one DELETE Of undo log , For each UPDATE sentence , It's the opposite UPDATE Of undo log , So when something goes wrong , You can roll back to the data state before the transaction .

meanwhile , undo log It's also MVCC( Multi version concurrency control ) The key to realization .

Reference resources

MySQL The secret of not losing data , It's hidden in its 7 In the log