summary

What is a reliable message final consistency transaction

The reliable message final consistency scheme is to send a message after the transaction initiator completes the local transaction , Transaction participants （ Message consumer ） Must be able to receive messages and handle transactions successfully , This scheme emphasizes that as long as the message is sent to the transaction participants, the final transaction must be consistent .
This scheme is completed by using message middleware , Here's the picture ：
Transaction initiator （ Message producer ） Send messages to message middleware , Transaction participants receive messages from message middleware , Between the transaction initiator and the message middleware , Transaction participants （ Message consumer ） And message middleware are all communicating through the network , Due to the uncertainty of network communication, it will lead to distributed transaction problems .

Therefore, the final consistency solution of reliable messages should solve the following problems ：

1. Atomicity of local transaction and message sending

The atomic problem of local transaction and message sending is ： After the local transaction is executed successfully, the transaction initiator must send the message , Otherwise, discard the message . That is to realize the atomicity of local transaction and message sending , Either they all succeed , Or they all failed , The atomicity of local transaction and message sending is the key to achieve the final consistency scheme of reliable messages .
Let's try this operation first , First message , Operating the database ：

begin transaction:
  // 1. send out MQ
  // 2. Database operation 
commit transaction;

In this case, the consistency between the database and the sent message cannot be guaranteed , Because it's possible to send a message successfully , Database operation failed .
You immediately come up with the second plan , Advanced row database operation , Send a message ：

begin transaction:
  // 1. Database operation 
  // 2. send out MQ
commit transaction;

There seems to be no problem in this case , If sent MQ The news failed , Will throw an exception , Cause database transaction rollback . But if the timeout is abnormal , Database rollback , but MQ In fact, it has been sent normally , It also leads to inconsistencies .

2. Reliability of message received by transaction participants

Transaction participants must be able to receive messages from the message queue , If you fail to receive the message, you can receive the message repeatedly .

3. The problem of repeated consumption of news

Because of the Internet 2 The storage , If a consumption node times out but the consumption is successful , At this time, the message middleware will deliver the message repeatedly , This leads to the repeated consumption of information .
To solve the problem of repeated message consumption, we need to realize the idempotence of transaction participants .

Solution

The previous section discussed the problems that need to be solved in the final consistent transaction scheme of reliable messages , This section discusses specific solutions .

Local message table scheme

The scheme of local message table was originally eBay Proposed , The core of this scheme is to ensure the consistency of data business operations and messages through local transactions , Then send the message to the message middleware through the customized task , The message to be confirmed is sent to the consumer successfully, and then the message is deleted .
Here is an example of sending points for registration ：
The following example has two microservice interactions , User service and point service , User service is responsible for adding users , The point service is responsible for increasing points .

The interaction process is as follows ：

1. User registration
   User services add users and add users in local transactions “ Integral message log ”.（ The user table and message table are consistent through local transactions ）
   Below is pseudo code ：

begin transaction;
  // 1. New users 
  // 2. Store the integral message log 
commit transaction;

In this case , The local database operation is in the same transaction as the store integral message log , Local database operation and message logging operation are atomic .

2. Scheduled task scan log
   How to ensure that messages are sent to message queues ？
   After the first step, the message has been written to the message log table , Independent threads can be started , Regularly scan the messages in the message log table and send them to the message middleware , Delete the message log after the message middleware feedback is sent successfully , Otherwise, wait for the next cycle of the scheduled task to retry .
3. News consumption
   How to ensure that consumers will be able to consume information ？
   Here you can use MQ Of ack（ That is, message confirmation ） Mechanism , Consumer monitoring MQ, If the consumer receives the message and the business processing is completed, then MQ send out ack（ That is, message confirmation ）, At this time, it indicates that the normal consumption message of the consumer is completed ,MQ Will no longer push messages to consumers , Otherwise, the consumer will try again and again to send the message to the consumer .
   The points service received “ Add points ” news , Start adding points , After the score is successfully increased, respond to the message middleware ack, Otherwise, the message middleware will repeatedly deliver secondary messages .
   Because the message will be delivered repeatedly , Points service “ Add points ” Functions need to be idempotent .

RocketMQ Transaction message scheme

In theory, it can be done through RabbitMQ Transaction message scheme substitution .

RocketMQ It is a distributed message middleware from Alibaba , On 2012 In open source , And in 2017 Officially became Apache Top projects . according to the understanding of , Including Alibaba cloud's message products and acquired subsidiaries , Alibaba Group's message products are all running in RocketMQ above , And in recent years, the promotion of the 11th National Congress of the Communist Party of China ,RocketMQ There are eye-catching performances .
Apache RocketMQ 4.3 Later versions officially support transaction messages , It provides convenient support for distributed transaction implementation . RocketMQ Transaction message design is mainly to solve Producer The atomicity of message sending and local transaction execution at the end ,RocketMQ In the design of broker And producer The two-way communication ability of the terminal , bring broker Born as a transaction coordinator ; and RocketMQ The storage mechanism provided by itself provides the persistence capability for transaction messages ;
RocketMQ The high availability mechanism and reliable message design of the system ensure that the final consistency of transactions can be achieved even when the system is abnormal . stay RocketMQ 4.3 After the implementation of the complete transaction message , In fact, it is an encapsulation of the local message table , Moved the local message table to MQ Inside , solve Producer The atomicity of message sending and local transaction execution at the end .