Full link voltage measurement

The core processes

The core process of full link voltage test implementation is as follows ：

Suddenly ： Determine the piezometric target

The pressure measurement target mainly includes the pressure measurement range 、 Strategy 、 Purpose , Often associated with business 、 Technical goals are closely related . for example ：

• Pressure measuring range ： User registration plus login , Prepare for large-scale innovation .
• Piezometric strategy ： High simulation production environment pressure measurement , Experience the real business peak in advance .
• The purpose of pressure measurement is ： Detect the service throughput limit , Verify architectural capabilities 、 Detect performance bottlenecks .

Step two ： Sort out the system architecture

Sort out the end-to-end request links 、 Technology Architecture 、 Hierarchical structure 、 Module partition , as well as RPC、 news 、 cache 、 The use of middleware such as database , Analyze potential bottlenecks , And targeted increase of monitoring indicators 、 Make emergency plan .

The system architecture diagram of this example is as follows ：

Step three ： Sort out the business model

The business model of pressure measurement is crucial to the accuracy of pressure measurement results . The link of full link compression test represents the business scope to be tested , The same link needs to construct a massive set of parameters to represent different behaviors of different users , The basic data of the system 、 The preheating condition of the system represents the state of the system . Link range 、 The access level of the link 、 Parameter set of link 、 Basic data 、 The preheating condition together constitutes the business model of pressure measurement .

Business models are usually combed from the following dimensions ：

• User behavior dimension
  • Determine the scope of the business interface 、 Target magnitude of interface 、 Parameter set of interface 、 Pressure curve, etc .
  • Determine the distribution of pressure measurement data according to business characteristics . For example, the size and region of users 、 Type and quantity of goods 、 Whether to manufacture hot businesses and commodities .
• System state dimension
  • According to the characteristics of business and scenario , Identify each component （ For example, caching ） The state of . For example, pull new scenes , Cache hit rate is very low , And daily peak scenes , The cache hit rate is very high , Different cache warm-up strategies need to be prepared according to different scenarios .
  • According to the characteristics of business and scenario , Determine the magnitude and scope of basic data . For example, pull new scenes , We need to consider the recall of old users , And daily peak scenes , Generally, prepare basic data of the same magnitude as active users .

All in all , Business model is strongly related to business , The business model of pressure measurement is crucial to the accuracy of pressure measurement results .

Step four ： Prepare the pressure test script

Write the pressure test script according to the business scenario , You can also reuse existing scripts directly , It is recommended to enter the script PTS scene , Convenient for scene debugging .

Step five ： Transform and upgrade the environment

Conduct full link voltage test in the production environment , The core is that online write operations cannot pollute normal business data . therefore , You need to make shadow database tables for storage , That is, the image of the normal business library table , Let the data of pressure measurement flow flow to the shadow library table , The normal business flow flows to the normal business database table , Logically isolate the two flows , Let it not affect each other .

Three prerequisites for pressure measurement of production environment ：

• The pressure measurement mark is not lost

Pressure measurement flow can be correctly identified in any link . Put pressure gauge on the flow inlet layer , The middleware identifies and continues to pass down the pressure gauge , Ensure that the voltage measurement mark on the whole link is not lost , In this way, the downstream application and storage can also receive the pressure gauge .

• The pressure measurement process is uninterrupted

The pressure measurement flow can be called normally , The whole process is not blocked , Return the expected business results . Business application layer , Corresponding transformation is also needed to support the full link . When the application layer identifies the pressure gauge , Parameter verification needs to be bypassed 、 Security verification and other verification logic , For example, mobile phone number format verification 、 User status verification 、 And some other special business verification logic .

• The pressure measurement data does not pollute

The pressure measurement data will not cause data pollution to the normal online business . Full link scenarios often include multiple read-write scenarios , In order to isolate the pressure measurement data , After the storage middleware recognizes the pressure gauge , Write data to the shadow library table , Distinguish from real data . In order to simulate the real scene more realistically , The basic data in the shadow database table （ For example, buyers 、 The seller 、 goods 、 Shop, etc ） It is constructed from real data plus fixed offset , Sampling will be performed during migration 、 Filter 、 Desensitization and other operations ensure data security , Generally, it is consistent with the real data in terms of data magnitude .

PTS The probe has the above three capabilities , Just deploy the probe on the top of the application 、 Configure the rules , No need to change the business code .

The upgrade scheme of the architecture diagram in this example is as follows ：

Step six ： Joint commissioning of normal flow

The joint debugging is usually completed by executing the function regression use case , It is necessary to mark the normal return flow with the flow indicator （ For example, add Header x-pts-test=2）, In this way, you can accurately locate the call link . The main concerns of this link are as follows ：

• Verify that the probe has no effect on normal business logic , The test results of the use cases are in line with expectations .
• Verify the adaptation of the probe to the dependent components , Without omission RPC call 、 The collected data is accurate ; The integrity of the call chain is the core of the security of the whole link pressure test data .
• Aggregate the call chain data collected by the probe （ Suggest 500+ above ）, Smooth different parameters 、 Call chain differences caused by different logical branches . Using the aggregated dependency topology to sort out component dependencies can greatly avoid component omission .
• According to the results of joint commissioning of normal flow , You need to sort out the scope of the shadow library table 、 Dependence on third-party services .

Step seven ： Prepare the pressure test data

Be careful   There is a high risk in the preparation of pressure measurement data , Please contact DBA、 Contact relevant personnel , Ensure relevant databases 、 Capacity of middleware 、 performance 、 Resources are sufficient to support the migration of pressure test data 、 Storage , And the subsequent pressure test plan .

1. Confirm the shadow library table range .

The scope of shadow database table is the database table used by the applications involved in the pressure test link . In the process of combing , You need to include the database name 、 Table name 、 Data magnitude 、 Core business fields （ Such as commodities ID、 user ID etc. ）, The relevance of fields between tables （ Foreign keys 、JSON References in fields are included ）.

  2. Confirm the offset field 、 Desensitization field .

Offset field ： Field offset can greatly ensure the security of business data . The offset field generally selects the user ID、 goods ID And other related fields , If it works to Sequence Class distribution ID Components , You also need to offset . Choose different offsets according to the actual growth of the business , I usually choose 10 Values that will not be used for more than years are used as offsets .

The specific offset needs to be determined according to business growth and data type , Common offset methods are as follows ：

explain   Desensitization field ： User data that is considered sensitive data in business , E.g. mobile number 、 password 、 User name, etc , Fields with different security levels will have different desensitization methods , Desensitize according to business requirements . Common desensitization methods include masking 、 Add salt 、 Gaussian noise, etc . It is necessary to ensure that the field value after desensitization can be accessed in the business process , If the calibration fails in the process of pressure measurement and joint commissioning , have access to Mock Rules bypass validation .

Create a new shadow library table .

explain   This step generally consists of DBA complete , Create a library table structure according to the shadow library table range .

4. Perform data migration .

explain   This step generally consists of DBA complete , Migration tools generally choose DataX, Migrate from the standby database to the shadow database table during low peak business hours , It is recommended to configure current limiting according to the actual situation . The amount of data migrated is generally consistent with the online data in terms of data magnitude .

5. Prepare interface parameter data .

Based on the basic data and pressure measurement model, construct the parameter set of the business interface . According to the different pressure measuring platforms , Supported format 、 Configuration methods are also different , Generally speaking, they support CSV File format , It can be constructed according to the requirements of each platform .

The business model of pressure measurement is very important to the accuracy of pressure measurement results , And pressure test data preparation is the core link of business model implementation . Pressure measurement data mainly includes basic data and link data .

• Basic data ： Including the library tables and data required for business operation , for example ： buyers 、 The seller 、 goods 、 Discounts, etc , The scale of basic data generally needs to be consistent with the actual business data .
• Link data ： Including interfaces requiring pressure measurement and diversified interface parameter sets , The parameter set of the interface request is generated based on the basic data . for example ： The interface of the product details page is https://xxx.com/item?itemId=xxx, The parameter set is a specific commodity ID Set .

There are usually two ways to prepare basic data: direct construction and data migration ：

• Direct structure ： It is constructed directly according to business rules , It is generally used in the preparation of a small amount of data , For example, data construction in the joint commissioning stage .
• Data migration ： Clean online data 、 sampling 、 Migrate to the shadow library table after offset , Good data completeness , High degree of simulation , Time saving and labor saving . It is recommended to use DataX Data migration .

Data preparation , The core principle is to ensure the mirror image 、 The software and hardware configuration of the shadow library table is consistent with that of the normal library table , At the same time, the configuration is simple and easy . This can ensure that the real problems of the online database table can be fully exposed during the pressure test .

There are the following ways to choose a data isolation strategy ：

• Shadow table isolation ： Create a shadow table with the same structure as the business table in the production library , Shadow table names are usually prefixed with the normal table names . Table level isolation is designed to allow reuse of a portion of read-only tables , But the difficulty of combing has increased .
• Shadow library isolation ： Create a shadow database with the same configuration as the source database on an instance , The shadow database name usually adds a fixed prefix to the normal database name , The table name remains unchanged . Library level isolation is the isolation of data sources , Isolation is relatively complete 、 Security .
• shadow Key Isolation ： Generally used in KV cache 、 On storage components （ for example Redis）, The probe will intercept the right KV cache 、 All operations of storage components , Automatically modify according to the flow meter Key And expiration time , Achieve the purpose of data isolation and data cleaning .

The isolation principle of other storage components is basically consistent with the above three ideas , According to your own business and architecture characteristics , Choose the best isolation method by yourself .

Step eight ： Joint pressure regulation and flow measurement

1. according to Step seven ： Prepare the pressure test data The library table combed in , Fill in the shadow rule on the console , Different rules require different fields .
2. according to Step six ： Joint commissioning of normal flow The third-party service dependencies combed in are configured on the console Mock The rules . If you need to use complex dynamic response results , You need to apply for deployment MockServer.

The mode of joint commissioning with normal flow is basically the same , In the process of joint commissioning, it is necessary to mark the pressure measurement flow with the flow indicator （ For example, add Header x-pts-test=1）, You can accurately locate the call chain . The main concerns of this link are as follows ：

• Verify whether the business logic is normal , The test results of use cases should meet the expectations . This link is greatly affected by basic data , It is easy to see that a certain field does not conform to some verification logic, resulting in business failure .
• Verify whether the call chain generated by the pressure measurement flow is consistent with the normal flow , If there are inconsistencies, relevant personnel need to intervene to investigate the reasons .
• Verify shadow isolation and Mock Whether the rules are valid , If there is test data writing in the formal table or normal data writing in the shadow table , Then relevant personnel are required to investigate the cause .

Step nine ： Small flow pressure test of single chain

Start the full link voltage test . Different business 、 Pressure measurement targets often correspond to different pressure measurement rhythms and methods , Can't be generalized . In addition to the following points , It also needs to be based on business 、 framework 、 Personnel, etc , Make different pressure test plans , Try to avoid online failures , Find more online problems .

• Formulate a clear pressure test plan 、 The pressure measurement passes the standard , Relevant personnel must support , A clear division of responsibilities , Unified command .
• Online pressure testing should be carried out during low peak hours , And formulate an emergency plan .
• It should have the ability to monitor the market , Pay close attention to relevant monitoring indicators .
• Follow the principle of step by step , Pressure measurement of single chain road > Small flow acceptance > Full link acceptance .

Conduct small flow pressure test on the production environment , Expose the most superficial problems , Ensure the correctness of the process .

Step 10 ： Pressure measurement of single chain road

Verify that all interfaces are free of interference 、 Performance baseline data without competition , Determine the performance of all interfaces SLA.

Step 11 ： Small flow pressure test of the whole link

Conduct small flow pressure test on the production environment , Expose the most superficial problems , Ensure the correctness of the process .

Step 12 ： Full link voltage test and acceptance

Conduct the full link voltage test of the composite scenario according to the flow ratio of the production environment . Detect mutual interference 、 Resource consumption levels and bottlenecks in competitive situations . It can be roughly divided into the following 5 Stages ：

1. Step pressurization and capacity planning .

Locate performance bottlenecks ; Get the performance baseline data and capacity of each application , Obtain the current limiting threshold .

1. Instantaneous pressurization .

Verify whether the system preheating is reasonable , For example, database connection 、RPC Connect 、 Business cache 、JIT Precompilation, etc .

1. Stability test .

Verify whether the use of system resources is reasonable , Whether there are memory leaks .

1. Trouble shooting .

Inject faults manually , Expose the stability of the architecture , Improve the robustness of the system .

5. Verify current limiting 、 Downgrade 、 Effectiveness of the plan , Produce the final deliverables .