Microservice high concurrency service governance

Add equipment

The most brutal way for local tyrants , For example, direct bidding 1000 Virtual machine , All major platforms have set up special personnel to cooperate , Mirror each module , Who can't bear it , Just add someone , Let cloud providers be stationed , Monitoring also allows them to provide , Add image , Do network configuration , All to them

No problem can be solved by money , The problem is that most companies don't have so much money to burn , ha-ha

Disaster preparedness

It's also a money burning thing
Another set of equipment with the same configuration , Once the service goes down , Load balancing switches to new disaster recovery nginx
The problem is
When the service is complex , All departments and external companies , There are all kinds of things connected , The topology is like a spider web .
Database data can be synchronized , What about caching data ？
request , Callback , Various configurations , All kinds of services accumulated over the years , All kinds of languages , All kinds of people have left , Nobody dared to move
You may have to pack the disaster recovery environment , Other dependent services ,ip The address and everything have to be sorted out

So disaster preparedness is not easy , It may be necessary to do a comprehensive reconstruction as a whole

cache

Page caching , Reduce database access times , Or directly nginx Page caching , Reduce the pressure
cdn, Put some pictures cdn, Reduce the time to access resources

Current limiting

The purpose of current limiting is to ensure the normal operation of the service , Others can pass , It can be business flow restriction , For example, the agent of the anchor adjusts the interface to reward the anchor , You have to limit the current once a second , Otherwise, there will always be problems with the lock of the database

• nginx Current limiting ,ip Current limiting
• tomcat Current limiting
• token Semaphore
• Counter
  My idea is aop Get a set of counters , all Controller Internal and external interfaces , Add this filter by default
  （ All external requests , You have to add ,）
  Method name do key, Numbers add up and subtract , Access interface +1, At the end -1
  And print the start time , End time , Time consuming , Concurrent , Total concurrency
  It's good , All requests are logged , When there are disputes about the docking of other systems , This is a basis

Some people think that doing this has a great impact on performance , It's really small
For those who are prone to over 10000 concurrent machines , My level is very low , I only consider single tomcat Three or five hundred concurrent （ A provider that provides traffic balance query service , It's called 2 platform tomcat Can achieve 1 ten thousand 2 Concurrent , I actually secretly pressed him 200 The amount of , Errors exceed 5% 了 , Sorry, that brother whose service is stuck inexplicably, I'm the one who pressed 500 ）

Downgrade

• Let go
  for instance , Video websites will judge advertisements and available resolutions based on user membership level , If there is an emergency , The number of users has increased , The authentication component will not be able to bear , Now , You can surrender , Give a general result , Direct discharge , No logical processing

• Refuse
  For example, direct return failure , And improve “ Please try again later ”
  For example, when paying , Databases are usually locked , Not enough to support single user high concurrent deduction , Isolation should be added , Directly demote , Try again later

Serial

Second kill what time , You can first tell him that the request is successful , But it doesn't mean that the rush purchase is successful , Serial execution of concurrent events , Turn into queuing execution
This can relieve the pressure on the server , Will not exceed the stock （ Provide inventory of the little sister's heart os： More than a little , In fact, the problem is not big ）
At this time, the queue , fifo , Something can be used ,

I have done many monitoring mq High frequency data demand ,
It may be an embarrassing situation that the later executed event arrives first than the first executed event , Therefore, it is generally cached for a period of time , Collision queue in cache , Then drop into the warehouse , Although it can't completely solve the problem of sequence , The probability is much lower , Database pressure is also much less ,200 One time execution , And one at a time 200 There is a big difference

Simple monitoring

How to know your concurrency
One is load balancing , perhaps nginx Statistics can be made over there
But the company is bigger , These things can't be checked by us , There are professional operation and maintenance loads
If we want to watch wechat , One spring cloud Health monitoring can be seen
But my idea is , According to the amount of logs ,logstash, perhaps filebeat turn logstash, Into the es, then Kibana Exhibition

logstash Very resource intensive , If the service itself consumes resources , Don't install it on the same device
es Put it in the array made of solid state , A few hundred G Your log is completely OK ,

Kibana Filter it out , You can see the interfaces （ I face all before Controller Did aop Log printing for ） The concurrency value

If it's alicloud , Log aggregation can be configured directly ,elk You don't need to build it yourself