I've come across a strange problem recently , Replace mysql db After the connection address of , Occasionally, simple sql Query statement report (2006,Mysql Server has gone away) error . The general troubleshooting process is recorded as follows .

Cause analysis

according to Mysql Description of the official website , The possible reasons are as follows ：

1.Mysql The service outage

It can be done by show status like ‘uptime’; To query the server running time ：

Troubleshoot server downtime .

2.Mysql Connection timeout

adopt show global variables like '%timeout'; Check the timeout parameters ：

You can see ,interactive_timeout and wait_timeout The parameters are 8h, You can eliminate the timeout problem .

3.mysql The request connection process is active kill

This situation is generally caused by dba Trigger , Take the initiative kill Slow search . adopt show global status like 'com_kill'; see ：

Exclude the connection process from being kill problem .

4.Your SQL statement was too large

The result set of the query exceeds max_allowed_packet Errors will also be reported during restriction . adopt show global variables like 'max_allowed_packet'; see ：

The parameter is 64m, Eliminate the problem that the result set is too large .

Other reasons

After eliminating the above reasons , There are also possible reasons ： Too many connections , Out of memory etc. .

Out of memory

according to stackOverflow The statement seen above , It may be caused by insufficient memory ：

But does this memory refer to program server memory or db Server memory ？ adopt free -h Check the application server memory ：

The memory available to applications is 105G, Completely enough .db The server cannot log in directly , I can't see memory usage , But if db There is a performance problem in the server memory ,dba There must be a reaction , Therefore, it can be basically ruled out .

Too many connections

Check the current number of open connections and the maximum number of connections used as follows ：

You can see , The current number of connections is 1178. The maximum number of connections is 1631, The time of appearance is not the latest , You can eliminate the problem of too many connections .

Continue to view other parameters ：

• threads_connected： The number of threads created to process the connection
• threads_running: The number of threads activated to process connections

Other parameters are basically normal . In business application scenarios , conversation session It is also closed after use , There is no more than 8h The connection of . thus , The cause of the problem cannot be found . Fortunately, this happens only occasionally , Does not affect service availability . Find out the reason later , Then update the solution .

The final reason is

Later, the reason was traced again , It is found that there is a problem with the database connection pool parameters . It's using python Of sqlAlchemy, As shown below ：

mysql_engine = create_engine(settings.get('mysql', 'host'),
                             pool_size=settings.getint('mysql', 'pool_size'),
                             max_overflow=10,
                             echo=settings.getboolean('mysql', 'echo'),
                             encoding=settings.get('mysql', 'charset'),
                             pool_recycle=20000
                             )

Session = sessionmaker(bind=mysql_engine)

The connection pool size here is 2, The maximum number of connections is 10, The connection keeps alive for 20000s, about 5.5h. And new db There is a connection and disconnection mechanism , Therefore, there will be more than... Connections 8 Hours , By db Active disconnection .

take pool_size It is amended as follows 2,max_overflow Change it to 100,pool_recycle by 60, No error will be reported after modification , Problem solving ！

appendix

mysql Database common parameters ：
show status like '% Following variables %'

• Aborted_clients The connection has died because the customer did not close it properly , Number of connections that have been dropped .
• Aborted_connects The attempt has failed MySQL The number of connections to the server .
• Connections Trying to connect MySQL The number of servers ( Cumulative data , Non current )
• Created_tmp_tables When executing a statement , The number of implicit temporary tables that have been created .
• Delayed_insert_threads The number of latency insertion processor threads in use .
• Delayed_writes use INSERT DELAYED Number of rows written .
• Delayed_errors use INSERT DELAYED There are some errors in writing ( Possible duplicate key values ) The number of rows .
• Flush_commands perform FLUSH The number of orders .
• Handler_delete Number of requests to delete rows from a table .
• Handler_read_first The number of requests to read the first row in the table .
• Handler_read_key The request number is based on the key read line .
• Handler_read_next The number of requests to read a line based on a key .
• Handler_read_rnd The number of requests to read a line based on a fixed location .
• Handler_update Number of requests to update a row in the table .
• Handler_write Number of requests to insert a row into the table .
• Key_blocks_used The number of blocks used for the keyword cache .
• Key_read_requests Number of requests to read a key value from the cache .
• Key_reads The number of times a key value is read from disk .
• Key_write_requests The number of times a keyword block was requested to be written to the cache .
• Key_writes The number of times a key block is physically written to disk .
• Max_used_connections The maximum number of simultaneous connections .
• Not_flushed_key_blocks Key blocks that have changed in the key cache but have not been cleared to disk .
• Not_flushed_delayed_rows stay INSERT DELAY The number of rows in the queue waiting to be written .
• Open_tables Number of open tables .
• Open_files The number of open files .
• Open_streams The number of open streams ( It is mainly used for logging ）
• Opened_tables The number of tables that have been opened .
• Questions The number of queries sent to the server .
• Slow_queries It costs more than long_query_time Number of time queries .
• Threads_connected The number of connections currently open .
• Threads_running The number of threads that are not sleeping .
• Uptime How many seconds the server worked

Reference material

[1]. https://stackoverflow.com/questions/7942154/mysql-error-2006-mysql-server-has-gone-away