Redis crash debugging

Reprint translation , Easy to sort and classify , Source address https://blog.csdn.net/weixin_34290000/article/details/90587202
Original English address ：http://antirez.com/news/43

background

Redis The code quality of has been praised by insiders , It can also have good stability under extremely high business pressure . But in extreme cases ,Redis It is also possible Crash Of . Sometimes for various reasons , System configuration problem , Disk space is full , Insufficient process permissions, etc , We may not be so lucky , There is one core Files can be used for debugging . This is the time ,Redis Several abnormal crash cases are provided Crash Report, Many times we are based on Crash Report, With some analysis, we can directly locate the problem .

Crash Report

In case of an abnormal breakdown ,Redis Will be set by signal handler To generate Crash Report, at present Redis To produce Crash Report The abnormal signals captured mainly include the following ：

• SIGSEGV
• SIGFPE
• SIGILL
• SIGBUS

this 4 This kind of signal should be able to contain most abnormal program crashes , The most common is SIGSEGV The paragraph is wrong , except 0 abnormal SIGFPE Sometimes you encounter .

When Redis Received above 4 One of the signals , It will be set up in sigsegvHandler() Generated in function Crash Report, as follows ,

=== REDIS BUG REPORT START: Cut & paste starting from here ===
[19179] 12 Apr 18:47:42.599 #     Redis 2.8.19 crashed by signal: 11
[19179] 12 Apr 18:47:42.599 #     Failed assertion: <no assertion failed> (<no file>:0)
[19179] 12 Apr 18:47:42.599 # --- STACK TRACE
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(logStackTrace+0x4a)[0x7f5be2d6895a]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(debugCommand+0x1b0)[0x7f5be2d69ad0]
/lib64/libpthread.so.0(+0xf500)[0x7f5be3c1a500]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(debugCommand+0x1b0)[0x7f5be2d69ad0]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(call+0x8a)[0x7f5be2d2f12a]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(processCommand+0x5dd)[0x7f5be2d3017d]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(processInputBuffer+0x4d)[0x7f5be2d3b86d]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(readQueryFromClient+0xf0)[0x7f5be2d3cb70]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(aeProcessEvents+0x13d)[0x7f5be2d2804d]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(aeMain+0x2b)[0x7f5be2d2833b]
/home/dejun.xdj/kvs-kernel/src/libredis-server.so(runRedis+0x4f)[0x7f5be2d31eaf]
/home/dejun.xdj/kvs-kernel/src/redis-server /home/dejun.xdj/local/redis/conf/redis_7071.conf *:1071(main+0x180)[0x405db0]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3d4ac1ecdd]
/home/dejun.xdj/kvs-kernel/src/redis-server /home/dejun.xdj/local/redis/conf/redis_7071.conf *:1071[0x4055e9]
[19179] 12 Apr 18:47:42.599 # --- INFO OUTPUT
...
...
[19179] 12 Apr 18:47:42.599 # --- CLIENT LIST OUTPUT
[19179] 12 Apr 18:47:42.599 # id=2 addr=127.0.0.1:30494 fd=5 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=debug read=0 write=0 type=admin next_opid=-1

[19179] 12 Apr 18:47:42.599 # --- CURRENT CLIENT INFO
[19179] 12 Apr 18:47:42.599 # client: id=2 addr=127.0.0.1:30494 fd=5 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=debug read=0 write=0 type=admin next_opid=-1
[19179] 12 Apr 18:47:42.600 # argv[0]: 'debug'
[19179] 12 Apr 18:47:42.600 # argv[1]: 'segfault'
[19179] 12 Apr 18:47:42.600 # --- REGISTERS
...
...
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           http://github.com/antirez/redis/issues

  Suspect RAM error? Use redis-server --test-memory to verify it.

Crash Report It mainly includes 4 part ,

• STACK TRACE, Call stack information at crash
• INFO OUTPUT, When it collapses Redis info Command output
• CLIENT OUTPUT, Including the crash CURRENT CLIENT, You can see the commands executed by the client when it crashes
• REGISTERS, Register information

One of the most helpful for debugging is STACK TRACE Information. , We directly take the above Crash Report To explain how to debug .

debugging

Redis Provides debug segfault The command is used to debug , Let's give it directly to Redis Send this command , You can generate a crash report similar to the above in the log （ Please do not use this command in the production environment ！！！）.

Explain before actual debugging , If -g Options , There is no debug symbol information in the executable , It is impossible to debug later , Considering that the impact of performance is very small ,Redis The default compilation is with -g Option .

above STACK TRACE The information tells us directly , Out core What's the point libredis-server.so in , Simple analysis shows that it is executing debugCommand A segment error occurred while , After the function + The address of the number band is the code offset in the function , As long as we know the starting address of the function, we can get core The code address in the function of , And then you can go through addr2line Get the specific source file name and line number corresponding to the address .

adopt nm The tool gets the starting address of the function ,

$nm -l /home/dejun.xdj/kvs-kernel/src/libredis-server.so  | grep debugCommand
000000000006d920 T debugCommand /home/dejun.xdj/kvs-kernel/src/debug.c:255
000000000007f3f0 T pfdebugCommand       /home/dejun.xdj/kvs-kernel/src/hyperloglog.c:1455

We can see debugCommand The starting address of is 0x6d920（ Hexadecimal ）, Plus offset 0x1b0, It can be known that core The specific address of is 0x6dad0, Then we can easily get the specific line number ,

$addr2line -e /home/dejun.xdj/kvs-kernel/src/libredis-server.so 0x6dad0
/home/dejun.xdj/kvs-kernel/src/debug.c:304

Referring to the source file, we can find debug.c Of 304 Illegal address access exists in line ,

*((char*)-1) = 'x';

summary

The above is just to illustrate the debugging process , Take a simple example , Sometimes it locates core The point of , More detailed analysis may be needed , combination info Output and client Output . In the absence of core File scenario ,Crash Report It can really help a lot , The above process , Interested students can directly make a script , Analyze logs directly , Automatic retrieval core Point information .

English source text ：http://antirez.com/news/43