Not so long ago, we in the company decided to allow users to send us notifications about errors that occurred in our software. No sooner said than done. But then there was the task of getting the backtrace of the current call stack of the program right in runtime. It turned out that there are several ways to solve this problem. This article is the result of my research on the issue of getting backtracks for programs written in C / C ++ and working on Linux and FreeBSD.

A bit of theory

Basically, getting a call chain is pretty simple. All necessary information is stored in the program stack. Modern compilers for calling functions form so-called stack frames . At the beginning of each frame is the address of the previous one. And immediately before the frame, the return address is saved, i.e. address of the instruction to be executed next, after the function is completed. Thus, all that needs to be done is to go through the list of frames and print the return addresses.
For example, this can be done as follows (example for amd64):

void * GetReturnAddress(int depth) {    void *res;    asm (        "mov %1, %%rcx\n"        "MOVE: mov 0x0(%%rbp), %%rax\n"        "loop MOVE\n"       "mov 0x8(%%rax), %rax\n"        "mov %%rax, %0\n" : "=m" (res) : "g" (depth) : "rax", "rcx");    return res; } 

The function could be written and shorter, because the return value is put into the rax register; it was possible to do without the variable res.
But, for me personally, the presence of assembly inserts is not the true path of the Jedi. So I went to look for another solution.

Gcc extensions

The first thing I came across was the function __builtin_return_address good-naturedly given to us by the creators of gcc. Here is an excerpt from her full description :
void * __builtin_return_address (unsigned int level) - returns the return address of the function. For level = 0, the function will return the return address of the current function, for level = 1, the return address of the function that called the current function, etc.
When using it, there is only one thing: the compiling function unfolds into a bunch of lines of assembler code (the farther along the stack we go, the more rows) and therefore, they cannot accept the variable as a parameter. Therefore, instead of a beautiful view of the form:
return __builtin_return_address (i);
you have to write ugly:

 switch(level) { case 0: return __builtin_return_address(1); case 1: return __builtin_return_address(2); …. } 

Already better. Go ahead.
')

backtrace

In Linux, the standard library provides the programmer with a whole range of functions that allow us to obtain the information we need. FreeBSD requires the libexecinfo library to be installed for this purpose. Here they are:
int backtrace (void ** buffer, int size) is a function that fills the caller's buffer backtrace.
char ** backtrace_symbols (void * const * buffer, int size) is a function that takes the result of the first function and translates the addresses of the functions into a textual representation.
void backtrace_symbols_fd (void * const * buffer, int size, int fd) - does the same as the previous one, but instead of allocating memory for strings via malloc, writes info directly to a file.
For each function that is included in the call stack, backtrace_symbols returns a string of the following form:

./prog(_Z6myfunci+0x1a) [0x8048840]
where: prog is the name of the binary
_Z6myfunci - coded function name
0x1a - offset inside the function
0x8048840 - the function address

You can find more detailed information, as well as an example of their use, in man backtrace . I just want to note that in order for backtrace_symbols to work properly, you need to compile the program with the -rdynamic option. This is due to the fact that the information about the name of the function backtrace_symbols is taken from the table of dynamic linking. And by default, only functions loaded from dynamic libraries get there. For the forced addition of all the functions in this table and need the above key.

dladdr

The disadvantage of the backtrace_symbols function is that it presents the result of its work as text. Those. if we want to perform any manipulations, for example, with the name of a function, we will have to parse this line. Again, not in Jedi! Why it is necessary, it will be clear a little later.
This is where the dladdr function comes in . Actually, it is her that is called backtrace_symbols inside. Its signature is very simple - the input is the address of the function, and the output is a structure like Dl_info :
int dladdr (void * addr, Dl_info * info);
In the case of a successful outcome of the dladdr call, the structure will contain all the same data as in the case of backtrace_symbols .
Well, almost fine. Now we have return addresses and even function names, albeit in a coded format (let's talk about solving this issue a bit later). Let's see what information you can still pull out. Can the name of the file with the source code and even the address of the line where the function is located? Actually, although it will have to be confused!

What to do with it?

In principle, the data that we already have is enough. Having the address, you can always find out the line number that originated the function call. The easiest way is to use the gdb debugger list command. If you have the same version of the program compiled with debug, list * <address> will show you the line number. And if you still have the source code next to it, then you will see this line “about a miracle!”
But the idea of ​​storing two versions of the program (with and without debugging) does not correspond to the Jedi aspirations for the ideal, and so I decided to study strip. I have known for a long time that he knows how to store binary files and debug information separately. It turned out to be quite simple:

1. We compile a program with debugs as usual (the -g key or, for fans, -g3 - then inline functions and various macros will be included in the debug).
2. Run objcopy --only-keep-debug a.out a.out.sym . Now all the info needed for comfortable work in gdb is in the a.out.sym file.
3. We strip a.out . Those. remove debag from a.out.

Now we can:

1. Link our a.out.sym with a.out with the objcopy command --add-gnu-debuglink = a.out.sym a.out . Then the debager will automatically load all the necessary info from a.out.sym, if it finds it in the same folder as the binary.
2. Manually download the a.out.sym file from gdb with the symbol-file a.out.sym command
   Now we have the opportunity to collect debug information for our software, but not to give it to the client. This can be done out of compassion (debugs take a rather large amount), or for security reasons (we complicate reverse engineering for hackers). But when it is necessary to debug something right at the client, you can simply upload a few missing .sym files.

But, if there is a desire to see not only line numbers, but also the source code itself, but you don’t want to upload it to the client (a legitimate desire for commercial software), you can use gdbserver, which allows you to debug the program remotely. For this you need:

1. On the client side, run gdbserver 127.0.0.1:2345 a.out
2. For our part, run gdb and execute the command target remote 127.0.0.1:2345 . In this case, all source files must be accessible with the same paths that were used at the time of compilation.

mangling / demangling

Finally, I will say a few words about the format for recording function names. In a nutshell, such a distortion of function names is necessary for the linker to resolve naming collisions. Well, or even easier, if there are two functions in the program with the same name, but different parameters (overloaded), then the linker needs to know exactly which of them to work with. To do this, the compiler encodes the name of the function by a special algorithm, assigning it a new unique name. In English, this process is called mangling, and the reverse is demangling .
To solve the problem of converting the encoded function name to the original format, you can again use the gcc-shny extension:
_{char * abi :: __ cxa_demangle (const char * mangled_name, char * output_buffer, size_t * length, int * status)}
This function, taking as input the encoded name of the function and the buffer, outputs the decoded name. An example of its use can be found here .

And finally

The funniest way to get the right information was to just ask gdb for it. The benefit of the latter allows us to do this (an example of the function is taken from here ).

 void print_trace() { char pid_buf[30]; sprintf(pid_buf, "%d", getpid()); char name_buf[512]; name_buf[readlink("/proc/self/exe", name_buf, 511)]=0; int child_pid = fork(); if (!child_pid) { dup2(2,1); // redirect output to stderr fprintf(stdout,"stack trace for %s pid=%s\n",name_buf,pid_buf); execlp("gdb", "gdb", "--batch", "-n", "-ex", "thread", "-ex", "bt", name_buf, pid_buf, NULL); abort(); /* If gdb failed to start */ } else { waitpid(child_pid,NULL,0); } } 

All we need to do is call the print_trace function and, voila, the call stack will print to stdout. In principle, the option is working, but very slow and requiring the installation of gdb.

That's all.
Pleasant debugging!