Using GDB when developing for ESP8266

TL; DR: we implemented a GDB server for ESP8266, which allows you to view the call stack and perform limited debugging.

Friends often ask me: what is it, develop software for embedded platforms. Usually, they mean limited resources (the size of the repository of the executable code, the amount of available memory, etc.) and sometimes the fact that such development requires specific skills.

However, nowadays, when the perception of embedded platforms has changed under the influence of wearable devices that can easily emulate my first personal computer and still have the resources to play music in the background, it's easy to forget about the problems that the developer faces every day embedded software.
')
Cesanta is developing a platform that lowers the threshold for entering the “magic” world of embedded programming, but when developing this platform, we are even more confronted with problems that interfere with the rest.

Today I would like to talk about the development tools, namely, about the part that is provided by almost any development environment; what is so easy to take for granted: call stack trace.

I'm not a big fan of debuggers. Moreover, I prefer to use printf for debugging. In my opinion, in many cases connecting a debugger brings more trouble than practical benefits.

For the last decade, I have written and maintained systems in C / C ++ / Python / Java / Scala / Go and I have used the debugger only in very rare cases. And even then, the debugger was used only as a means to understand the call chain that led to this crash.

Since for most languages ​​the call stack is a built-in functionality, such situations mostly happened when debugging C / C ++ programs.

Coming back today: start a built-in development, select ESP8266 as a platform, leave me without a debugger and I will be in a difficult position.

Suddenly this happened:

• Debug output is too expensive. The only interface with the device is a serial port connection, which is also used to do something else.
• The slow cycle of “editing / compiling / flashing / launching” forces one to pray, so that this time the cards would go as it should.
• The memory simply ends if you write too much logging code.

And I realized that if I want to finish something, I need this damn stack.

To begin with: why do we even have this problem? What is the story with ESP8266?

ESP8266

ESP8266 is a damn cool piece of silicon; and its coolest part is the price: you can get a fully functional fee for $ 3.60. And you do not need expensive boards for development, you can use it right with a minimum set of tools.

This product began its journey as a WiFi module that understands AT commands via the serial port. Just connect to the Arduino - and you can start.

But using it as a daughterboard is just the beginning of the story. The board itself has a 32 bit processor, 32Kb IRAM, 80Kb DRAM and 512Kb (or more) flash. Quickly enough, Espressif began distributing an SDK that allows programs to be written directly to the device. The capabilities of the device are superior to those of well-known AVRs with comparable energy consumption.

How, then, did a little-known Chinese company achieve all this? I can not judge the quality of the ASIC and radio module, but they seem quite good for my inexperienced look. But the quality of the SDK and documentation makes one wonder.

The core of the ESP8266EX is a very interesting Xtensa processor developed by Tensilica. Tensilica was bought by Cadence in 2013. This company is best known for its configurable processor cores.

And they provide great tools like compiler, debugger, emulator ...

The problem is that Espressif cannot deliver these tools along with its SDK. Moreover, even if you buy the Xtensa SDK directly from Cadence, the exact parameters used in the production of the ESP8266EX can only be determined by the heap of generated files. It is difficult to even understand whether it is possible to reverse all this until the trial license of the Xtensa SDK has ended. And even if it is possible, the game is worth the candle only if ESP8266 is all that interests you.

And while Espressif uses Xtensa tools to build its binary libraries, the most appropriate choice for the user is using the GCC port. The specific architecture used in the ESP8266 is called lx106.

The configurable nature of the Xtensa platform implies that the actual set of features (including a set of instructions) of different devices built on the basis of the Xtensa CPU is quite different. This greatly complicates the reuse of tools and understanding how it all should work.

The first feature of lx106 is that this configuration does not use one of the most important Xtensa features: the register window. This has a major impact on the calling convention used by ESP8266 and, as a result, on all tools.

There is an actively developing GCC port supported by Max Filippov, available on the github . He is still far from perfect, but he is developing by leaps and bounds. Hats off to Max for his dedication to the community.

The existing debugging options for ESP8266 are quite modest.

You can try on-chip-debugger (either via xt-ocd from Xtensa or via openocd , but this requires JTAG connectivity and ESP with enough divorced pins (i.e., this approach will not work on ESP-01).

Qemu port is still in its infancy.

View call stack

If all we need is to be able to trace the stack, then the simplest way would be to implement this functionality directly into the code, i.e. something like libunwind. Unfortunately, there is no port of such a library for lx106. Moreover, given that most of the code is compiled without a frame pointer, the implementation of such a reasonable size library will be .... let's say, complicated.

GDB solves a problem by analyzing the code, instruction by instruction, finding the function prologue, rolling back stack changes, etc. What if you try to feed GDB the contents of memory and let it do the heavy work?

GDB server protocol

GDB supports remote debugging using a simple text protocol. It can work over a network or serial port. A simple description of the protocol is here .

The two main commands we need to support are:

• 'g' - unload the contents of the registers
• 'm' - read a pack of bytes from memory

Thus, all we need to do is write the code that executes if an exception occurs, find out the state of the registers and implement the GDB protocol on the serial port.

Get control

At first, I tried to directly change the low-level vector of exceptions in the Xtensa CPU, but without success.
Then I noticed that the _xtos_set_exception_handler function is mentioned in the linker script. XTOS is a very thin layer provided by the Xtensa SDK.

It turned out that _xtos_set_exception_handler allows you to register a C-function that will be called if the specified exception occurs.

ICACHE_FLASH_ATTR void gdb_init() { char causes[] = {EXCCAUSE_ILLEGAL, EXCCAUSE_INSTR_ERROR, EXCCAUSE_LOAD_STORE_ERROR, EXCCAUSE_DIVIDE_BY_ZERO, EXCCAUSE_UNALIGNED, EXCCAUSE_INSTR_PROHIBITED, EXCCAUSE_LOAD_PROHIBITED, EXCCAUSE_STORE_PROHIBITED}; int i; for (i = 0; i < (int) sizeof(causes); i++) { _xtos_set_exception_handler(causes[i], gdb_exception_handler); } } 

The low-level exception handler saves the state of the registers in the structure on the stack and calls the C-handler, passing it the address of this structure as a parameter.

Since Xtensa is parameterizable, it’s not so easy to understand in the documentation what is related to what. The Xtensa documentation is very general and although much can be understood from the code available for other Xtensa configurations, one cannot be sure that this applies to lx106.

In the end, I got confused, and decided to just write down certain values ​​in the registers to see where they eventually appear. I managed to find a2-a16 registers, but a1 (stack pointer) seems to be overwritten by the contents of a0 (return address).

Later, I found a couple of links that confirmed my guess and explained the loss of register a1.

• Defining a UserFrame structure
• XTOS Exception Handler

Well, putting it all together:

 struct xtos_saved_regs { uint32_t pc; /* instruction causing the trap */ uint32_t ps; uint32_t sar; uint32_t vpri; /* current xtos virtual priority */ uint32_t a0; /* when __XTENSA_CALL0_ABI__ is true */ uint32_t a[16]; /* a2 - a15 */ }; 

The LITBASE register is absent, but it seems that the low-level exception handler does not change it and therefore you can simply give GDB its current value.

The key feature here is that despite the absence of a stack pointer, it can be calculated at the address of the xtos_saved_regs structure passed to handler C. This is 256 bytes below the stack pointer.

Now, we can simply disable interrupts and wait for requests from GDB

 /* The user should detach and let gdb do the talkin' */ ICACHE_FLASH_ATTR void gdb_server() { printf("waiting for gdb\n"); /* * polling since we cannot wait for interrupts inside * an interrupt handler of unknown level. * * Interrupts disabled so that the user (or v7 prompt) * uart interrupt handler doesn't interfere. */ xthal_set_intenable(0); for (;;) { int ch = gdb_read_uart(); if (ch != -1) gdb_handle_char(ch); } } 

Communication with GDB

Now you need to understand what format the response to the command 'g' expects GDB.
It depends on the specific GDB. We need to use the port under lx106.

Register descriptions can be found in the gdb / regformats / reg-xtensa.dat file here .

From it we get:

 struct regfile { uint32_t a[16]; uint32_t pc; uint32_t sar; uint32_t litbase; uint32_t sr176; uint32_t sr208; uint32_t ps; }; 

There are a couple of less interesting technical subtleties regarding the GDB protocol and secure memory access (part of the memory is not available for byte addressing), but in general this is all.

Let's see what happened:

 #0 0x40242557 in crash (v7=<optimized out>, this_obj=18445899648779419648, args=18446462599806581592) at user/v7_esp.c:371 #1 0x4023c321 in i_eval_call (v7=v7@entry=0x3fff5c28, a=a@entry=0x3fff96f0, pos=pos@entry=0x3ffffe94, scope=<optimized out>, this_object=<error reading variable: can't compute CFA for this frame>, is_constructor=<optimized out>, is_constructor@entry=0) at user/v7.c:9977 #2 0x40239962 in i_eval_expr (v7=0x3fff5c28, v7@entry=<error reading variable: can't compute CFA for this frame>, a=0x3fff96f0, a@entry=<error reading variable: can't compute CFA for this frame>, pos=0x3ffffe94, pos@entry=<error reading variable: can't compute CFA for this frame>,scope=<optimized out>) at user/v7.c:9595 #3 0x4023bcf0 in i_eval_stmt (v7=<error reading variable: can't compute CFA for this frame>, a=<error reading variable: can't compute CFA for this frame>, pos=<error reading variable: can't compute CFA for this frame>, pos@entry=0x3ffffe94, scope=<optimized out>, brk=<optimized out>,brk@entry=0x3ffffe90) at user/v7.c:10487 #4 0x4023bd4a in i_eval_stmts (v7=<error reading variable: can't compute CFA for this frame>, a=<error reading variable: can't compute CFA for this frame>, pos=0x3ffffe94, pos@entry=<error reading variable: can't compute CFA for this frame>, end=15, scope=<optimized out>, brk=<error reading variable: can't compute CFA for this frame>) at user/v7.c:10053 #5 0x4023b104 in i_eval_stmt (v7=<optimized out>, a=a@entry=0x3fff96f0, pos=pos@entry=0x3ffffe94, scope=<optimized out>, brk=<optimized out>, brk@entry=0x3ffffe90) at user/v7.c:10088 #6 0x4024140a in v7_exec_with (v7=<optimized out>, res=res@entry=0x3fffff30, src=<optimized out>, w=<optimized out>) at user/v7.c:10607 #7 0x4024148a in v7_exec (v7=<optimized out>, res=res@entry=0x3fffff30, src=<optimized out>) at user/v7.c:10631 #8 0x402421c4 in process_js (cmd=<optimized out>) at user/v7_cmd.c:66 #9 0x4024234a in process_command (cmd=cmd@entry=0x3ffebc14 <recv_buf$3591> "crash()") at user/v7_cmd.c:128 #10 0x402423f7 in process_prompt_char (symb=<optimized out>) at user/v7_cmd.c:163 #11 0x40244a59 in rx_task (events=<optimized out>) at user/v7_uart.c:151 #12 0x40000f49 in ?? () #13 0x40000f49 in ?? 

This is a stack of code collected with -Og -g3

Working with -Os is not yet possible. Also note the “error reading variable: can't compute CFA for this frame” . It seems lx106 GDB requires some improvements (well, or I missed something).

Note: The problem with CFA was fixed in gdb 7.9.1, available in the “lx106-g ++ - 1.21.0” branch of this repository . Thanks to Angus for pointing out that it is worth trying the new gdb. He, however, does not fix problems with -Os.

If I find time, I will continue to work and add the ability to set breakpoints and resume execution after an exception, but the current implementation solves a pressing problem: displaying the call stack. I hope you find this useful.

The source code (GPLv2) is here .
And instructions for use here .

Enjoy.