Debugging multithreaded programs based on FreeRTOS

Debugging multitasking programs is not easy, especially if you come across this for the first time. After the joy of starting the first task or the first demo of the program, from infinitely exciting observation of the LEDs, each of which blinks in its own task, there comes a time when you realize that you understand quite a bit (you don’t understand it all ) what's really going on. Classics of the genre: “I allocated as many as 3KB to the operating system and launched only 3 tasks with a stack of 128B, and for the fourth reason I don't have enough memory for some reason” or “How much stack should I allocate to the task? How much is enough? And so much? Many solve these problems through trial and error, so in this article I decided to combine most of the points that, at present, greatly simplify my life and allow me to more consciously debug multi-threaded programs based on FreeRTOS.

This article is intended, first of all, for those who only recently began to learn FreeRTOS, but it is likely that readers well familiar with this operating system will find something interesting for themselves here. In addition, despite the fact that the article is aimed at developers of embedded software, it will also be interesting for application programmers, since A lot of words will be said about FreeRTOS as such, regardless of microcontroller romance.

In this article I will talk about the following points:

1. Setting up OpenOCD to work with FreeRTOS.
2. Do not forget to include hooks.
3. Static or dynamic memory allocation?
4. The tale of the parameter configMINIMAL_STACK_SIZE.
5. Monitoring the use of resources.

Setting up OpenOCD to work with FreeRTOS

The first thing you may encounter when using FreeRTOS is the absence of any useful information in the Debug window:
')


It looks as sad as possible. Fortunately, OpenOCD supports FreeRTOS debugging, you just need to configure it correctly:

1. Add the file FreeRTOS-openocd.c to the project
2. Add flags to the linker (Properties> C / C ++ Build> Settings> Cross ARM C ++ Linker> Miscellaneous> Other linker flags):
   
   

   -Wl,--undefined=uxTopUsedPriority 

3. Add flags to debugger (Run> Debugs configurations> Debugger> Config options):
   
   

    -c "$_TARGETNAME configure -rtos auto" 

4. Uncheck Run> Debugs configurations> Startup> Set breakpoint at main.

After these settings, the Debug window will display all existing threads with details, i.e. we will always have access to information about the state of a particular process and what it is currently busy with:

Do not forget to include hooks

If our program has fallen into some hard_fault_handler (), then with the settings from the previous paragraph, we can understand from which task we got there. However, we will not know anything about the reasons for this fall.



For example, in the picture above, we see that the error occurred during the execution of the YellowLedTask task. The first thing we do is in the debug, we start to walk line by line along the endless loop of the task to clarify the place of the fall. Suppose we learned that the program breaks down during the execution of the dummy () function (by the way, there is a way to immediately understand what function we have broken, you can read about it in this article ). We begin to check the function body for errors or typos. An hour passes, the eye begins to twitch, and we are sure that the function is written correctly as firmly as we are sure that the chair on which we sit exists. So what's the deal? But the fact is that the error that occurred may not have anything to do with your function, and the problem lies precisely in the operation of the OS. And here we come to the aid of hooks.

The following hooks exist in FreeRTOS:

 /* Hook function related definitions. */ #define configUSE_IDLE_HOOK 0 #define configUSE_TICK_HOOK 0 #define configCHECK_FOR_STACK_OVERFLOW 2 #define configUSE_MALLOC_FAILED_HOOK 1 #define configUSE_DAEMON_TASK_STARTUP_HOOK 0 

The most important debugging programs are configCHECK_FOR_STACK_OVERFLOW and configUSE_MALLOC_FAILED_HOOK.

The configCHECK_FOR_STACK_OVERFLOW parameter can be enabled with a value of 1 or 2 depending on which stack overflow detection method you want to use. Read more about it here . If you enabled this hook, you will need to define a function.
void vApplicationStackOverflowHook (TaskHandle_t xTask, signed char * pcTaskName), which will be executed every time when the stack allocated for a task is not enough for it to work, and most importantly, you will see it in the call stack of a specific task. Thus, to solve the arisen problem, you only need to increase the size of the stack allocated for the task.


Parameter configUSE_MALLOC_FAILED_HOOK is enabled 1, like most configurable FreeRTOS parameters. If you enabled this hook, you will need to define the void vApplicationMallocFailedHook () function. This function will be called when there is not enough free space on the heap allocated for FreeRTOS to accommodate the next entity. And, again, the main thing is that we will see all this in the call stack. Therefore, all we need to do to solve this problem is to increase the size of the heap allocated for FreeRTOS.


Now, if we run our program again, then when it crashes in hard_fault_handler () we will see the reason for this crash in the Debug window:



By the way, if you have ever found an interesting use of configUSE_IDLE_HOOK, configUSE_TICK_HOOK or configUSE_DAEMON_TASK_STARTUP_HOOK, then it would be very interesting to read about it in the comments)

Static or dynamic memory allocation?

So, we figured out how to monitor stack overflow and heaps in FreeRTOS, and now it's time to talk about the eternal - about memory.

In this section we will consider the following FreeRTOS parameters:

 /* Memory allocation related definitions. */ #define configSUPPORT_STATIC_ALLOCATION 0 #define configSUPPORT_DYNAMIC_ALLOCATION 1 #define configTOTAL_HEAP_SIZE 100000 #define configAPPLICATION_ALLOCATED_HEAP 0 

In FreeRTOS, memory for creating tasks, semaphores, timers, and other RTOS objects can be allocated both statically (configSUPPORT_STATIC_ALLOCATION) and dynamically (configSUPPORT_DYNAMIC_ALLOCATION). If you enable dynamic memory allocation, you also need to specify the size of the heap that RTOS can use (configTOTAL_HEAP_SIZE). In addition, if you want the heap to be located in a particular place, and not automatically located in the memory by the linker, then you need to enable the configAPPLICATION_ALLOCATED_HEAP parameter and define the uint8_t array ucHeap [configTOTAL_HEAP_SIZE]. And do not forget that for dynamic memory allocation, in the folder with the FreeRTOS files you need to add the file heap_1.c, heap_2.c, heap_3.c, heap_4.c or heap_5.c, depending on which version of the memory manager is more suitable for you.

In order to estimate how much memory you can give to the FreeRTOS heap, after building the project you need to look at the size of the .bss section. It displays the amount of RAM needed to store all static variables. For example, I have a controller with a 128K RAM, I gave FreeRTOS 50KB and after assembling the project the .bss section takes 62304B. This means that in my project static variables at 12,304 bytes + 50,000 bytes are statically allocated to the OS heap. It is necessary to remember that a couple of kilobytes must be stored for the main () stack, and as a result we get that a bunch of FreeRTOS can be increased by (128,000 - 62304 - 2000) bytes.

The advantages of each approach to memory allocation can be read here , and a detailed comparative description of various memory managers is presented here .

As for my opinion, at this stage of development, I see no point in using static memory allocation, therefore, in the above config, static memory allocation is turned off. And that's why:

1. Why self-allocate a buffer for the stack and the StaticTask_t structure, if the operating system supports as many as 5 different memory managers for every taste and color, which will figure out where, what and how to create, and even tell you if they haven't succeeded? In particular, for most programs under microcontrollers, heap_1.c is more than fully suitable .
2. You may need some kind of third-party library, written in a very optimal and capacious manner, but using inside itself malloc (), calloc () or new [] (). And what to do? To abandon it in favor of less optimal (this is even if there is a choice)? Or you can simply use dynamic memory allocation with heap_2.c or heap_4.c . The only thing you need to do is override the corresponding functions so that the memory allocation takes place using the FreeRTOS tools in the heap provided to it:
   
   
   code snippet

    void* malloc(size_t size) { return pvPortMalloc(size); } void* calloc(size_t num, size_t size) { return pvPortMalloc(num * size); } void free(void* ptr) { return vPortFree(ptr); } void* operator new(size_t sz) { return pvPortMalloc(sz); } void* operator new[](size_t sz) { return pvPortMalloc(sz); } void operator delete(void* p) { vPortFree(p); } void operator delete[](void* p) { vPortFree(p); } 

In my projects I use only dynamic memory allocation with heap_4.c, giving the maximum possible memory for the OS heap, and always redefine the functions malloc (), calloc (), new (), etc., regardless of whether they are used at the moment or not.

Ratuya for dynamic memory allocation, I, of course, do not deny that there are tasks for which the static memory allocation is an ideal solution (this, you can also discuss in the comments).

The tale of the parameter configMINIMAL_STACK_SIZE

The value of the parameter configMINIMAL_STACK_SIZE is calculated NOT in bytes, but in words! Moreover, the word size varies from one OS port to another and it is defined in the file portmacro.h by the portSTACK_TYPE file. For example, in my case, the word size is 4 bytes. Thus, the fact that the configMINIMAL_STACK_SIZE parameter in my configuration is 128 means that the minimum stack size for the task is 512 bytes.

That's all I wanted to say.

Resource Usage Monitoring

It would be great to have answers to such questions as:

• Did I adequately select the stack size for the task? Is it too much? Maybe too little?
• And how much CPU time is required to complete my task?
• And how much does the real heap allocated for the OS use? The program is already at the limit, or is there another where to turn?

In this section I will give an example of how you can implement a simple monitoring of resources, which will help to get unambiguous answers to all the above questions.

FreeRTOS has a toolkit that allows you to collect resource usage statistics on the fly, which includes the following parameters:

 #define configGENERATE_RUN_TIME_STATS 0 #define configUSE_TRACE_FACILITY 0 #define configUSE_STATS_FORMATTING_FUNCTIONS 0 

I will talk about the value of each parameter a little further, but first we need to create a task, let's call it MonitorTask, an infinite loop of which will collect statistics at config :: MonitorTask :: SLEEP_TIME_MS interval and send it to the terminal.

After the task is created, we need to set the configUSE_TRACE_FACILITY parameter to 1, after which the function will be available to us:

 UBaseType_t uxTaskGetSystemState(TaskStatus_t* const pxTaskStatusArray, const UBaseType_t uxArraySize, uint32_t* const pulTotalRunTime) 

The pxTaskStatusArray should be sizeof (TaskStatus_t) * uxTaskGetNumberOfTasks (), i.e. it must be large enough to contain information about all existing tasks.

By the way, about the structure of TaskStatus_t. What kind of information regarding each task can we get? But this:


So now we are ready to describe the infinite loop of the MonitorTask task. It might look like this:


Suppose that in my program besides MonitorTask, there are a few more tasks with these parameters, where configMINIMAL_STACK_SIZE = 128:


Then, after launching the program, I will see the following information in the terminal:



Wow, this is pretty good! So let's see what we see in this log.

• We see the names of all existing tasks. In addition to the tasks described in the TaskConfig.h file, we also see the IDLE task, which is created automatically when the RTOS scheduler starts (its purpose is written here ).
• We see the status of each task, where B = Blocked, R = Ready, S = Suspended, D = Deleted.
• We see the priority of each task.
• We see the minimum amount of free space on the stack, since the creation of the task. And here it becomes obvious to us that for most of the tasks, we allocated too much stack. For example, for the LoggerTask task, a stack of 256 words was allocated, but in reality it uses only 40. Thus, a stack of 64 words is sufficient for the functioning of the task. Here you have the start of optimization.
• We see the current and minimum (from the start of the scheduler) free space in the heap. In our simple example, these values ​​are equal, but in more complex programs, these two variables are, of course, different. Thus, we understand that out of 100K given to FreeRTOS, it uses less than 10K, therefore we have more than 90K of free memory in our hands.
• And finally, we see the amount of time elapsed since the start of the scheduler in milliseconds.

Applying this knowledge to the TasksConfig.h file and lowering the value of the configMINIMAL_STACK_SIZE parameter from 128 to 64, we get the following picture:



Super! Now each task has an optimal supply of free space on the stack: not too large, and not too small. In addition, we freed up almost 3KB of memory.

And now it's time to talk about what we have not yet seen in the resulting log. We do not see how much CPU time each task uses, i.e. how much time the task was in running state. To find out, we need to set the configGENERATE_RUN_TIME_STATS parameter to 1 and add the following definitions to the FreeRTOSConfig.h file:

 #if configGENERATE_RUN_TIME_STATS == 1 void vConfigureTimerForRunTimeStats(void); unsigned long vGetTimerForRunTimeStats(void); #define portCONFIGURE_TIMER_FOR_RUN_TIME_STATS() vConfigureTimerForRunTimeStats() #define portGET_RUN_TIME_COUNTER_VALUE() vGetTimerForRunTimeStats() #endif 

Now we need to start an external timer that counts down the time (preferably in microseconds, because some tasks may take less than a millisecond time, but we still want to know about everything). Let's add the file of MonitorTask.h with the declaration of two static functions:

 static void config_timer(); static unsigned long get_counter_value(); 

In the MonitorTask.cpp file, we write their implementation:

 void MonitorTask::config_timer() { _timer->disable_counter(); _timer->set_counter_direction(cm3cpp::tim::Timer::CounterDirection::UP); _timer->set_alignment(cm3cpp::tim::Timer::Alignment::EDGE); _timer->set_clock_division(cm3cpp::tim::Timer::ClockDivision::TIMER_CLOCK_MUL_1); _timer->set_prescaler_value(hw::config::MONITOR_TIMER_PRESQ); _timer->set_autoreload_value(hw::config::MONITOR_AUTORELOAD); _timer->enable_counter(); _timer->set_counter_value(0); } unsigned long MonitorTask::get_counter_value() { static unsigned long _counter = 0; _counter += _timer->get_counter_value(); _timer->set_counter_value(0); return (_counter); } 

And in the main.cpp file we will write the implementation of the functions vConfigureTimerForRunTimeStats () and vGetTimerForRunTimeStats (), which we declared in FreeRTOSConfig.h:

 #if configGENERATE_RUN_TIME_STATS == 1 void vConfigureTimerForRunTimeStats(void) { tasks::MonitorTask::config_timer(); } unsigned long vGetTimerForRunTimeStats(void) { return (tasks::MonitorTask::get_counter_value()); } #endif 

Now, after starting the program, our log will be like this:



Comparing the values ​​of Total RunTime and System Uptime, we can conclude that only a third of the time our program is busy performing tasks, and 98% of the time is spent on IDLE, and 2% on all other tasks. What is our program doing the remaining two thirds of the time? This time is spent on the work of the scheduler and switching between all tasks. Sad but true. Of course, there are ways to optimize this time, but this is a topic for the next article.

As for the configUSE_STATS_FORMATTING_FUNCTIONS parameter, it is very minor, most often it is used in various demo programs provided by the FreeRTOS developers. Its essence lies in the fact that it includes two functions:

 void vTaskList(char* pcWriteBuffer); void vTaskGetRunTimeStats(char* pcWriteBuffer); 

Both of these functions are NOT part of FreeRTOS. Inside themselves, they call the same uxTaskGetSystemState function that we used above, and add already formatted data to pcWriteBuffer. The developers themselves do not recommend using these functions (but, of course, they do not prohibit them), pointing out that their task is rather a demonstration, and it is obvious that they can use the uxTaskGetSystemState function directly, as we did.

That's all. As always, I hope that this article was useful and informative)

To build and debug the demo of the project described in the article, the Eclipse + GNU MCU Eclipse bundle (formerly GNU ARM Eclipse) + OpenOCD was used.

Third Pin Company Blog