M0AGX / LB9MG

Amateur radio and embedded systems

Preserving debugging breadcrumbs across reboots in Cortex-M

Debugging embedded systems during development even with the best tools can be hard. Certainly a good debug probe makes life easier but what do you do after the product is shipped? What if the customer complains that something strange is happening sometimes or a bug makes the device reboot but only once a week? You make the firmware gather diagnostic information for you. This is the first post in series.

Catastrophic firmware faults are probably impossible to handle by the MCU itself but there are many kinds of faults that can be handled by the firmware and leave enough information to fix them afterwards. These include bugs like:

  1. Hard faults (on ARM)
  2. Invalid memory writes (on ARM)
  3. Runtime assertion failures
  4. Stack overflows (not all)

Intentional reboots (like configuration changes or bootloader requests) can also be logged by the firmware. This post is written with Cortex-M in mind but some of it can also be applicable to other MCUs.

How to store information across reboots?

In order to diagnose a crash or reboot some information has to be made available to the firmware that is running after the crash. Very little data is actually needed to diagnose most of the faults so storing debugging breadcrumbs in RAM is an easy option. Microcontroller RAM retains all information across reboots (only the values at power-on are undefined), however all "regular" variables will be initialized by the startup code (to either zeros or their specified values). So how can you "cheat" C code to not initialize some data?

The basic options are:

  1. make a special section in the linker file and place your debugging breadcrumbs there
  2. tell the linker that the RAM is smaller and starts somewhere else :)

I chose the second option, because of its simplicity. In my MCU I reserved 64 bytes at the beginning of RAM. The MEMORY RAM entry in the linker script has to be modified. In case of the GNU linker the ORIGIN must be 64 bytes larger (hex!) and the LENGTH must be 64 bytes smaller (decimal!).

EFM32GG linker script example

1
2
3
4
5
6
7
MEMORY
{
  FLASH (rx) : ORIGIN = 0x00000000, LENGTH = 1048576
  /* Original setup: RAM (rwx)  : ORIGIN = 0x20000000, LENGTH = 131072 */
  /* Beginning of RAM (64 bytes) will be reserved for crash info & bootloader interaction:*/
  RAM (rwx)  : ORIGIN = 0x20000040, LENGTH = 131008
}

STM32L151 linker script example

1
2
3
4
5
6
7
8
MEMORY
{
  FLASH (rx)      : ORIGIN = 0x08000000, LENGTH = 128K
  /* Original setup: RAM (xrw)       : ORIGIN = 0x20000000, LENGTH = 16K */
  /* Beginning of RAM (64 bytes) will be reserved for crash info & bootloader interaction:*/
  RAM (xrw)       : ORIGIN = 0x20000040, LENGTH = 16320
  MEMORY_B1 (rx)  : ORIGIN = 0x60000000, LENGTH = 0K
}

Changing the start of RAM and its size makes the linker and C code totally ignore this part of memory. Of course the memory is still there but untouched by any code. Accessing this block is as simple as assigning the start of RAM address to a pointer. For example: uint32_t *block = (uint32_t*)0x20000000;.

It is important that the remaining RAM is word-aligned (ie. is multiple of 4), otherwise the linker may place variables in an unaligned way, which can lead to strange problems.

Diagnostic module

The diagnostic module is responsible for storing and restoring debugging breadcrumbs across reboots. It also handles intentional reboots and requests to start the bootloader.

Header

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#pragma once
#include <stdbool.h>
#include <stdint.h>

/* The crash info section is at the beginning of the RAM,
 * that is not initialized by the linker to preserve
 * information across reboots.
 */

#define CRASH_INFO_SECTION_LENGTH 64

typedef enum {
    crash_magic_none = 0,
    crash_magic_bootloader_entry = 0xB000B000,
    crash_magic_hard_fault       = 0xCAFEBABE,
    crash_magic_debug_mon        = 0x01020304,
    crash_magic_reboot_requested = 0x0ABCDEF,
    crash_magic_stack_overflow   = 0xBADBEEF,
} crash_magic_t;

typedef struct {
    uint32_t magic;
    uint32_t uptime;
    union {
        struct {
            uint32_t reg_CFSR;
            uint32_t reg_HFSR;
            uint32_t reg_DFSR;
            uint32_t reg_AFSR;
            uint32_t reg_MMAR;
            uint32_t reg_BFAR;
            uint32_t reg_PC;
            uint32_t reg_LR;
        } hard_fault;

        char task_name[4*8]; //used during stack overflow

        uint32_t reset_request_line;
    };
    uint8_t unused[20];
    uint32_t xor_checksum;
} crash_info_t;

_Static_assert(sizeof(crash_info_t) == CRASH_INFO_SECTION_LENGTH, "Wrong size");

void crash_handler_init(void);
const crash_info_t* crash_handler_get_info(void);

#define SYSTEM_RESET() system_reset_func(__LINE__)
__attribute__((noreturn)) void system_reset_func(uint32_t line);
#ifdef BOOTLOADER_BUILD
bool system_check_bootloader_request_flag(void);
#else
__attribute__((noreturn)) void system_request_bootloader_entry(void);
#endif

Central part is struct crash_info_t that holds a magic value, checksum and values that depend on the kind of fault. reg_CFSR and others are specific to Cortex-M. Each of the union members depends on the magic value.

At runtime any code can call crash_handler_get_info to get the debugging struct and store it permanently (eg. flash) or send it somewhere for processing (telemetry!).

The code

This module supports the following scenarios:

  1. Firmware requests reboot and bootloader entry
  2. Bootloader checks if the application requested its start
  3. Firmware requests reboot
  4. Stack overflow reported by FreeRTOS

More features will be described in future posts. The code comes from an EFM32 project, hence the em_dbg.h and em_device.h includes.

Most important data is the _crash_info_ram pointer to the uninitialized part of RAM (made by changes to the linker script). To distinguish junk from a proper debugging information the init code first checks the RAM address for one of the magic values and then computes an XOR checksum. If both checks are okay the data is copied to a regular variable and made available to the rest of the code, the original data is erased so it is not counted twice (for example after a brief power cut the RAM can retain data).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
#include "crash.h"
#include <em_dbg.h>
#include <em_device.h>
#ifndef BOOTLOADER_BUILD
#include "rtos.h"
#endif
#include <stdio.h>
#include <string.h>

static volatile crash_info_t *_crash_info_ram = (crash_info_t*)0x20000000; //beginning of RAM - see linker script
static crash_info_t _previous_crash_info;
static bool _previous_crash_info_valid = false;

__attribute__((always_inline)) static inline uint32_t calculate_checksum(void){
    uint32_t checksum = 0;
    uint32_t *p = (uint32_t*)_crash_info_ram;
    for (uint32_t i = 0; i < (sizeof(crash_info_t)/sizeof(uint32_t))-1; i++){
        checksum ^= *p;
        p++;
    }
    return checksum;
}

#ifndef BOOTLOADER_BUILD
void crash_handler_init(void){
    if (_crash_info_ram->magic == crash_magic_hard_fault ||
            _crash_info_ram->magic == crash_magic_stack_overflow ||
            _crash_info_ram->magic == crash_magic_reboot_requested){

        uint32_t xor_checksum = calculate_checksum();

        if (xor_checksum == _crash_info_ram->xor_checksum){
            memcpy(&_previous_crash_info, (void*)_crash_info_ram, sizeof(crash_info_t));
            _previous_crash_info_valid = true;
        } else {
                        //crash not valid
        }
    }
    memset((void*)_crash_info_ram, 0, sizeof(crash_info_t));
}
#endif

const crash_info_t* crash_handler_get_info(void){
    if (_previous_crash_info_valid){
        return &_previous_crash_info;
    }
    return NULL;
}

#ifndef BOOTLOADER_BUILD
void vApplicationStackOverflowHook(TaskHandle_t xTask __attribute__((unused)), signed char *pcTaskName){
    strcpy((char*)_crash_info_ram->task_name, (char*)pcTaskName);
    _crash_info_ram->magic = crash_magic_stack_overflow;
    _crash_info_ram->uptime = GLOBAL_uptime_seconds;
    _crash_info_ram->xor_checksum = calculate_checksum();
    __DSB(); //make sure all data is really written into the memory before doing a reset

    if (DBG_Connected()){
        __BKPT(1);
    }
    NVIC_SystemReset();
}

__attribute__((noreturn)) void system_reset_func(uint32_t line){
    _crash_info_ram->magic = crash_magic_reboot_requested;
    _crash_info_ram->uptime = GLOBAL_uptime_seconds;
    _crash_info_ram->reset_request_line = line;
    _crash_info_ram->xor_checksum = calculate_checksum();
    __DSB();
    NVIC_SystemReset();
    __builtin_unreachable();
}
#endif

#ifdef BOOTLOADER_BUILD
bool system_check_bootloader_request_flag(void){
    if (_crash_info_ram->magic == crash_magic_bootloader_entry){
        uint32_t xor_checksum = calculate_checksum();
        if (xor_checksum == _crash_info_ram->xor_checksum){
            memset((void*)_crash_info_ram, 0, sizeof(crash_info_t));
            return true;
        }
    }
    return false;
}
#else
__attribute__((noreturn)) void system_request_bootloader_entry(void){
    memset((void*)_crash_info_ram, 0, sizeof(crash_info_t));
    _crash_info_ram->magic = crash_magic_bootloader_entry;
    _crash_info_ram->xor_checksum = calculate_checksum();
    __DSB();
    NVIC_SystemReset();
    __builtin_unreachable();
}
#endif

Reset request

The application may want to perform a valid reset, for example when applying some new settings. In this case the macro SYSTEM_RESET(), which adds caller line number to system_reset_func function. Line number is a very easy way to distinguish reset sources, as the chance is very low that a reset is called from the very same line in different source files. The function sets appropriate magic value, stores caller line number, calculates the checksum and requests system reset from the interrupt controller.

The DSB instruction makes the CPU wait until all data is fully written to RAM. Depending on the exact Cortex-M and final chip configuration. There may be some caching or buffering involved so even though the code was executed and the data checksummed, it may not have enough time to get into RAM, while the whole MCU was being reset.

Bootloader entry request and check

The bootloader must also have RAM address modified in the linker script the same way as the main application. Bootloaders often do entry checks at power on (eg. press and hold a button while powering on to enter the bootloader). Being able to reset and enter the bootloader makes it easier for the user and also allows for fully remotely controller firmware updates.

At startup the bootloader checks the memory for valid magic value and checksum. If it does not match, then the memory is left alone to allow the application read it so a reboot after a crash (which always leads through the bootloader) will not destroy debugging breadcrumbs.

Stack overflow detection

This feature uses FreeRTOS standard task stack overflow hook, stores task name, calculates the checksum and issues a software reset.

Summary

Most of this code has to be customized depending on your application type but the principle of keeping a small piece of RAM uninitialized at startup stays the same. Especially the part dealing with reading crash info data when it becomes available has to be implemented. This depends a lot on the kind of device. Some ideas:

  • Device with a display and operator panel - make a menu called "Diagnostics" and display data from the struct. Whenever problems arise - ask the final user for a photo of the screen.
  • Device with lots of flash memory or an SD card - write data from the struct (as an ASCII text file) to the SD card.
  • Device with a GSM module or another wireless - send the crash info (alongside with device serial number) to the server.

In next posts I will show how to diagnose hard faults and memory corruption.