Chapter 9: Debugging, Verification, and the AI Era

Updated on 2026-08-30ai-generated

Writing embedded C is only half the battle. The other half is figuring out why your code doesn't work on real hardware—or worse, why it works sometimes and fails other times. Debugging embedded systems requires different tools and a different mindset than desktop debugging. This chapter covers hardware debugging, crash analysis, and why verification is becoming more valuable than code generation.

The Embedded Debugging Mindset

On desktop, you can add a printf, run the program, and see the output in a terminal. If the program crashes, you get a stack trace. If memory leaks, a profiler shows you where. These tools largely don't exist in embedded systems.

Embedded debugging is different:

No console by default. You may need to configure a UART just to see debug output.
No process termination. A crash doesn't end the program; the hardware keeps executing corrupted code until a watchdog resets it.
Timing-dependent bugs. Race conditions and timing issues may only appear under specific hardware conditions.
Limited visibility. You can't always inspect variables while the system runs at full speed.

The debugging mindset shifts from "find the bug" to "understand the system state at the moment of failure." You reconstruct what happened by examining registers, memory, and peripheral states.

Hardware Debugging: JTAG and SWD

Most ARM Cortex-M MCUs provide hardware debugging through JTAG or SWD (Serial Wire Debug). A debug probe (ST-Link, J-Link, CMSIS-DAP) connects your PC to the MCU and provides:

Breakpoints: Pause execution at a specific instruction
Watchpoints: Pause when a memory address is read or written
Single-stepping: Execute one instruction at a time
Register inspection: Read CPU registers and peripheral registers
Memory inspection: Read/write RAM and flash

Breakpoints in Embedded Code

// Set a breakpoint at the beginning of this function
void process_packet(uint8_t *data, uint16_t len) {
    // When execution pauses here, inspect:
    // - data pointer (valid address?)
    // - len value (expected range?)
    // - Stack pointer (overflow?)
    
    if (len > MAX_PACKET_SIZE) {
        // Watchpoint on len: triggers when len changes
    }
}

Hardware vs. Software Breakpoints

Software breakpoints replace an instruction with a breakpoint opcode. They are unlimited but require the code to be in RAM or writable flash. Hardware breakpoints use debug registers in the CPU; most Cortex-M MCUs have 4-8 hardware breakpoints. Use hardware breakpoints for flash-resident code.

Watchpoints: Trapping Memory Accesses

A watchpoint triggers when a specific memory address is accessed. This is invaluable for finding who corrupts a variable:

// Problem: config.uart_baud changes unexpectedly
// Set a write watchpoint on &config.uart_baud
// When triggered, the debugger shows the exact instruction that wrote it

Debugging Without a Debugger: UART and GPIO Toggling

Sometimes you don't have a debug probe. Two techniques work everywhere:

UART Debug Output

Configure a UART early in startup and route debug prints through it:

// debug.h
#define DEBUG_LEVEL_NONE    0
#define DEBUG_LEVEL_ERROR   1
#define DEBUG_LEVEL_WARN    2
#define DEBUG_LEVEL_INFO    3
#define DEBUG_LEVEL_VERBOSE 4

#define DEBUG_LEVEL DEBUG_LEVEL_INFO

#if DEBUG_LEVEL >= DEBUG_LEVEL_ERROR
    #define DEBUG_ERROR(fmt, ...) debug_print("ERROR: " fmt "\n", ##__VA_ARGS__)
#else
    #define DEBUG_ERROR(fmt, ...)
#endif

void debug_init(void);
void debug_print(const char *fmt, ...);

// Usage
DEBUG_ERROR("UART init failed: %d", status);
DEBUG_WARN("Retrying SPI transaction (%d attempts left)", retries);
DEBUG_INFO("System started, version %s", VERSION_STRING);

The macro approach allows compile-time removal of debug output by changing DEBUG_LEVEL. Zero overhead when disabled.

GPIO Toggling for Timing Analysis

Toggle a GPIO pin at key points in your code and observe with a logic analyzer or oscilloscope:

#define DEBUG_PIN_SET()   (GPIOA->BSRR = GPIO_PIN_8)
#define DEBUG_PIN_CLEAR() (GPIOA->BSRR = (GPIO_PIN_8 << 16))

void ISR_handler(void) {
    DEBUG_PIN_SET();
    // ISR work...
    DEBUG_PIN_CLEAR();
}

This tells you:

How long the ISR takes (pulse width)
How often it runs (pulse frequency)
Whether deadlines are met (pulse timing vs. expected)

Analyzing Crashes: Hard Faults

When an ARM Cortex-M CPU encounters an unrecoverable error, it triggers a Hard Fault. Common causes:

Dereferencing a null or invalid pointer
Executing from a non-executable memory region
Unaligned access (when unaligned access is disabled)
Stack overflow (writing past the stack limits)
Divide by zero (on some MCUs)

The Hard Fault Handler

The default hard fault handler is often an infinite loop:

void HardFault_Handler(void) {
    while (1) {
        // Stuck here: no indication of what went wrong
    }
}

A better handler captures the crash state:

typedef struct {
    uint32_t r0;
    uint32_t r1;
    uint32_t r2;
    uint32_t r3;
    uint32_t r12;
    uint32_t lr;      // Link register (return address)
    uint32_t pc;      // Program counter (where the fault occurred)
    uint32_t psr;     // Program status register
    uint32_t bfar;    // Bus fault address (if applicable)
} crash_info_t;

static crash_info_t crash_info;

void HardFault_Handler(void) {
    __asm volatile(
        "MRS %0, R0\n"
        "MRS %1, R1\n"
        "MRS %2, R2\n"
        "MRS %3, R3\n"
        "MRS %4, R12\n"
        "MOV %5, LR\n"
        "MRS %6, PSR\n"
        : "=r"(crash_info.r0), "=r"(crash_info.r1),
          "=r"(crash_info.r2), "=r"(crash_info.r3),
          "=r"(crash_info.r12), "=r"(crash_info.lr),
          "=r"(crash_info.psr)
    );
    
    // Extract the stacked PC (pushed by hardware before entering ISR)
    uint32_t *stack_ptr = (uint32_t *)__get_MSP();
    crash_info.pc = stack_ptr[6];  // PC was pushed at offset 6
    crash_info.bfar = SCB->BFAR;   // Bus fault address
    
    // Store crash info in non-volatile memory or output via UART
    debug_print("HARD FAULT! PC=0x%08X LR=0x%08X\n", 
                crash_info.pc, crash_info.lr);
    
    // Optionally reset the system
    NVIC_SystemReset();
}

With the PC (program counter) captured, you can look up the instruction in the disassembly and find which C line caused the fault.

Stack Overflow Detection

Stack overflow is a common cause of hard faults. Detect it by filling the stack with a pattern and checking it periodically:

// In startup code or early main
extern uint32_t _stack_start;
extern uint32_t _stack_end;

void stack_pattern_fill(void) {
    uint32_t *p = &_stack_start;
    while (p < &_stack_end) {
        *p++ = 0xDEADBEEF;
    }
}

uint32_t stack_usage(void) {
    uint32_t *p = &_stack_start;
    while (*p == 0xDEADBEEF && p < &_stack_end) {
        p++;
    }
    return (uint32_t)(&_stack_end - p) * 4;
}

// Check periodically or in a debug command
if (stack_usage() > STACK_WARNING_THRESHOLD) {
    DEBUG_WARN("Stack usage: %lu bytes", stack_usage());
}

If the pattern is completely overwritten, the stack overflowed.

Logic Analyzers and Oscilloscopes

For timing-critical debugging, a logic analyzer is invaluable:

// Toggle a debug pin at key events
void uart_isr(void) {
    DEBUG_PIN_SET();
    uint8_t data = UART->DR;
    ring_buffer_push(&rx_buf, data);
    DEBUG_PIN_CLEAR();
}

The logic analyzer shows:

ISR duration: Pulse width while the pin is high
ISR frequency: Time between rising edges
Response time: Delay from a trigger event to the ISR execution
Protocol analysis: Decode UART, SPI, I2C signals directly

Oscilloscopes show analog characteristics: voltage levels, rise/fall times, noise, and signal integrity issues that digital analyzers miss.

The Verification Mindset for the AI Era

Large language models and code generators are increasingly capable of writing embedded C. They can generate register configurations, driver skeletons, and even complete modules. What they cannot do:

Verify timing behavior on real hardware. AI doesn't know if an ISR takes 3 microseconds or 300.
Observe race conditions. AI cannot run your code on hardware and see the intermittent failure.
Understand system-level interactions. AI sees code in isolation, not the emergent behavior of multiple peripherals and ISRs.
Debug hardware-specific issues. Signal integrity, clock glitches, power supply noise—these require physical measurements.

This shifts the embedded engineer's role from writing code to verifying behavior:

What AI Can Handle (Offload)

Generating register initialization code from datasheet specifications
Writing repetitive driver boilerplate
Converting between vendor HAL patterns
Generating test cases and mock implementations
Documenting code and explaining existing implementations

What Requires Human Verification (Focus Here)

Defining system architecture and timing requirements
Reviewing AI-generated code for correctness and edge cases
Testing on real hardware and analyzing failures
Optimizing ISR timing and interrupt priorities
Certifying safety-critical systems
Debugging race conditions and timing issues

The Verification Workflow

1. Define requirements and constraints
   ↓
2. Generate initial implementation (AI or manual)
   ↓
3. Review code for correctness (static analysis, manual review)
   ↓
4. Test on hardware (debugger, logic analyzer)
   ↓
5. Measure timing and resource usage
   ↓
6. Iterate until verified
   ↓
7. Document assumptions and limitations

This workflow works whether step 2 is done by AI, a colleague, or yourself. The value lies in steps 1, 3, 4, and 5—the verification steps.

What to Actually Learn vs. Look Up

Look up (as needed):

Specific debugger commands for your IDE
JTAG/SWD pinout and wiring
Vendor-specific fault registers and their meanings
Logic analyzer protocol decoders

Deeply understand:

How to capture crash state from a hard fault handler
How to detect and diagnose stack overflow
The difference between hardware and software breakpoints
How to use GPIO toggling for timing analysis
The verification workflow: requirements → implementation → testing → measurement
What AI tools can and cannot do in embedded development

Key Takeaways

Embedded debugging requires different tools. No console, no process termination, no easy stack traces.
JTAG/SWD provides hardware-level visibility. Breakpoints, watchpoints, and register inspection.
UART debug output and GPIO toggling work everywhere. No debug probe required.
A good hard fault handler captures the crash state. PC, registers, and fault address enable post-mortem analysis.
Stack overflow is a common crash cause. Pattern filling detects it before it causes failures.
Verification is the critical skill. AI can write code; only humans can verify it on real hardware.

Chapter 10 will introduce RTOS concepts, showing how the bare-metal patterns from earlier chapters scale to multi-tasking systems with a real-time operating system.