A few weeks ago, I attended small event organized by Tasking. They showed their compilers, debuggers, and testing tools. It was all really interesting, but what really stood out to me is just how capable hardware-based debuggers are today. They can do some really quite incredible things.
Hardware Debugger?
With “hardware debugger” I mean the hardware box that serves to connect an embedded processor to the debugger software running on a PC. The connection between the debugger hardware and the board is usually called a probe, and it requires a connection on the board. Sometimes the debugger box and the connection to the board are collectively called a debug probe.
The capabilities of the debug box vary greatly, from just a simple connector to a machine with significant built-in capability. I remember the first time I noticed what they could do, when Green Hills launched the Time Machine debugger back in 2003. With a few gigabytes of onboard RAM, it could record a slice of execution time from a target system, and then the debugger on the PC could use that trace to facilitate reverse debugging. It should be noted that Lauterbach had launched a similar capability a few years earlier.
Current examples include the “Blue box” line from Tasking, Lauterbach PowerDebug, IAR I-jet, and Green Hills Probe. There are even more out there, every time I go to the embedded world it seems I stumble on some new player.
The debug probes rely on the debug unit present in the targeted chip – the debugger needs a way to access the target system over a back door and perform tasks like setting hardware breakpoints, configuring hardware tracing, reading and writing registers, etc. This is provided by the on-chip debug unit. The precise capabilities differ between vendors.
Often, there are “development” versions of chips available with the most extensive debug support, while “production” versions have many fewer features. This reduces cost and the attack surface (after all, a hardware debugger is a very nice back door for hackers).
More than One Target
The first debug probes were built to connect to a single processor core, implemented as a single chip on a board. Later on, multicore processors appeared and debuggers started supporting that. Today, hardware debuggers support heterogeneous multicore processors (such as main cores plus the core in a hardware security model, or big cores plus real-time cores).
The next logical step is to support multiple chips (controllers/processors/SoCs/MCUs/whatever) at once. The processors can be on the same board or on multiple different boards. I find the idea of debugging multiple chips on a single board quite compelling. It is very common to have little firmware-running devices around the main processor even in quite small and limited system, for example power-management ICs (PMICs).
This is pretty cool. It means that hardware debuggers can offer the kind of multiple-target multiple-architecture debug that we previously could only do with virtual platforms. The debugger can even stop all the attached SoCs at once – the kind of system-level break that used to be the exclusive domain of virtual platforms (with some delay due to communications latencies, obviously).
Another maybe less known capability of modern hardware debuggers is the ability to attach to communications buses like CAN and discrete IO lines. This allows the debugger to correlate code execution traces with network and hardware activity traces. In real hardware. In real time. It is pretty astonishing technology. The connections can also be used to drive signals and even serve as a simple network packet injector tool (for serious network work, there are specialized tools that do a more comprehensive job).
Sampling at High Speed
Tracing has always been a feature of hardware debuggers (with help from the debug unit in the target system). Classic tracing is based on the chip emitting trace events when something happens, i.e., it is triggered from the logic inside the target chip. However, if you do not have tracing support in the target – such as when using a production device – what can you do?
The answer is that the basic debug hardware that is almost always there can still pull values out of the target, even is push-based tracing is no longer available. Pulling values basically means sampling. The debugger will pull (or sample or read) values based on some external criterion, typically time.
To make sampling work, you need the debug hardware to operate very quickly. Ordering memory reads from the debugger on the PC has terrible latency. The solution is to let the debug hardware do the job. How?
Hardware debuggers have contained their own internal programmable processor cores for a long time. I remember working with the debugger team at Wind River in the early 2010s and realizing that the PowerPC processor inside the hardware debuggers was more powerful than most of the (quite simple) chips that they were used to debug!
However, even firmware on a fast processor can be too slow. For that reason, high-end hardware debuggers contain FPGAs, which allows the debugger solution to implement functionality like high-speed sampling in hardware. No firmware needed. With the FPGA, it is possible to sample multiple memory locations (i.e., software variables) at rates of up to 10000 Hz or faster! We are looking at sampling rates of upwards of a million samples per second. Impressive.
Debugger Instead of Network
Another development worth noting is that hardware debuggers are moving beyond classic debug work to serve as an interface between testing tools and the hardware. Typically, a software (unit) testing tool employs a test harness running on the target system. The test harness communicates with the testing tool running on a development PC to receive test cases and report results. This communication is traditionally done using a network connection or serial port – which requires that the software stack running on the target system has a functioning driver stack and available hardware ports for the network being used.
Using a debugger avoids the driver stack entirely. Instead, the testing tool talks to the testing harness on the target by writing and reading memory locations via the debugger. Of course, it does require that your
It’s Cool
The Tasking event helped me realize just how capable hardware debuggers are today. They are doing things that I would previously have said were infeasible to do with hardware and really the unique domain of virtual platforms.
Yeah, that’s wild that production chips intentionally strip out debug features to cut costs and reduce attack surface. Makes sense when you think about it, but I never really considered the security angle before.