A comment on my old blog post about the history of reverse execution gave me a pointer to a fairly early example of replay debugging. The comment pointed at a 2002 blog post which in turn pointed at a 1999 LWN.net text which almost in passing describes a seemingly working record-replay debugger from 1995. The author was a Michael Elizabeth Chastain, of whom I have not managed to find any later traces.
A new record, replay, and reverse debugger has appeared, and I just had to take a look at what they do and how they do it. “rr” has been developed by the Firefox developers at Mozilla Corporation, initially for the purpose of debugging Firefox itself. Starting at a debugger from the angle of attacking a particular program does let you get things going quickly, but the resulting tool is clearly generally useful, at least for Linux user-land programs on x86. Since I have tried to keep up with the developments in this field, a write-up seems to be called for.
In this final part of my series on the history of reverse debugging I will look at the products that launched around the mid-2000s and that finally made reverse debugging available in a commercially packaged product and not just research prototypes. Part one of this series provided a background on the technology and part two discussed various research papers on the topic going back to the early 1970s. The first commercial product featuring reverse debugging was launched in 2003, and then there have been a steady trickle of new products up until today.
Originally published in January 2012. Post updated 2012-09-28 with a revised timeline for Lauterbach CTS. Post updated 2016-04-05 to include Mozilla RR. Post updated 2016-12-26 to add Simulics. Post updated 2017-10-08 to add Microsoft WinDbg.
Last week, I finally got the last “OK” from the maintainers of GDB, the Gnu Debugger, indicating that my contribution to the GDB project was accepted. This is my first contribution to an open-source project, and the piece of code that has my name on it is positively puny. It is actually not really code at all, it is just a piece of documentation, for the extensions to the GDB-MI command set needed to support reversible debugging. The actual code doing the work was contributed by a colleague of mine, Tomas Holmberg, credit where credit is due.
An unplanned and unexpected bonus with my trip to the FDL 2009 conference was the co-located S4D conference. S4D means System, Software, SoC and Silicon Debug, and is a conference that has grown out of some recent workshops on the topic of debugging, as seen from the perspective of hardware designers (mostly). S4D was part of the same package as FDL and DASIP, entrance to one conference got you into the other two too. As I did not know about S4D until quite late in the process, this was a great opportunity for me to look at what they were doing.