reverse execution – Observations from Uppsala

Reversing out of Reverse

The Intel Simics simulator version 7 removed a long-standing feature from the simulator framework. Reverse execution is no longer available. In its place, in-memory snapshots were introduced, which arguably offer most of the benefits at a lower implementation cost. What happened? I’ve been asked about the reasoning behind the chance on several occasions since I left Intel. I’d like to share my perspective on the decision, as it highlights the challenges of turning an idea into a robust, shippable feature.

Borland Turbo Debugger – Reverse Execution in 1992

Thanks to a tip from “Derek” on a previous blog post about a replay debugger from 1995, I was made aware of the reverse execution ability that was available in the Borland Turbo Debugger version 3.0 from 1992! This is the oldest commercial instance of “reverse” that I have found (so far), and definitely one of the oldest incarnations of the idea overall. Thanks to Google and the Internet, I managed to find a scanned copy of the manual of the product, which provided some additional information. Note that the debugger only does reverse execution, but not reverse debugging since you cannot run in reverse to stop at a breakpoint.

Continue reading “Borland Turbo Debugger – Reverse Execution in 1992”

Simulics – A New Commercial Reverse Debugger

simulics_logo_just_logo A new entry just showed up in the world of reverse debugging – Simulics, from German company Simulics. It does seem like the company and the tool are called the same. Simulics is a rather rare breed, the full-system-simulation-based reverse debugger. We have actually only seen a few these in history, with Simics being the primary example. Most reverse debuggers apply to user-level code and use various forms of OS call intercepts to create a reproducible run. Since the Simulics company clearly comes from the deeply embedded systems field, it makes sense to take the full-system approach since that makes it possible to debug code such as interrupt handlers.

I have also updated my history of commercial reverse debuggers to include Simulics.

Continue reading “Simulics – A New Commercial Reverse Debugger”

Reverse Debug with Hardware in the Loop

Last year (2015), a paper called “Don’t Panic: Reverse Debugging of Kernel Drivers” was presented at the ESEC/FSE (European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering) conference. The paper was written by Pavel Dovgalyuk, Denis Dmitriev, and Vladimir Makarov from the Russian Academy of Sciences. It describes a rather interesting approach to Linux kernel device driver debug, using a deterministic variant of Qemu along with record/replay of hardware interactions. I think this is the first published instance of using reverse debugging in a simulator together with real hardware.

Continue reading “Reverse Debug with Hardware in the Loop”

Intel Blog: Finding a Linux Kernel bug by running Simics on Simics

intel sw small I love bug and debug stories in general. Bugs are a fun and interesting part of software engineering, programming, and systems development. Stories that involve running Simics on Simics to find bugs are a particular category that is fascinating, as it shows how to apply serious software technology to solve problems related to said serious software technology. On the Intel Software and Services blog, I just posted a story about just that: debugging a Linux kernel bug provoked by Simics, by running Simics on a small network of machines inside of Simics. See https://blogs.intel.com/evangelists/2016/05/30/finding-kernel-1-2-3-bug-running-wind-river-simics-simics/ for the full story.

Continue reading “Intel Blog: Finding a Linux Kernel bug by running Simics on Simics”

Wind River Blog: Reverse Execution (not Debugging)

There is a new post at my Wind River blog, where I go back to the basics of reverse execution in Simics and what it can do. The post is not about reverse debugging, about which I have written quite a bit (see for example my series of blog posts: 1, 2, 3, 4, 5, 6), but about the core of reverse execution. I.e., moving the system state back in time in a variety of ways. There is an accompanying video demo on Youtube.

Wind River Blog: Collaborating with Recording Checkpoints

There is a new post at my Wind River blog, about how some new features in Simics 4.8 improve the collaboration power of Simics checkpoints. For the first time, Simics checkpoint can now carry a piece of history (slice of time), which also makes reverse execution and reverse debug work with checkpoints in a logical way.

Continue reading “Wind River Blog: Collaborating with Recording Checkpoints”

Reverse Execution History Updates

After some discussions at the S4D conference last week, I have some additional updates to the history and technologies of reverse execution. I have found one new commercial product at a much earlier point in time, and an interesting note on memory consistency.

Continue reading “Reverse Execution History Updates”

Paper & Talk at S4D 2012: Reverse Debug

I am going to the S4D conference for the third year in a row. This year, I have a paper on reverse debugging, reviewing the technology, products, and history of the idea. I will probably write a longer blog post after the conference, interesting things tend to come up.

Youtube Movie on Reverse Execution (and a small bit of Reverse Debug)

We just uploaded a short movie about reverse execution and reverse debugging to Youtube, to the Wind River official channel. In the short time available in this demo, we really only show reverse execution. Reverse debug, as I define it, is not used much at all, as explaining what goes on when you start to put breakpoints into a program and analyze its behavior takes a surprising amount of time.

Continue reading “Youtube Movie on Reverse Execution (and a small bit of Reverse Debug)”

Reverse History Part Three – Products

In this final part of my series on the history of reverse debugging I will look at the products that launched around the mid-2000s and that finally made reverse debugging available in a commercially packaged product and not just research prototypes. Part one of this series provided a background on the technology and part two discussed various research papers on the topic going back to the early 1970s. The first commercial product featuring reverse debugging was launched in 2003, and then there have been a steady trickle of new products up until today.

Originally published in January 2012. Post updated 2012-09-28 with a revised timeline for Lauterbach CTS. Post updated 2016-04-05 to include Mozilla RR. Post updated 2016-12-26 to add Simulics. Post updated 2017-10-08 to add Microsoft WinDbg. Post updated 2018-07-28 to add Borland Turbo Debugger.

Continue reading “Reverse History Part Three – Products”

Reverse History Part Two – Research

This is the second post in my series on the history of reverse execution, covering various early research papers. It is clear that reverse debugging has been considered a good idea for a very long time. Sadly though, not a practical one (at the time). The idea is too obvious to be considered new. Here are some of the papers that I have found, going back before reverse debugging got started for real in actual products (around 2003) as well later on for interesting research papers that did not make it into products. It is worth noting that products/useful software has become more common in recent times as the way that reverse debugging ideas get expressed.

Continue reading “Reverse History Part Two – Research”

Reverse History Part One

For some reason, when I think of reverse execution and debugging, the sound track that goes through my head is a UK novelty hit from 1987, “Star Trekkin” by the Firm. It contains the memorable line “we’re only going forward ’cause we can’t find reverse“. To me, that sums up the history of reverse debugging nicely. The only reason we are not all using it every day is that practical reverse debugging has not been available until quite recently. However, in the past ten years, I think we can say that software development has indeed found reverse. It took a while to get there, however. This is the first of a series of blog posts that will try to cover some of the history of reverse debugging. The text turned out to be so long that I had to break it up to make each post usefully short. Part two is about research, and part three about products.
Continue reading “Reverse History Part One”

Wind River Blog Post: Determinism vs Variability

I have a new post at my Wind River blog, about variability and determinism and how these two concepts interact. In short, even a deterministic simulator can expose great variability in a software workload and target system behavior.

Contributing to the Reversible GDB

Last week, I finally got the last “OK” from the maintainers of GDB, the Gnu Debugger, indicating that my contribution to the GDB project was accepted. This is my first contribution to an open-source project, and the piece of code that has my name on it is positively puny. It is actually not really code at all, it is just a piece of documentation, for the extensions to the GDB-MI command set needed to support reversible debugging. The actual code doing the work was contributed by a colleague of mine, Tomas Holmberg, credit where credit is due.

Continue reading “Contributing to the Reversible GDB”