reverse debugging – Observations from Uppsala

Reversing out of Reverse

The Intel Simics simulator version 7 removed a long-standing feature from the simulator framework. Reverse execution is no longer available. In its place, in-memory snapshots were introduced, which arguably offer most of the benefits at a lower implementation cost. What happened? I’ve been asked about the reasoning behind the chance on several occasions since I left Intel. I’d like to share my perspective on the decision, as it highlights the challenges of turning an idea into a robust, shippable feature.

“Studying the Advancement in Debugging Practice” / 2016 Paper

I find debugging a very interesting topic of study, so when I stumbled on the paper “Studying the advancement in debugging practice of professional software developers”, I had to do a blog on its contents.

Intel Blog Post: Simics in the DARPA Cyber Grand Challenge

The US Defense Advanced Projects Agency (DARPA) ran a “Cyber Grand Challenge” in 2016, where automated cyber-attack and cyber-defense systems were pitted against each other to drive progress in autonomous cyber-security. The competition was run on physical computers (obviously), but Simics was used in a parallel flow to check that competitors’ programs were not trying to undermine the infrastructure of the competition rather than compete fairly inside the rules of the competition.

Microsoft REPT: You CAN Reverse from a Core Dump!

There are some things in computing that seem “obviously” true and that “clearly” make it “impossible” to do some things. One example of this is the idea that you cannot go backwards in time from the current state of a program or computer system and recover previous state by just reversing the semantics of the instructions in the program. In particular, that you cannot take a core dump from a failed system and reverse-execute back from it – how could you? In order to do reverse debugging and reverse execution, you “have to” record the state at the first point in time that you want to be able to go back to, and then record all changes to the state. Turns out I was wrong, as shown by a recent Usenix OSDI paper.

Borland Turbo Debugger – Reverse Execution in 1992

Thanks to a tip from “Derek” on a previous blog post about a replay debugger from 1995, I was made aware of the reverse execution ability that was available in the Borland Turbo Debugger version 3.0 from 1992! This is the oldest commercial instance of “reverse” that I have found (so far), and definitely one of the oldest incarnations of the idea overall. Thanks to Google and the Internet, I managed to find a scanned copy of the manual of the product, which provided some additional information. Note that the debugger only does reverse execution, but not reverse debugging since you cannot run in reverse to stop at a breakpoint.

Continue reading “Borland Turbo Debugger – Reverse Execution in 1992”

Undo Reverse Debugger “Pivots” to Record-Replay

I just found a story about Undo software that was rather interesting from a strategic perspective. “Patient capital from CIC gives ‘time travelling’ company Undo space to pivot“, from the BusinessWeekly in the UK. The article describes a change from selling to individual developers, towards selling to enterprises. This is an important business change, but it also marks I think a technology thinking shift: from single-session debug to record-replay.

Continue reading “Undo Reverse Debugger “Pivots” to Record-Replay”

A new (and old) Reverse Debugger – Microsoft WinDbg

A blog post from Undo Software informed me that Microsoft has rather quietly released a reverse debugger tool for Windows programs – WinDbg with Time Travel Debug. It is available in the latest preview of WinDbg, as available through the Windows Store, for the most recent Windows 10 versions (Anniversary update or later). According to a CPPcon talk about the tool (Youtube recording of the talk) the technology has a decade-long history internally at Microsoft, but is only now being released to the public after a few years of development. So it is a new old thing 🙂

Continue reading “A new (and old) Reverse Debugger – Microsoft WinDbg”

Simulics – A New Commercial Reverse Debugger

simulics_logo_just_logo A new entry just showed up in the world of reverse debugging – Simulics, from German company Simulics. It does seem like the company and the tool are called the same. Simulics is a rather rare breed, the full-system-simulation-based reverse debugger. We have actually only seen a few these in history, with Simics being the primary example. Most reverse debuggers apply to user-level code and use various forms of OS call intercepts to create a reproducible run. Since the Simulics company clearly comes from the deeply embedded systems field, it makes sense to take the full-system approach since that makes it possible to debug code such as interrupt handlers.

I have also updated my history of commercial reverse debuggers to include Simulics.

Continue reading “Simulics – A New Commercial Reverse Debugger”

UndoDB Recording during Automatic Testing

UndoDB is an old player in the reverse debugging market, and have kept at it for ten years. Last year, they released the Live Recorder record-replay function. Most recently, they have showed an integration between the recorder function and Jenkins, where the idea is that you record failing runs in your CI system and replay them on the developer’s machine. Demo video is found on Youtube, see https://www.youtube.com/watch?v=ap8552P5vss.

Continue reading “UndoDB Recording during Automatic Testing”

Reverse Debug with Hardware in the Loop

Last year (2015), a paper called “Don’t Panic: Reverse Debugging of Kernel Drivers” was presented at the ESEC/FSE (European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering) conference. The paper was written by Pavel Dovgalyuk, Denis Dmitriev, and Vladimir Makarov from the Russian Academy of Sciences. It describes a rather interesting approach to Linux kernel device driver debug, using a deterministic variant of Qemu along with record/replay of hardware interactions. I think this is the first published instance of using reverse debugging in a simulator together with real hardware.

Continue reading “Reverse Debug with Hardware in the Loop”

A Replay Debugger from 1995!

2016-06-05_21-29-16 A comment on my old blog post about the history of reverse execution gave me a pointer to a fairly early example of replay debugging. The comment pointed at a 2002 blog post which in turn pointed at a 1999 LWN.net text which almost in passing describes a seemingly working record-replay debugger from 1995. The author was a Michael Elizabeth Chastain, of whom I have not managed to find any later traces.

Continue reading “A Replay Debugger from 1995!”

Intel Blog: Finding a Linux Kernel bug by running Simics on Simics

intel sw small I love bug and debug stories in general. Bugs are a fun and interesting part of software engineering, programming, and systems development. Stories that involve running Simics on Simics to find bugs are a particular category that is fascinating, as it shows how to apply serious software technology to solve problems related to said serious software technology. On the Intel Software and Services blog, I just posted a story about just that: debugging a Linux kernel bug provoked by Simics, by running Simics on a small network of machines inside of Simics. See https://blogs.intel.com/evangelists/2016/05/30/finding-kernel-1-2-3-bug-running-wind-river-simics-simics/ for the full story.

Continue reading “Intel Blog: Finding a Linux Kernel bug by running Simics on Simics”

rr– The Mozilla Reverse Debugger

A new record, replay, and reverse debugger has appeared, and I just had to take a look at what they do and how they do it. “rr” has been developed by the Firefox developers at Mozilla Corporation, initially for the purpose of debugging Firefox itself. Starting at a debugger from the angle of attacking a particular program does let you get things going quickly, but the resulting tool is clearly generally useful, at least for Linux user-land programs on x86. Since I have tried to keep up with the developments in this field, a write-up seems to be called for.

Continue reading “rr– The Mozilla Reverse Debugger”

UndoDB Live Recorder: Record/Replay for Live Systems

I have a long-standing interested in debugging in general and reverse debugging in particular and the related idea of record-replay debug (see a series of blog posts I did a few years ago on the topic: history 1, history 2, history 3, S4D report, updates, Simics reverse execution, and then Lab Cloud record/replay). Recently, I found out that Undo Software, one of the pioneers in the field, had released a product called “Live Recorder“. So I went to check it out by reading their materials and comparing it to what we have seen before.

Continue reading “UndoDB Live Recorder: Record/Replay for Live Systems”

Wind River Blog: Reverse Execution (not Debugging)

There is a new post at my Wind River blog, where I go back to the basics of reverse execution in Simics and what it can do. The post is not about reverse debugging, about which I have written quite a bit (see for example my series of blog posts: 1, 2, 3, 4, 5, 6), but about the core of reverse execution. I.e., moving the system state back in time in a variety of ways. There is an accompanying video demo on Youtube.