debugging – Observations from Uppsala

Reversing out of Reverse

The Intel Simics simulator version 7 removed a long-standing feature from the simulator framework. Reverse execution is no longer available. In its place, in-memory snapshots were introduced, which arguably offer most of the benefits at a lower implementation cost. What happened? I’ve been asked about the reasoning behind the chance on several occasions since I left Intel. I’d like to share my perspective on the decision, as it highlights the challenges of turning an idea into a robust, shippable feature.

Intel Blog: Catching a Tricky Bug by Running Simics on Simics

I recently published a long post on the Intel Community Blog, talking about how my colleague Evgeny solved a nicely complicated bug using Simics-on-Simics. The bug involved UEFI, an operating system, SMM, SMI, and virtualization. Just another day in the office (or more like a year, given how long it took to get this one resolved).

Intel Blog Post: “Quit Thinking and Look” – Mea Culpa Chasing a Performance Bug

I have written before about the debug advice to “Quit thinking and look.” It means that you should not form conclusions prematurely. Stop and look at what is going on instead of guessing and cooking up theoretical scenarios. Sound advice that I completely failed to follow in the case that I just chronicled on my Intel Blog: https://software.intel.com/en-us/blogs/2020/03/18/quit-thinking-and-look-chasing-simics-performance

“Studying the Advancement in Debugging Practice” / 2016 Paper

I find debugging a very interesting topic of study, so when I stumbled on the paper “Studying the advancement in debugging practice of professional software developers”, I had to do a blog on its contents.

Intel Blog Post: Simics in the DARPA Cyber Grand Challenge

The US Defense Advanced Projects Agency (DARPA) ran a “Cyber Grand Challenge” in 2016, where automated cyber-attack and cyber-defense systems were pitted against each other to drive progress in autonomous cyber-security. The competition was run on physical computers (obviously), but Simics was used in a parallel flow to check that competitors’ programs were not trying to undermine the infrastructure of the competition rather than compete fairly inside the rules of the competition.

Borland Turbo Debugger – Reverse Execution in 1992

Thanks to a tip from “Derek” on a previous blog post about a replay debugger from 1995, I was made aware of the reverse execution ability that was available in the Borland Turbo Debugger version 3.0 from 1992! This is the oldest commercial instance of “reverse” that I have found (so far), and definitely one of the oldest incarnations of the idea overall. Thanks to Google and the Internet, I managed to find a scanned copy of the manual of the product, which provided some additional information. Note that the debugger only does reverse execution, but not reverse debugging since you cannot run in reverse to stop at a breakpoint.

Continue reading “Borland Turbo Debugger – Reverse Execution in 1992”

Undo Reverse Debugger “Pivots” to Record-Replay

I just found a story about Undo software that was rather interesting from a strategic perspective. “Patient capital from CIC gives ‘time travelling’ company Undo space to pivot“, from the BusinessWeekly in the UK. The article describes a change from selling to individual developers, towards selling to enterprises. This is an important business change, but it also marks I think a technology thinking shift: from single-session debug to record-replay.

Continue reading “Undo Reverse Debugger “Pivots” to Record-Replay”

A new (and old) Reverse Debugger – Microsoft WinDbg

A blog post from Undo Software informed me that Microsoft has rather quietly released a reverse debugger tool for Windows programs – WinDbg with Time Travel Debug. It is available in the latest preview of WinDbg, as available through the Windows Store, for the most recent Windows 10 versions (Anniversary update or later). According to a CPPcon talk about the tool (Youtube recording of the talk) the technology has a decade-long history internally at Microsoft, but is only now being released to the public after a few years of development. So it is a new old thing 🙂

Continue reading “A new (and old) Reverse Debugger – Microsoft WinDbg”

Intel Blog: Finding a Linux Kernel bug by running Simics on Simics

intel sw small I love bug and debug stories in general. Bugs are a fun and interesting part of software engineering, programming, and systems development. Stories that involve running Simics on Simics to find bugs are a particular category that is fascinating, as it shows how to apply serious software technology to solve problems related to said serious software technology. On the Intel Software and Services blog, I just posted a story about just that: debugging a Linux kernel bug provoked by Simics, by running Simics on a small network of machines inside of Simics. See https://blogs.intel.com/evangelists/2016/05/30/finding-kernel-1-2-3-bug-running-wind-river-simics-simics/ for the full story.

Continue reading “Intel Blog: Finding a Linux Kernel bug by running Simics on Simics”

rr– The Mozilla Reverse Debugger

A new record, replay, and reverse debugger has appeared, and I just had to take a look at what they do and how they do it. “rr” has been developed by the Firefox developers at Mozilla Corporation, initially for the purpose of debugging Firefox itself. Starting at a debugger from the angle of attacking a particular program does let you get things going quickly, but the resulting tool is clearly generally useful, at least for Linux user-land programs on x86. Since I have tried to keep up with the developments in this field, a write-up seems to be called for.

Continue reading “rr– The Mozilla Reverse Debugger”

Hardware debug and measurement in the IBM POWER8

ibm power doodle

I have read some recent IBM articles about the POWER8 processor and its hardware debug and trace facilities. They are very impressive, and quite interesting to compare to what is usually found in the embedded world. Instead of being designed to help with software debug, it seems the hardware mechanisms in the Power8 are rather focused on silicon bringup and performance analysis and verification in IBM’s own labs. As well as supporting virtual machines and JIT-based systems!

Continue reading “Hardware debug and measurement in the IBM POWER8”

Microsoft Catapult – Real Interesting Research at Real Scale

At the ISCA 2014 conference (the biggest event in computer architecture research), a group of researchers from Microsoft Research presented a paper on their Catapult system. The full title of the paper is “A Reconfigurable Fabric for Accelerating Large-Scale Datacenter Services“, and it is about using FPGAs to accelerate search engine queries at datacenter scale. It has 23 authors, which is probably the most I have ever seen on an interesting paper. There are many things to be learnt from and discussed about this paper, and here are my thoughts on it.

Continue reading “Microsoft Catapult – Real Interesting Research at Real Scale”

Wind River Blog: UEFI on Simics

Simics can run and debug UEFI BIOSes, and that is the topic of my latest blog at Wind River. UEFI is actually pretty interesting once you get to know it, and building a good debug experience for UEFI took a bit of work. Still, it was built as just another target for the standard uniform Simics debugger, which is not the way most other UEFI and BIOS debuggers are built. I guess in that in the past, debugging a BIOS required such specialized tools that it made sense to also build a custom specialized frontend for the purpose. With a simulator as the backend, things do become simpler and more uniform, and Eclipse CDT is a actually a very good basis for a debugger for any kind of C and C++ code.

For more reading on UEFI itself, I can recommend the 2011 Intel Tech Journal on the topic.

I Planted a Bug for Myself to Find…

butterfly I have a silly demo program that I have been using for a few years to demonstrate the Simics Analyzer ability to track software programs as they are executing and plot which threads run where and when. This demo involves using that plot window to virtually draw text, in a way akin to how I used to make my old ZX Spectrum “draw” things in the border. But when I brought it up in a new setting it failed to run properly and actually starting hanging on me. Strange, but also quite funny when I realized that I had originally foreseen this very problem and consciously decided not to put in a fix for it… which now came back to bite me in a pretty spectacular way. But at least I did get an interesting bug to write about.

Continue reading “I Planted a Bug for Myself to Find…”

Book Review: Debugging

Debugging – the 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems by David Agans was published in 2002, based on several decades of practical experience in debugging embedded systems. Compared to the other debugging book I read this Summer, Debugging is much more a book for the active professional currently working on embedded products. It is more of a guidebook for the practitioner than a textbook for students that need to learn the basics.

Continue reading “Book Review: Debugging”