Swap ‘n’ play is the technology of taking a machine state out of a fast virtual platform, and transferring it to a detailed virtual platform. This enables a user to boot a machine and position a workload using a fast platform, and then get detailed performance results (or other information from a detailed run, such as chasing low-level timing bugs) using the detailed model. This is an old and proven idea, and I think it was in fact the main reason for the first implementation of “virtual platform checkpointing” that I know of, in SimOS in the early 1990s (see Stanford technical report CSL-TR-94-631). It is also something I have seen implemented in Simics for at least a decade, and used by computer architecture researchers.

When I read an ARM IQ article about Carbon’s technology, Bill Neifert introduces the technology in a way that does not acknowledge the long history of the idea:

Carbon Design Systems has recently introduced a best of both worlds approach to virtual platforms. Relying upon a new technology which Carbon calls “Swap ‘n Play”, Carbon’s newest generation of virtual platforms now enable a user to run a platform at 100s of MIPS using ARM Fast Models and then swap over to a 100% accurate, Carbon model implementation of that same platform at any software breakpoint.

It is certainly new to Carbon and the ARM FastSim simulators, but this way of putting it makes it sound like this is all brand new.

Another proven use of abstraction-level swapping is also described in the article: sampled simulation for computer architecture evaluations:

Instead of actually swapping however, the system continues running in the fast functional mode. The checkpoint manager is use to create additional Swap ‘n Play checkpoints at set time increments, say every 0.1 seconds of real-time. At each time increment, the checkpoint is created and execution continues in the Fast Model system. …  Each one of these checkpoints can be executed as a separate, 100% accurate simulation and distributed to a farm of computers.

This somewhat simplifies the problem, as computer architecture experience shows that using regular samples tend to skew results. Tools like Simpoint have been used for a long time to do this sampling in a more sophisticated and effective way. However, as a simplified explanation to introduce the idea to a new audience this makes sense.

While it is certainly interesting to see Carbon implement classic checkpointing and abstraction level transfers, it seems that they have not read or do not acknowledge the many available publications on this from the computer architecture field. EDA is in many ways a world of its own, with surprisingly few bridges to the research done in computer architecture, for example. That technologies are crossing over is good, but it is also good to be aware of the history and legacy of a technology before billing it as “new”. It is good, useful, great to have, certainly, but not original or new.

Update: I did receive some feedback from Carbon about this topic, and they were aware of prior work. It was just hard to fit it into the format of the IQ article, which I can see makes some sense based on my experience with popular editorials. Still, I think it would have made the presentation in the IQ article and their blog a bit deeper to point out that this is an old proven technology with almost two decades of active use behind it.

Once upon a time, I was young man in high school where our little computer club got a new PC with a color screen and a floating-point coprocessor. One fun little program I wrote was a simple gravity simulator, where a number of point-size assigned various mass flew around interacting with each other. We used that program and tried to set up initial setting for sizes, speeds, and directions of bodies that would result in some kind of stable system. More often that not, all we managed to create were comets that came in, took a sharp corner around a “star” and disappeared out into the void again. Still, it was great fun. And when I discovered Angry Birds Space it felt like a chance to try that again. Overall, “space” as my son calls it is a great spin on the Angry Birds idea. However, the way it is sold does not make me too happy.

Angry Birds Space is certainly different from previous iterations of the idea. Instead of a flat ground with constant gravity pull, we have space where gravity comes from planets (and in some levels, there is no gravity at all, just free floating objects). Birds will be pulled in by the planets, and trajectories can easily turn into multiple orbits of a planet before crashing down on some pigs or structures.

Rovio has made celestial mechanics game-friendly by simplification and cartoonification. A total gravity simulation across the entire playing area would probably both be too hard to play and also somewhat of a nightmare to implement efficiently. Not to mention the issue that it would limit the level design. Instead,gravity fields have been disconnected from any mass causing them. They are just bubbles that pull things to the center, with a clear outer boundary beyond which nothing happens. We won’t train future space pilots on Angry Birds Space, but it still makes it possible to do quite a few things that look really cool and that feel like you are trying to send a probe to Saturn by taking a sling shot path around Mars…

For example, flying around a planet a few times to hit a pig:

Or, pushing a bird through one gravity field to sling itself into the next:

The use of space also opens up for level design where you fire birds “backwards”to get at the pigs on the other side of the planet:

I managed to get some pigs stuck in a “lagrangian point” between two planets:

On some levels, there is no gravity at all, and you are rather looking for chain-reactions between blocks floating around in space. Unfortunately, space is not space, but rather some kind of viscuous fluid… blocks slow down and stop after a while, in a decidedly non-spacey way. guess that is also to facilitate level design, but I would thought it way cooler if chain reactions could have gone on for as long there was momentum left in any block.

he game theming with sound and graphics is very classic sci-fi, and the soundtrack is actually very amusing (not that I care much about it, since I usually play games with no sound on, waiting for the kids to fall asleep). There are a quite a few special levels (“Golden Eggstroids”) in the game, and the ones I have seen so far have all been takes on classic 1970′s and 1980′s computer games. For example, here is something a bit inspired by Bubble Bobble:

Overall, a brilliant game even if some things are not quite how I would have liked them.

However, for all of this goodness, the way the game is being sold really leaves a bad taste. The main game you buy costs 1 USD (or equivalent). This provides you with quite a few levels, and the levels are being expanded post-release in the great tradition of earlier Angry Birds games. I have been positively surprised by how generous Rovio has been in updating Angry Birds and Angry Birds Seasons over time. But now it seems that someone has decided it is time to get some more cash out of gamers, in ways that I really do not like.

With the “Danger Zone”, we have a whole set of levels that you have to buy in addition to the base game. These are also “in-app” purchases, which means that they only apply to one device. If you have multiple devices (like we do), you have to buy this on each individually. The Mighty Eagle has also become a limited-use purchase, rather than its old unlimited use variant:

Twenty bucks for eagles? That is silly!

I am not saying that the people at Rovio do not deserve their money – they have given me and the world countless hours of entertainment at a very low price – but I would much rather simply pay some five dollars up-front and have all game features unlocked forever, rather than being nickled-and-dimed over time. This subscription or addtional-content model is all the rage in gaming today from what I understand (I have no time to play PC-based games), and it just seems like a way to trick users into parting with more money by doing it in smaller chunks. Totally legal, but it still feels like a bad practice. But then again, it might just be me not liking this particular aspect of the modern economy.

The most interesting part of this is how a simulator can provide an easy way to get information out of target software, without all the software and driver overhead associated with doing the same on a real target. In this case, all that is needed is a single memory-mapped location that can written to be software – which can be put into user-mode-accessible locations if necessary.

Some hardware actually exposes something similar, with hardware trace ports that is part of the on-chip debug blocks like ARM’s CoreSight. However, in hardware, that still requires some form of hardware connection to the chip to be available to get the data to where it can be processed. A key advantage of a simulator is that in the most extreme cases, we could simulate a computer system put into an inaccessible location (like inside a aero engine) along with its physical environment (very hot gases and lots of hot moving parts) – and do code coverage on the software in a way that just could be done in hardware due to the total absence of communications path to the physical hardware.

It is also an interesting personal thing for me, as that simple device we used for this is the only piece of simulated hardware in the Simics distribution that I created. Everything else I do is bolted on to the outside as demos, and demos don’t typically ship as part of the product.

Make no mistake. The big long-term trend is still towards more layers. Higher-level programming languages, operating systems, libraries and middleware, dynamic languages, interpreted languages, and virtual-machine-based languages all add software layers to provide more powerful abstractions and increased programmer productivity. Dynamic and virtualized systems aim to divorce even operating systems from the hardware and make processors an abstraction rather than bedrock. This happens in all areas of computing, including embedded systems.

However, there are some areas of computing where that bucks this trend.  When maximum performance and minimum latency really matters, it seems that software architects and programmers realize that they have to remove or at least get around most of the abstraction layers. Real control is sometimes really necessary. APIs are de-abstracted to something quite close to the hardware.

One particular area where this is happening is in networking – with the advent of 10 gigabit per second (10G) Ethernet and higher speeds, software has to be brutally efficient to produce and consume packets at the rates that the network can deliver them. Network speeds have been increasing faster than processor speeds in recent years, which affects not only the core infrastructure of the Internet, but also regular application programs that need to take advantage of high data rates.

I have recently seen quite a few examples of this.

Luigi Rizzo wrote a nice and accessible article on the subject in the March 2012 issue of Communications of the ACM, called “Revisiting Network I/O APIs: The Netmap Framework“. In it, he notes that at 10 Gbps Ethernet, maximum packet rates are close to 15 million packets per second. 15 million events per seconds leaves a typical 2 to 3 gigahertz general-purpose processor with only a few hundred cycles to handle each event. At such data rates, the driver and network stack overhead in a typical OS is many times bigger than the time to process a single packet. Basically, the old sockets API and its layered implementation just cannot keep up. There is too much abstraction in the way. The solution proposed is the netmap API, which basically uses the abstraction of raw packets and work on a set of abstractions very close to how modern Ethernet controllers are implemented. Netmap aims to process batches of packets, not just individual packets, and to avoid all copying costs by directly putting received data packets into user space. Netmap also makes use of multicore hardware, having a single network interface map different hardware queues to different host cores (this is a standard function available in many Ethernet controllers since some years back).

The way that netmap provides software with very direct access to hardware is reminiscent of other APIs that are used in industry. Freescale has an API called Netcomm, for accelerating network code on their QorIQ and PowerQUICC SoCs. Intel has the DPDK (Data Plane Development Kit) API, which is used with quite a few different Intel platforms and Ethernet cards. Both these APIs let programmers use the hardware acceleration functions, in a way that does not rely on standard OS APIs for networking. The APIs are also necessary to access other kinds of hardware acceleration, like encryption, for which there is no standard API available at all. I have not worked extensively with either API, but the overall pattern is clear.  You cut out the OS drivers, go directly to the hardware, and rely on polling rather than interrupts to synchronize software and hardware.  Network interfaces (or queues in an individual interface) are tied to particular processor cores in multicore processors to reduce latencies and guarantee throughput.

This makes perfect sense.  I did some experiments a few years ago with device drivers in Linux that makes me believe that OS overhead can be very significant (blog post). In these experiments with a simple hardware accelerator, I finally connected my user-level software directly to the hardware (using the mmap() function in Linux).  I also switched from being interrupt-driven to polled, as that provided a much more efficient way to use really fast hardware. The details are captured in a Wind River whitepaper.

Yet another example of this kind of static allocation of work to cores and short paths from software to hardware is found in the Microsoft Sora software-defined-radio project. In Sora, they even mange to convince Windows XP to provide short latencies and tasks bound to cores – showing that it seems possible to retrofit real-time behavior to almost any OS as long as you have a few cores to spare.

Overall, an interesting trend. It is clear that not all software requires very short latency or the ability to handle millions of events per second. But is it equally clear that when this is the case, reducing the number of abstraction layers is the way to go. There are recurring attempts to find some way to build an abstraction that provides this kind of control on top of or inside existing many-layered systems, but for some reason I just like this reduction to the simplest possible. It just seems simpler.

I have a blog post about this up on the Wind River tools blog. Pretty amazing to see us model an eight bit machine in 2012 – just proves how long-lived some hardware systems are.

When reading the slides I really wish I had been there, this must have been a blast live. Still, there is an interview video posted online taken at the event (it failed to play properly for me).

Two things stand out in the talk:

• The poetic notion that a 1000-core processor is like discovering a new space. It is something new, not just more of the same.
• Synchronization is bad.

I think this second point is why the talk is titled “everything you know is wrong”. When teaching parallel programming, synchronization is a core part of the subject. When designing parallel languages and systems, the hardest problem is how to handle synchronization (or hide it so that it does not have to be explicitly handled).

Instead of finding better ways to synchronize, David Ungar makes the point that we need to find ways to eliminate synchronization altogether. Any synchronization will sooner or later become a bottleneck. I see where he is coming from. A talk I heard in 2009 made much the same point:

They also had the opportunity to test their solution [for parallel Erlang] on a Tilera 64-core machines. This mercilessly exposed any scalability limitations in their system, and proved the conventional wisdom that going beyond 10+ cores is quite different from scaling from 1 to 8… The two key lessons they learned was that no shared lock goes unpunished, and data has to be distributed as well as code.

In the same vein, it seems that all big system parallelization efforts turn into a hunt for locks and the splitting up of code and data into units that can run in parallel without having to synchronize. The “upper limit” of this process is clearly a system with no synchronization points at all.

He makes the valid point that even “lock-free” algorithms depend on synchronization, albeit a simple form of it. They do want a single atomic instruction – and that means that the hardware has to do some kind of synchronization to guarantee this atomicity across cores sharing memory (at least at the hardware level). I had not really considered that before, but it is true.

I would add that message-passing shared-nothing systems like Erlang are not exempt either – sending a message is a synchronizing operation, and it is quite possible to build synchronized systems in Erlang that do not scale either. It is all about communication vs computation, and making sure each thread can do as much work as possible without having to stop and wait for other threads.

Dropping synchronization as a concept makes the system non-deterministic, and that is the part that makes this hard to do. Most programmers do not like the idea that their programs do not have defined results, but there is a fundamental conflict here. Synchronized or high-performance, choose one. I have been working alongside very smart programmers for a long time trying to build parallel simulation systems. As long as there is a requirement for determinism and precise repeatability, performance from parallelism is very hard to obtain. Relax determinism, and performance is much easier to achieve. But maybe the entire value of the program is lost in the process?

I am reminded of an old Dilbert cartoon, where Wally asks the boss “Is it OK to do things wrong if we are really really fast?” Maybe it is, in the world of a thousand cores…

Ungar’s core idea seems to be to move towards a more approximate or statistical idea of program correctness. That makes sense, and is applicable in quite a few domains.

In a case like web searching, is there really a “right” result? For a human, all you really want is a useful result. Behind this is the idea of eventually consistent databases: it does not hurt if data is not quite the same in all places in a system, as long as updated data eventually reaches all parts of it. Better to available and fast and produce a useful result, than to be perfectly right but only be able to serve a portion of the user base.

In “soft” real-time systems we accept that we will occasionally miss a deadline, drop a phone call, have a bump in the flight of airliner, skip a frame in a movie playback, etc – as long as the results are not fatal but just an inconvenience. We rely on humans being forgiving, adaptable, and able to cope with the occasional problem. Traditionally, computers are not like that, in a binary a world a single error tends to be absolute.

Obviously, there are systems that just have to be right – I would not like control systems for aircraft to occasionally hiccup, nor my bank to play roulette with my money on a massively parallel machine… and the same goes for basic data structures. David Ungar brings up the example of a parallel hash table, and compares a few different implementations that all actually strive to give a correct results. The actual order of things in lists would seem to be different depending on how races play out, but with checking and patching making sure that the results are all correct (containing all data in the right buckets with no duplicates). That is familiar territory in parallel programming – correct final result, different way to get there (in terms of the exact order in which threads insert values into the table).

The techniques used in the hash table example remind me of the old idea of optimistic concurrency control – make a change and then check if it was invalidated, rather than checking before you make the change. This is benchmarked against compare-and-swap atomic instructions, and the performance trade-off is not clear. Even very slight synchronization will hurt performance, even though patching things up after a mistake has been done should use more cycles for retries and redoing work.

Overall, I think the core point David Ungar really makes is our thinking about right or wrong needs to go from a binary concept to shades-of-grey. In the future, programmers will not create programs that are fast but wrong, but rather programs which are sufficiently correct and still fast.

How to debug such a thing is still a bit of mystery though…

Ticket to Ride

Chasing high scores is fairly typical for computer games – playing against human beings you are motivated to win, even if you win by scoring a measly 75 points… while against the computer it becomes about beating your own old scores. Unfortunately, this also turns repetitive after a while, due to some small design flaws that really should be easy to fix.

What I have found is that I believe there is an optimum high-score strategy in the game, due to the very uneven spread of tickets and a very bad computer player.

My way to play it to get a high score (not necessarily a highscore, though) is to essentially build the exact same network each time, and then stop and pull as many tickets as a I can that fits into this network. When lucky, 15 tickets can fit inside and none miss, and the score will be 250 or more. If not lucky, some tickets will miss, and the score will be lower.

The reason this style of play works is that the computer player is blind to what you do and does not try to stop you from connecting up obvious points. In a real game, if you see someone starting to build a route that looks familiar, you will tend to play a blocking move. In the iPod version, you can have two 20-car segments separated by a single two-segment track… and the computer just ignores it and keeps plugging away at its own goal. Also, when playing with humans, you cannot just say “aw, I got some godawful starting tickets, let’s all start over with something better”. With just yourself and the computer, that’s very natural.

The critical network is this:

Tracks in green are definitely on the critical network, while the yellow parts depend on exactly which tickets show up. Sometimes, Boston is included, sometimes not. The red circles mark cities which are never on any tickets, and which thus are useless.

The connect the corners strategy is good even in the normal real-world Ticket to Ride – but with observant human opponents, very hard to pull of. Also, it seems to me that the Tickets in place in the iPod version is different from what’s in the original board game, and they are bit less balanced.

What I Days of Wonder should do is to copy Angry Birds and just release some free updates containing a ton of new tickets. If there were more tickets connecting from mid-land (Helena, Salt Lake City, Las Vegas) to the unused cities on the east coast, if would be much more variable. Also, the computer components could well be made to be a bit more aggressive and blocking.

In the iPod version of Carcassonne, the computer styles of play vary much more between different personalities, and some of them are very good at playing offensively and blocking you. A similar development could make Ticket to Ride for iPod a bit more exciting to play. As it is right now, it feels like the game has been conquered and won and is pretty pointless to play. The chance of grabbing a higher score is rapidly diminishing, as the random luck needed to get the just right set of tickets with the computer getting a non-conflicting set and being fairly slow in building feels very small. It will happen if you just play many times enough – but where is the fun in that?

2003. The embedded tools company Green Hills launched their
Time Machine feature in their well-known MULTI debugger. I consider this the start of commercial reverse debugging, as it was the first
commercial-grade product to include reverse debugging. The implementation was based on tracing the execution of a program on actual hardware, using a debug probe and a “JTAG” debug interface. The trace box would capture several gigabytes of execution data, and then the debugger performed operations based on this trace. To check a backwards breakpoint, you scan back over the trace until you find a matching state or operation (such a memory access or instruction address that is being executed). The main limitation of the method is that the trace buffer can only capture a few seconds of execution on a typical 100s of MHz embedded processor. It only works for a single processor, and it does not capture IO actions (except as memory-mapped IO). It is system-level and cross-target.

Extending this kind of trace to multicore has proven hard, since getting a synchronized trace out of several processors is very hard. There might be debug hardware coming out in the next few years that can indeed support a time-stamped consistent trace of multiple cores, and with such hardware, the Time Machine approach could well be extended into multicore.

2005. Simics 3.0 was launched by Virtutech (later acquired by Wind River and Intel) with full-system reverse execution and reverse debugging. The Simics approach was also unique, being based on a full-system simulator. By simulating the entire target, it is trivial to reverse (and put reverse breakpoints on) changes to memory, persistent storage like disks, and hardware devices. Since all device models in the simulator are deterministic in their implementation, re-executing hardware events like interrupts and IO outputs is just as easy as re-executing code on the main processor, something that had eluded all previous approaches. Recording is used at the interface between the simulator and the outside world, such as user interaction over graphics displays and serial ports and connections to the real-world network. The software stack is unmodified and system-level, and the simulator can handle multiple processors and even multiple machines in a network as a unit. The use case is normally cross-target (even if a system identical to the host can be simulated, it would work like a cross target logically). Time is handled by counting clock cycles on all processors in the system, and reverse debugging can position the simulation at any point in time based on the virtual time.

There is a cost in execution speed from simulation rather than direct execution, and an intrusion effect from running on a simulator rather than on a physical machine. This affects the timing of events, even with a software stack that is not modified. Still, the fact that you can run a complete real software stack with no modifications needed before starting to run the target system is fairly rare in the world of reverse debuggers.

Simics shipped with a modified gdb that talked gdb serial to Simics and accessed reverse execution with some new debugger commands as well as extensions to the gdb serial protocol. This was offered to the gdb community, but not accepted. However, prompted by this, the gdb community started to discuss reverse execution. Some interesting old threads can still be found, such as http://sourceware.org/ml/gdb/2005-05/msg00225.html. Clearly, at that point in time Virtutech did not really explain how Simics worked, and there were some pretty bad proposals floated in the community for how to do reverse. In the end, the gdb serial design did turn out in the right way, assuming
the remote debugger would reverse itself and gdb would just ask it to do so. This separation of concerns is important to creating practical reverse debugging solutions that can use any debugger backend.

2005. Also in 2005, Lauterbach launched the Context Tracking System, CTS. Lauterbach is a big player in the embedded debug market, with their TRACE32 debugger. CTS can be seen as their reply to the Time Machine debugger. CTS is also based on a trace from a hardware unit or from an instruction-set simulator. However, from the available information is also appears to be more limited – you can step back and go back in time and replay forward, but there is no mention of actual backwards breakpoints (even today, six years later).  Thus, I count this as record-replay rather than reverse debug. It is cross-target, system-leve, and uniprocessor like Time Machine.

2006. Undo Software launched the first Linux-targeting host-based reverse debugger, UndoDB. It is described as a bidirectional debugger (the same terminology as the Boothe 2000 PLDI paper). It is user-level, does do reverse breakpoints (and data breakpoints, also known as watchpoints, which is really useful). It handles multiple threads (at least in 3.0), but from the description of the recording technology used I believe they have to serialize their execution. The implementation is based on checkpoint and re-execution, with recording of all non-deterministic events like IO. There is a feature to move to a certain point in time, based on “simulated nanoseconds”. These are not really nanoseconds, but values which are guaranteed to increase even between two instructions (which probably means that they are sub-nanoseconds and on a > 1GHz CPU single-cycle instructions will indeed take less than one nanosecond).

There is a nice description of how it works on their online man page. It is worth noting that they call it “gdb”, but the command set is distinct from what gdb introduced with its reverse execution in 2009. They use the “b” prefix for backwards commands rather than “r” for reverse.  In some way, UndoDB is in direct competition with the gdb reverse target, but it is much much faster and has more features.

2008. The Rogue Wave (at the time, it was an independent company) TotalView debugger gained support for reverse debugging, with the ReplayEngine add-on. TotalView is an old mainstay in the HPC market, having been around since at least 1993. Indeed, it was developed initially for the BBN Butterfly computer, and thus it might have had a touch with reverse execution as far back as the 1987 paper cited in my previous blog post.

Judging from the available materials, TotalView can clearly can step back in various ways. However, it is not clear that it triggers breakpoints when going backwards. Thus, it has to count as record-replay debugging rather than reverse debugging. The base of the implementation is extensive instrumentation of the the runtime system of the target computer.  The implementation builds on the fact that the target programs tend to b clustered programs that use MPI to communicate – and thus a large part of the communication between threads is explicit and easily intercepted and recorded.  There is also an existing infrastructure of checkpoint and restart for parallel programs using MPI to support fault tolerance that was used as the base of the implementation.  Finally, in a slightly ugly hack, they make each multi-threaded program run on a single processor by a big lock. In this way, all that needs to be replayed is the interleaving of threads on a single processor, a far more tractable problem compared to trying to replicate a true parallel execution in a new session.

2008. VmWare officially launched a record-replay debugger based on their virtual machine technology with VmWare Workstation 6.5. Single-processor, system-level (but really only supported for user-level debugging), cross target (since the VM is not really the absolutely same hardware as the host), time model is based on the virtual machine which I believe is cycles-based. Mostly used for record-replay debug of non-deterministic software bugs, but could also do reverse debugging including reverse data breakpoints. Based on snapshot and deterministic re-execution, plus recording of all non-deterministic device accesses (not all devices in the VmWare hardware emulation layer are deterministic). Going back to a snapshot was a very heavy operation (I tried it) since you had to restore the entire target memory (quickly got into gigabytes). The hardware supported in the VM was quite limited, and things like CD-ROMs and floppies could not be part of a record/replay session. Replay logs could be moved between hosts.

The VmWare reverse debug functionality was removed from VmWare workstation version 8 in 2011, since it required a large investment and was not apparently used by very many VmWare users. This indicates that trying to build developer-oriented functionality into a technology base that is fundamentally driven by the need of deployed virtual machines was hard. There are contradictions between these two goals, as the determinism and control needed for a good reverse debugger is not necessarily consistent with maximum performance for running virtual machines in a production setting.

2009. gdb 7.0 added support for reverse execution (a work that began in 2006). The built-in “record” target supports reverse debugging on user-level single-threaded programs on the same host. The command set for reverse debugging is fairly full-featured, but is a bit quirky with a “set direction” command that makes regular run-control commands work in reverse. The record technology is quite slow since it basically records the effect of each and every instruction run in the program.

In addition to its built-in target, gdb can also control external reversible debug systems over the gdb serial protocol. This made the changes to gdb-serial created by Virtutech for Simics in 2005 part of the mainline gdb release. Several tools support the command set, including VmWare, UndoDB, and Simics. There was also a set of MI commands added to basically let Eclipse use gdb as a backend for reverse debug, including using it to control external tools via gdb-serial. How this happened is quite a long story, and I made a small contribution to the gdb code base myself in the process. Read about this here.

2009. Eclipse CDT added support for reverse execution, using gdb 7.0 reverse as the initial backend. As noted above, this lets Eclipse also use other reverse debugging backends (Eclipse uses the gdb-MI interface to gdb to control the debug session). This is noteworthy since it meant that the buttons to control reverse execution are now part of the CDT, making it much easier to use Eclipse to build a frontend to any reversible backend. Eclipse is not really a debugger, just an interface to a debugger.

2009. Microsoft Visual Studio got record-replay debugging with IntelliTrace. It is strictly about replay debugging, including the nice ability to send traces around between developers. There are no backwards breakpoints. The support is limited to programs running on top of the .net runtime system, meaning that it does not apply to classic Windows software. Using the CLR virtual machine as the implementation basis should make the implementation easier, cleaner, and faster compared to a machine-level native solution. It is user-level, single-threaded, and host-based. Time concept is unknown.

2011. Adobe demonstrated (not launched) reverse debugging in their Flash Builder programming environment. A nice video is posted on the Adobe website. Seems to be based on the virtual machine that flash runs on, and includes what looks like pretty powerful backwards data analysis tools. In a blog post, the developer describes some of the features, which to me seem to indicate some pretty heavy recording.

Final notes.In researching these commercial tools, there also seems to be a lost one. A company called Visicomp launched a Java debugger called RetroVue in 2002 which supposedly did allow backwards debugging in some way. However, it seems that this tool was not really practical, being too slow for actual use. It seems to have disappeared since without anyone picking up its legacy. The technology was apparently pretty much like the Omniscient Debugger presented in 2003 and which I described in the blog post on reverse execution research.

When searching through the literature using the ACM Digital Library (and some Yahoo and Google web searches), I find quite a large body of research literature dating to the late 2000s. Apparently, the appearance of commercial reverse debuggers have inspired research into the topic – and our increasingly powerful machines have made heavier solutions feasible (at least for single programs).

Disclaimer – this is not really an exhaustive survey of the field, but some of the more interesting papers that I have stumbled on.

1973. The first actual mention of reverse debugging I have found is a short 1973 Communications of the ACM article called “Reversible Execution” by Marvin Zelkowitz. Basically, this paper proposed to apply a backtracking facility built into a PL/I compiler to support AI research (a precursor to Prolog execution semantics from what it sounds like) to also replay parts of a normal program for debugging. This in essence would enable replay of a program execution, based on a log of how the program ran and the values of variables changed. Basically, a forward-only record-replay solution for user-level single-threaded programs.

1987. Thomas LeBlanc and John Mellor-Crummey, “Debugging Parallel Programs with Instant Replay“, IEEE Transactions on Computers. This paper covers record-replay debugging of parallel user-level programs on the BBN Butterfly. They record all interaction between processes (threads) by intrumenting the OS calls that are used to communicate and synchronize between processes. Programs also sometimes had to be changed so that their communication protocols allowed the instrumentation to work properly. Thus, the approach did not apply to unchanged code either at source or binary level. Processes still compute their results, only the order or interactions are enforced. In this way, the overhead was kept usefully low. The approach does not work well for programs that deal with asynchronous interrupts and hardware interaction by the software, since there is no good way to hook and replay such events in their implementation.

The most important concept here is the use of re-execution of code to reconstruct values of a program, assuming deterministic computation as long as asynchronous interactions can be controlled. This is the basis of all reverse execution approaches based on checkpointing and re-execution. Not sure if they invented the idea, but this is the earliest distinct mention I have seen of this.

1988. Stuart Feldman and Channing Brown, “IGOR: a system for program debug via reversible execution“, 1988 ACM SIGPLAN and SIGOPS workshop on Parallel and distributed debugging (PADD) . This paper looks at the problem of how to debug user-level single-threaded code on a UNIX-like system. The programs need to be recompiled to add a small amount of support, in particular to help with restarting
execution. It is essentially a record-replay approach, but capable of placing the execution of a program at any point in its past execution. They save regular checkpoints during program execution (every tenth of a second or so seemed appropriate in their testing). The checkpoints only include the data that has been changed since the previous checkpoint (they added an OS facility to mark all pages that a program writes between checkpoints). A unique feature is the ability to look at the value of a variable as it is stored in successive checkpoints, and to “go to the point in time where variable X has value V” (assuming a variable is monotonically increasing or decreasing, like an outermost loop counter). To do the detailed positioning, they use an interpreter for the target system, essentially mixing native and interpreted execution in their debugger.

1996. Mark Russinovich (who I believe is the man behind the recent novel “Zero Day“) and Bryce Cogswell, “Replay for Concurrent Non-Deterministic Shared-Memory Applications“, PLDI 1996. In this paper, instrumented system libraries and binary code modification is used to enable deterministic replay of multithreaded applications. The binary modification is used to implement an instruction counter to provide a logical time-base, and instrumented system libraries are used to force a serialized (but interleaved) execution of the program. It does not run the program in parallel on a multiprocessor, but rather runs it on  a single processor. User-level, instrumented and intrusive, parallel on single processor.

2000. Bob Boothe, “Efficient Algorithms for Bidirectional Debugging“, PLDI 2000. I actually saw this paper being presented live at the PLDI conference in Vancouver. At the time, I was working on embedded compilers and debuggers, and our little group from IAR looked at what was being presented at said “yeah, might work on a big desktop, but you can’t do that in a real machine”. Why was that the case? Essentially, what Boothe did was to use the Unix fork call to spawn off checkpoints of a program as it was executing. This was a smart way to achieve checkpointing using existing OS facilities. In addition, he recorded the results of system calls to replay them during replay.

Time was given by counters in target program (compiled-into it as part of the debugging build). Most of the paper was spent on basic algorithms for how to do “previous” (step back 1 line), “before” (inverse of finish), “buntil” (backwards until, looking at data conditionals to determine when to stop). His use of “backwards” rather than “reverse” as the name of the concept does make this paper a bit harder to find. While limited to a single threaded user-level process, this paper is probably the first to introduce reverse debugging in the sense of the primary user interface being the ability to step backwards and check for breakpoints backwards in time. A good idea in the UI design was the undo command that jumped the debugger back to the last position where you stopped. Having worked with reverse debugging quite a lot myself, this operation makes a lot of sense as an addition to standard run-control commands.

2002. Tankgut Akgul and Vincent Mooney, “Instruction-level Reverse Execution for Debugging“, at PASTE 2002. This paper is an odd one, as it was neither really reverse nor practical. The approach is based on doing a static analysis of the assembly code of a program to generate code that could reverse operations based on some logging of destructive operations. Interesting, but not really practical as I see it. One noteworthy tidbit in the paper is the measurement that logging only destructive operations is some 50 x more efficient than logging all changes.

2002. George Dunlap, Samuel King, Sukru Cinar, Murasa Bazrai, and Peter Chen, “ReVirt: enabling intrusion analysis through virtual-machine logging and replay“, at OSDI 2002. This paper uses the User-Mode Linux (UML) paravirtual machine to checkpoint and replay an operating system. I think this is the first use of a true virtual machine to achieve reverse execution, even though it is a paravirtual machine rather than a full virtual machine. The replay starts from a single checkpoint at the start of execution of the guest system, so moving to a certain point in time can be very time-consuming. It does support reverse breakpoints. To move to a certain point in time, they count executed instructions, not actual time. The use of a paravirtual solution precludes the use of many real device drivers and debugging issues related to interaction with most real devices. It is uniprocessor, system-level, unmodified target (except the change to make the OS itself run paravirtualized on top of UML).

2003. Bil Lewis and Mireille Ducasse, “Using events to debug Java programs backwards in time“, OOPSLA 2003. This paper targets programs running on top of a Java Virtual Machine, resulting in an Omniscient Debugger. They instrument the target program to log all state changes. A big lock is used to serialize all threads into a single interleaved execution. The implementation is admitted to be almost worst-case inefficient, but the debugger was still considered useful for some real work. The debug UI does feature backwards breakpoints, so it qualifies as actual reverse debugging.

Note that in principle, backwards debugging and record-replay should be possible to implement quite efficiently for a language virtual machine like the JVM, if you can modify the VM itself. This has indeed been done today, for example by Microsoft for their CLR and in the use of full-system simulators (and other simulators) to implement reverse debugging.

2005. Samuel King, George Dunlap, and Peter Chen, “Debugging Operating Systems with Time-Traveling Virtual Machines“, USENIX 2005. This paper builds on the foundation of ReVirt, but adds checkpoints during a run to make the replay to a certain point in time more efficient. They also introduce the differential storing of disk state in the checkpointing (just like done in Simics). Overall, it is very similar to the replay approach taken by VmWare a few years later. Still single-processor. The paper contains a nice list of example bugs for which reverse debugging work much better than cyclic debugging. They also reference Simics as something that might do something with reverse, but do not quite know what (this is right after the launch of Simics reverse execution as discussed in the next post).

2007. Paul Brook and Daniel Jacobowitz, “Reversible Debugging“, GCC Developer’s Summit 2007. This paper presented some early work on implementing reverse debugging inside the Qemu full-system simulator. Logically, it follows in the steps of the Simics reversible debugger announced in 2005 (see the third post in this series of blogs). They use gdb as their frontend, and the paper introduced the idea of setting the default execution direction of the debugger. This UI choice was later adopted in the reversible gdb 7.0 released in 2009. The paper contains a good discussion of some of the practical issues involved in performing operations like reverse-step to go back a line, or stepping back up through a function call. It is a good explanation of the basic problems.

The actual reversing is implemented in Qemu using a checkpoint and deterministic replay, along with logging of any operation that is not deterministic. Their main test case is actually using the semihosting variant of Qemu where IO calls are intercepted and passed on to the host. Just like all other such solutions, they record the return values and do not repeat side-effects. It is system-level, single processor, unmodified target software, and time is based on counting executed instructions.

The Qemu architecture makes enforcing reversibility across all devices fairly difficult, since each device manages its own interaction with the user. One interesting tidbit in the paper is that Qemu by default triggers timer interrupts based on host time, but these had to be made fully virtual to be reversible.

Summary The above is a small selection from all the papers published on this topic, but I think they capture the most common methods used. Basically, almost all of the approaches require some kind of change to the target software to enable reversibility. Interestingly, the first few commercial solutions for reverse debug that appeared did not take that approach, but rather targeted unmodified programs using a different set of fundamental techniques. See the next blog post for that part of reverse history.

Let’s start with background and definitions.

To me, the key defining factor of reverse debugging is the ability to apply breakpoints in reverse – essentially, to be able to go to the previous occurence of a breakpoint just as well as going to the next. The implementation allowing this might vary.

The contrast to reverse debugging is classic cyclic debugging where you run and rerun a program with a problem, looking at variables and setting breakpoints during each run to zoom in on an issue. For cyclic debugging to work, you pretty much require each run to behave the same way. The program has to be fundamentally deterministic. This is
usually the case for non-interactive single-threaded programs, but not the case for real-time programs, parallel programs, or programs that involve some kind of asynchronous input/output (reading and writing files is a special case of IO that can indeed be deterministic since there is usually no interference from the environment in those operations).

Thus, for non-deterministic programs and rare errors, reverse debug is what you want. Run once, hit the error, reverse to diagnose it.

In addition to reverse debug and cyclic debug, we also have record-replay debug. In such a system, you record an execution and later replay it forward. Debugging is strictly forward: you cannot step back in time instruction by instruction, nor trigger breakpoints backwards in time. Record-replay debug is a way to allow cyclic debugging on non-deterministic highly variable program runs. This can be implemented with a lot less debugger complexity than a true reverse debugger, even if the underlying runtime system is almost identical to reverse debugging.

So, how do you implement reverse debugging?

Reverse execution is one way to implement reverse debug. In reverse execution, the execution system has the ability to move backwards in time and put the entire system in the state it was in at some previous point in time.  You can step the system back instruction by instruction or cycle by cycle.  You can also continue the execution
forward from any point in time, throwing away the history of asynchronous inputs to actually take a new execution path.

Typically, reverse execution is implemented by using checkpoints of previous system states plus a deterministic way to re-execute the system from those checkpoints. The key technical problems to be solved is how to checkpoint the system and how to make re-execution deterministic (note that determinism does not imply invariant or predictable behavior, just that you can reliable recreate a particular execution). You also need to record and replay anything that you cannot re-execute, such as user interactions or network communications with the world outside the controlled system.

Schematically, it works like this, note how simulation time always moves forward even though the logical system time sometimes moves backwards:

Record-replay debug is usually implemented in a way similar to this, without the ability to go back in time and usually without the intermediate checkpoints.

Trace-based debugging is based on recording everything a system does into a trace, and once the recording is done, debugger works on the trace rather than using the log to drive an actual system.  Reverse debug is implemented by recreating the state of a system by reading the trace, and finding points in the trace where breakpoint conditions are true. This is also known as post-mortem debug, since you debug after the target system has finished executing (typically). A log can be captured by a hardware device, or by software being instrumented to log everything that is going on. The technology can also be used to implement record-replay debugging.

There are some other important distinctions between different approaches to reverse debugging that we need to keep in mind as we review the history of the field.

• System or user-level? Do you debug a system, including the OS, or just a user-level application running on top of the operating system? User-level debug can often be solved in simpler ways than system-level debug, but also suffers from some limitations. There are also times where system-level is simpler.
• Single-threaded or multi-threaded? Can you debug multiple threads, processes, processors, or just a single processor or user-level thread? A single thread of control greatly simplifies the problem.
• Cross-target or host-based? Do you debug programs running on the same host  as the debugger, or can it also target remote targets or cross targets (such as embedded systems)?
• Instrumented or unchanged programs? Quite a few solutions for reverse debug tried over the years involve compiling programs in a special way, using instrumented OS libraries, or other implementation variants where the program debugged is not identical to the eventual deployed program.

Given this background, the next post will cover early research, and the third post the beginning of commercial products.

Note that on Stack Overflow, reverse debug does not seem to be particularly big thing still. The reverse-debugging tag has all of 10 members, while other topics like C# and Android programming has millions… so maybe reverse is not quite mainstream yet. Or maybe it is just the case that Stack Overflow has more web-style developers than low-level developers in their membership.

Essentially, as the (embedded) software systems we are developing get more and more complex, we need to update our debuggers to keep up with the times. This means moving beyond looking at single processors, threads, programs, or even nodes – and making debuggers that support working with multiple contexts at once. For example, debugging a couple of different boards in a rack from a single debug session. This would mean supporting multiple processor architectures, execution modes, and operating systems inside a single debugger session. Debuggers need to orient themselves towards handling software that is already running in a target, rather than starting the target software. The debugger becomes more of a passive observer of the target system than an active component.

Take a look!

I have been thinking about this subject lately, after using a BlackBerry Torch 9800 as my work phone for a few months.  I like the device a lot, but there are certainly some rough edges and some places where there is a UI conflict between touching the screen and pressing the buttons. At the same time, I am using both an iPod Nano 3G, and a couple of iPod Touches. I used to have SonyEricsson Symbian-based P900, P990i, and G900 smart phones which also were combined touch/press devices with a stylus.

I think it is clear that using a physical keyboard is preferable to an on-screen keyboard for typing serious amounts of text.  On the iPod, entering a URL or search term in a browser just feels fiddly, compared to the ease of composition with the physical keyboard on the
BlackBerry.  To me, I cannot get over the feeling that an iPod Touch or iPhone is really best as a consumption device, but that a BlackBerry is a superb creation device.  On a BB, I can type quite long emails, while an iOS device feels more appropriate for the occasional short messages.  To me, this is an important aspect, as I tend use the smart phone as a two-way email communications device.

The slide-out keyboard solution on the BB Torch (and many other phones from many different vendors) seems just right for this, giving you a full-screen device for reading tasks, with a full keyboard when needed.

On my old G900, I used T9 with a 0 to 9 keypad, and that worked surprisingly well.  A problem both with the BB and definitely with an on-screen keyboard is the fact that using it single-handedly is very hard.  On a screen, you do not get feedback from your fingers where you are on the keyboard, and the full keyboard of the BB is a bit too small to reliably use with a single hand.  I noted this in my G900 review three years ago, and it still holds true. The G900 looked like this:

Since a primary use of our iPods is gaming, I have noted some different UI principles for games.  There seem to be one class of games where touch makes perfect sense, like Angry Birds.  Press and draw to load, swipe to move the display, pinch to zoom – perfect and logical. Another class of games are obviously ports of mouse-based games, like Plants vs Zombies.  In such games, the touch screen is essentially used as it was back in stylus-time. It is just another way to generate single-point clicks.

Finally, we have the category of games that really would work best with a physical controller, like PacMan. Displaying a four-way control key setup on the screen does not get close to the right feeling.

I think the SonyEricsson Xperia Play was a brilliant idea – imagine an Apple device with that little built-in game controller.  Seems that SonyEricsson has not quite managed to execute on the idea, but in an ecosystem that generates variation like Android, this kind of device should have a place.

However, I have a hard time seeing the swiss-army-knife design of a phone having both a slide out keyboard and a slide-out game controller. Would be neat, but mechanically I don’t think it would work very well.

On to navigation.  The idea of touch gestures (in particular swipes) to navigate around the UI as popularized by Apple in the first iPhone is great in many ways.  My youngest child has used it since she was two, and it just works quite naturally.  However, there are still cases where a plain old navigation key works better. And having both is best.

Holding a device in one hand and scrolling a web page is faster on the navigation key on my BB than using touch on an iPod. The old G900 was even more efficient, just hold the down key – no physical motion needed at all. The iPod would be so much better if it could just have a scroll wheel on the side or something so you could use it one-handed
for reading without having to put a finger on top of the screen (and quite often accidentally clicking links in the process).

I also like the menu key on the BB (and on my old SonyEricsson phones), as a way to quickly get to the most important functions. This seems hard to emulate on a touch screen in a good way. In their touch-enabled phones, BB has tried it with a press-and-hold action – if you press down on the screen, a little action palette opens. Which does not quite work for me.

Another great advantage of some well-choosen physical buttons is that they offer immediate access to certain functions.  On a touch-screen device, I find that you have to dig through several steps to get into any function.  Home screens with most commonly accessed functions in all honor, but a direct key to dial a call or activate the camera is faster.  It cannot support all functions, obviously, but it lets you bring up the most common and important functions quickly. This is an aspect where my iPod Nano is much better than an iPod touch. On the iPod Nano, you can have it in a pocket and just hit the device to move to the next song or pause. On an iPod Touch, you have to look at the screen and find the right spot to hit.

On the other hand, the Nano would have been even better with a couple of buttons to increase/decrease volume rather than the scroll wheel. When out running, having to hold the things in both hands to make simple adjustments is really annoying. It also does not work with gloves on, which is a big problem with all touch device as well as touch-wheels – they require naked fingers, which is annoying in a climate where gloves are on for at least half the year. As I said before, someone should put a design center in Luleå or Novosibirsk or Alaska, and then see what kinds of devices come out.

A full keyboard like the BB also have the nice property of providing many points of access to shortcuts.  It might seem quite backwards of me, but I like the speed of action you get with the keyboard shortcuts in the BB email application. It has a lot in common with classic computer UIs, actually. It is hard to afford immediate access to 20+ functions on a touch screen without creating a very cluttered interface.  On the other hand, a touch interface done right makes it simple to access system-level things quickly by just touching the status icons on the screen – you get out of the application faster than if you had to use the keyboard to navigate up to some button.  For some reason, neither the iOS devices nor the BB does this right, while my old SonyEricsson smartphones all let me access detailed status by hitting the battery icon or network icons.

However, after singing the praise of the keyboard and touch screen in union, I must admit that there is a big downside to a combined touch/press device. UI design is harder when you have to both afford a good keyboard-controlled interface and a touch interface. The BB Torch I have suffers from this quite badly – having to afford both touch and non-touch interaction, as well as non-touch versions of the device itself does mean that the UI is not as streamlined and elegant as what you can get on a pure touch device. I often find myself jumping between keyboard and touch as I cannot quite complete what I want to do using just one or the other interaction mode, which really should not be necessary. Only having to think about a single type of input does seem to make things simpler for a UI designers, even if it also sometimes precludes creating truly efficient applications.

The Fujitsu Scientific and Technical Journal is the Fujitsu equivalent of IBM’s Journal of Research and Development. Thankfully, the FSTJ is still free while IBM erected a paywall around their articles. Number 2 of 2011 had the theme of servers, and there is an article about Quality Assurance for  Stable Server Operation by Masafumi Matsuo, Yuji Uchiyama, and Yuichi Kurita.

The article describes the process of ensuring that the final servers that are shipped to customers (from what seems to be the Sparc-based line of Fujitsu computers, even though it might actually apply equally to their mainframes and x86-based servers) are as stable as possible. Apart from designing things right, this also requires testing the fault handling and recovery operations.

I found it noteworthy that they do a lot of configuration testing, where various hardware and software configurations are played off against each other. In this way, corner cases are explored and coverage of the actual configurations that customers will be using becomes more likely (it is always dangerous to only test on one or a few configurations). They push memory system and processor loads to very high levels to ensure continued operation even in extreme cases, and also try to push the actual chips to make sure they will operate reliably in a wide range of environmental conditions. Indeed, a large focus is placed on pure physical reliability, as that is the basis for system reliability.

The best part, however, is on page four of the article, where they show the physical fault injection robot that is applied to the chips mounted on boards. This robot  shorts out individual pins on chips, clamping them to zero volts. It goes over all pins, and the test system checks what happens to the system in each case. Some kind of exhaustive testing going on here.

Neat. I have heard other stories of physical fault injection, including complex mechanisms like passing computer boards through irradiation chambers all the way to brutally simple tasks like putting an axe into a board to break it or pulling boards out of racks to simulate a sudden catastrophic failure. I would like to see more of just how these things are done in the real world. I suspect there are quite a few interesting robotics setups out there that do fault injection.

In any case, the article offered an interesting glimpse of many of the techniques used to make computer systems robust and reliable. Recommended.

It ends by noting that deeply consolidated SoC designs and aggressive dynamic power management are challenging from a testing and observation perspective, as well as creating more single points of failure. If a single system on chip fails, that system is all gone…

I have recently seen a number of examples of what Microsoft does with the user feedback data they collect from their massive installed base. I am not talking about Google-style personal information collection, but rather anonymous collection of user interface and error data in a way that is more designed to built better products than targeting ads.

The first paper is “Debugging in the (very) large: ten years of implementation and experience” by Kinshumann et al, Communications of the ACM, July 2011. This paper describes how Microsoft uses of the data they collect from Windows Error Reporting (you know, the little dialog boxes that appear every once in a while on Windows when a program has crashed or frozen, or Windows restored from a crash).

Microsoft has a number of heuristics that look at the data collected, grouping the bug reports into buckets. Ideally, each bucket corresponds to a single root cause for possibly quite different errors. They automatically analyze the errors and generate metadata about the error reports that can be used to generate statistics and allow database queries to be performed over all collected error
reports.  Heuristics include walking through chains of threads blocked on synchronization objects to determine which one is the actual cause of a hang, and finding the most likely thread and stackframe for containing the root cause of an error.  Heuristics are applied both on the client and the server, but mostly on the server. Technically very hard to do right, I can appreciate the huge amount of work that has gone into engineering this.

With this huge pile of information, a new debugging method becomes available: statistics-driven bug finding and prioritization at large scale.  The introduction to the paper puts it very well:

Beyond mere debugging from error reports, WER enables a new form of statistics-based debugging. WER gathers all error reports to a central database. In the large, programmers can mine the error report database to prioritize work, spot trends, and test hypotheses. Programmers use data from WER to prioritize debugging so that they fix the bugs that affect the most users, not just the bugs hit by the loudest customers. WER data also aids in correlating failures to co-located components. For example, WER can identify that a collection of seemingly unrelated crashes all contain the same likely culprit—say a device driver—even though its code was not running at the time of failure.

For a product manager like me, used to working with individual bug reports in bug reporting systems and trying to manually assess the importance of each error, this is nothing short of a dream.  Instead of trying to guess how many users can be impacted by a bug, Microsoft can run queries against the error report database and get a fairly accurate idea of how common a certain error is in the user base.  This has allowed them to address the most common errors first, leading to Windows and Office becoming more stable for more users in recent generations.  They can also pinpoint which device drivers are causing the most issues, and putting pressure on vendors to clean up their act.

I wonder where else you really apply this idea of statistical debugging. You need a large user base, in systems which are connected to the Internet so you can collect data, and who are comfortable with providing direct feedback to you as a vendor.  Apparently, Apple has the same kind of feature built into iOS, with more than 100 million users which seem not to be too interested in strong privacy.  Presumably, Google can do the same thing with Android, at least its use in phones. Mozilla has a crash reporter, so I guess it makes sense in the consumer space.

But when your user base counts in thousands of seats and half of these are in defense sector beyond air-gaps, it is harder to apply. Products that call home are not taken to kindly in the professional field, as secrecy and confidentiality is very important to big companies. Industrial embedded products like telecom infrastructure might have sufficient volume of code and computer hardware to form a basis for statistical reporting – as long as operators agree to provide the information to the hardware vendors.

Another example of how Microsoft makes use of their collected data is in UI design. The blog post “Improvements in Windows Explorer“, by Steven Sinofsky, from the Windows 8 blog discusses how Windows Explorer has evolved over the years, and how it is now getting a radical redesign based on usage data.  Microsoft is an enviable position here, having collected information about what millions of users are doing.  Definitely beats inspiration or trying with a few users in a classic user interface lab.

I have seen quite a few people criticize this blog post from a variety of angles – from the fact that they are not data-driven enough and keep rarely-used buttons in the ribbon to the fact that they remove somebody’s favorite function.  It is also the case that the measurements can only tell you which functions people are using from what is available today – if you want to invent new things, data like this might not be very helpful.

Fortunately, Microsoft also seems to have taken a clue from Linux and is allowing much more user customization than before. For me, this is great news, as I seem to have a user profile quite far from the mainstream.  We have not seen Windows 8 in its final form just yet, but hopefully this approach will be applied to other parts of that GUI overhaul too.  There are professional Windows users who need an OS that makes even very esoteric operations easy to access, and customizations of things like the start menu possible.  Hopefully, we do not get washed away by the flood of data from regular users.

For some reason, I feel that bug reporting is not as sensitive to the user style as GUI design – Windows and driver bugs would seem to be more evenly distributed as they depend more on hardware than on software. At least it seems to me that Windows is more stable today
than it was a couple of years ago.

As David Alan Grier would have said, this kind of event tends to serve to build the professional identity of a group of people. Computer science is not a profession per se, but it is clear that the Uppsala computer science students (almost 2000 has started since 1981) has a particular culture that comes back to us very easily when amongst our peers. I think it is based in the art and practice of programming.

The talks and discussions during the dinner often went back to the defining experiences of our student days, and these experiences were mostly about night hacks and classic labs. A speaker told us about how in 1983 they tried to program a simple computer built by the department itself, and how it turned out that 3 out 4 machines were broken. There idea that they expected the program to just work the first time it loaded made me look up my favorite debugging quote. Somebody reminded me (19 years after the deed) about the time I made a Prolog program exhaust all memory on a Sun server, by performing an exhaustive search for a problem with no solution).

I remembered how some students I taught (while still an undergraduate myself, a common practice at DV) spent 24 hours a day for almost a week in a lab room trying to get their operating systems to boot on some MIPS-based lab machines. In particular, the group that was gripped by ambition and tried to turn on the MMU. Each time they reset the machine and tried to boot their OS, they would see a serial terminal spewing out diagnostics text and then stopping cold… as they failed again to make it work (they passed the course anyway).

This all indicates the fundamental importance of programming to computer science students. That is also what we believed was our core mission when I was a student – to go out into the world and create great software. Many still do, even if quite a few of us have left day-to-day coding to become project leaders and outright managers. Can’t say I program all that much myself, apart from some demos and virtual machine scripts, but I still find the topic incredibly interesting and important.

The event also touched on institutional memory and the longevity of data. The very ambitious anniversary committee had produced a brand new version of the classic “Manualen”, a song book first produced in 1995. In it, I found a text I wrote in 1995 about what a computer scientist actually does (a bit pompous, as can be expected from a proud student) as well as a photo of myself from a 1996 cover of the DV student magazine “Blurgel”. However, in both cases, these had been reproduced from paper copies. There was no digital memory in place of these 15-year-old pieces of data.

I actually still have both paper copies in good shape and the data – on a multisession CD-R created in 1999 on a Mac. My current computers all being Windows-based cannot extract the data. I know a Mac can still read them, but it is not clear that the data can be used today. It is likely in some old version of Aldus PageMaker and with no file extensions to guide you as to what each file is. The images are greyscale or black-and-white high-resolution TIFF files I believe (from the era before PNG), once again with no file extensions to hint at what is what. It underscores the importance of actual running programs and systems as a way to access our digital past. The data might all be there and readable, but with no software to interpret it, what can you do? I actually noted the same four years ago for the somewhat late 25 years anniversary.

Overall, a very memorable evening, and kudos to the organizers for putting it all together.

Jan Bosch is a professor in the software engineering center at Chalmers in Gothenburg. He is a dutchman who has worked all over the world. He used to run Nokia’s research labs in Finland, and then moved to Intuit in Silicon valley. Now, he is back in Scandinavia at Chalmers. What I think he tried to do was to impress the audience with the software development ethos and style of Silicon Valley. I got a feeling that he wanted to make it felt that Europe is maybe behind on modern software development.

However, I am not always sure how the experience from web companies like Google and Amazon can translate into the development of software for systems like telecoms and vehicles (which formed a large part of the audience).

The main points of the talk, and my comments:

Speed is more important than anything else. If we can get 10% efficiency improvement out of our , it should be used to cut lead times by 10%, not reduce cost and keep the same time. The earlier we get software out, the bigger our revenue. This means that you should focus engineering efforts on shortening lead times rather than reducing man hours worked- and if you get the lead times shorter, you will automatically get efficiency improvements. I must say I agree with this point.

We should turn R&D into an experimental system – quickly implement features and ideas and test them in some way. Maybe not with external customers, internal users might do just as well (that’s how Apple works, for example, by evaluating many candidate designs by prototypes that are never seen by the outside). This is a good idea, but I think it might be hard to find the resources to do this in many cases – Apple is kind of extreme in that it rides high, is immensely profitable, and release only a few products each year.

Once we have experiments, product planning should be based on data, not on opinion. Sometimes, vision is needed, but that should be turned into small experiments to validate ideas, not big long-term full-blown plans based on opinion. This sounds very nice – but getting hard data can be very hard in practice.

Initially,  I felt that this would conflict with the concept of the “BHAG” – Big Hairy Audacious Goals – from the book “Built to Last“. There, vision that transcends the current state of affairs is crucial to build products that really change the world. You cannot base that on data – if you ask people what they need, they will not reply with what they do not know. But I guess that if you have a great idea, it is a good idea to vet it with some quick experiments to ascertain that it makes sense at all, before betting the firm on it. So, I guess there is no real conflict here.

Quickly test new features with customers. He sees the world switching to a model where the producer decides when an updrade is applied to the market, not the consumers. Already in place for things like Facebook, Google, and Apple iOS devices. I wonder though if this works for professional software and systems. The customers that I know have to plan the rollout of new versions carefully in order to not disrupt projects that rely on them. When someone pays for a piece of software, they tend to want to have something to say about new features too. Consider the uproar that happens every time Facebook change their UI – if you had paid a hundred thousand dollars for the system, I think Facebook would have listened rather more to their users than they do now. As long as there is no alternative, people will grudgingly accept and learn to live with it – even though the new version might be strictly worse for them than the old version.

When users do have a choice, they often stay with older version. Just consider the vast numbers of PCs out there still running Windows XP. In some way, I feel that this freedom to choose and control your own computing environment is being threatened by this trend of the web.

If you can recruit willing beta testers in your user community, this is great. But you need quite a few customers in place to be able to have enough beta testers to have anything like a representative sample. For a consumer application with tens or hundreds of millions of users, this is not too hard. For a professional piece of software tooling with less than a hundred different customers, you tend to get three loud voices – and we are essentially back to opinion rather than data. It would have been interesting to discuss this with Jan, but I never managed to grab him during the day to discuss this. There is also the time it takes to upgrade a user and have them test a new version – even if the upgrade does not require any changes to their existing code or systems, they still might need training on the new system to fully use it.

The counter to this is obvious: if things are this complicated to use and deploy, maybe we should think about making deployment and use simpler?  And not hide behind “it is hard and we have always done things this way”. Make no mistake – I would love to be able to this, especially with things like debuggers and other daily tools of the programming trade.

Still, I do think that stability is sometimes a hard requirement. When you have a system that you want to maintain for decades, it is very sound to freeze the versions of critical tools used to develop it and stick to these, to minimize the risk of failure due to changes. If the system is certified, you basically have to. Amazon does not build flight control systems. Their system going down is certainly going to cause economic havoc, but nobody will die.

Create an after market – customers will come to expect new features over time, and might well pay for them. This “app market” idea for software features keeps coming up in current discussions on architecture and software design. My guess is that it will actually work, once professional users get influenced enough by the overall consumerization of IT to take certain patterns for granted. Selling new software for car engine control or new features into a telecom switch sounds pretty sane – and is sometimes already practiced today.

Get to know the customer- at Intuit, each engineer famously follows a customer for one day each year to understand their life and get empathy with their users. Knowing your customer is key, the better you know what your customer wants, the more they will take to your product. This idea is something that I have actually seen being tried in the past, with mixed results. Once again, in a business-to-business world this requires some selling to happen, and customers might be sensitive to information leakage and secrecy. However, often the value of having a product expert available to them for a few days to help them work better and run their systems more efficiently is  great idea. If your business has a consulting or services arm, this can be part of regular business – as long as you let core development engineers consult a bit and not keep them all hidden inside the company.

When it came down to practical software development practice, Jan Bosch was all about agile, automatic building, testing, and deploying, and standard modern practice. He retold the Amazon “Two Pizza Rule” – i.e., keep groups small. Keep groups self-governing, based on quantitative assessments of their output, and guide them with goals, not detailed requirements. That certainly works for a certain class of highly motivated and skilled labor that tends to concentrate in Silicon Valley – but in the real world, we also have to deal with less able developers that do need leadership to perform and do the right thing. Not everyone can be totally selective in hiring, unfortunately.

Still, a provocative talk that really did get me thinking about how we do things. Which I think was Jan’s goal to start with, so I guess that means mission accomplished. If you ever get the chance to listen to Jan, take it!