Ticket to Ride

Chasing high scores is fairly typical for computer games – playing against human beings you are motivated to win, even if you win by scoring a measly 75 points… while against the computer it becomes about beating your own old scores. Unfortunately, this also turns repetitive after a while, due to some small design flaws that really should be easy to fix.

What I have found is that I believe there is an optimum high-score strategy in the game, due to the very uneven spread of tickets and a very bad computer player.

My way to play it to get a high score (not necessarily a highscore, though) is to essentially build the exact same network each time, and then stop and pull as many tickets as a I can that fits into this network. When lucky, 15 tickets can fit inside and none miss, and the score will be 250 or more. If not lucky, some tickets will miss, and the score will be lower.

The reason this style of play works is that the computer player is blind to what you do and does not try to stop you from connecting up obvious points. In a real game, if you see someone starting to build a route that looks familiar, you will tend to play a blocking move. In the iPod version, you can have two 20-car segments separated by a single two-segment track… and the computer just ignores it and keeps plugging away at its own goal. Also, when playing with humans, you cannot just say “aw, I got some godawful starting tickets, let’s all start over with something better”. With just yourself and the computer, that’s very natural.

The critical network is this:

Tracks in green are definitely on the critical network, while the yellow parts depend on exactly which tickets show up. Sometimes, Boston is included, sometimes not. The red circles mark cities which are never on any tickets, and which thus are useless.

The connect the corners strategy is good even in the normal real-world Ticket to Ride – but with observant human opponents, very hard to pull of. Also, it seems to me that the Tickets in place in the iPod version is different from what’s in the original board game, and they are bit less balanced.

What I Days of Wonder should do is to copy Angry Birds and just release some free updates containing a ton of new tickets. If there were more tickets connecting from mid-land (Helena, Salt Lake City, Las Vegas) to the unused cities on the east coast, if would be much more variable. Also, the computer components could well be made to be a bit more aggressive and blocking.

In the iPod version of Carcassonne, the computer styles of play vary much more between different personalities, and some of them are very good at playing offensively and blocking you. A similar development could make Ticket to Ride for iPod a bit more exciting to play. As it is right now, it feels like the game has been conquered and won and is pretty pointless to play. The chance of grabbing a higher score is rapidly diminishing, as the random luck needed to get the just right set of tickets with the computer getting a non-conflicting set and being fairly slow in building feels very small. It will happen if you just play many times enough – but where is the fun in that?

2003. The embedded tools company Green Hills launched their
Time Machine feature in their well-known MULTI debugger. I consider this the start of commercial reverse debugging, as it was the first
commercial-grade product to include reverse debugging. The implementation was based on tracing the execution of a program on actual hardware, using a debug probe and a “JTAG” debug interface. The trace box would capture several gigabytes of execution data, and then the debugger performed operations based on this trace. To check a backwards breakpoint, you scan back over the trace until you find a matching state or operation (such a memory access or instruction address that is being executed). The main limitation of the method is that the trace buffer can only capture a few seconds of execution on a typical 100s of MHz embedded processor. It only works for a single processor, and it does not capture IO actions (except as memory-mapped IO). It is system-level and cross-target.

Extending this kind of trace to multicore has proven hard, since getting a synchronized trace out of several processors is very hard. There might be debug hardware coming out in the next few years that can indeed support a time-stamped consistent trace of multiple cores, and with such hardware, the Time Machine approach could well be extended into multicore.

2005. Simics 3.0 was launched by Virtutech (later acquired by Wind River and Intel) with full-system reverse execution and reverse debugging. The Simics approach was also unique, being based on a full-system simulator. By simulating the entire target, it is trivial to reverse (and put reverse breakpoints on) changes to memory, persistent storage like disks, and hardware devices. Since all device models in the simulator are deterministic in their implementation, re-executing hardware events like interrupts and IO outputs is just as easy as re-executing code on the main processor, something that had eluded all previous approaches. Recording is used at the interface between the simulator and the outside world, such as user interaction over graphics displays and serial ports and connections to the real-world network. The software stack is unmodified and system-level, and the simulator can handle multiple processors and even multiple machines in a network as a unit. The use case is normally cross-target (even if a system identical to the host can be simulated, it would work like a cross target logically). Time is handled by counting clock cycles on all processors in the system, and reverse debugging can position the simulation at any point in time based on the virtual time.

There is a cost in execution speed from simulation rather than direct execution, and an intrusion effect from running on a simulator rather than on a physical machine. This affects the timing of events, even with a software stack that is not modified. Still, the fact that you can run a complete real software stack with no modifications needed before starting to run the target system is fairly rare in the world of reverse debuggers.

Simics shipped with a modified gdb that talked gdb serial to Simics and accessed reverse execution with some new debugger commands as well as extensions to the gdb serial protocol. This was offered to the gdb community, but not accepted. However, prompted by this, the gdb community started to discuss reverse execution. Some interesting old threads can still be found, such as http://sourceware.org/ml/gdb/2005-05/msg00225.html. Clearly, at that point in time Virtutech did not really explain how Simics worked, and there were some pretty bad proposals floated in the community for how to do reverse. In the end, the gdb serial design did turn out in the right way, assuming
the remote debugger would reverse itself and gdb would just ask it to do so. This separation of concerns is important to creating practical reverse debugging solutions that can use any debugger backend.

2005. Also in 2005, Lauterbach launched the Context Tracking System, CTS. Lauterbach is a big player in the embedded debug market, with their TRACE32 debugger. CTS can be seen as their reply to the Time Machine debugger. CTS is also based on a trace from a hardware unit or from an instruction-set simulator. However, from the available information is also appears to be more limited – you can step back and go back in time and replay forward, but there is no mention of actual backwards breakpoints (even today, six years later).  Thus, I count this as record-replay rather than reverse debug. It is cross-target, system-leve, and uniprocessor like Time Machine.

2006. Undo Software launched the first Linux-targeting host-based reverse debugger, UndoDB. It is described as a bidirectional debugger (the same terminology as the Boothe 2000 PLDI paper). It is user-level, does do reverse breakpoints (and data breakpoints, also known as watchpoints, which is really useful). It handles multiple threads (at least in 3.0), but from the description of the recording technology used I believe they have to serialize their execution. The implementation is based on checkpoint and re-execution, with recording of all non-deterministic events like IO. There is a feature to move to a certain point in time, based on “simulated nanoseconds”. These are not really nanoseconds, but values which are guaranteed to increase even between two instructions (which probably means that they are sub-nanoseconds and on a > 1GHz CPU single-cycle instructions will indeed take less than one nanosecond).

There is a nice description of how it works on their online man page. It is worth noting that they call it “gdb”, but the command set is distinct from what gdb introduced with its reverse execution in 2009. They use the “b” prefix for backwards commands rather than “r” for reverse.  In some way, UndoDB is in direct competition with the gdb reverse target, but it is much much faster and has more features.

2008. The Rogue Wave (at the time, it was an independent company) TotalView debugger gained support for reverse debugging, with the ReplayEngine add-on. TotalView is an old mainstay in the HPC market, having been around since at least 1993. Indeed, it was developed initially for the BBN Butterfly computer, and thus it might have had a touch with reverse execution as far back as the 1987 paper cited in my previous blog post.

Judging from the available materials, TotalView can clearly can step back in various ways. However, it is not clear that it triggers breakpoints when going backwards. Thus, it has to count as record-replay debugging rather than reverse debugging. The base of the implementation is extensive instrumentation of the the runtime system of the target computer.  The implementation builds on the fact that the target programs tend to b clustered programs that use MPI to communicate – and thus a large part of the communication between threads is explicit and easily intercepted and recorded.  There is also an existing infrastructure of checkpoint and restart for parallel programs using MPI to support fault tolerance that was used as the base of the implementation.  Finally, in a slightly ugly hack, they make each multi-threaded program run on a single processor by a big lock. In this way, all that needs to be replayed is the interleaving of threads on a single processor, a far more tractable problem compared to trying to replicate a true parallel execution in a new session.

2008. VmWare officially launched a record-replay debugger based on their virtual machine technology with VmWare Workstation 6.5. Single-processor, system-level (but really only supported for user-level debugging), cross target (since the VM is not really the absolutely same hardware as the host), time model is based on the virtual machine which I believe is cycles-based. Mostly used for record-replay debug of non-deterministic software bugs, but could also do reverse debugging including reverse data breakpoints. Based on snapshot and deterministic re-execution, plus recording of all non-deterministic device accesses (not all devices in the VmWare hardware emulation layer are deterministic). Going back to a snapshot was a very heavy operation (I tried it) since you had to restore the entire target memory (quickly got into gigabytes). The hardware supported in the VM was quite limited, and things like CD-ROMs and floppies could not be part of a record/replay session. Replay logs could be moved between hosts.

The VmWare reverse debug functionality was removed from VmWare workstation version 8 in 2011, since it required a large investment and was not apparently used by very many VmWare users. This indicates that trying to build developer-oriented functionality into a technology base that is fundamentally driven by the need of deployed virtual machines was hard. There are contradictions between these two goals, as the determinism and control needed for a good reverse debugger is not necessarily consistent with maximum performance for running virtual machines in a production setting.

2009. gdb 7.0 added support for reverse execution (a work that began in 2006). The built-in “record” target supports reverse debugging on user-level single-threaded programs on the same host. The command set for reverse debugging is fairly full-featured, but is a bit quirky with a “set direction” command that makes regular run-control commands work in reverse. The record technology is quite slow since it basically records the effect of each and every instruction run in the program.

In addition to its built-in target, gdb can also control external reversible debug systems over the gdb serial protocol. This made the changes to gdb-serial created by Virtutech for Simics in 2005 part of the mainline gdb release. Several tools support the command set, including VmWare, UndoDB, and Simics. There was also a set of MI commands added to basically let Eclipse use gdb as a backend for reverse debug, including using it to control external tools via gdb-serial. How this happened is quite a long story, and I made a small contribution to the gdb code base myself in the process. Read about this here.

2009. Eclipse CDT added support for reverse execution, using gdb 7.0 reverse as the initial backend. As noted above, this lets Eclipse also use other reverse debugging backends (Eclipse uses the gdb-MI interface to gdb to control the debug session). This is noteworthy since it meant that the buttons to control reverse execution are now part of the CDT, making it much easier to use Eclipse to build a frontend to any reversible backend. Eclipse is not really a debugger, just an interface to a debugger.

2009. Microsoft Visual Studio got record-replay debugging with IntelliTrace. It is strictly about replay debugging, including the nice ability to send traces around between developers. There are no backwards breakpoints. The support is limited to programs running on top of the .net runtime system, meaning that it does not apply to classic Windows software. Using the CLR virtual machine as the implementation basis should make the implementation easier, cleaner, and faster compared to a machine-level native solution. It is user-level, single-threaded, and host-based. Time concept is unknown.

2011. Adobe demonstrated (not launched) reverse debugging in their Flash Builder programming environment. A nice video is posted on the Adobe website. Seems to be based on the virtual machine that flash runs on, and includes what looks like pretty powerful backwards data analysis tools. In a blog post, the developer describes some of the features, which to me seem to indicate some pretty heavy recording.

Final notes.In researching these commercial tools, there also seems to be a lost one. A company called Visicomp launched a Java debugger called RetroVue in 2002 which supposedly did allow backwards debugging in some way. However, it seems that this tool was not really practical, being too slow for actual use. It seems to have disappeared since without anyone picking up its legacy. The technology was apparently pretty much like the Omniscient Debugger presented in 2003 and which I described in the blog post on reverse execution research.

When searching through the literature using the ACM Digital Library (and some Yahoo and Google web searches), I find quite a large body of research literature dating to the late 2000s. Apparently, the appearance of commercial reverse debuggers have inspired research into the topic – and our increasingly powerful machines have made heavier solutions feasible (at least for single programs).

Disclaimer – this is not really an exhaustive survey of the field, but some of the more interesting papers that I have stumbled on.

1973. The first actual mention of reverse debugging I have found is a short 1973 Communications of the ACM article called “Reversible Execution” by Marvin Zelkowitz. Basically, this paper proposed to apply a backtracking facility built into a PL/I compiler to support AI research (a precursor to Prolog execution semantics from what it sounds like) to also replay parts of a normal program for debugging. This in essence would enable replay of a program execution, based on a log of how the program ran and the values of variables changed. Basically, a forward-only record-replay solution for user-level single-threaded programs.

1987. Thomas LeBlanc and John Mellor-Crummey, “Debugging Parallel Programs with Instant Replay“, IEEE Transactions on Computers. This paper covers record-replay debugging of parallel user-level programs on the BBN Butterfly. They record all interaction between processes (threads) by intrumenting the OS calls that are used to communicate and synchronize between processes. Programs also sometimes had to be changed so that their communication protocols allowed the instrumentation to work properly. Thus, the approach did not apply to unchanged code either at source or binary level. Processes still compute their results, only the order or interactions are enforced. In this way, the overhead was kept usefully low. The approach does not work well for programs that deal with asynchronous interrupts and hardware interaction by the software, since there is no good way to hook and replay such events in their implementation.

The most important concept here is the use of re-execution of code to reconstruct values of a program, assuming deterministic computation as long as asynchronous interactions can be controlled. This is the basis of all reverse execution approaches based on checkpointing and re-execution. Not sure if they invented the idea, but this is the earliest distinct mention I have seen of this.

1988. Stuart Feldman and Channing Brown, “IGOR: a system for program debug via reversible execution“, 1988 ACM SIGPLAN and SIGOPS workshop on Parallel and distributed debugging (PADD) . This paper looks at the problem of how to debug user-level single-threaded code on a UNIX-like system. The programs need to be recompiled to add a small amount of support, in particular to help with restarting
execution. It is essentially a record-replay approach, but capable of placing the execution of a program at any point in its past execution. They save regular checkpoints during program execution (every tenth of a second or so seemed appropriate in their testing). The checkpoints only include the data that has been changed since the previous checkpoint (they added an OS facility to mark all pages that a program writes between checkpoints). A unique feature is the ability to look at the value of a variable as it is stored in successive checkpoints, and to “go to the point in time where variable X has value V” (assuming a variable is monotonically increasing or decreasing, like an outermost loop counter). To do the detailed positioning, they use an interpreter for the target system, essentially mixing native and interpreted execution in their debugger.

1996. Mark Russinovich (who I believe is the man behind the recent novel “Zero Day“) and Bryce Cogswell, “Replay for Concurrent Non-Deterministic Shared-Memory Applications“, PLDI 1996. In this paper, instrumented system libraries and binary code modification is used to enable deterministic replay of multithreaded applications. The binary modification is used to implement an instruction counter to provide a logical time-base, and instrumented system libraries are used to force a serialized (but interleaved) execution of the program. It does not run the program in parallel on a multiprocessor, but rather runs it on  a single processor. User-level, instrumented and intrusive, parallel on single processor.

2000. Bob Boothe, “Efficient Algorithms for Bidirectional Debugging“, PLDI 2000. I actually saw this paper being presented live at the PLDI conference in Vancouver. At the time, I was working on embedded compilers and debuggers, and our little group from IAR looked at what was being presented at said “yeah, might work on a big desktop, but you can’t do that in a real machine”. Why was that the case? Essentially, what Boothe did was to use the Unix fork call to spawn off checkpoints of a program as it was executing. This was a smart way to achieve checkpointing using existing OS facilities. In addition, he recorded the results of system calls to replay them during replay.

Time was given by counters in target program (compiled-into it as part of the debugging build). Most of the paper was spent on basic algorithms for how to do “previous” (step back 1 line), “before” (inverse of finish), “buntil” (backwards until, looking at data conditionals to determine when to stop). His use of “backwards” rather than “reverse” as the name of the concept does make this paper a bit harder to find. While limited to a single threaded user-level process, this paper is probably the first to introduce reverse debugging in the sense of the primary user interface being the ability to step backwards and check for breakpoints backwards in time. A good idea in the UI design was the undo command that jumped the debugger back to the last position where you stopped. Having worked with reverse debugging quite a lot myself, this operation makes a lot of sense as an addition to standard run-control commands.

2002. Tankgut Akgul and Vincent Mooney, “Instruction-level Reverse Execution for Debugging“, at PASTE 2002. This paper is an odd one, as it was neither really reverse nor practical. The approach is based on doing a static analysis of the assembly code of a program to generate code that could reverse operations based on some logging of destructive operations. Interesting, but not really practical as I see it. One noteworthy tidbit in the paper is the measurement that logging only destructive operations is some 50 x more efficient than logging all changes.

2002. George Dunlap, Samuel King, Sukru Cinar, Murasa Bazrai, and Peter Chen, “ReVirt: enabling intrusion analysis through virtual-machine logging and replay“, at OSDI 2002. This paper uses the User-Mode Linux (UML) paravirtual machine to checkpoint and replay an operating system. I think this is the first use of a true virtual machine to achieve reverse execution, even though it is a paravirtual machine rather than a full virtual machine. The replay starts from a single checkpoint at the start of execution of the guest system, so moving to a certain point in time can be very time-consuming. It does support reverse breakpoints. To move to a certain point in time, they count executed instructions, not actual time. The use of a paravirtual solution precludes the use of many real device drivers and debugging issues related to interaction with most real devices. It is uniprocessor, system-level, unmodified target (except the change to make the OS itself run paravirtualized on top of UML).

2003. Bil Lewis and Mireille Ducasse, “Using events to debug Java programs backwards in time“, OOPSLA 2003. This paper targets programs running on top of a Java Virtual Machine, resulting in an Omniscient Debugger. They instrument the target program to log all state changes. A big lock is used to serialize all threads into a single interleaved execution. The implementation is admitted to be almost worst-case inefficient, but the debugger was still considered useful for some real work. The debug UI does feature backwards breakpoints, so it qualifies as actual reverse debugging.

Note that in principle, backwards debugging and record-replay should be possible to implement quite efficiently for a language virtual machine like the JVM, if you can modify the VM itself. This has indeed been done today, for example by Microsoft for their CLR and in the use of full-system simulators (and other simulators) to implement reverse debugging.

2005. Samuel King, George Dunlap, and Peter Chen, “Debugging Operating Systems with Time-Traveling Virtual Machines“, USENIX 2005. This paper builds on the foundation of ReVirt, but adds checkpoints during a run to make the replay to a certain point in time more efficient. They also introduce the differential storing of disk state in the checkpointing (just like done in Simics). Overall, it is very similar to the replay approach taken by VmWare a few years later. Still single-processor. The paper contains a nice list of example bugs for which reverse debugging work much better than cyclic debugging. They also reference Simics as something that might do something with reverse, but do not quite know what (this is right after the launch of Simics reverse execution as discussed in the next post).

2007. Paul Brook and Daniel Jacobowitz, “Reversible Debugging“, GCC Developer’s Summit 2007. This paper presented some early work on implementing reverse debugging inside the Qemu full-system simulator. Logically, it follows in the steps of the Simics reversible debugger announced in 2005 (see the third post in this series of blogs). They use gdb as their frontend, and the paper introduced the idea of setting the default execution direction of the debugger. This UI choice was later adopted in the reversible gdb 7.0 released in 2009. The paper contains a good discussion of some of the practical issues involved in performing operations like reverse-step to go back a line, or stepping back up through a function call. It is a good explanation of the basic problems.

The actual reversing is implemented in Qemu using a checkpoint and deterministic replay, along with logging of any operation that is not deterministic. Their main test case is actually using the semihosting variant of Qemu where IO calls are intercepted and passed on to the host. Just like all other such solutions, they record the return values and do not repeat side-effects. It is system-level, single processor, unmodified target software, and time is based on counting executed instructions.

The Qemu architecture makes enforcing reversibility across all devices fairly difficult, since each device manages its own interaction with the user. One interesting tidbit in the paper is that Qemu by default triggers timer interrupts based on host time, but these had to be made fully virtual to be reversible.

Summary The above is a small selection from all the papers published on this topic, but I think they capture the most common methods used. Basically, almost all of the approaches require some kind of change to the target software to enable reversibility. Interestingly, the first few commercial solutions for reverse debug that appeared did not take that approach, but rather targeted unmodified programs using a different set of fundamental techniques. See the next blog post for that part of reverse history.

Let’s start with background and definitions.

To me, the key defining factor of reverse debugging is the ability to apply breakpoints in reverse – essentially, to be able to go to the previous occurence of a breakpoint just as well as going to the next. The implementation allowing this might vary.

The contrast to reverse debugging is classic cyclic debugging where you run and rerun a program with a problem, looking at variables and setting breakpoints during each run to zoom in on an issue. For cyclic debugging to work, you pretty much require each run to behave the same way. The program has to be fundamentally deterministic. This is
usually the case for non-interactive single-threaded programs, but not the case for real-time programs, parallel programs, or programs that involve some kind of asynchronous input/output (reading and writing files is a special case of IO that can indeed be deterministic since there is usually no interference from the environment in those operations).

Thus, for non-deterministic programs and rare errors, reverse debug is what you want. Run once, hit the error, reverse to diagnose it.

In addition to reverse debug and cyclic debug, we also have record-replay debug. In such a system, you record an execution and later replay it forward. Debugging is strictly forward: you cannot step back in time instruction by instruction, nor trigger breakpoints backwards in time. Record-replay debug is a way to allow cyclic debugging on non-deterministic highly variable program runs. This can be implemented with a lot less debugger complexity than a true reverse debugger, even if the underlying runtime system is almost identical to reverse debugging.

So, how do you implement reverse debugging?

Reverse execution is one way to implement reverse debug. In reverse execution, the execution system has the ability to move backwards in time and put the entire system in the state it was in at some previous point in time.  You can step the system back instruction by instruction or cycle by cycle.  You can also continue the execution
forward from any point in time, throwing away the history of asynchronous inputs to actually take a new execution path.

Typically, reverse execution is implemented by using checkpoints of previous system states plus a deterministic way to re-execute the system from those checkpoints. The key technical problems to be solved is how to checkpoint the system and how to make re-execution deterministic (note that determinism does not imply invariant or predictable behavior, just that you can reliable recreate a particular execution). You also need to record and replay anything that you cannot re-execute, such as user interactions or network communications with the world outside the controlled system.

Schematically, it works like this, note how simulation time always moves forward even though the logical system time sometimes moves backwards:

Record-replay debug is usually implemented in a way similar to this, without the ability to go back in time and usually without the intermediate checkpoints.

Trace-based debugging is based on recording everything a system does into a trace, and once the recording is done, debugger works on the trace rather than using the log to drive an actual system.  Reverse debug is implemented by recreating the state of a system by reading the trace, and finding points in the trace where breakpoint conditions are true. This is also known as post-mortem debug, since you debug after the target system has finished executing (typically). A log can be captured by a hardware device, or by software being instrumented to log everything that is going on. The technology can also be used to implement record-replay debugging.

There are some other important distinctions between different approaches to reverse debugging that we need to keep in mind as we review the history of the field.

• System or user-level? Do you debug a system, including the OS, or just a user-level application running on top of the operating system? User-level debug can often be solved in simpler ways than system-level debug, but also suffers from some limitations. There are also times where system-level is simpler.
• Single-threaded or multi-threaded? Can you debug multiple threads, processes, processors, or just a single processor or user-level thread? A single thread of control greatly simplifies the problem.
• Cross-target or host-based? Do you debug programs running on the same host  as the debugger, or can it also target remote targets or cross targets (such as embedded systems)?
• Instrumented or unchanged programs? Quite a few solutions for reverse debug tried over the years involve compiling programs in a special way, using instrumented OS libraries, or other implementation variants where the program debugged is not identical to the eventual deployed program.

Given this background, the next post will cover early research, and the third post the beginning of commercial products.

Note that on Stack Overflow, reverse debug does not seem to be particularly big thing still. The reverse-debugging tag has all of 10 members, while other topics like C# and Android programming has millions… so maybe reverse is not quite mainstream yet. Or maybe it is just the case that Stack Overflow has more web-style developers than low-level developers in their membership.

Essentially, as the (embedded) software systems we are developing get more and more complex, we need to update our debuggers to keep up with the times. This means moving beyond looking at single processors, threads, programs, or even nodes – and making debuggers that support working with multiple contexts at once. For example, debugging a couple of different boards in a rack from a single debug session. This would mean supporting multiple processor architectures, execution modes, and operating systems inside a single debugger session. Debuggers need to orient themselves towards handling software that is already running in a target, rather than starting the target software. The debugger becomes more of a passive observer of the target system than an active component.

Take a look!

I have been thinking about this subject lately, after using a BlackBerry Torch 9800 as my work phone for a few months.  I like the device a lot, but there are certainly some rough edges and some places where there is a UI conflict between touching the screen and pressing the buttons. At the same time, I am using both an iPod Nano 3G, and a couple of iPod Touches. I used to have SonyEricsson Symbian-based P900, P990i, and G900 smart phones which also were combined touch/press devices with a stylus.

I think it is clear that using a physical keyboard is preferable to an on-screen keyboard for typing serious amounts of text.  On the iPod, entering a URL or search term in a browser just feels fiddly, compared to the ease of composition with the physical keyboard on the
BlackBerry.  To me, I cannot get over the feeling that an iPod Touch or iPhone is really best as a consumption device, but that a BlackBerry is a superb creation device.  On a BB, I can type quite long emails, while an iOS device feels more appropriate for the occasional short messages.  To me, this is an important aspect, as I tend use the smart phone as a two-way email communications device.

The slide-out keyboard solution on the BB Torch (and many other phones from many different vendors) seems just right for this, giving you a full-screen device for reading tasks, with a full keyboard when needed.

On my old G900, I used T9 with a 0 to 9 keypad, and that worked surprisingly well.  A problem both with the BB and definitely with an on-screen keyboard is the fact that using it single-handedly is very hard.  On a screen, you do not get feedback from your fingers where you are on the keyboard, and the full keyboard of the BB is a bit too small to reliably use with a single hand.  I noted this in my G900 review three years ago, and it still holds true. The G900 looked like this:

Since a primary use of our iPods is gaming, I have noted some different UI principles for games.  There seem to be one class of games where touch makes perfect sense, like Angry Birds.  Press and draw to load, swipe to move the display, pinch to zoom – perfect and logical. Another class of games are obviously ports of mouse-based games, like Plants vs Zombies.  In such games, the touch screen is essentially used as it was back in stylus-time. It is just another way to generate single-point clicks.

Finally, we have the category of games that really would work best with a physical controller, like PacMan. Displaying a four-way control key setup on the screen does not get close to the right feeling.

I think the SonyEricsson Xperia Play was a brilliant idea – imagine an Apple device with that little built-in game controller.  Seems that SonyEricsson has not quite managed to execute on the idea, but in an ecosystem that generates variation like Android, this kind of device should have a place.

However, I have a hard time seeing the swiss-army-knife design of a phone having both a slide out keyboard and a slide-out game controller. Would be neat, but mechanically I don’t think it would work very well.

On to navigation.  The idea of touch gestures (in particular swipes) to navigate around the UI as popularized by Apple in the first iPhone is great in many ways.  My youngest child has used it since she was two, and it just works quite naturally.  However, there are still cases where a plain old navigation key works better. And having both is best.

Holding a device in one hand and scrolling a web page is faster on the navigation key on my BB than using touch on an iPod. The old G900 was even more efficient, just hold the down key – no physical motion needed at all. The iPod would be so much better if it could just have a scroll wheel on the side or something so you could use it one-handed
for reading without having to put a finger on top of the screen (and quite often accidentally clicking links in the process).

I also like the menu key on the BB (and on my old SonyEricsson phones), as a way to quickly get to the most important functions. This seems hard to emulate on a touch screen in a good way. In their touch-enabled phones, BB has tried it with a press-and-hold action – if you press down on the screen, a little action palette opens. Which does not quite work for me.

Another great advantage of some well-choosen physical buttons is that they offer immediate access to certain functions.  On a touch-screen device, I find that you have to dig through several steps to get into any function.  Home screens with most commonly accessed functions in all honor, but a direct key to dial a call or activate the camera is faster.  It cannot support all functions, obviously, but it lets you bring up the most common and important functions quickly. This is an aspect where my iPod Nano is much better than an iPod touch. On the iPod Nano, you can have it in a pocket and just hit the device to move to the next song or pause. On an iPod Touch, you have to look at the screen and find the right spot to hit.

On the other hand, the Nano would have been even better with a couple of buttons to increase/decrease volume rather than the scroll wheel. When out running, having to hold the things in both hands to make simple adjustments is really annoying. It also does not work with gloves on, which is a big problem with all touch device as well as touch-wheels – they require naked fingers, which is annoying in a climate where gloves are on for at least half the year. As I said before, someone should put a design center in Luleå or Novosibirsk or Alaska, and then see what kinds of devices come out.

A full keyboard like the BB also have the nice property of providing many points of access to shortcuts.  It might seem quite backwards of me, but I like the speed of action you get with the keyboard shortcuts in the BB email application. It has a lot in common with classic computer UIs, actually. It is hard to afford immediate access to 20+ functions on a touch screen without creating a very cluttered interface.  On the other hand, a touch interface done right makes it simple to access system-level things quickly by just touching the status icons on the screen – you get out of the application faster than if you had to use the keyboard to navigate up to some button.  For some reason, neither the iOS devices nor the BB does this right, while my old SonyEricsson smartphones all let me access detailed status by hitting the battery icon or network icons.

However, after singing the praise of the keyboard and touch screen in union, I must admit that there is a big downside to a combined touch/press device. UI design is harder when you have to both afford a good keyboard-controlled interface and a touch interface. The BB Torch I have suffers from this quite badly – having to afford both touch and non-touch interaction, as well as non-touch versions of the device itself does mean that the UI is not as streamlined and elegant as what you can get on a pure touch device. I often find myself jumping between keyboard and touch as I cannot quite complete what I want to do using just one or the other interaction mode, which really should not be necessary. Only having to think about a single type of input does seem to make things simpler for a UI designers, even if it also sometimes precludes creating truly efficient applications.

The Fujitsu Scientific and Technical Journal is the Fujitsu equivalent of IBM’s Journal of Research and Development. Thankfully, the FSTJ is still free while IBM erected a paywall around their articles. Number 2 of 2011 had the theme of servers, and there is an article about Quality Assurance for  Stable Server Operation by Masafumi Matsuo, Yuji Uchiyama, and Yuichi Kurita.

The article describes the process of ensuring that the final servers that are shipped to customers (from what seems to be the Sparc-based line of Fujitsu computers, even though it might actually apply equally to their mainframes and x86-based servers) are as stable as possible. Apart from designing things right, this also requires testing the fault handling and recovery operations.

I found it noteworthy that they do a lot of configuration testing, where various hardware and software configurations are played off against each other. In this way, corner cases are explored and coverage of the actual configurations that customers will be using becomes more likely (it is always dangerous to only test on one or a few configurations). They push memory system and processor loads to very high levels to ensure continued operation even in extreme cases, and also try to push the actual chips to make sure they will operate reliably in a wide range of environmental conditions. Indeed, a large focus is placed on pure physical reliability, as that is the basis for system reliability.

The best part, however, is on page four of the article, where they show the physical fault injection robot that is applied to the chips mounted on boards. This robot  shorts out individual pins on chips, clamping them to zero volts. It goes over all pins, and the test system checks what happens to the system in each case. Some kind of exhaustive testing going on here.

Neat. I have heard other stories of physical fault injection, including complex mechanisms like passing computer boards through irradiation chambers all the way to brutally simple tasks like putting an axe into a board to break it or pulling boards out of racks to simulate a sudden catastrophic failure. I would like to see more of just how these things are done in the real world. I suspect there are quite a few interesting robotics setups out there that do fault injection.

In any case, the article offered an interesting glimpse of many of the techniques used to make computer systems robust and reliable. Recommended.

It ends by noting that deeply consolidated SoC designs and aggressive dynamic power management are challenging from a testing and observation perspective, as well as creating more single points of failure. If a single system on chip fails, that system is all gone…

I have recently seen a number of examples of what Microsoft does with the user feedback data they collect from their massive installed base. I am not talking about Google-style personal information collection, but rather anonymous collection of user interface and error data in a way that is more designed to built better products than targeting ads.

The first paper is “Debugging in the (very) large: ten years of implementation and experience” by Kinshumann et al, Communications of the ACM, July 2011. This paper describes how Microsoft uses of the data they collect from Windows Error Reporting (you know, the little dialog boxes that appear every once in a while on Windows when a program has crashed or frozen, or Windows restored from a crash).

Microsoft has a number of heuristics that look at the data collected, grouping the bug reports into buckets. Ideally, each bucket corresponds to a single root cause for possibly quite different errors. They automatically analyze the errors and generate metadata about the error reports that can be used to generate statistics and allow database queries to be performed over all collected error
reports.  Heuristics include walking through chains of threads blocked on synchronization objects to determine which one is the actual cause of a hang, and finding the most likely thread and stackframe for containing the root cause of an error.  Heuristics are applied both on the client and the server, but mostly on the server. Technically very hard to do right, I can appreciate the huge amount of work that has gone into engineering this.

With this huge pile of information, a new debugging method becomes available: statistics-driven bug finding and prioritization at large scale.  The introduction to the paper puts it very well:

Beyond mere debugging from error reports, WER enables a new form of statistics-based debugging. WER gathers all error reports to a central database. In the large, programmers can mine the error report database to prioritize work, spot trends, and test hypotheses. Programmers use data from WER to prioritize debugging so that they fix the bugs that affect the most users, not just the bugs hit by the loudest customers. WER data also aids in correlating failures to co-located components. For example, WER can identify that a collection of seemingly unrelated crashes all contain the same likely culprit—say a device driver—even though its code was not running at the time of failure.

For a product manager like me, used to working with individual bug reports in bug reporting systems and trying to manually assess the importance of each error, this is nothing short of a dream.  Instead of trying to guess how many users can be impacted by a bug, Microsoft can run queries against the error report database and get a fairly accurate idea of how common a certain error is in the user base.  This has allowed them to address the most common errors first, leading to Windows and Office becoming more stable for more users in recent generations.  They can also pinpoint which device drivers are causing the most issues, and putting pressure on vendors to clean up their act.

I wonder where else you really apply this idea of statistical debugging. You need a large user base, in systems which are connected to the Internet so you can collect data, and who are comfortable with providing direct feedback to you as a vendor.  Apparently, Apple has the same kind of feature built into iOS, with more than 100 million users which seem not to be too interested in strong privacy.  Presumably, Google can do the same thing with Android, at least its use in phones. Mozilla has a crash reporter, so I guess it makes sense in the consumer space.

But when your user base counts in thousands of seats and half of these are in defense sector beyond air-gaps, it is harder to apply. Products that call home are not taken to kindly in the professional field, as secrecy and confidentiality is very important to big companies. Industrial embedded products like telecom infrastructure might have sufficient volume of code and computer hardware to form a basis for statistical reporting – as long as operators agree to provide the information to the hardware vendors.

Another example of how Microsoft makes use of their collected data is in UI design. The blog post “Improvements in Windows Explorer“, by Steven Sinofsky, from the Windows 8 blog discusses how Windows Explorer has evolved over the years, and how it is now getting a radical redesign based on usage data.  Microsoft is an enviable position here, having collected information about what millions of users are doing.  Definitely beats inspiration or trying with a few users in a classic user interface lab.

I have seen quite a few people criticize this blog post from a variety of angles – from the fact that they are not data-driven enough and keep rarely-used buttons in the ribbon to the fact that they remove somebody’s favorite function.  It is also the case that the measurements can only tell you which functions people are using from what is available today – if you want to invent new things, data like this might not be very helpful.

Fortunately, Microsoft also seems to have taken a clue from Linux and is allowing much more user customization than before. For me, this is great news, as I seem to have a user profile quite far from the mainstream.  We have not seen Windows 8 in its final form just yet, but hopefully this approach will be applied to other parts of that GUI overhaul too.  There are professional Windows users who need an OS that makes even very esoteric operations easy to access, and customizations of things like the start menu possible.  Hopefully, we do not get washed away by the flood of data from regular users.

For some reason, I feel that bug reporting is not as sensitive to the user style as GUI design – Windows and driver bugs would seem to be more evenly distributed as they depend more on hardware than on software. At least it seems to me that Windows is more stable today
than it was a couple of years ago.

As David Alan Grier would have said, this kind of event tends to serve to build the professional identity of a group of people. Computer science is not a profession per se, but it is clear that the Uppsala computer science students (almost 2000 has started since 1981) has a particular culture that comes back to us very easily when amongst our peers. I think it is based in the art and practice of programming.

The talks and discussions during the dinner often went back to the defining experiences of our student days, and these experiences were mostly about night hacks and classic labs. A speaker told us about how in 1983 they tried to program a simple computer built by the department itself, and how it turned out that 3 out 4 machines were broken. There idea that they expected the program to just work the first time it loaded made me look up my favorite debugging quote. Somebody reminded me (19 years after the deed) about the time I made a Prolog program exhaust all memory on a Sun server, by performing an exhaustive search for a problem with no solution).

I remembered how some students I taught (while still an undergraduate myself, a common practice at DV) spent 24 hours a day for almost a week in a lab room trying to get their operating systems to boot on some MIPS-based lab machines. In particular, the group that was gripped by ambition and tried to turn on the MMU. Each time they reset the machine and tried to boot their OS, they would see a serial terminal spewing out diagnostics text and then stopping cold… as they failed again to make it work (they passed the course anyway).

This all indicates the fundamental importance of programming to computer science students. That is also what we believed was our core mission when I was a student – to go out into the world and create great software. Many still do, even if quite a few of us have left day-to-day coding to become project leaders and outright managers. Can’t say I program all that much myself, apart from some demos and virtual machine scripts, but I still find the topic incredibly interesting and important.

The event also touched on institutional memory and the longevity of data. The very ambitious anniversary committee had produced a brand new version of the classic “Manualen”, a song book first produced in 1995. In it, I found a text I wrote in 1995 about what a computer scientist actually does (a bit pompous, as can be expected from a proud student) as well as a photo of myself from a 1996 cover of the DV student magazine “Blurgel”. However, in both cases, these had been reproduced from paper copies. There was no digital memory in place of these 15-year-old pieces of data.

I actually still have both paper copies in good shape and the data – on a multisession CD-R created in 1999 on a Mac. My current computers all being Windows-based cannot extract the data. I know a Mac can still read them, but it is not clear that the data can be used today. It is likely in some old version of Aldus PageMaker and with no file extensions to guide you as to what each file is. The images are greyscale or black-and-white high-resolution TIFF files I believe (from the era before PNG), once again with no file extensions to hint at what is what. It underscores the importance of actual running programs and systems as a way to access our digital past. The data might all be there and readable, but with no software to interpret it, what can you do? I actually noted the same four years ago for the somewhat late 25 years anniversary.

Overall, a very memorable evening, and kudos to the organizers for putting it all together.

Jan Bosch is a professor in the software engineering center at Chalmers in Gothenburg. He is a dutchman who has worked all over the world. He used to run Nokia’s research labs in Finland, and then moved to Intuit in Silicon valley. Now, he is back in Scandinavia at Chalmers. What I think he tried to do was to impress the audience with the software development ethos and style of Silicon Valley. I got a feeling that he wanted to make it felt that Europe is maybe behind on modern software development.

However, I am not always sure how the experience from web companies like Google and Amazon can translate into the development of software for systems like telecoms and vehicles (which formed a large part of the audience).

The main points of the talk, and my comments:

Speed is more important than anything else. If we can get 10% efficiency improvement out of our , it should be used to cut lead times by 10%, not reduce cost and keep the same time. The earlier we get software out, the bigger our revenue. This means that you should focus engineering efforts on shortening lead times rather than reducing man hours worked- and if you get the lead times shorter, you will automatically get efficiency improvements. I must say I agree with this point.

We should turn R&D into an experimental system – quickly implement features and ideas and test them in some way. Maybe not with external customers, internal users might do just as well (that’s how Apple works, for example, by evaluating many candidate designs by prototypes that are never seen by the outside). This is a good idea, but I think it might be hard to find the resources to do this in many cases – Apple is kind of extreme in that it rides high, is immensely profitable, and release only a few products each year.

Once we have experiments, product planning should be based on data, not on opinion. Sometimes, vision is needed, but that should be turned into small experiments to validate ideas, not big long-term full-blown plans based on opinion. This sounds very nice – but getting hard data can be very hard in practice.

Initially,  I felt that this would conflict with the concept of the “BHAG” – Big Hairy Audacious Goals – from the book “Built to Last“. There, vision that transcends the current state of affairs is crucial to build products that really change the world. You cannot base that on data – if you ask people what they need, they will not reply with what they do not know. But I guess that if you have a great idea, it is a good idea to vet it with some quick experiments to ascertain that it makes sense at all, before betting the firm on it. So, I guess there is no real conflict here.

Quickly test new features with customers. He sees the world switching to a model where the producer decides when an updrade is applied to the market, not the consumers. Already in place for things like Facebook, Google, and Apple iOS devices. I wonder though if this works for professional software and systems. The customers that I know have to plan the rollout of new versions carefully in order to not disrupt projects that rely on them. When someone pays for a piece of software, they tend to want to have something to say about new features too. Consider the uproar that happens every time Facebook change their UI – if you had paid a hundred thousand dollars for the system, I think Facebook would have listened rather more to their users than they do now. As long as there is no alternative, people will grudgingly accept and learn to live with it – even though the new version might be strictly worse for them than the old version.

When users do have a choice, they often stay with older version. Just consider the vast numbers of PCs out there still running Windows XP. In some way, I feel that this freedom to choose and control your own computing environment is being threatened by this trend of the web.

If you can recruit willing beta testers in your user community, this is great. But you need quite a few customers in place to be able to have enough beta testers to have anything like a representative sample. For a consumer application with tens or hundreds of millions of users, this is not too hard. For a professional piece of software tooling with less than a hundred different customers, you tend to get three loud voices – and we are essentially back to opinion rather than data. It would have been interesting to discuss this with Jan, but I never managed to grab him during the day to discuss this. There is also the time it takes to upgrade a user and have them test a new version – even if the upgrade does not require any changes to their existing code or systems, they still might need training on the new system to fully use it.

The counter to this is obvious: if things are this complicated to use and deploy, maybe we should think about making deployment and use simpler?  And not hide behind “it is hard and we have always done things this way”. Make no mistake – I would love to be able to this, especially with things like debuggers and other daily tools of the programming trade.

Still, I do think that stability is sometimes a hard requirement. When you have a system that you want to maintain for decades, it is very sound to freeze the versions of critical tools used to develop it and stick to these, to minimize the risk of failure due to changes. If the system is certified, you basically have to. Amazon does not build flight control systems. Their system going down is certainly going to cause economic havoc, but nobody will die.

Create an after market – customers will come to expect new features over time, and might well pay for them. This “app market” idea for software features keeps coming up in current discussions on architecture and software design. My guess is that it will actually work, once professional users get influenced enough by the overall consumerization of IT to take certain patterns for granted. Selling new software for car engine control or new features into a telecom switch sounds pretty sane – and is sometimes already practiced today.

Get to know the customer- at Intuit, each engineer famously follows a customer for one day each year to understand their life and get empathy with their users. Knowing your customer is key, the better you know what your customer wants, the more they will take to your product. This idea is something that I have actually seen being tried in the past, with mixed results. Once again, in a business-to-business world this requires some selling to happen, and customers might be sensitive to information leakage and secrecy. However, often the value of having a product expert available to them for a few days to help them work better and run their systems more efficiently is  great idea. If your business has a consulting or services arm, this can be part of regular business – as long as you let core development engineers consult a bit and not keep them all hidden inside the company.

When it came down to practical software development practice, Jan Bosch was all about agile, automatic building, testing, and deploying, and standard modern practice. He retold the Amazon “Two Pizza Rule” – i.e., keep groups small. Keep groups self-governing, based on quantitative assessments of their output, and guide them with goals, not detailed requirements. That certainly works for a certain class of highly motivated and skilled labor that tends to concentrate in Silicon Valley – but in the real world, we also have to deal with less able developers that do need leadership to perform and do the right thing. Not everyone can be totally selective in hiring, unfortunately.

Still, a provocative talk that really did get me thinking about how we do things. Which I think was Jan’s goal to start with, so I guess that means mission accomplished. If you ever get the chance to listen to Jan, take it!

The paper describes a simulation for a network-on-chip based homogeneous system containing a “ARM-subset” ISS instances with local instruction and data caches, some local RAM, and also some shared RAM. Each core runs its own local software load, there is no SMP operating system. All communication between cores is over shared memory, using explicit operations across the NoC. All cores run a single cycle before they check communications from their neighbors.

This last point is crucial to understanding why this is feasible at all – in general, simulating a general shared-memory multiprocessor machine on a shared-memory multiprocessor falls down on the synchronization overhead. If your simulation semantics dictate that you synchronize every cycle anyway, and you do not try to optimize each core simulator, there is clearly decent room for parallel execution. By including the cache, they increase scalability, since there is more work per target cycle that can be run in isolation.

After reading the article, I am impressed by their work – just getting this to work is pretty good work. But there are quite a few questions which are not really answered in the article and which are crucial to understanding just how well GPGPUs could be used for this kind of ISS work.

• The targeted level of abstraction is a bit confusing. The authors claim it is “instruction accurate and not cycle accurate”, but still simulate caches and cycle-based communications across the NoC. If I read the paper right, communications will take a varying number of cycles depending on the distance for messages to travel. This is more detailed than a typical “instruction accurate” simulator.
• The target system does not run an OS – that might (but I do not know) be an advantage for their approach, since it probably implies less variation in the instruction flow in cores, potentially enhancing the amount of time that all ISSes in a thread group in the GPU can execute the same instruction. This would seem crucial, as if each ISS was running a totally different program, the instruction execution part of the code would be running serialized.
• They should really try to run the same kind of simulation on a high-end x86 CPU like an Intel Sandy Bridge with 8 or more hardware threads. I wonder if their scaling might not work just as well there – and with a much faster serial execution engine. This should give  a much more relevant point of comparison for GPU vs CPU execution of the simulator than…
• the comparison object they use right now, a JIT-accelerated multicore simulation using OVP seems pretty irrelevant since it is not doing the same thing at all. That simulator does not simulate the caches or NoC, just a large number of isolated processors. They also do not run a parallel program on OVP, but rather a large number of single-core fibonacci and dhrystone programs. Thus, the fact that OVP uses a large temporal decoupling time slice does not matter for semantics. It just does not seem like a very relevant comparison point. OVP and their simulator try to solve different problems – fast execution of general code vs. performance profiling of massively parallel machines.
• As I understand it, the given “S-MIPS” numbers in the evaluation tell us the total number of MIPS that we get out across all target cores. That seems to peak around 2000 – which isn’t necessarily that fantastic if we compare to high-performance ISS work in general where a few GIPS is definitely achievable. It is pretty good considering the level of detail here, though, where i would expect a normal ISS + cache simulator to produce at most a few MIPS. Once again, the authors need to be a bit more precise as to what they compare to what.
• Not having an MMU and not implementing any interrupts or exceptions in the target machines avoids a large part of the complexity of a real ISS. That complexity might well be too much for the quite rigid execution environment of a GPGPU.
• They missed that Simics, unique among instruction-accurate mainstream simulators, is parallel since version 4.0.

So, overall, this paper does not really tell us much whether a GPGPU can be used for instruction-set simulation in general. It does tell us that it might be doable, but there are many crucial complications which are not addressed.

He has a wonderful line from an anonymous Convec C3800 programmer who accounted his experience with the optimizing compiler as “having to program as if the compiler was my ex-wife’s lawyer“. I can recognize and understand the sentiment. The C language, in particular, is full of corner cases where behavior is “undefined” or “implementation dependent”. The exact behavior of some statements is also pretty complex – and as a compiler writer, you end up having to work with the rules of the language like a lawyer works the law – the thinking is not all that different. And if you take this approach to the extreme, the description of the compiler as an adversarial lawyer is spot-on.

Unfortunately, it is quite easy in C (and trivial in C++) to wander into such territory without noticing and without doing anything that is obviously wrong for an average programmer. For example, the program I described in my recent post about a bug demo is strictly speaking using undefined behavior, and the compiler is theoretically free to generate a program that reformats the harddrive or does nothing at all. In practice, I don’t think compiler designers strive to generate truly creative unexpected results. Rather, if you detect unclear statement in the program, you would emit a warning.

Still, I can identify with this kind of adversarial relationship with a compiler. It really feels like being in a game with a rules lawyer, where you read the rules (programming language book) and try to figure out how to play the better than your opponent.

First, “Rethink your project planning with a virtual platform“, which talks about how virtual platforms can change your entire project planning. Essentially, by reducing project friction and risks related to hardware availability, software integration, and show-stopper bugs, you can make projects work much better.

Then we have “Transporting bugs with virtual checkpoints“, which is a shorter, popular science, version of the paper I published last year at S4D. This describes how you can use checkpointing in a virtual platform to communicate bugs across time, space, and teams.

It is common practice in multicore computing today to dynamically change the clock frequency of a processor and turn cores on and off in order to adjust the compute power available to the current workload. Such operations tend to be limited in scope, as processors have minimum clock frequencies that make sense, and often the memory system requires all cores to be at the same frequency. Operating systems also tend to want to work with homogeneous sets of cores, as that makes scheduling reasonably straight-forward. This is probably what has kept the idea of “small + large” cores of the same ISA out of the mainstream of SMP design, despite all its advantages in principle.

Now, Nvidia has managed to implement some of that idea in Kal-El.

The key observation is that if you can turn cores on and off, once you get down to a single active core, any system is by definition homogeneous across all cores regardless of what that core is. Changing the nature of this core should then be much easier, since there is only a single core to contend with.

What Nvidia does in Kal-El is to add a fifth low-power core to the main group of four high-performance cores. The fifth core is architecturally identical (ARM Cortex-A9), so that the system state can be moved from the high-performance to the low-performance cores without undue complexities. Indeed, this is all done in hardware, so the OS (typically, Android) thinks it is running on a homogeneous quad-core. When the system is lightly loaded and the OS decides to only have a single core on, the hardware can detect the load is really light, and effectively change the nature of the active core to a low-power-optimized version.

Once more compute power is needed, the hardware invisible slips back to the first high-power core, and then the OS can start increasing clocks and turning on cores as usual. It is effectively the same as a regular ARM Cortex-A9 quad-core setup, but with better low-power performance. The following graph from the Nvidia white paper shows it pretty clearly (red text is my added comment):

Note the slope of the green line: that core is not a good one if you want high performance. It is optimized to scale within a range of low compute-power requirements, rather than provide the best performance per watt at the high end. Using Variable SMP, Nvidia lets us have both.

Neat.

More reading:

• ArsTechnica has a short summary
• There does not seem to be much more right now, everyone is really just reiterating the points from the white paper.

It is way too common to be so busy running around being inefficient that there is no time to think about how to become more efficient. Change also requires some discipline to actually keep pushing at habits until they change for the better.

All of this can be illustrated by tying shoes.

It is pretty very nice demo in which you first start a period program A, which prints the value of an incrementing counter every target second.  You then run a supposedly unrelated program B, resulting in the values that program A prints to become corrupted.  Perfect to show off reverse execution and data breakpoints in reverse as you go from the point where the corrupted value is printed to the piece of code that overwrote the variable.

But then I ported the demo to a new platform… and the bug didn’t work anymore. My bug had caught a bug and was now not working, or at least not they way I expected it to. What had happened?

Very simple. I changed the compiler I used. Since my bug relied on an unspecified behavior in C, the change was totally valid and really expected.  Still, it was interesting to see how things played out… in the end, we got a different bug from the same code thanks to the change.

The code is essentially the following, with some simplifications that make it easier to read for those not familiar with VxWorks, and ignoring all the code to start tasks initially.

// Global variables
int     iDataArray[100];
int     myWdISRcount;
WDOG_ID myWatchDogId;

// Periodic task - program A
void myWdISR(void)
{
  /* Increment ISR invocation count */
  myWdISRcount = myWdISRcount+1;
  printf("wd Fired %d times\n",myWdISRcount);

  /* Start off next invocation */
  wdStart (
    myWatchDogId,
    WD_INTERVAL,
    (FUNCPTR) myWdISR,
    (int) NULL
  );   
}

// Overwrite code - program B
int myCompletelySafeRoutine(void)
{
  uint32_t *a,i;
  a = iDataArray;   
  // This loop writes one word beyond the
  // limit of the iDataArray
  for (i=0; i<=100; i++) {
     a[i] = 0x7fffffff;
   }   
   return OK;
}

In the original setup, compiled for a Power Architecture target, iDataArray ended up right before myWdISRcount in memory.  Thus, the buffer overflow changed the value of the counter from something like 10 to 0x7fffffff.  Very noticeable in the printouts from the periodic task.

When I changed to an x86 target (using a compiler from the same family, but obviously with a different code generator since the target was different), the variable order in memory changed and it seems that we got iDataArray placed last.  Suddenly, the effect of running the safe code was that nothing happened at all.  A bit annoying for a demo.  Some small source-code changes and a recompile later, the effect was instead to crash the target with a triple fault (page fault inside a page fault handler). Seems the program now managed to corrupt some kernel state. While impressive as a bug, it was not quite what I was looking for.

I then changed the compiler type to compiler 2, and the data layout changed once more.  This produced a very useful bug, but it took me a while to actually understand this.  Now, when program B was run, program A stopped.  This looked like a bug in my program, and I actually started trying to fix this – until I realized that this was the bug I was looking for.  Running program B kills program A is just as good a bug as corrupting a counter value, after all.  In this case, the array overrun hits the myWatchDogId variable, and when that gets corrupted, the wdStart call ignores the request since it does not recognize the ID it gets.

So, in the end, I got a bug that was just as good as the first, and arguably a bit more intruiging. It is still obviously a contrived example – but I think that a good demo or lab exercise can be artificial as long as it gets the point across.  Judging from how people who have done the lab reacts, the goal seems to have been
achieved.

The moral of the story is really that compilers are free to change things which are explicitly implementation-defined or not specified at all in the C standard. That is a good thing as it gives the compiler freedom to optimize the code. If you want to control how variables are laid out in memory, I guess you have to resort to linker scripts or similar – but that was too much pain for me in this case. Just changing things around until I got a good bug, and then freezing the binary (and not recompiling it ever again) is a sufficient strategy.

The result: not exactly useful. Netscape 4 was bad back then, and it does not work at all with the current style of web coding.