2003. The embedded tools company Green Hills launched their
Time Machine feature in their well-known MULTI debugger. I consider this the start of commercial reverse debugging, as it was the first
commercial-grade product to include reverse debugging. The implementation was based on tracing the execution of a program on actual hardware, using a debug probe and a “JTAG” debug interface. The trace box would capture several gigabytes of execution data, and then the debugger performed operations based on this trace. To check a backwards breakpoint, you scan back over the trace until you find a matching state or operation (such a memory access or instruction address that is being executed). The main limitation of the method is that the trace buffer can only capture a few seconds of execution on a typical 100s of MHz embedded processor. It only works for a single processor, and it does not capture IO actions (except as memory-mapped IO). It is system-level and cross-target.

Extending this kind of trace to multicore has proven hard, since getting a synchronized trace out of several processors is very hard. There might be debug hardware coming out in the next few years that can indeed support a time-stamped consistent trace of multiple cores, and with such hardware, the Time Machine approach could well be extended into multicore.

2005. Simics 3.0 was launched by Virtutech (later acquired by Wind River and Intel) with full-system reverse execution and reverse debugging. The Simics approach was also unique, being based on a full-system simulator. By simulating the entire target, it is trivial to reverse (and put reverse breakpoints on) changes to memory, persistent storage like disks, and hardware devices. Since all device models in the simulator are deterministic in their implementation, re-executing hardware events like interrupts and IO outputs is just as easy as re-executing code on the main processor, something that had eluded all previous approaches. Recording is used at the interface between the simulator and the outside world, such as user interaction over graphics displays and serial ports and connections to the real-world network. The software stack is unmodified and system-level, and the simulator can handle multiple processors and even multiple machines in a network as a unit. The use case is normally cross-target (even if a system identical to the host can be simulated, it would work like a cross target logically). Time is handled by counting clock cycles on all processors in the system, and reverse debugging can position the simulation at any point in time based on the virtual time.

There is a cost in execution speed from simulation rather than direct execution, and an intrusion effect from running on a simulator rather than on a physical machine. This affects the timing of events, even with a software stack that is not modified. Still, the fact that you can run a complete real software stack with no modifications needed before starting to run the target system is fairly rare in the world of reverse debuggers.

Simics shipped with a modified gdb that talked gdb serial to Simics and accessed reverse execution with some new debugger commands as well as extensions to the gdb serial protocol. This was offered to the gdb community, but not accepted. However, prompted by this, the gdb community started to discuss reverse execution. Some interesting old threads can still be found, such as http://sourceware.org/ml/gdb/2005-05/msg00225.html. Clearly, at that point in time Virtutech did not really explain how Simics worked, and there were some pretty bad proposals floated in the community for how to do reverse. In the end, the gdb serial design did turn out in the right way, assuming
the remote debugger would reverse itself and gdb would just ask it to do so. This separation of concerns is important to creating practical reverse debugging solutions that can use any debugger backend.

2005. Also in 2005, Lauterbach launched the Context Tracking System, CTS. Lauterbach is a big player in the embedded debug market, with their TRACE32 debugger. CTS can be seen as their reply to the Time Machine debugger. CTS is also based on a trace from a hardware unit or from an instruction-set simulator. However, from the available information is also appears to be more limited – you can step back and go back in time and replay forward, but there is no mention of actual backwards breakpoints (even today, six years later).  Thus, I count this as record-replay rather than reverse debug. It is cross-target, system-leve, and uniprocessor like Time Machine.

2006. Undo Software launched the first Linux-targeting host-based reverse debugger, UndoDB. It is described as a bidirectional debugger (the same terminology as the Boothe 2000 PLDI paper). It is user-level, does do reverse breakpoints (and data breakpoints, also known as watchpoints, which is really useful). It handles multiple threads (at least in 3.0), but from the description of the recording technology used I believe they have to serialize their execution. The implementation is based on checkpoint and re-execution, with recording of all non-deterministic events like IO. There is a feature to move to a certain point in time, based on “simulated nanoseconds”. These are not really nanoseconds, but values which are guaranteed to increase even between two instructions (which probably means that they are sub-nanoseconds and on a > 1GHz CPU single-cycle instructions will indeed take less than one nanosecond).

There is a nice description of how it works on their online man page. It is worth noting that they call it “gdb”, but the command set is distinct from what gdb introduced with its reverse execution in 2009. They use the “b” prefix for backwards commands rather than “r” for reverse.  In some way, UndoDB is in direct competition with the gdb reverse target, but it is much much faster and has more features.

2008. The Rogue Wave (at the time, it was an independent company) TotalView debugger gained support for reverse debugging, with the ReplayEngine add-on. TotalView is an old mainstay in the HPC market, having been around since at least 1993. Indeed, it was developed initially for the BBN Butterfly computer, and thus it might have had a touch with reverse execution as far back as the 1987 paper cited in my previous blog post.

Judging from the available materials, TotalView can clearly can step back in various ways. However, it is not clear that it triggers breakpoints when going backwards. Thus, it has to count as record-replay debugging rather than reverse debugging. The base of the implementation is extensive instrumentation of the the runtime system of the target computer.  The implementation builds on the fact that the target programs tend to b clustered programs that use MPI to communicate – and thus a large part of the communication between threads is explicit and easily intercepted and recorded.  There is also an existing infrastructure of checkpoint and restart for parallel programs using MPI to support fault tolerance that was used as the base of the implementation.  Finally, in a slightly ugly hack, they make each multi-threaded program run on a single processor by a big lock. In this way, all that needs to be replayed is the interleaving of threads on a single processor, a far more tractable problem compared to trying to replicate a true parallel execution in a new session.

2008. VmWare officially launched a record-replay debugger based on their virtual machine technology with VmWare Workstation 6.5. Single-processor, system-level (but really only supported for user-level debugging), cross target (since the VM is not really the absolutely same hardware as the host), time model is based on the virtual machine which I believe is cycles-based. Mostly used for record-replay debug of non-deterministic software bugs, but could also do reverse debugging including reverse data breakpoints. Based on snapshot and deterministic re-execution, plus recording of all non-deterministic device accesses (not all devices in the VmWare hardware emulation layer are deterministic). Going back to a snapshot was a very heavy operation (I tried it) since you had to restore the entire target memory (quickly got into gigabytes). The hardware supported in the VM was quite limited, and things like CD-ROMs and floppies could not be part of a record/replay session. Replay logs could be moved between hosts.

The VmWare reverse debug functionality was removed from VmWare workstation version 8 in 2011, since it required a large investment and was not apparently used by very many VmWare users. This indicates that trying to build developer-oriented functionality into a technology base that is fundamentally driven by the need of deployed virtual machines was hard. There are contradictions between these two goals, as the determinism and control needed for a good reverse debugger is not necessarily consistent with maximum performance for running virtual machines in a production setting.

2009. gdb 7.0 added support for reverse execution (a work that began in 2006). The built-in “record” target supports reverse debugging on user-level single-threaded programs on the same host. The command set for reverse debugging is fairly full-featured, but is a bit quirky with a “set direction” command that makes regular run-control commands work in reverse. The record technology is quite slow since it basically records the effect of each and every instruction run in the program.

In addition to its built-in target, gdb can also control external reversible debug systems over the gdb serial protocol. This made the changes to gdb-serial created by Virtutech for Simics in 2005 part of the mainline gdb release. Several tools support the command set, including VmWare, UndoDB, and Simics. There was also a set of MI commands added to basically let Eclipse use gdb as a backend for reverse debug, including using it to control external tools via gdb-serial. How this happened is quite a long story, and I made a small contribution to the gdb code base myself in the process. Read about this here.

2009. Eclipse CDT added support for reverse execution, using gdb 7.0 reverse as the initial backend. As noted above, this lets Eclipse also use other reverse debugging backends (Eclipse uses the gdb-MI interface to gdb to control the debug session). This is noteworthy since it meant that the buttons to control reverse execution are now part of the CDT, making it much easier to use Eclipse to build a frontend to any reversible backend. Eclipse is not really a debugger, just an interface to a debugger.

2009. Microsoft Visual Studio got record-replay debugging with IntelliTrace. It is strictly about replay debugging, including the nice ability to send traces around between developers. There are no backwards breakpoints. The support is limited to programs running on top of the .net runtime system, meaning that it does not apply to classic Windows software. Using the CLR virtual machine as the implementation basis should make the implementation easier, cleaner, and faster compared to a machine-level native solution. It is user-level, single-threaded, and host-based. Time concept is unknown.

2011. Adobe demonstrated (not launched) reverse debugging in their Flash Builder programming environment. A nice video is posted on the Adobe website. Seems to be based on the virtual machine that flash runs on, and includes what looks like pretty powerful backwards data analysis tools. In a blog post, the developer describes some of the features, which to me seem to indicate some pretty heavy recording.

Final notes.In researching these commercial tools, there also seems to be a lost one. A company called Visicomp launched a Java debugger called RetroVue in 2002 which supposedly did allow backwards debugging in some way. However, it seems that this tool was not really practical, being too slow for actual use. It seems to have disappeared since without anyone picking up its legacy. The technology was apparently pretty much like the Omniscient Debugger presented in 2003 and which I described in the blog post on reverse execution research.

When searching through the literature using the ACM Digital Library (and some Yahoo and Google web searches), I find quite a large body of research literature dating to the late 2000s. Apparently, the appearance of commercial reverse debuggers have inspired research into the topic – and our increasingly powerful machines have made heavier solutions feasible (at least for single programs).

Disclaimer – this is not really an exhaustive survey of the field, but some of the more interesting papers that I have stumbled on.

1973. The first actual mention of reverse debugging I have found is a short 1973 Communications of the ACM article called “Reversible Execution” by Marvin Zelkowitz. Basically, this paper proposed to apply a backtracking facility built into a PL/I compiler to support AI research (a precursor to Prolog execution semantics from what it sounds like) to also replay parts of a normal program for debugging. This in essence would enable replay of a program execution, based on a log of how the program ran and the values of variables changed. Basically, a forward-only record-replay solution for user-level single-threaded programs.

1987. Thomas LeBlanc and John Mellor-Crummey, “Debugging Parallel Programs with Instant Replay“, IEEE Transactions on Computers. This paper covers record-replay debugging of parallel user-level programs on the BBN Butterfly. They record all interaction between processes (threads) by intrumenting the OS calls that are used to communicate and synchronize between processes. Programs also sometimes had to be changed so that their communication protocols allowed the instrumentation to work properly. Thus, the approach did not apply to unchanged code either at source or binary level. Processes still compute their results, only the order or interactions are enforced. In this way, the overhead was kept usefully low. The approach does not work well for programs that deal with asynchronous interrupts and hardware interaction by the software, since there is no good way to hook and replay such events in their implementation.

The most important concept here is the use of re-execution of code to reconstruct values of a program, assuming deterministic computation as long as asynchronous interactions can be controlled. This is the basis of all reverse execution approaches based on checkpointing and re-execution. Not sure if they invented the idea, but this is the earliest distinct mention I have seen of this.

1988. Stuart Feldman and Channing Brown, “IGOR: a system for program debug via reversible execution“, 1988 ACM SIGPLAN and SIGOPS workshop on Parallel and distributed debugging (PADD) . This paper looks at the problem of how to debug user-level single-threaded code on a UNIX-like system. The programs need to be recompiled to add a small amount of support, in particular to help with restarting
execution. It is essentially a record-replay approach, but capable of placing the execution of a program at any point in its past execution. They save regular checkpoints during program execution (every tenth of a second or so seemed appropriate in their testing). The checkpoints only include the data that has been changed since the previous checkpoint (they added an OS facility to mark all pages that a program writes between checkpoints). A unique feature is the ability to look at the value of a variable as it is stored in successive checkpoints, and to “go to the point in time where variable X has value V” (assuming a variable is monotonically increasing or decreasing, like an outermost loop counter). To do the detailed positioning, they use an interpreter for the target system, essentially mixing native and interpreted execution in their debugger.

1996. Mark Russinovich (who I believe is the man behind the recent novel “Zero Day“) and Bryce Cogswell, “Replay for Concurrent Non-Deterministic Shared-Memory Applications“, PLDI 1996. In this paper, instrumented system libraries and binary code modification is used to enable deterministic replay of multithreaded applications. The binary modification is used to implement an instruction counter to provide a logical time-base, and instrumented system libraries are used to force a serialized (but interleaved) execution of the program. It does not run the program in parallel on a multiprocessor, but rather runs it on  a single processor. User-level, instrumented and intrusive, parallel on single processor.

2000. Bob Boothe, “Efficient Algorithms for Bidirectional Debugging“, PLDI 2000. I actually saw this paper being presented live at the PLDI conference in Vancouver. At the time, I was working on embedded compilers and debuggers, and our little group from IAR looked at what was being presented at said “yeah, might work on a big desktop, but you can’t do that in a real machine”. Why was that the case? Essentially, what Boothe did was to use the Unix fork call to spawn off checkpoints of a program as it was executing. This was a smart way to achieve checkpointing using existing OS facilities. In addition, he recorded the results of system calls to replay them during replay.

Time was given by counters in target program (compiled-into it as part of the debugging build). Most of the paper was spent on basic algorithms for how to do “previous” (step back 1 line), “before” (inverse of finish), “buntil” (backwards until, looking at data conditionals to determine when to stop). His use of “backwards” rather than “reverse” as the name of the concept does make this paper a bit harder to find. While limited to a single threaded user-level process, this paper is probably the first to introduce reverse debugging in the sense of the primary user interface being the ability to step backwards and check for breakpoints backwards in time. A good idea in the UI design was the undo command that jumped the debugger back to the last position where you stopped. Having worked with reverse debugging quite a lot myself, this operation makes a lot of sense as an addition to standard run-control commands.

2002. Tankgut Akgul and Vincent Mooney, “Instruction-level Reverse Execution for Debugging“, at PASTE 2002. This paper is an odd one, as it was neither really reverse nor practical. The approach is based on doing a static analysis of the assembly code of a program to generate code that could reverse operations based on some logging of destructive operations. Interesting, but not really practical as I see it. One noteworthy tidbit in the paper is the measurement that logging only destructive operations is some 50 x more efficient than logging all changes.

2002. George Dunlap, Samuel King, Sukru Cinar, Murasa Bazrai, and Peter Chen, “ReVirt: enabling intrusion analysis through virtual-machine logging and replay“, at OSDI 2002. This paper uses the User-Mode Linux (UML) paravirtual machine to checkpoint and replay an operating system. I think this is the first use of a true virtual machine to achieve reverse execution, even though it is a paravirtual machine rather than a full virtual machine. The replay starts from a single checkpoint at the start of execution of the guest system, so moving to a certain point in time can be very time-consuming. It does support reverse breakpoints. To move to a certain point in time, they count executed instructions, not actual time. The use of a paravirtual solution precludes the use of many real device drivers and debugging issues related to interaction with most real devices. It is uniprocessor, system-level, unmodified target (except the change to make the OS itself run paravirtualized on top of UML).

2003. Bil Lewis and Mireille Ducasse, “Using events to debug Java programs backwards in time“, OOPSLA 2003. This paper targets programs running on top of a Java Virtual Machine, resulting in an Omniscient Debugger. They instrument the target program to log all state changes. A big lock is used to serialize all threads into a single interleaved execution. The implementation is admitted to be almost worst-case inefficient, but the debugger was still considered useful for some real work. The debug UI does feature backwards breakpoints, so it qualifies as actual reverse debugging.

Note that in principle, backwards debugging and record-replay should be possible to implement quite efficiently for a language virtual machine like the JVM, if you can modify the VM itself. This has indeed been done today, for example by Microsoft for their CLR and in the use of full-system simulators (and other simulators) to implement reverse debugging.

2005. Samuel King, George Dunlap, and Peter Chen, “Debugging Operating Systems with Time-Traveling Virtual Machines“, USENIX 2005. This paper builds on the foundation of ReVirt, but adds checkpoints during a run to make the replay to a certain point in time more efficient. They also introduce the differential storing of disk state in the checkpointing (just like done in Simics). Overall, it is very similar to the replay approach taken by VmWare a few years later. Still single-processor. The paper contains a nice list of example bugs for which reverse debugging work much better than cyclic debugging. They also reference Simics as something that might do something with reverse, but do not quite know what (this is right after the launch of Simics reverse execution as discussed in the next post).

2007. Paul Brook and Daniel Jacobowitz, “Reversible Debugging“, GCC Developer’s Summit 2007. This paper presented some early work on implementing reverse debugging inside the Qemu full-system simulator. Logically, it follows in the steps of the Simics reversible debugger announced in 2005 (see the third post in this series of blogs). They use gdb as their frontend, and the paper introduced the idea of setting the default execution direction of the debugger. This UI choice was later adopted in the reversible gdb 7.0 released in 2009. The paper contains a good discussion of some of the practical issues involved in performing operations like reverse-step to go back a line, or stepping back up through a function call. It is a good explanation of the basic problems.

The actual reversing is implemented in Qemu using a checkpoint and deterministic replay, along with logging of any operation that is not deterministic. Their main test case is actually using the semihosting variant of Qemu where IO calls are intercepted and passed on to the host. Just like all other such solutions, they record the return values and do not repeat side-effects. It is system-level, single processor, unmodified target software, and time is based on counting executed instructions.

The Qemu architecture makes enforcing reversibility across all devices fairly difficult, since each device manages its own interaction with the user. One interesting tidbit in the paper is that Qemu by default triggers timer interrupts based on host time, but these had to be made fully virtual to be reversible.

Summary The above is a small selection from all the papers published on this topic, but I think they capture the most common methods used. Basically, almost all of the approaches require some kind of change to the target software to enable reversibility. Interestingly, the first few commercial solutions for reverse debug that appeared did not take that approach, but rather targeted unmodified programs using a different set of fundamental techniques. See the next blog post for that part of reverse history.

Let’s start with background and definitions.

To me, the key defining factor of reverse debugging is the ability to apply breakpoints in reverse – essentially, to be able to go to the previous occurence of a breakpoint just as well as going to the next. The implementation allowing this might vary.

The contrast to reverse debugging is classic cyclic debugging where you run and rerun a program with a problem, looking at variables and setting breakpoints during each run to zoom in on an issue. For cyclic debugging to work, you pretty much require each run to behave the same way. The program has to be fundamentally deterministic. This is
usually the case for non-interactive single-threaded programs, but not the case for real-time programs, parallel programs, or programs that involve some kind of asynchronous input/output (reading and writing files is a special case of IO that can indeed be deterministic since there is usually no interference from the environment in those operations).

Thus, for non-deterministic programs and rare errors, reverse debug is what you want. Run once, hit the error, reverse to diagnose it.

In addition to reverse debug and cyclic debug, we also have record-replay debug. In such a system, you record an execution and later replay it forward. Debugging is strictly forward: you cannot step back in time instruction by instruction, nor trigger breakpoints backwards in time. Record-replay debug is a way to allow cyclic debugging on non-deterministic highly variable program runs. This can be implemented with a lot less debugger complexity than a true reverse debugger, even if the underlying runtime system is almost identical to reverse debugging.

So, how do you implement reverse debugging?

Reverse execution is one way to implement reverse debug. In reverse execution, the execution system has the ability to move backwards in time and put the entire system in the state it was in at some previous point in time.  You can step the system back instruction by instruction or cycle by cycle.  You can also continue the execution
forward from any point in time, throwing away the history of asynchronous inputs to actually take a new execution path.

Typically, reverse execution is implemented by using checkpoints of previous system states plus a deterministic way to re-execute the system from those checkpoints. The key technical problems to be solved is how to checkpoint the system and how to make re-execution deterministic (note that determinism does not imply invariant or predictable behavior, just that you can reliable recreate a particular execution). You also need to record and replay anything that you cannot re-execute, such as user interactions or network communications with the world outside the controlled system.

Schematically, it works like this, note how simulation time always moves forward even though the logical system time sometimes moves backwards:

Record-replay debug is usually implemented in a way similar to this, without the ability to go back in time and usually without the intermediate checkpoints.

Trace-based debugging is based on recording everything a system does into a trace, and once the recording is done, debugger works on the trace rather than using the log to drive an actual system.  Reverse debug is implemented by recreating the state of a system by reading the trace, and finding points in the trace where breakpoint conditions are true. This is also known as post-mortem debug, since you debug after the target system has finished executing (typically). A log can be captured by a hardware device, or by software being instrumented to log everything that is going on. The technology can also be used to implement record-replay debugging.

There are some other important distinctions between different approaches to reverse debugging that we need to keep in mind as we review the history of the field.

• System or user-level? Do you debug a system, including the OS, or just a user-level application running on top of the operating system? User-level debug can often be solved in simpler ways than system-level debug, but also suffers from some limitations. There are also times where system-level is simpler.
• Single-threaded or multi-threaded? Can you debug multiple threads, processes, processors, or just a single processor or user-level thread? A single thread of control greatly simplifies the problem.
• Cross-target or host-based? Do you debug programs running on the same host  as the debugger, or can it also target remote targets or cross targets (such as embedded systems)?
• Instrumented or unchanged programs? Quite a few solutions for reverse debug tried over the years involve compiling programs in a special way, using instrumented OS libraries, or other implementation variants where the program debugged is not identical to the eventual deployed program.

Given this background, the next post will cover early research, and the third post the beginning of commercial products.

Note that on Stack Overflow, reverse debug does not seem to be particularly big thing still. The reverse-debugging tag has all of 10 members, while other topics like C# and Android programming has millions… so maybe reverse is not quite mainstream yet. Or maybe it is just the case that Stack Overflow has more web-style developers than low-level developers in their membership.

As David Alan Grier would have said, this kind of event tends to serve to build the professional identity of a group of people. Computer science is not a profession per se, but it is clear that the Uppsala computer science students (almost 2000 has started since 1981) has a particular culture that comes back to us very easily when amongst our peers. I think it is based in the art and practice of programming.

The talks and discussions during the dinner often went back to the defining experiences of our student days, and these experiences were mostly about night hacks and classic labs. A speaker told us about how in 1983 they tried to program a simple computer built by the department itself, and how it turned out that 3 out 4 machines were broken. There idea that they expected the program to just work the first time it loaded made me look up my favorite debugging quote. Somebody reminded me (19 years after the deed) about the time I made a Prolog program exhaust all memory on a Sun server, by performing an exhaustive search for a problem with no solution).

I remembered how some students I taught (while still an undergraduate myself, a common practice at DV) spent 24 hours a day for almost a week in a lab room trying to get their operating systems to boot on some MIPS-based lab machines. In particular, the group that was gripped by ambition and tried to turn on the MMU. Each time they reset the machine and tried to boot their OS, they would see a serial terminal spewing out diagnostics text and then stopping cold… as they failed again to make it work (they passed the course anyway).

This all indicates the fundamental importance of programming to computer science students. That is also what we believed was our core mission when I was a student – to go out into the world and create great software. Many still do, even if quite a few of us have left day-to-day coding to become project leaders and outright managers. Can’t say I program all that much myself, apart from some demos and virtual machine scripts, but I still find the topic incredibly interesting and important.

The event also touched on institutional memory and the longevity of data. The very ambitious anniversary committee had produced a brand new version of the classic “Manualen”, a song book first produced in 1995. In it, I found a text I wrote in 1995 about what a computer scientist actually does (a bit pompous, as can be expected from a proud student) as well as a photo of myself from a 1996 cover of the DV student magazine “Blurgel”. However, in both cases, these had been reproduced from paper copies. There was no digital memory in place of these 15-year-old pieces of data.

I actually still have both paper copies in good shape and the data – on a multisession CD-R created in 1999 on a Mac. My current computers all being Windows-based cannot extract the data. I know a Mac can still read them, but it is not clear that the data can be used today. It is likely in some old version of Aldus PageMaker and with no file extensions to guide you as to what each file is. The images are greyscale or black-and-white high-resolution TIFF files I believe (from the era before PNG), once again with no file extensions to hint at what is what. It underscores the importance of actual running programs and systems as a way to access our digital past. The data might all be there and readable, but with no software to interpret it, what can you do? I actually noted the same four years ago for the somewhat late 25 years anniversary.

Overall, a very memorable evening, and kudos to the organizers for putting it all together.

The result: not exactly useful. Netscape 4 was bad back then, and it does not work at all with the current style of web coding.

So when Software Engineering Radio episode 177 interviewed the i chief architect Steve Will, I was hooked. It turned out that IBM i was cooler than I imagined. Here are my notes on why I think that IBM i is one of the most interesting systems out there in real use.

Independence of Processor

IBM i uses a byte code format for application programs. This byte code (known as technology-independent machine interface, or TIMI) is quite unlike what we have in the JVM or CLR. First of all, it predates the JVM by about 15 years. The first generation of systems, the IBM Series 38, came out in 1980. Second, the TIMI code contains many higher-level operations like database accesses, making it possible to generate far better executable code than if it was just plain API calls. Third, it is compiled before execution, and not just-in-time.

The TIMI was designed as the designers even in 1972 realized that processors will come and go, but software will remain. I would guess that the IBM experience with the System/360 and migrating software to it from older system had something to do with this.

Over its life time, IBM i has gone from the original System 38 CPU to the AS/400 CPU to a customized 64-bit POWER CPU (still called AS/400), and now to a completely standard Power7 processor. Indeed, I did not realize until now that IBM merged the pSeries and iSeries hardware in 2008. Today, IBM i is just a software stack running on a hardware platform that can just as well run AIX or Linux! That is quite a journey for a system over 30 years, and proves that the original design was amazingly sound.

There seems to have been at least one slight imperfection in continuity. The original design used 48-bit pointers, which was very far-sighted for a design team in 1972, when the biggest machines around used 24-bit pointers (like designing a 128-bit pointer into a system of today as the default). Still, this did become too small, and a change was made to 64-bits in 1995 (when the almost-PowerPC RS64 became the processor for the AS/400).  Apparently, this required some side-band information about a program to allow patching even the TIMI code in the right way.

Integrated Database

The i OS (I guess Apple just beat IBM to the trademark, since iOS would seem the natural name for the OS, right?) is an integrated environment that tries to do a lot of things for the user that would normally require third-party software. In particular, it has a database integrated, which can both be accessed from i APIs and lately over SQL. It is branded as “DB2″ and DB2-targeting programs sees no difference between it and DB2 running on AIX. But according to Steve, the core is not DB2 but the database that was built into IBM i from the start.

Integral Security

Where IBM i really stands out is in the decision to forgo the traditional concept of a file system and instead rely on an object storage concept. This has tremendous advantages for security. Both since access rights are powerful and attached to objects, and by avoiding all the dangers of a typical file system. For example, there is no way to make a document executable. Programs are programs, data objects are data objects, and you cannot make a Windows .exe masquerade as a .jpg. All users are associated with a user profile indicating what they can do and work with.

This does require some special treatment for users like programmers. Programmers are always a problem, since they need to create new programs. Same thing with the no-execute protection in recent Windows operating systems and just-in-time compilers. The i solution is to have a special programmer role with special permissions.

Importing APIs

Just like IBM zEnterprise (the latest name for the heritage from System/360), the IBM i system has been modernized in recent years by adding support for many standard APIs and concepts from mainstream computing. They can run Java and use JDBC, for example. IBM does not seem to hesitate to help programmers reuse code written for other platforms on their heavy-duty machines.

A funny part for i is that they had to add a virtual file system in order to make Java happy. Apparently, a JVM cannot work or run most programs without accessing a file system. So, Java pretty much assumes the machine has a file system. Typically a true assumption – except on IBM i.Thus, IBM i Java machine simulates your average hierarchical file system on top of its real data storage mechanisms!

It is also interesting to note the choice in programming languages added to the platform. Java is a given, but IBM has made a big splash around PHP! Turns out that many business applications are migrating towards that kind of web-based platform. PHP replacing COBOL? Not sure that is an improvement…

Internal Tuning

One design goal of IBM i from the outset was to create a system that would be easy to use. In particular, the need for system administrators should be minimal. I don’t know how well this works when it comes to dealing with adding users and things like that, but I guess that if you use roles appropriately, it will be hard to mess that up.

What is more interesting and subtle is the extent to which IBM i tries to avoid needing system administrators around to tune the machine performance. Normally, if you have a large database, you will need to manually tune and tweak the system for maximum performance. In IBM i, the idea is that the system takes care of that for you. There must be a lot interesting algorithms at work in the core of the system for this to work, but apparently it does work.

For example, the handling of the storage hierarchy is transparent to programs. A program allocates an object, but has no idea if it lives in RAM or on disk. The system moves things around as needed to reach the performance needed (you set goals for each subsystem). When solid-state drives were added to i a few years ago, that just introduced yet another level of the storage hierarchy, and the OS core took care of managing it. User programs did not need to change at all. That is pretty cool!

My gut feeling is that this is thanks to the higher-level APIs compared to many other systems, which gives the system a clearer view of what a program is trying to achieve. Working on system-defined objects with known types sure beat trying to make sense of uninterpreted strings of bytes coming out of your typical file-system-oriented program.

Overall, IBM i impresses me by implementing a series of unique and innovative technologies that is largely different from the more well-known UNIX-style of OS design that rose in parallel to the development of IBM i. It demonstrates that there are technical alternatives to the mainstream, and that doing things differently can indeed be a very goo idea. Refreshing, in a world where too many things are me-too designs that just follow the majority herd of thinking.

The title of this blog post is based on one fun little tidbit in the article. Steve actually keeps a functioning BBC Micro system around in the shape of an emulator.  But the emulator is not running on his current PC directly, but on top an emulated Acorn Archimedes machine.  A machine from 1981 emulated on top of a machine from 1987 that is in turn emulated on top of a machine from this year.  A real-life example of how nested emulation can help save our digital heritage (an interesting topic I blogged about before).

He also has a physical BBC Micro around, and it is impressive just how high-quality construction could be back then. I still believe the old IBM keyboards are the best ever made, while the cheap laptop I am writing this on probably costs less to manufacture than that keyboard did… obviously at the expense of keyboard feeling.  Grumble grumble kurmudgeon me, right? Steve loves the way the BBC Micro keyboard was designed, and claims that it could hold up to 10 years of abuse by children. In contrast, my old ZX Spectrum had some 4 keyboard membranes worn out in the six years or so I used it. But that is what cheap gives you.

He also touches on the topic of just how inaccessible computing is getting in terms of getting to grips with the machine and the hardware level. Back in the 1980s, you could pretty easily build things and attach them to your home computer.  Slow buses and low-level access and no stupid operating system getting in your way facilitated that.  He proposes people using a PIC development kit, but maybe an Arduino is a better choice.  I like LEGO Mindstorms, but that is indeed a very high-level view of hardware and robotics.  Not at all like the direct programming experience I had with my first ZX Spectrum back in 1984.

It is also funny to read how 1983-era 16-bit processors were not keeping up with memory – too complex instructions made the machine use memory too rarely. How the world had changed!

Recommended reading!

Why do I think it would have been impossible?

In the white paper, some techniques are cited which were published in the late 1990s. Thus, they weren’t even around when the Commodore 64 was new. We have come a long way in our understanding of digital signal and sound processing since 1982. Thanks to the incredible increase in available computing power, many techniques have been invented over the past few decades that were not available to the first generation of home computer hackers.

A more interesting reason is the amount of compute power needed to prepare the samples. Due  to the tiny memory of the Commodore 64 (in 32kB you cannot store more than a few seconds of audio samples at any kind of reasonable resolution and sampling frequency), as well as the way the samples are played, the audio is heavily compressed and preprocessed before being loaded into the Commodore 64.

This process is reported as having taken 25 minutes on a modern 2 GHz PC. How long would that have taken on a Commodore 64 in 1982? The clock frequency is 2000 x higher. If we assume this is plain floating point math, the PC can get about 1 operation through each cycle on average. The C64 would probably require at least 1000 cycles per floating point operation, emulating it using 8-bit integer math. That gives us a factor of 2,000,000 (give or take an order of magnitude). Thus, the computation would have taken  about 95 years (or as little as 10 years, or as much as 1000 years) back in 1982 (I don’t think this kind of operation is easily parallelized). It would also have required an unheard-of incredibly large memory of several megabytes.

OK, so maybe if you were  a billionaire and could have rented a few IBM mainframes you could have done it. Just to stand before a bemused crowd showing the little box making music. Would have been much easier to just buy an early CD player which played music at much higher quality.

But don’t get me wrong – I really like this work. It is impressive to see what you can do in terms of media replay on a small and slow device if you are willing to put incredible effort into the compression of the media. The way that the author finds ways to achieve various effects within the confines of the Commodore 64.

When retro computing is at its best, it shows how human ingenuity can overcome limitations and get things done even when it should not be possible. This kind of engineering spirit has been crucial many times in the past, in particular when saving space missions from seemingly impossible odds. If you only work with systems with power in abundance and few hard limits, you just don’t develop that kind of creativity.

You really should read the article to get the full story. The short summary is that while the basic principle of the graphics display is very simple to simulate, the effect of rewriting the display contents as it is being drawn on the CRT is quite difficult to get right.

I found this picture of the system online, for reference of what the graphics might look like:

The hardware of the TRS-80 works pretty much like my old ZX Spectrum did: a bit of memory is used to hold image data (single buffer), and then the video hardware simply reads this memory as the CRT scan goes by to display the right picture. There is no locking of the display memory during this time, so the processor can race the video hardware and modify the contents of a location in memory between scan lines. This is indeed done, and that’s what makes simulating the machine much much harder.

On the TRS-80, racing the scan makes it possible to (for example) increase the apparent vertical resolution of the display (since each graphics “pixel” actually consists of four vertical pixels that could not be individually addressed). On the ZX Spectrum, you could increase the apparent vertical color resolution. On the Commodore 64, I think you change the graphics palette during redraw to allow more colors to be displayed simultaneously.

For a simulator, this is pure pain. The result of graphics code written for those machines essentially depends on making a very precise simulation of the timing of all processor instructions, as well the behavior of the video hardware. As noted in the article, you need to model the memory access contention resulting from the processor writing as the display memory simultaneously with the video hardware reading it. You need to know the cycle count of each pixel, and the setup time between each row of pixels. To account for effects like dithering on a modern perfectly stable LCD display, you have to apply filters to the basic bitmap, simulating the fuzziness of the old cheap TVs that these computers used to drive. If you want to simulate the horrible tricks used to make music on the ZX Spectrum’s 1-bit sound output (as I recall it, you made noise by flipping a bit in an OUT instruction on the Z-80 CPU), you probably need to do a bit of analog waveform simulation.

Indeed, it seems to me that one enabler for today’s virtual platforms is that we have hardware that is not entangled in low-level timing like this. Instead, thanks to the variability of execution time in modern processors, hardware is asynchronous and tends to use interrupts or status bits in registers to report when a requested operation is complete. You do not see code of the type “do X, wait precisely Y cycles, and then do Z” anymore, and that really helps in changing the target system timing to enable fast simulation – which is what any fast virtual platform has to do, replacing a real processor with variable timing with an ISS with pretty simple instruction timing. It also means that hardware models can be simpler too, since they do not model how the hardware achieves its work, only what that work is. To loop back to the article prompting this blog post, in a modern virtual platform, all you would simulate is the specified effect of setting graphics bytes in memory. Not the incidental effect of how it is drawn onto a TV, scan-line by scan-line.

There are many fascinating aspects to the machine and its utter simplicity, but one that struck me as I read the paper was that so many of the fundamental ideas of programming and practical computing were invented then and there. Indeed, the EDSAC was designed as a machine to experiment with programming, rather than as a machine for maximal computing performance.

In particular, the way programs were entered into the machine was much more sophisticated (or desperate due to a lack of computing resources) than any other contemporary system. In hindsight, the “initial orders” of the EDSAC merged three major inventions:

• The use of a “bootloader”, a program that starts the machine and then loads the “real” program into memory. Other machines of that era seems to have had special loader hardware, or just used Hollerith punched cards and put the results into memory.
• The use of a symbolic code for instructions on the input tapes, not just a binary. Thus, the initial order also constituted an assembler. The binary instruction format of the EDSAC was never seen outside of the machine itself.
• The relocation and rewriting of the program as it was being loaded, essentially making it the first program loader and linker.

All of this was achieved in 40 instructions and 1 constant in the last version. A very impressive feat of programming, maybe one of the most impressive in the history of computing.

For those unfamiliar with the Stretch machine, it was a supercomputer developed by IBM which introduced many of the performance techniques and basic computer technologies that we all use today (most of them handed down to us via the IBM System/360). For example, it was the first to use 8-bit bytes and 64-bit floating point. It also introduced memory protection, memory interleaving, and instruction prefetching.

More relevant for my blog is the fact that the Stretch used the world’s first pipelined main processor, complete with interlocks to maintain program-order semantics. When developing this pipeline, Frederick Brooks claims that IBM developed a program to simulate the pipeline. This simulator was used to test the performance of the pipeline design on various test programs (this was before they were called benchmarks), and tune the design accordingly. The simulator was created by Harwood Kolsky. There is no firm date for the pipeline simulator, but based on the development time of the Stretch, it can be dated somewhere around 1960.

Thus, the simulation-driven approach to computer architecture is about 50 years old by now. Should have gone to ISCA and used this as an excuse for a party I guess…

It is also interesting to note that the Stretch computer acquired a co-processor in 1962, to do cryptology work. This machine was the one-off IBM 7950 “Harvest” and was tailored for the needs of the NSA in the US. It was a seriously special-purpose hardware unit adding a few instructions to the Stretch machine, and beating any other machine at the time by about 50 to 200 on the particular NSA workloads.  Sounds like the kind of performance claims that Tensilica and other application-customized processors claim. 50 years ago.

Please be kind to any mistakes… I am trying to paint a broad picture for a computer-history-ignorant audience here.

The silly thing is that I don’t have a strong recollection of it. My most vibrant memory from that tumultuous Fall is from December, as we saw tanks roll through the streets of Bucharest on TV. I saw it on the American Armed Forces Network, to be precise, a night on the base with my friends from Keflavík Naval Air Station, also known to the Icelanders as “the base”. This was during the year I lived in Iceland, and went to school on the AT Mahan High School on NAS Keflavik. Sometimes I would sleep over on the base with some friends, despite being a “neutral” (non-NATO). But I digress…

The impact of the wall coming down was felt much more clearly the next year, when me and some friends drove down to Berlin. The wall was gone, but the space it had left was still there. The trade in East Bloc memorabilia was rampant, and we had several East German military caps and some cool medals with us as we drove home. Indeed, for the next few years, East Bloc and Soviet stuff was all the rage.  I actually found the cap when we cleaned out some boxes of old memorabilia the other week, guess it was fate. Or just coincidence.

It feels strange to know that the freshmen in university this year were not even born then. They have no memory of the cold war, and newspaper articles carry sidebars about “this was the DDR”… to the youngsters of today, DDR means dual-data-rate RAM, not the Deutsche Demokratische Republik.

1989 was also the year that I got my first Macintosh. An SE, with a 8 MHz 68000 processor and 2 MB of RAM, and 20 MB of HDD. It is dizzying to compare it with the machine I have at home today, a cheap Core i7 Dell. The processor is probably around 2000 times faster (300x clock, and at least 6x more efficient per clock), the memory is 4000 times bigger, the HDD 30000 times larger. Still, it does not feel as magic as that first Macintosh did… The next year, I got myself an SE/30, with a 16 MHz 68030 and an 68882 math co-processor. Screaming machine at the time, total clunker by today’s standards.

Finally, what other big events have I seen, events that you can peg a date and a memory on? The second biggest one is the killing of Olof Palme on Feb 28, 1986. Then we have “September 11″ which was quite a shock. Chernobyl. U137. I have no personal relationship to the other two big recent disasters of Sweden, the Estonia disaster and the Thailand Tsunami, but they could have counted otherwise.

Still, the end of the cold war is a big deal, and the way in which it opened up Europe to what it is today.

The article is highly recommended for its long historical look at network processing and network processing offload. As the balance  between speeds of networks, processors, memory, and interconnects between network cards and the rest of the system has changed over the years, it is an idea that occasionally (four or five times since the 1970s) has made sense. However, in the end, Mike thinks that it usually does not, and for a machine with multiple cores and a modern fast interconnect, it is hard to see how a hardware accelerator can actually help speed things up much when the coordination between the hardware and the software is accounted for. Even if there would appear to be a big bottleneck somewhere today, we can be sure that it wil be removed in the next generation of hardware, rendering the market window for an accelerator quite short.

I read this article as another great motivation for the need to carefully consider the functional design of the hardware-software interface for acceleration devices. For simple data-pumping or media-processing units, this looks easy. For something as complex as TCP/IP processing, it is not. I think the key is that for TCP, we have something that is much more like control-plane processing than data-plane processing, and that is harder to efficiently integrate between hardware and software. Also, there is not really that much work left to offload once data copies have been architected in the right way (and I read Mike’s article to say that we now know how to do this in a sufficently few-copies way that software is close to optimal in architecture).

From a market perspective, it would also indicate that the acceleration circuits that are in common use today are by definition those that make sense. Having hardware-accelerated graphics and video decoders does seem to help build more efficient and attractive computer systems, as do cryptography accelerators. With this view, it will be interesting to see which of all the accelerators found in modern networking SoCs like those from Freescale and Cavium will survive the test of time. I am willing to put a small bet that pattern-matching engines for traffic inspection is one of them. Apart from that, hard to say.

So go read that article before you start designing your next brilliant accelerator for a common expensive operation.

It also reminds me of a whitepaper I wrote early this year on how to evaluate performance of a hardware accelerator in the context of a full system with a full software stack, considering the details of the hardware-software interface.

The Spectrum came with a built-in BASIC interpreter that are up the bottom 16kB of the 64kB addressing space. The BASIC was actually fairly powerful and easy-to-use, and included a very fun programming textbook. I just reread that textbook, and it is quite strikingly well-written and manages to cover both basic computer-science-style programming and deep close-to-the-machine and real-time programming in a compact 150 pages. There is no credit to a particular author in the book I have (Swedish translation by a group of people at Ord & Form here in Uppsala), but an online scan credits Steven Vickers.

Overall, the book does a very good job of explaining both the Sinclair BASIC interpreter, as well as some general programming concepts. And how the machine works deep down. Here are some highlights.

Explaining Functions

The way that functions are explained made me laugh out loud (quoting the original English according to the online resource I found):

Consider the sausage machine. You put a lump of meat in at one end, turn a handle, and out comes a sausage at the other end. A lump of pork gives a pork sausage, a lump of fish gives a fish sausage, and a load of beef a beef sausage.

Functions are practically indistinguishable from sausage machines but there is a difference: they work on numbers and strings instead of meat. You supply one value (called the argument), mince it up by doing some calculations on it, and eventually get another value, the result.

This is a very good analogy to introduce functions, in a fairly gender-neutral way and assuming no mathematical background on part of the user. I wonder if I ever cared to define any functions… as I recall it, GOSUB was as far as I got in sophistication before leaving BASIC and starting writing everything in Z80 assembler.

Reading a Changing Counter

Later on in the book, a suggestion is made to get a better time counter. Essentially, by reading a 24-bit value incrementing at 50 Hz, using three separate PEEK commands:

(65536*PEEK 23674+256*PEEK 23673+PEEK 23672)/50

This code does subject you to the issue of reading during an update and counter wrap from one byte to the next. This complication is cheerfully noted after a complete program using this feature as a by-the-way-the-code-is-correct by a small loop. There are many such small gems in the book, essential tricks and caveats that still apply to programming close to the hardware and doing real-time programming.

We have No Clock

The ZX Spectrum, as most computers of its day, did not have a real-time clock on board. Instead, you had to time things using the 50Hz screen refresh interrupt (which is what the code in previous section reads) or by just adjusting your code to wait appropriately. One example program is supposed to show a clock updating every second. However, getting to a second is a bit tricky when there is no firm time-base to work on. The suggesting instead is this:

This clock will run down after about 55.5 hours because of line 60, but you can easily make it run longer. Note how the timing is controlled by line 210. You might expect PAUSE 50 to make it tick one a second, but the computing takes a bit of time as well and has to be allowed for. This is best done by trial and error, timing the computer clock against a real one, and adjusting line 210 until they agree. (You can’t do this very accurately; an adjustment of one frame in one second is 2% or half an hour in a day.)

So what we have here is a classic “how long must I wait to get a steady pace of execution for my periodic code?” issue familiar to anyone using an RTOS and trying to determine how long to sleep a process for until the next periodic interval starts. Once again, illustrated in ten lines of BASIC and a few lines of text.

Repeatability and Randomness

In Chapter 11, randomness using the built-in random function is introduced. This was a 16-bit periodic pseudo-random system where you could set the random seed to put start the random generation at any particular point in a 100% repeatable manner. Indeed, the book introduces the concept of doing this in order to ensure repeatable execution of a program relying on randomness. This sounds familiar from my previous post on simulation determinism:

If you had a program with RND in it and it also had some mistakes that you had not found, then it would help to use RANDOMIZE like this so that the program behaved the same way each time you ran it.

Delving into Details

The BASIC manual does not try to hide how the BASIC interpreter works internally, rather on the contrary, including a list of all system variables so that they can be POKEd and changed if need be. It also explains the data layout of all variable types (the Spectrum BASIC used 32-bit integers, impressive for an 8-bit machine, and 40-bit floating point numbers).

The only useful information for me in the way I used the machine as a pure assembly-language target was the memory layout of the screen. Which is bizarre to say the least. It had a separate black-and-white bitmap and color overlay for every 8×8 character. And each scan line was NOT consecutive with the previous in memory, but rather offset by 256 bytes. I think the idea was to make it easy to insert a single character into the bitmap by just incrementing the high byte of the address by one for each pixel line.

To make that work, the screen was divided into three pieces of 8 character lines/2048 bytes each (8 character lines of 8 pixel lines each, each with 32×8 bits of data).  The sprite drawing routines that would work with that to plot a 16×16 sprite in any screen position was amazing code, to say the least. I independently invented unrolled loops to make this run fast. And misused the POP instruction to pull in data to draw (it took 11 cycles, rather than 12 as a more usual register-indirect load would do) — provided that you turned off interrupts or data corruption would ensue.

These were fun programming times. It shaped my programming habits for a decade, culminating in an infamous 68000 assembler hack in my first undergrad year at Uppsala, with some 50 pages of very tight and complicated assembly language making use of most features of the 68k instruction set… poor TA.

Final Notes

On it goes like that. The chapters are short but very much to the point, and a large number of fairly subtle concepts are introduced. The Spectrum was a good school for a budding programmer, with no compiler, no safety net, but also no way to break the machine in a way that a reboot would not cure.

I actually recommend seeking out the aforementioned online repository for the Spectrum programming book, it is almost still in shape to be used as an introduction to real-time and embedded programming.

I am currently trying to figure out what is a good ZX Spectrum emulator to relive these old times if I can find time. The hardware unit I have depends on an analog tape drive to load software… and I gave up analog sound back in 1993. As if the magnetic tapes which have been in storage for 20 years have any chance of containing useful data anyway.

However, moving to Vista has made the printing bit harder.

In Windows XP, there were drivers available for the printer, since it was fairly recent when XP was released. In Vista, no such luck. So you have to resort to using “LPR” printing (optional install), and using the generic “Microsoft Imagesetter” as the printer profile. This, somewhat surprisingly, works pretty well.

With one exception: Acrobat.

It seems that Acrobat is trying so hard to be smart about printing that it gets confused by the Imagesetter bit, and decides that the thing on the other end is not a Postscript printer. And thus, it needs to have a 600 dpi bitmap of the page being printed sent to it. Needless to say, my old printer with its upgraded 12 MB of RAM (it came with 4, and I scavenged 8 more MB from some old dead PC that passed through my hands in the late 1990′s) usually chokes on this.

When my last XP machine was retired, this did indeed create a problem, since all official online forms tend to be Acrobat-based.

However, by accident and luck I decided to try the Foxit Reader as an Acrobat Reader alternative. This has turned out to be the perfect to solution to my printing woes. With Foxit, a PDF file prints as a small nice regular vector graphics file that my LaserWriter has little problem printing. It makes printing PDFs feasible and reliable again, and means that I do not have to go out and figure out which new printer to buy. It is kind of cool to have such a decade-old technology icon at home, and still in working order.

A final note: 12 MB in my printer. My first hard drive back in 1990 had 20 MB on it. My new desktop Core i7 machine just got upped to 9 GB of RAM. Back in 1991, I had my high school’s most powerful home computer: a Macintosh SE/30 with all of 5 MB of RAM (which cost a fortune at the time).

Strongly recommended thread at stackoverflow: http://stackoverflow.com/questions/102714/what-was-your-first-home-computer is about your first home computer. Some good product shots, and also some really funny things inserted.

Living in a decently old city does provoke a certain interest in history and how to access the past. Note that Uppsala, like everything in Sweden, is fairly recent on the grand scale of things. We have a cathedral from the 1200s, and some older churches going back to the 1000s, but before that there is not much to boast about. It is not like Rome or Egypt or China with thousands of years of civilization and with impressive buildings that still stand. But I digress…

The real problem with archives and digital technology is that it is really hard to preserve a large portion of the intellectual data that we produce today. When I was an archivist at Smålands Nation here at the university in the 1990s, it was real fun to browse through the old papers we had in the archive (only stuff from the past century, all the really old and valuable stuff was in professional hands at the national archives). Reading a menu from a party held in 1945, or the time plan for a large formal celebration in the 1950s, or a guest list from such an event was actually quite intruiging. You could see how tastes and traditions change, even as students all believe they are maintaining ages-old traditions faithfully… Browsing the member list in old handwritten ledgers was also great fun.

In another fifty year’s time, what will we have left from today’s activities? The party plans are now all just printouts from a throwaway word document, the member list a database on a PC, and unless you make a conscious effort, memories and printed papers will quickly fade (you do need to use the appropriate laser toner on low-acid archive-quality paper if you want the things to last even a decade, let alone a hundred years) and the students fifty years from now will have no idea on we ran our parties, who were in the nation, and other such facts of daily life. “Print it out” is easy to say but hard to do. Much of today’s digital data does not really print out in a useful way. A wiki, for example, can be printed as a snapshot. But the links and the edit history is gone.

But imagine if we could retain these digital documents accessible, in their living digital form. Having a proper database available for use and query is so much more interesting and powerful than a stack of papers. Being able to look at the audit history of a wiki, or the commit history of a program’s source code would offer a much deeper insight into the people and processes that produced the end results compared to just looking at a static snapshot.

What does it take to do that? A huge and known problem in digital archiving is the physical media access. All great archival institutions in the world have scores of different tape readers, disk readers, some new, some antique, and try to use these to pry the bits of equally antiquated storage media. This is bad, just like we have problems looking at old movies as the reams of film are literally falling apart. Clay tablets start to look positively sophisticated and a smart choice compared to our very brittle technologies.

Second, once we have the bits, what can we do with them? Now we have to tackle the file formats, and having pried them open, to find the fonts and character sets which are suitable for displaying them. Rendering a MacWrite document from 1985 on a modern machine will likely not quite give the same wonderful black-and-white 72-dpi view that you got back then… Or figuring out that for a while, we used 7-bit ASCII to write Swedish texts by replacing [, ], |, {, }, \ with å, ä, ö, Å, Ä, Ö (not in exactly that order — but I used to be fluent in reading and writing that!). Even worse, decoding a database file or purely binary CAD/CAM file is going to require some serious reverse engineering of the old programs that created them… unless you could run these same old programs in some way.

At least to me, an “obvious” solution is to use virtual platform technology to make keep the old software stacks running. Some archivists are apparently working on OS emulation of various kinds to keep the old software running, but I do think that it is much simpler and more general to just simulate the entire machine hardware. This will result in the smallest risk of error, and the greatest fidelity to the original software as oddities such as word lengths and character sets can be faithfully reproduced. Simply because the SW-HW interface in a machine is the best document and narrowest interface in the entire machine. Old tapes and disks should be turned into files stored on the mass storage of current machines, and these files can then be migrated to new machines as they come online. As long as you maintain the copies of the materials by reproducing them on new hard drives, they will not rot.

It will also have the property that the entire look and feel of the old systems remains accessible. Which is a blessing in that it maintains our digital heritage and lets computer historians also go back and see how old software looked and worked. It is also considered a curse by practising archivists who do not particularly like the idea of learning how to operate strange old operating systems with arcane command lines. I think this is not necessarily a big problem. Just like people today study and learn ancient languages to learn more about ancient cultures as embodied in their written records, I can see historians of the future learning to use old operating systems and programming languages to study the ancient culture of our time as embodied in our computerized records.

Creating these kinds of virtual platforms is quite different from the dominant virtual platform thinking in the commercial market place today which is focused on modeling new and future hardwares. Rather, we need to study an existing artefact and make a very good model of it. It is a different type of task, requiring slightly different types of tools.

Of particular importance is that the simulation platform totally insulates simulation models from the underlying machine. You need to be able to port the simulation to new operating systems, machines, and architectures without changing a line in the source code of the models. This means making sure to locate all host dependencies inside the simulation framework, and making sure that models are completely portable regardless of word lengths, endianness, and other aspects. You want to be able to take a model that you run today on an x86-64 on Linux and run it on some future 128-bit middle-endian platform with a completely different type of operating system. Basically, the simulation platform has to be a complete virtual machine in its own right. Probably, over time, we are going to add more layers of virtual platforms to get around really major shifts in computer system architecture. There is nothing saying that you can only virtualize once, already the IBM S/370 showed that you could virtualize recursively and to an arbitrary depth.

I do think that some tools that we have today are perfectly useful for this kind of undertaking.

The business aspects are going to be interesting, though. We would like to use the technologies already available in the sophisticated commercial virtual platform tools, that’s kind of given. But the archival market place is not the most lucractive… and that they would like to have very good insurance that the tools remain available. Including source-code escrow that would make the requirements of 25-years military projects look positively light.

Developing a completely new totally open-source solution sounds like a nice idea in theory, but is probably a bit too much work. It also takes no advantage of all the commercial technology available today. Maybe as the virtual platform market matures, the industry can come up with some way to provide this technology for the greater benefit of society. Basically, doing a bit of social work where we really have the tools to help.

I have no good solutions to that to offer right now.

But imagine how cool it would be to, in fifty years from now, gather the old team from Virtutech around some kind of display device and watch an ancient Simics 4.0 run on top of a virtual quadcore x86 with Linux 2.6… all running on top of Simics 29.0 (estimating one major version every two years) on some future I-have-no-idea-what-it-will-be hardware and software system. That really is something that would be cool to see happen.