The approach taken for their checkpointing system is to save the entire state of  the running simulation program (i.e., an entire host operating system process), and later recreate the process in the same state. This gets around the need program all simulation models to explicitly support checkpointing, but it also limits the applicability of checkpointing severely. Of the checkpointing operations described in my previous post, it only supports the bringing up of a checkpoint into the same model, on the same machine (or a machine that runs a completely identical software stack down to the exact versions of all libraries, drivers, etc., which is not very likely to happen). This is honestly admitted in the paper, I appreciate that.

Process-based checkpointing in this way is also used by Cadence, and just like Cadence’s solution, the problem that appears when implementing it in practice is how to handle all the OS resources opened by a process. The process itself is not enough. The resources are typically files open for input and output, connections to debuggers, and other things that reach out of the virtual world of the simulation into the real world of the host machine. The solution is just like Cadence’s solution: provide callbacks for the SystemC side of such connections that can save the state of the connection in some way, and restart the connection when a checkpoint is opened. Doing this right does require some care in what to do in which order, and the paper nicely explains this.

For example, you need to close down all connections before taking a checkpoint, and then restore them after taking it. This is a fairly destructive operation, compared to the Simics-style checkpointing where all you do is interrogate the state of all objects and save that. I had not thought of that before, but it makes perfect sense.

To me, this also points to an important architectural issue in SystemC simulations in general: what are these simulation modules doing opening files on their own anyway? In my opinion, a simulation model should never do such low-level thing, it should operate using only the defined simulation system API and simulation-level connections to other simulation modules. If you need to load test data, do that by putting it into some storage system managed by the simulator itself. In all my professional life, I have considered file I/O to be a fundamental service provided by the application framework I use, nothing that my actual payload code should have to deal with. For example, in Simics, we tend to solve the loading and checkpointing of test data vectors using a memory “image”, which supports checkpointing in itself. Then a script just loads a file into the memory image, and a device reads the memory and moves transactions into the simulated system. This means that file I/O is totally absent from the model, and also that the test data can be managed and inspected by existing infrastructure.

The cost of taking a checkpoint is quantified in the paper, and it is not as bad as one could fear. The size of  a checkpoint easily reaches 100s of MB for even small target systems, but saving and loading that today does not take all that long. Still, going to a system with only 16 target processors (8 ARM, 8 DSP, and 8 times 64 MB of target memory = 512 MB) generates a checkpoint taking 1.7 GB to store and 30 seconds to save.

Once again, it is interesting to compare this to the Simics-style approach, where only differences are saved for target memories, and only target memory that is in use needs to be saved at all. This makes for usually far more compact checkpoints, which save and open in a few seconds in most cases. A checkpoint taking 30 seconds to open in Simics is rare indeed, and basically requires a target containing many gigabytes of target memory that is all in use (not host memory).

The paper does raise a novel point for why checkpointing is good: it saves you the time to setup a debugger. It is certainly true that setting up a debug environment for a session does take time, but it is not made clear just how it is recreated. My impression is that really what is going on here is that the debugger remains alive and loaded, and the simulation goes back and forth in time using checkpoints. Which is perfectly valid and nice.

In Simics, we solve this problem in two ways: when it comes to setting up the internal debugger when opening a checkpoint, the solution is to use a script to configure it. On purpose and by design, Simics checkpoints do not store the simulation session state, only the target system state, as that is the only way to be portable over time and across widely different machines. it would be very strange to open a checkpoint a few years after it was created, by some random user, start a simulation, and have it stop just because there was some breakpoint still in place for some long-forgotten reason. In any case, the simulator-side of the debugger has to be adopted in all systems to accept checkpointing.

Another strange point in the paper that I would have liked to ask the authors about is the idea that you can always change the target software in the middle of a session. But what is the value of checkpointing then? Since the idea is saving the cost of booting a machine and setting up a debug session, it is hard to see what is gained by booting a machine, keeping the hardware state, but replacing the software state. The expensive operation is presumably setting up the software state? At least it is in my experience.

I commend the paper for actually running something more than the archetypal “ARM+DSP”, by running eight copies of an ARM+DSP subsystem. That is at least starting to look like something interesting to simulate.

Reviewer Notes for the Paper

Finally, I have some small critiques on the academic paper itself, of the kind that I tend to provide to paper authors when active as a reviewer for conferences.

The SoC conference paper itself is well-written, but I have point out that the authors for some reason ignore the history of checkpointing in full-system simulation. The seminal work here was the checkpointing system used for changing the level of abstraction from fast functional simulation to cycle-accurate simulation in the SimOS system developed at Stanford already in the early 1990s. This has later been continued in both IBM Mambo and Virtutech Simics, and remains the most powerful way of doing checkpointing until this day. I don’t think the authors were completely ignorant of this work, even though finding good references can be a bit difficult. Even so, here are some to add to future versions of that paper:

• SimOS: A Fast Operating System Simulation Environment, Stanford CS Technical Report CSL-TR-94-631, 1994. The earliest mention of using checkpointing in a full-system simulator (for switching from fast to detailed simulation).
• “Design and validation of a performance and power simulator for PowerPC systems“, from the IBM Journal of Research and Development in 2003. Mentions that IBM Mambo does checkpointing like SimOS.
• SIMFLEX: A Fast, Accurate, Flexible Full-System Simulation Framework for Performance Evaluation of Server Architecture, ACM SIGMETRICS Performance Evaluation Review, 2004. Also see the Simflex homepage. Shows an ambitious use of checkpointing in computer architecture simulations.

It is also a bit disingenuous to dismiss the issue of just how the process checkpointing is performed by a reference to an early requirements paper from the BLCR project.  All that says is that there are lots of possible variants of implementation, nothing on what was done in this particular instance. It would have been nice to have had some more details: does the approach use a kernel module or not?

Let’s look at this in more detail.

Definitions

A clock-cycle (CC) model in this discussion is something that attempts to provide a cycle-by-cycle depiction of the behavior of a computer system. Usually, such models are driven by a cycle-by-cycle clock, as that is the easiest way to write and structure them.
A cycle-accurate (CA) model is a CC model where the depiction is “the same” as what would happen in the real system provided they both started from the same state.

What is ARM Doing?

ARM seems to be passing on the tools and technologies they acquired when they bought Axys back in 2004. These tools are CC-oriented, and are aimed at hardware architects (and some really-low-level software work). They make it possible to evolve a target design cycle by cycle in the simulator to get a very accurate picture of the target behavior. I think this fits very well for Carbon, as they generate cycle-driven very accurate models by essentially compiling the actual RTL implementation of a piece of logic, processor, or device into something a bit faster than plain HDL simulation. Carbon models are a natural fit for the Axys tools.

Basically, it sounds as if ARM decided that manually creating CC level CA models for their latest processors for use in the Axys tools (SoC Designer) was too much work and too hard to validate. Thus, they pass the whole thing on to Carbon and seem to expect Carbon to generate CA models for use with SoC designer straight from the actual ARM implementation RTL. Carbon will have the old CC/CA models written by Axys (and later ARM), and then generate new models for new generations of ARM chips like the Cortex A9. I quote:

“The model generation flow will be optimized and validated using the RTL code, ensuring speed and accuracy. The processor models will also leverage the Carbon model application programming interface (API) to offer a direct connection to the ARM RealView(R) Debugger. Carbon-generated models of ARM IP will offer our customers the fastest, most-accurate path for firmware development and architectural exploration.” (press release)

And:

ARM made this decision, Cornish said, because it’s become increasingly difficult and time-consuming to develop cycle-accurate models. “We recognized it would make more sense to work with a specialist like Carbon that has technology for generating models directly from RTL,” he said. (SCDSource News Piece on the deal)

Feasibility of Construction

The core argument here is really how easy or feasible it is to build CA models of a processor core (or any other really complex piece of logic). There are several interesting views to consider.

1. The ARM statement is basically saying that building CA models of a processor core is very hard. It is hard to get right, hard to validate, and hard to maintain. So why even try? Better to generate it from the RTL and let experts at doing that do the work.
2. In my PhD thesis from 2002, I concluded that building an accurate model of a processor from public information and reverse engineering is very very difficult, and cited a number of computer architecture and real-time systems attempts to build models that all turned out to have accuracy issues. I did not know much about EDA then — and ESL did not really exist. But I think that still holds water: constructing a model of a processor is hard.
3. In the SCDSource article, I make the statement that “Building cycle-accurate (CA) models is very difficult, as you need to understand and describe the implementation details of complicated hardware units. … It is quite easy to end up with something that is essentially an alternative implementation to the actual chip RTL. It is especially difficult for third parties, as it requires access to the device and processor core designers to explain the design.” Which is essentially saying that you need to get inside the processor design group to get the information.
4. The common knowledge that all great processor design teams, from the DEC Alpha to Intel x86s to AMD Opterons to IBM Power to Freescale Power to Infineon TriCore to Sun Niagara use internal cycle-detailed simulators as their main design tools to prototype and decide how to design pipelines, memory systems, and system platforms. In this case, the simulator comes before the processor, not the other way around.
5. Tensilica has, as Grant Martin points out in comments at SCDSource, tools that generate both the processor and an accurate model at the same time from the same information base.
6. CoWare’s LisaTek tools for describing and generating application-specific processors also claim to generate accurate models from the LISA source files in a way similar to Tensilica but based on a user describing a completely custom design in a third-party tool. In the case of Tensilica, the tool and the design come from the same company.

So where does this leave us? It makes it clear that in order to build a good cycle-accurate model you need access to internal information and the processor design/processor design team. The CA model can be built either:

1. By synthesizing from the RTL, Carbon-style.
2. By synthesizing from some more abstract design description, Tensilica or LisaTek-style.
3. By the design team as part of the design process.
4. By some poor guy working after the fact from specs and test cases.

I think the ARM-Carbon deal (and all practical experience as well) invalidates the fourth variant. Essentially, that is what Axys had to do: build models after the fact, separate from the CPU design flow. This is a property of how ARM design processors and the fact that Axys began life outside of ARM (my guess, nota bene). It is what computer architecture researchers often want to do but fall down on over and over again. In fact, a common question from computer architecture newbies is if Virtutech Simics has correct models of processors like the Intel Pentium4 or Core 2 available to use as starting points in research. It would be nice, but sorry, we do not.

But the other three variants do make sense, and will all result in some kind of decent model. Which one you end up doing depends on the style of your design and quite likely the complexity of the processor and system design. In the end, any truly revolutionary design (think Sun Rock, for example) will need to write a custom simulator as tools will not have the concepts in them to model all ideas. It seems that simple “standard” designs that fit in the categories of “custom RISC” or “custom DSP” and that do not break new ground in computer architecture can probably be designed using tools that allow processor and simulator generation. I think that most heavy-duty general-purpose processor cores will have to do either the design-model or RTL-generation path, while more accelerator-style cores can use the tools approach.

As a final note, there could really be two different problems being addressed here regarding “cycle accuracy”, and that this might contribute to different levels of feasibility:

• Using the simulator to validate and optimize software performance can tolerate some errors in details as long as errors do not accumulate (see for example the “timing anomalies” or “unbounded long timing effects” found in WCET research). It is about understanding the software behavior versus the processor design (or complex accelerator design versus input data), in small focused spots of execution.
• Using the simulator to validate a chip design including buses and other devices that can be bus masters. This ought to require a higher level of accuracy, as the penalty for errors would potentially seem greater. And this is also where ARM’s SoC designer fit in, rather than as a tool to understand the software behavior. The scope here is larger and there is usually no idea of zooming in on detail at particular points in time.

So where does this land us?

I guess that CC/CA models can be built if you have a nice inside track to the design team, and that the only sensible way to use them is as a zoom device for the places in your code where you absolutely need the details. Most of the time (say 90-95-99%) software does not need CC models, but rather something that is functionally accurate and that runs really really fast so that all software can at least be executed. That is something a CC model will never be able to do, at least not for systems using non-trivial operating systems requiring a few billion instructions just to boot…