First of all, what makes register design such a special task? My guess is because it is a higher-level aspect of the design: it describes an interface, which can be implemented in many different ways in hardware. Regular HDLs do not help you reason about register layouts in a good way. The register specification should also be used in other places than the hardware: in address definitions for device drivers, in manuals, and other documentation. All of this indicates that the information needs to be explicit, high-level, and in a format that facilitates automatic processing by tools. It basically screams for a domain-specific language (DSL) at some level of ambition (read my old post about how languages are grown from problems for more background on just how simple a DSL can be).

If you look at the register design tool vendors that Gary interviewed, you can essentially see two different approaches to how to support the task of register design. In particular, the input language differs quite radically between the tools.

There is domain-specific programming language approach. A register-design DSL makes it easy to work with register designs, and pretty hard to do anything else (imagine writing a sorting algorithm in a register-design language – I don’t think it can be done, and if it can be done, the result has to be absolutely contorted). The DSL approach assumes a willingness  on the part of a user to learn a new language, but it seems that for this domain, the languages are simple enough that learning them saves you time in the end. Especially if you maintain large register maps that keep changing or is updated and extended across generations of chips (indeed, ease of maintenance is an undersold aspect of most DSLs in my experience).

The underlying assumption of the DSL approach is that the entry language is not too important, as long as you can export data into other formats. To me, this makes sense. From a single source with sufficient descriptive power, you can ideally generate both HDL code to implement the decoder in hardware, as well as documentation, header files, and maybe even skeletons for virtual platform models. And standard formats like IP-XACT to feed into any other tool.

It does not matter if there is a single tool using each input language, since the outputs are what matters. This leaves the vendors free to invent on the input side, and provide a really powerful tool.

The second approach is transformation-based. Its idea is to not use a dedicated input language, but rather powerful import functions to read whatever specifications already exist in text files, Microsoft Word documents, Microsoft Excel spreadsheets, Framemaker documentation files, IP-XACT, or whatever you can come up with.  The assumption is that the tool is really about transforming data and not about compiling a language. Register designs kind of already exist, and since there is no standard language to compile, you just import whatever there is. It makes the assumption that the user base is not willing to learn a new language to handle register design. A funky side-effect is that we might end up actually doing programming work in Word docs.

Still, the transformation approach ends up generating the same outputs as the DSL approach. In  both cases, some of the outputs can be considered “industry standards”, like IP-XACT, while others are decidedly ad-hoc, like Excel sheets. To me, this demonstrates that the true value in standards is to enable tool interoperability, and not so much as a way to provide inputs.

Indeed, there is a clear difference between good input languages and good exchange formats. An input language tends to drive for expressive power and human readability, and it is OK to have a heavy compilation process associated with it. An exchange format like IP-XACT does not need to be human-readable, nor efficient to code in. It needs to be easy to parse and compile, so that as many tools as possible can work on it.

A typical example from the field of register design is dealing with repeated groups of registers (such as banks of registers for a DMA channel) and parametrized register designs. The input languages used by the tools that Gary Stringham covers all allow this – while the standard for register design exchange, IP-XACT, only uses a flat compiled map. It just lists all registers with sizes and offsets, not how these offsets were arrived at or if there is some logical grouping or iterated structure. Quite OK for a tool to work on, but not very useful as a repository for the actual information.

This distinction is worth to keep in mind.

I definitely side with the DSL idea, as that fits my idea of a good tool, but I can see why many people find the transformation from other formats attractive. In all cases, the goal is to have an original design specification that is as clear, succinct, and flexible as possible.

TI deploys two totally different platforms for hardware and software development, which makes perfect sense.  The goals are so different between a high-speed software development platform and performance-accurate hardware design platform that trying to force them together would likely just create a bad compromise that is bad for everybody.

Additionally, FPGAs are used to create timing-dependent low-level code, where you need both timing accuracy and decent speed.  It is worth noting that the performance model is mostly “dataless” – it models the timing of actions and their dependencies, but not their values and computations.

The different models serve different purposes, require different levels of effort to use, and become available at different times during the project. The SystemC performance model is always available first and is always the simplest to create and use. The virtual platform is the next to become available. It is used for software development and has very little timing accuracy.  TI uses Virtio technology to create this model rather than SystemC.

Given the number of ultimately failed attempts I have seen at making timing and function available in the same model but as orthogonal concerns, this observation in the article is very insightful:

It would appear the choice of two different technologies for the virtual platform and the performance model is inefficient, wasting potential code reuse. However, the two have completely different (almost fully orthogonal) requirements, and at module level almost no code reuse is possible.

Maybe this is an impossible dream in the general case.

One somewhat surprising statement in the article is that there is no real software available to use in the SoC design phase. Often, virtual platforms are sold as being able to use “the real software” when designing hardware. But in the case of TI, the software is mostly written by their customers, with little available for TI to use. Thus, they are forced to design their own test cases to drive the hardware design process.

The requirements on the simulation technology are first and foremost ease in creating test cases and models and credibility of results. The emphasis on test-case creation is a consequence of the complexity of the devices and of the way in which an SoC platform such as OMAP-2 is used: because the whole motivation is to be able to move from marketing requirements to RTL freeze and tape-out in a very short time; and because in many cases large parts of the software will be written by the end customer and not by the SoC provider (Texas Instruments, in this article), the performance-area-power tradeoff of a proposed new SoC must be achieved without the aid of “the software.”

The platform they built is all based on clock-cycle-level interfaces (CC), which is very natural when the primary use case is hardware design.

The primary component optimized in the TI design process is the on-chip interconnect structure, called the “NoC” in the article. Each SoC variant is built from a set of (usually already existing) devices and processor cores. The main work of the integration is designing an appropriate NoC for the SoC. The NoC design is crucial to the actual performance level the final SoC product will have.

By playing with the topology, the level of concurrency, and the level of pipelining in the NOC, it’s possible to create SoCs from the same basic modules with quite different capabilities.

The only real instruction-set simulators used are CC-level models of DSPs, used for software optimization taking but contention into account. No models of the ARM control cores are used. Mostly, processors are represented by stochastic or trace-driven traffic generators that put transactions on buses but do not actually run any real code.

The stochastic processor models are very powerful and provide traffic that is very similar to a real processor.  A very elegant property of such models is that it is very easy to change the parameters of the model to model quite different software/processor scenarios. Compared to writing real test programs for a full ISS, this is much faster and allows for the exploration of more alternatives.

The stochastic models are used along side function-graph breakdowns of software, essentially models that say that an application does A, then B, then C, and that maybe D can happen in parallel. This model of an application is connected to the hardware simulation and can control when things happen and what goes on in parallel. It amounts to a simple model of what an RTOS would do, to some extent.

Configurability is a key theme throughout the OMAP architecture exploration platform. SystemC being what it is, it is limited to configuration at start-up time, but that is perfectly sensible for an architecture exploration use case where you want to setup and platform and test its performance. Dynamic reconfiguration during a run is not that important.  TI has spent a great deal of effort in making the system easy to configure using parameter files.

The article goes into many more fascinating details on the models used.  I can only say one thing: read it, if you have any interest in these kinds of issues.

Good work, James!

This is a small detail, but one where I have always had a feeling that some fundamental assumption was missing in my discussions with various people from the hardware design community. It always seemed that hardware designers assumed a different basic design – and Grant Martin explained it very well just what that was. They only check for interrupts at the beginning of a time slice. Which makes interrupts less precise  versus the code, but also makes the core interpreter fairly simple since all it has to do is to churn through instructions.

There is another solution, which is employed in Simics, where the processor can take an interrupt at any point in a time quantum. To do this, the processor needs to be aware of what is going to happen. The essentials of the solution is to have devices call the processor and tell it that they intend to interrupt it at some point T in time. The processor simulator then makes sure to stop and give the device model a chance to act at that exact point in time. It is obvious that this solution is easily generalized to cover all time callbacks needed to drive device work. A significant part of the responsibility for running the event-driven simulation is moved into the processor core.

Making the event queue visible to the processor also gives the processor a chance to hypersimulate, or skip idle time. Since it knows the next point in time that something will happen (either the end of a time quantum or an event posted by a device), it can very easily, safely, and repeatably jump forward in time without any impact on simulation semantics.

When dealing with multiple processors, this means that each processor will have precise interrupts from the devices that are close to it. Timers and IO interrupts tend to work closely with a certain processor for a prolonged period of time. Interrupts between processors suffer a time-quantum delay sometimes, but that is no worse than the solution of checking all interrupts at time-quantum boundaries.

Qemu uses a solution which is a mix of the two. According to the 2005 Usenix paper, devices do call into the processor to announce an interrupt, but this is handled by “soon” returning to the processor main loop. Processors are not responsible for keeping track of interrupts, making it very imprecise and not very repeatable when interrupts will happen.

Thus, we can see that there are a few different ways to implement interrupts in virtual platforms. Each approach comes from a different tradition and features different trade-offs.

I was a bit surprised by the comment in the Tensilica chapter that only correctly synchronized programs will work on a temporally decoupled simulation. In my experience, temporal decoupling is transparent to software functionality – all software runs. The perceived timing of operations can be different, and some tightly-coupled code might behave in suboptimal ways, but it certainly runs and works. And lets you observe  parallel code errors.

Temporal decoupling is necessary in any fast platform, and its effect on semantics are really minor. With the simple tweak of having a processor know when interrupts might happen, it will also not affect the device-processor interface very much, maintaining very tight synchronization between processors and their controlled hardware.

On one hand, we have the task of supporting the design of new hardware bits, for the purpose of creating it. On the other hand, we have the task of describing a particular design for the purpose of simulating it. These two are not necessarily the same.

To use an analogy with building a house, a design language helps the architect create the house (piece of hardware). Since the architect relies on craftsmen and experts (compilers) to do detailed design (how to put in windows, where to put light switches, etc.), the high-level description does not contain all the details of the house. However, if you are trying to simulate the house (piece of hardware) so that its inhabitants (software) don’t see the difference to the real thing, the details are sometimes what matters most. For example, the precise way to operate the stove in the house is very important for familiarity, but is a detail most likely left out of the architect’s initial drawings.

A design language can leave many things unspecified to be filled in by a compiler, but these things can be absolutely core to a description language. In particular, programming register maps tend to be created as a not-too-important side activity in hardware design. They do not really need to be visible in higher-level ESL languages, as they can obviously be filled in later by a tool or a human. But for a description language, they are absolutely core.

A description language can also leave out many parts of the hardware. If the software being used or written does not use certain modes or functions of a piece of hardware, those pieces can be ignored and implemented as dummies. That means that support for dummies is very important in description languages. But dummies make little sense in a design language, as you are unlikely to design a chip with lots of area spent on dummy functions that do nothing.

A description language can also ignore crucial aspects like power constraints and synthesis constraints. These are guidelines for a compilation step that has no bearing on the description of the hardware — the description language should describe what ended up happening, not the if, please, what, and buts that guided how we got there.

For virtual platform creation, you seem to need a bit of both. I maintain that most of a VP is based on old hardware that exists, which calls for languages with strong description abilities. That’s the space that Simics DML was designed for. For the small part of the hardware that is novel would be nice to have some way to convert from a design language to a virtual platform. Here, I don’t really see any usable current tools or languages — SystemC is really more a design language, but if you want a virtual platform model, you have to use it as a description language. There is no automagic getting to a fast abstract model from a design-oriented description. That’s why we need new, higher level systems, that can push out decent descriptions from a design.

I have seen the “write 1 clears” solution before in real hardware, but I was not aware of the other two variants. The idea of having a “write mask” in one half of a 32-bit word is really clever.

However, this got me thinking about what the fundamental issue here really is.

As I see it, it is the fact that the processor cannot address small enough units atomically. The read-modify-write that was used to start the discussion in the Embedded Bridge #37 was needed in order to get the current state of a configuration register, change some setting that only occupied a few bits in it, and write back the result to the register. The way most configuration registers that I have seen in practice works.

But if each setting could be given its own register, the problem would go away. Each operation would target a unique address, achieving the same effect as the bit-wise masks or write-1 solutions proposed. The core problem is that hardware tends to share settings into registers, as it has been considered too expensive to put information that might cover a range as small as [0,1] into a 32-bit register. Probably, since there is a lack of addresses for registers, you cannot have 1000 settings cause each simple device to use up 1000 words of physical addresses.

But is that really an issue, if we look forward?

It seems to me that, as 64-bit instruction sets and addressing systems penetrate down into more and more embedded systems, a simple solution would be to throw address space at the problem. I don’t think it is uneconomical to allocate huge chunks of memory space to each device, giving each setting its own register, when you have 64 bit virtual addresses to work with. There is no way you can fill up a physical memory system (guess that will some day come back to haunt me)… even the highest-end machines today only use something like 40 bits for actually addressing physical memories.

The software would be simpler and more robust, with virtually no cost.

Another solution that I have also seen starting to appear is to dispense with register settings altogether, and rather define a command API that the processor “calls” by putting in command packets into some memory area. This does require quite a bit of silicon for a decoder, but it provides for a much higher level of interaction with devices. As hardware devices get defined in successively higher-level languages (C, C++, UML, MatLab, …), and their programming interfaces and associated drivers get autogenerated, this solution makes eminent sense.

The approach taken for their checkpointing system is to save the entire state of  the running simulation program (i.e., an entire host operating system process), and later recreate the process in the same state. This gets around the need program all simulation models to explicitly support checkpointing, but it also limits the applicability of checkpointing severely. Of the checkpointing operations described in my previous post, it only supports the bringing up of a checkpoint into the same model, on the same machine (or a machine that runs a completely identical software stack down to the exact versions of all libraries, drivers, etc., which is not very likely to happen). This is honestly admitted in the paper, I appreciate that.

Process-based checkpointing in this way is also used by Cadence, and just like Cadence’s solution, the problem that appears when implementing it in practice is how to handle all the OS resources opened by a process. The process itself is not enough. The resources are typically files open for input and output, connections to debuggers, and other things that reach out of the virtual world of the simulation into the real world of the host machine. The solution is just like Cadence’s solution: provide callbacks for the SystemC side of such connections that can save the state of the connection in some way, and restart the connection when a checkpoint is opened. Doing this right does require some care in what to do in which order, and the paper nicely explains this.

For example, you need to close down all connections before taking a checkpoint, and then restore them after taking it. This is a fairly destructive operation, compared to the Simics-style checkpointing where all you do is interrogate the state of all objects and save that. I had not thought of that before, but it makes perfect sense.

To me, this also points to an important architectural issue in SystemC simulations in general: what are these simulation modules doing opening files on their own anyway? In my opinion, a simulation model should never do such low-level thing, it should operate using only the defined simulation system API and simulation-level connections to other simulation modules. If you need to load test data, do that by putting it into some storage system managed by the simulator itself. In all my professional life, I have considered file I/O to be a fundamental service provided by the application framework I use, nothing that my actual payload code should have to deal with. For example, in Simics, we tend to solve the loading and checkpointing of test data vectors using a memory “image”, which supports checkpointing in itself. Then a script just loads a file into the memory image, and a device reads the memory and moves transactions into the simulated system. This means that file I/O is totally absent from the model, and also that the test data can be managed and inspected by existing infrastructure.

The cost of taking a checkpoint is quantified in the paper, and it is not as bad as one could fear. The size of  a checkpoint easily reaches 100s of MB for even small target systems, but saving and loading that today does not take all that long. Still, going to a system with only 16 target processors (8 ARM, 8 DSP, and 8 times 64 MB of target memory = 512 MB) generates a checkpoint taking 1.7 GB to store and 30 seconds to save.

Once again, it is interesting to compare this to the Simics-style approach, where only differences are saved for target memories, and only target memory that is in use needs to be saved at all. This makes for usually far more compact checkpoints, which save and open in a few seconds in most cases. A checkpoint taking 30 seconds to open in Simics is rare indeed, and basically requires a target containing many gigabytes of target memory that is all in use (not host memory).

The paper does raise a novel point for why checkpointing is good: it saves you the time to setup a debugger. It is certainly true that setting up a debug environment for a session does take time, but it is not made clear just how it is recreated. My impression is that really what is going on here is that the debugger remains alive and loaded, and the simulation goes back and forth in time using checkpoints. Which is perfectly valid and nice.

In Simics, we solve this problem in two ways: when it comes to setting up the internal debugger when opening a checkpoint, the solution is to use a script to configure it. On purpose and by design, Simics checkpoints do not store the simulation session state, only the target system state, as that is the only way to be portable over time and across widely different machines. it would be very strange to open a checkpoint a few years after it was created, by some random user, start a simulation, and have it stop just because there was some breakpoint still in place for some long-forgotten reason. In any case, the simulator-side of the debugger has to be adopted in all systems to accept checkpointing.

Another strange point in the paper that I would have liked to ask the authors about is the idea that you can always change the target software in the middle of a session. But what is the value of checkpointing then? Since the idea is saving the cost of booting a machine and setting up a debug session, it is hard to see what is gained by booting a machine, keeping the hardware state, but replacing the software state. The expensive operation is presumably setting up the software state? At least it is in my experience.

I commend the paper for actually running something more than the archetypal “ARM+DSP”, by running eight copies of an ARM+DSP subsystem. That is at least starting to look like something interesting to simulate.

Reviewer Notes for the Paper

Finally, I have some small critiques on the academic paper itself, of the kind that I tend to provide to paper authors when active as a reviewer for conferences.

The SoC conference paper itself is well-written, but I have point out that the authors for some reason ignore the history of checkpointing in full-system simulation. The seminal work here was the checkpointing system used for changing the level of abstraction from fast functional simulation to cycle-accurate simulation in the SimOS system developed at Stanford already in the early 1990s. This has later been continued in both IBM Mambo and Virtutech Simics, and remains the most powerful way of doing checkpointing until this day. I don’t think the authors were completely ignorant of this work, even though finding good references can be a bit difficult. Even so, here are some to add to future versions of that paper:

• SimOS: A Fast Operating System Simulation Environment, Stanford CS Technical Report CSL-TR-94-631, 1994. The earliest mention of using checkpointing in a full-system simulator (for switching from fast to detailed simulation).
• “Design and validation of a performance and power simulator for PowerPC systems“, from the IBM Journal of Research and Development in 2003. Mentions that IBM Mambo does checkpointing like SimOS.
• SIMFLEX: A Fast, Accurate, Flexible Full-System Simulation Framework for Performance Evaluation of Server Architecture, ACM SIGMETRICS Performance Evaluation Review, 2004. Also see the Simflex homepage. Shows an ambitious use of checkpointing in computer architecture simulations.

It is also a bit disingenuous to dismiss the issue of just how the process checkpointing is performed by a reference to an early requirements paper from the BLCR project.  All that says is that there are lots of possible variants of implementation, nothing on what was done in this particular instance. It would have been nice to have had some more details: does the approach use a kernel module or not?

The main theme of the interview is the need to shift business thinking from small details to entire systems. From operations research where you spend lots of time understanding some process or department in great detail, to a system-level thinking where you focus on what an entire enterprise is doing.

For me, this struck a chord in my system-level heart… in my world of computer systems and virtual platforms, system-level is what it is so hard to get engineers to. Far too much time is spent (in my opinion) understanding, modeling, and tweaking subsystems. Far too little effort is spent on understanding the whole, how things fit together in practice, taking software, hardware, and software system evolution over time into account. The analogy is not perfect, but there are more things that are alike than are not.

The most interesting analysis that Russell Ackoff fires off from his perspective is that of comparing companies and architecture. An architect knows the whole of a building, but does not entirely go into details on just how it is to be built. He/she trusts the carpenters, bricklayers, and other workers to know how best to solve their local problems. Basically, applying hierarchical abstraction to the task of constructing an actual building.

This got me thinking some of why this is the case. I think it could be because building things (castles, cathedrals, houses, walls, pyramids, canals, …) must have been among the most complex tasks undertaken for a very long time in human history. Thanks to this long history, we have perfected the abstraction and division of labor in construction. Buildings are built in a certain way, by a certain set of crafts, since that method has been proven to work well for a very long time. So just like in the case of the design patterns craze in the late 1990′s, architecture might have something to teach us about how to build hardware/software systems too.

Note that for some reason, I cannot find a link to the podcast on the BBC homepage. But if you subscribe in iTunes or similar, I think you will find it. Something is not as user-friendly as it could be.

The paper will explain how we did Simics-style checkpointing in SystemC, using the GreenSocs GreenConfig mechanisms to obtain an approximation for the Simics attribute system.

It is an approach that does not have the limitations of the “save the entire simulation process” method employed by Cadence (and I think also CoWare) in their SystemC checkpointing solution. It does require you to mark all relevant state in your models, but the benefit from doing so is that regardless of how you change the code of a model, you can still use the same old checkpoints. It is also portable across hosts. We did have to do some patching to the OSCI SystemC kernel to draw out and reset all relevant state from the kernel. The OSCI kernel does not provide sufficient interfaces to checkpoint its state in its vanilla form.

The conference takes place on September 22 to 24, in Sophia Antipolis in France. Now all I have to do is figure out how to get there in the most convenient way. I expect this to be as much fun as the other EDA conferences I have been to recently (I seem to only go to such events nowadays, nothing left on the old embedded circuit for me it seems).

By the way, the FDL logo is really pretty. I think all long-running events should spend the time to create a recognizable logo. My old real-time conferences used to just have plain text and the IEEE and ACM logos.

As is often the case, after the panel has ended, I realized several good and important points that I never got around to making… and of those one struck me as worthy of a blog post in its own right.It is the issue of how high-level synthesis can help software design.

At the end of the panel, the last comment from Kees Vissers of Xilinx pointed out that high-level synthesis is a very powerful way to build hardware. I think his point was that hardware and software are not that different… but the remark also got me thinking. If it is the case that high-level synthesis currently makes hardware creation easier, can’t it also be applied to software creation? In particular, if you have a HLS description of a piece of hardware, can’t you also generate its driver software?

I think that makes eminent sense, since one of the hard parts of doing device drivers is just getting the use of the programming registers of a device right. The programming register interface is a really strange thing if you think about it. It is not native to either software or hardware, really.

In hardware, you communicate between devices using fifos or signals or packet-based mechanisms which do not in general look like programming register writes. You move data by sending a stream of data directly, not word-by-word addressing registers. Similarly, on the software side, software units call each other using functions (or higher-level OS abstractions like signals or network packets). They do not put data into addressed registers…

Today, high-level synthesis as practiced in industry involves describing the function of a device in pretty abstract terms, so that the compiler can make smart decisions on the implementation details. It also makes it easier to try different alternatives in the implementation, trading size, speed, and power consumption against each other. Different types of concurrency and pipelining can be explored.

However, once we get to the hardware-software interface, we get rudely dropped into a world of manual detailing of an interface with no tool support to explore it or validate it. Why should that really be the case? I think that the hardware-software interface requires just as much care as the internals of the device. After all, it is the external face of the device, and if that is too hard to use, users will not get the full benefit of the device. Here are some previous posts on the nature of interfaces and why they matter: 1, 2, 3, 4.

So I would propose a different take on this, where you apply synthesis at a higher-level, and generate the hardware internals, the programming register interface, and the software driver from the same source. The interface you design for a device would be a set of function calls expressed in software terms, and thus relatively easy to use from software. Let’s call this SHLS, Software High-Level Synthesis. Or maybe Software-Level Synthesis, SLS.

I am well aware that most operating systems do not provide an interface to device drivers consisting of function calls… but rather rely on pretty non-semantic methods like read/write/ioctl. However, that is easy to overcome by generating a user-level interface library in addition to the raw device driver. Obviously, a tool like this would need some adaptation to apply to each operating system targeted. But that does not feel like something that cannot fairly easy be handled by a template system. Compared to the complexity of synthesizing hardware this feels pretty basic.

If you hide the software-hardware interface inside a black box like this, it can also be implemented in quite different and interesting ways. For example, you could imagine using a small memory-mapped buffer where you enter commands and data and then ask the hardware to “parse” it, rather than using discrete registers with immediate effects. Or you could optimize the coding of the relevant hardware operations into a bit-compacted representation no human would like, but which are no problem for the machine.

Another option is obviously to just stop at the hardware-software interface, but let the tool help the hardware designer build the programming register interface and explore various options for it. Not having to invent a register layout from scratch should make that job much easier.

TLM is often considered to be SystemC TLM-2.0. Most of the statements from the EDA companies are to the effect that SystemC TLM-2.0 solves the problem of combining models from different sources. Scratching the surface of this happy picture, it is clear that it is not that simple…

The issue is that even if all agree on using the TLM-2.0 standard and its default standard generic memory-mapped bus protocol and payload for the memory-map part of their device models, there are other interfaces which are not standard at this point in time.

For example, there is no standard way to model interrupts between devices. So any time you have interrupts in a system (which tends to be always), you need to write custom wrappers between modules to convert different ways of modeling interrupts. Even worse, the standard way to do it is to use SystemC signals, which are definitely not TLM abstractions. They take a detour through the SystemC kernel, which is quite costly.

The defining property (from a simulation execution perspective) of TLM is that your simulation modules talk directly to each other through direct function calls, rather than passing over whatever simulation kernel you happen to be using. Essentially, TLM tends to convert simulators into being much more like “regular programs”, with fewer references to the simulation kernel and its event and time handling. In my world, unless you are doing direct function calls, you are not doing TLM.

Note that this state of things in the SystemC world is likely to change for the better over time. GreenSocs announced at DAC that they are working with Virtutech and an unnamed other partner to create a set of TLM interfaces for other interconnects, such as signals (interrupts under another name), serial, and Ethernet.

But apart from all the technicalities of SystemC TLM-2.0 and how it works, the big question is just what to use TLM for, and how. Here, everyone seems to try to turn TLM into their own use cases. The most obvious application is doing fast virtual platforms, but you also have TLM use as the basis for hardware synthesis, validation, golden reference models, architectural exploration, and pretty much all other EDA design tasks.

Even so, the most important message for me is that the EDA industry is actually starting to get interesting in TLM. It is no longer a quaint odd thing done by some peripheral start-up companies, but rather a mainstream technology that everyone has to pay attention to.

Finally, I want to point out that TLM is not just SystemC. TLM is a general idea that has been in active use since the late 1960s. It is the obvious way to model a computer, if all you are concerned about is how it looks to the software. Another current example is the Simics style of TLM (and here), which is similar to but different in details from the SystemC implementation.

My entry is called “Virtual Platforms unites Hardware, Software Engineers“, and is presented by Richard Goering (who used to be with SCDSource), in his “Industry Insights” section of the Cadence community of blogs. Richard Goering has a personal post pointing in the same direction, about EDA tackling embedded software. Worth reading.

Note that I do not say that hardware and software engineers should use the same programming languages as a result of using virtual platforms. Programming languages efficient for hardware design are quite different from those efficient for virtual platform creation, which are in turn different from good software engineering languages.  In some cases, some of them coincide, but in general, I believe in using the best tool for each job, and a programming language is just a tool. And the more designed it is for its task, the better. Some older posts of mine on this topic:

• DSL: Purpose-built languages
• DSL: The tyranny of syntax
• Multicore programming and DSLs
• What is the obsession with C in EDA?
• Kunle Olukotun on DSLs
• Modeling hardware at a high level for software development

And there is the ChipDesign article from last year about using virtual platforms in the hardware design process all the way out to customers.

As part of explaining why this is cool and what it means, I have a guest blog posting over at Cadence’s blog site, called “Way Worse than the Real Thing”. The blog is posted under the general “TeamESL” “personality” on the blog site, which is used for people external to Cadence in the ESL space.

It all comes down to a simple classic tradeoff that I usually illustrate like this (using more neutral ground than computer systems; and with credit to Peter Magnusson who had this slide already in place when I joined Virtutech back in 2002):

What this is telling you is simple:

• You simulate something very large using large units, i.e., low level of detail; or
• You simulate something quite small using small units, i.e., high level of detail.

I wanted to test the idea that by using less detail, you can run larger test cases and therefore obtain better coverage of overall landscape than diving in and counting cycles in some small part of it. In the end, this made me cross the trillion instruction line — since each experiment took a few hundred billion target instructions to complete, repeating and tweaking during the development work definitely add up to more than a trillion instructions.

And this is where I have put my little finger close to my mouth and say:

‘I want one trillion instructions’

So what did I get from these trillion instructions?

An interesting study in how operating system overhead can have a big impact on the profitability of hardware accelerators. By running hundreds of test cases with different assigned computation latencies of a hardware accelerators, as well as different driver models for my hardware (all running under Linux on my favorite MPC8641D), a key diagram emerged:

Read the paper for all the details, but the key thing to note is that with a poor driver architecture, making the hardware 100 times faster resulted in zero gain in system performance. Had this experiment been performed on a bare-bones platform without a full operating system in place, I am fairly certain that the faster hardware would have been considered much more worthwhile.

In the end, I resorted to a driver variant where I had user-level code directly access the device programming interface via an mmap()-mapped memory region. Not pretty, essentially this was bare-metal programming wrapped inside a big cosy Linux package, but it sure was efficient compared to doing a kernel/user mode switch for each hardware operation. But even here, it turned out that making the hardware very very fast as opposed to just very fast had no benefit. It proves to me that the software has to be taken into account in full in order to properly evaluate an idea for a hardware design.

You could say that the poor results for acceleration here were due to my inept Linux driver programming skills, but that just underscores the key result: you have to take the software into account. If the conclusion is that a better Linux device driver programmer is needed, you have still decided that the key system bottleneck is not just the speed of the hardware, but how it is used. And that is exactly what system design needs to be about.

As an aside, playing around with a complete system like this, and automatically run large volumes of test with varying parameters was a really interesting experience. I must admit that getting to these trillions of instructions required  a few hours of simulation time, but nothing that could not be solved by leaving a computer running over lunch or a long meeting. The machine was modeled using standard Simics “software timing”, i.e., without any particular cache or pipeline or bus details, and it seems that that is usually all you need. Had I increased the level of detail and slowed things down by a factor of ten or a hundred, I would never have covered such a large set of test cases and been able to evaluate as many different variants of drivers and hardware speeds.

IBM did it before me

Finally, I found it interesting that an analogous experience about the effect of creating a complete software stack and testing what looks like a very good hardware idea was reported in an IBM paper from a few years ago, in “Application of full-system simulation in exploratory system design and development“, by Peterson et al, in the IBM Journal of Research and Development. Look at the section about the “MIP Morphing” feature, which is essentially cache locking. They do use a fairly detailed simulator for the end evaluation of their performance – but the key message is that by running a full software stack, they realized that just managing the feature was too hard in a realistic software environment to make it worthwhile:

Initially, the MIP morphing feature was well received by internal development and HPCS customers alike. The team was aware of the need to both manage this hardware feature at the OS level and provide portable abstractions to the programmer to exploit this feature in a productive way. …

And then:

The implementation effort was facilitated by Mambo, allowing the OS team to prototype the MIP morph idea in a controlled development environment. Taking the prototyping effort to this level of realism uncovered many complexities in supporting the MIP morph in a virtualized manner. ..

And finally:

By prototyping the software support that was needed at the OS level and exposing the usage issues at the application programmer’s level, the magnitude of the problem was exposed at its fullest. Further, the improvement in performance did not show a sufficient payback for the immense effort that would be required at the software level to support the idea, and as a result it was dropped from further consideration.

It seems that whatever you do, IBM did it first… and it validates the idea of full-system simulation and that software is king today.

The beginning of the argument was found in an EDA Design Line article titled “Unified Verification for Hardware and Embedded Software Developers” by Lauro Ritazzi of Eve USA. In it, he makes the following claim:

While some may have achieved the scope of jump-starting software development, they only address application programs that do not require an accurate representation of the underling hardware design. They fall short when testing the interaction of the embedded software with hardware, such as firmware, device drivers, operating systems and diagnostics. For this testing, embedded software developers need an accurate model of the hardware to validate their code, while hardware designers need fairly complete software to fully validate their application specific integrated circuit (ASIC) or SoC.

The interesting part here is really that jump-start is just for applications, and that OS and drivers require more details than a fast virtual platform can supply. I do not quite agree with this. But let’s first see what Frank Schirrmeister said:

the majority of the software development on virtual platforms is spent on firmware, device drivers, operating system porting and diagnostics. And that is not – as one could assume – on cycle accurate models, but on functionally accurate models with only essential timing, the type of models called loosely timed (LT) in SystemC.

I totally agree with this. As is evident from many different public use cases, OS, BSP, and driver development is a big use of virtual platforms. For example, last summer, Freescale announced the QorIQ P4080 with pretty good software support in terms of Linux and VxWorks operating systems, as well as some middleware stacks. All developed on Simics using an even more timing-abstracted model of the hardware.

However, Frank then makes the following claim that I have a harder time with:

In contrast, application software is developed more often than not using completely hardware independent techniques, including cross compilation from the host development machine using development kits like Apple’s iPhone development kit.

This is to some extent true, but as time goes on, I think this type of development environment is going to be less useful. Traditionally, OS vendors have had tools like VxSim and OSE SoftKernel in place to help customers “run code on their desktop”, while using the API of the operating system of choice. However, such solutions have lots of problems in how close they can get to the target.

• If you have any kind of third-party binary-only application, or want to use an existing binary component without lots of complex recompilation, you need a virtual platform running the underlying OS. You cannot squeeze that into a host-compiled API simulator.
• You are not using the same compiler and code-generation settings and build settings as you are for your actual target, and this can (read: will) introduce nasty compiler version issues.
• It forces you to maintain an additional build variant for your code, which can be pretty expensive for a complex build.
• You are not using the real OS scheduler, device drivers, and interrupt structure found on the target system. This can have a huge impact, especially for multithreaded multiprocessor systems.
• The API simulator needs to be kept in synch with the real software stack, and customized in the same way for any particular target. This is hard to get right (even though it has been done).
• The API simulator does not handle heterogeneous systems very well, such as chips or boards or racks mixing two or more different OS kernels in the same system (like a DSP and a main processor OS).
• API simulation completely falls apart when the OS is no longer the lowest level of the software stack, but you also have a hypervisor layer underneath the OSes on your target system. An API simulator simply cannot represent this kind of case.
• Using a virtual platform and the real target binaires also fits with the very important “fly what you test, test what you fly” principle of embedded software development.

For various subsets of these reasons, I see many users picking up virtual platforms as a way to streamline application development. For example, NASA recently selected a virtual platform based on Simics to develop the software for the new Orion spacecraft. That is going to be a complete software stack, not just OS and drivers, which tend to to be fairly off-the-shelf component for these kinds of systems. Most of the effort is on the application level, and the platform used is a virtual platform.

However, note that there are cases where a fast virtual platform like we are discussing here is not sufficient to validate all aspects of the code. I think the main reason we see different viewpoints on this, is that we are looking at very different types of software-hardware integration.

In a blog post I wrote last year on the dead-ness of cycle-accurate simulation, Grant Martin of Tensilica pointed out that some software desperately needs cycle-accuracy as it is intimately dependent on the timing of the hardware. This is certainly true for some aspects of drivers, and more so for the really early boot code.

Here, FPGA-based hardware-accelerated simulation of the actual design in VHDL or Verilog makes eminent sense as a way to get the details perfectly right. But that is only one part of a much greater system development puzzle, and it really only applies to very small subsystems as  it is kind of hard to fit much more than a single chip inside a hardware acceleration unit. Just as Frank Schirrmeister says, hardware accelerated simulation is very important. The nice article on the IBM z10 development that I blogged about earlier says exactly that: for some parts of the validation, there is no way around using the actual hardware RTL design.

And in the end, you have to test the timing and analogue aspects of a design on physical hardware anyway. There should not be too many suprises at this stage, if you have used all of the cool current tools right. But there surely will be some — even a VHDL simulation is a simulation, and not reality, after all.

Thanks to Simon Kågströms post (and the even better second-generation with screenshots) about using Eclipse for the Linux kernel, I have a much nicer work environment now for my ongoing work in learning Linux device drivers on PowerPC, which has helped me work my way through several hard-to-figure-out system calls. Here is a screenshot that I found pretty cool… the tool has found the definition and comments for the IRQ registration function:

2009-Feb-01:

I had to rebuild my indexing from scratch in the past weekend, and as a result, I have a word of warning: you have to create a “C project” in Eclipse, if you accidentally create a “Project”, the Eclipse workspace file will have the wrong name (.project instead of .cproject), and the autoconf-to-eclipse script will not work.

In practice, threading as a paradigm for software models of hardware circuits connected to a programmable processor brings more problems than it provides benefits in terms of “natural” modeling.

As I see it, the main alternative modeling paradigm is to use a classic event-driven system, where all activity is triggered by events and run the associated code to completion. This makes execution occur in a series of simulation steps in various part of the system, rather than as a set of (pseudo) concurrent tasks.

Threaded Problems

The most common complaint with threading is performance. This has become very clear in the case of using SystemC for transaction-level modeling. All advice in how to do good and fast TLM coding tells us to use SC_METHODs, which are essentially callbacks that are not active objects in their own right. Note that SystemC models found in the wild are often built on SC_THREADs despite this advice, as that is the “easiest” way to do things. Some convenience systems part of the OSCI TLM-2.0 library also rely on threads to convert between AT-style asynchronous and LT-style synchronous function calls (which is pretty unavoidable, but not applicable in the realm of high-performance simulation for virtual platforms).

Furthermore, using threading as a paradigm (even cooperative single-active-thread cooperative threads like in SystemC or classic MacOS) bring with it the problems of concurrent programming, in that you suddenly need to care about protecting data structures against conflicting accesses, worry about deadlocks, and similar concurrent programming issues. Without threads, all such issues go away.

Note that using threading as a modeling paradigm with truly concurrent execution of models will make the execution have all the problems of parallel programs, especially non-deterministic execution and hard-to-find bugs. At least a cooperative multitasking system tends to be deterministic in the way it goes wrong.

Threading as a hardware model programming style therefore makes concurrent multithreaded simulation harder rather than easier to achieve. Especially if the semantics of the simulation system specifies an interleaved model of execution as the semantics, which is the case for SystemC. In this cases, there is no way to really make SystemC parallel without adding parallelism as some extra library.

However, one of the biggest practical problems with threading is the problem of inspecting, changing, and checkpointing simulation state. With threads, you end up having state stored in local variables on the stacks in the system, as well as in processor registers, the program counter, and other places that are hard to get to from the outside.  This is not just me saying this, I found this well said in the sampalib white paper :

Using threads means that part of the simulation state is in stacks, which may limit the ability to persist the state of the simulation in checkpoints.

Using wait() implies context switch which are costly in terms of simulation speed, and thus often discouraged in guidelines for modeling SystemC™ models

To furthermore drive this point, all librariesfor general program state serialization that I have seen (for C++ and Java, for example) also rely on explicit state stored in objects, and explicitly do not support the “transient” state held in local variables and the program counter. Essentially, only heap-allocated objects are handled in serialization solutions.

Event-Driven Solutions

An event-driven transaction-level hardware simulation is coded in a different way from a naive threaded implementation (but not that differently from a more sophisticated threaded program).

Each device model has to make its state explicit as a set of variables, and preferably also declare these for access for an external tool using something like GreenSocs GreenControl or Simics Attributes. It also has to expose a set of functions to be called when events happen or other devices in the simulation system send a transaction into the device model.

Additionally, you should encapsulate all state in a model inside the model object and not expose it for direct access from the outside. A pure object-oriented style with accessor functions for everything is required for best modularity.

The advantages of this model are clear:

• Concurrency problems are reduced, since each function call will run to completion before any other object or function is activated. There is no need to worry about shared data variables, as they should not exist.
• Checkpointing and inspection is facilitated, since all state is now explicit and declared.
• Performance is typically increased, since there is no need to do context switches between threads. Locality is also increased by having functions run to completion before returning.
• True concurrency is easier to achieve, since each model can quite easily be considered a local-state, shared-nothing, explicit message-passing component similar to Erlang threads. This makes it possible for the simulation scheduler to run multiple models concurrently on multiple host threads. For more on this topic, see my SiCS Multicore Days 2008 presentation on how Simics was threaded.

The downside is that some people consider the programming more complicated. Which is really a matter of appearance over substance: event-driven programming tends to be more robust and easier to follow in the long run, since threaded programming makes things a bit too implicit.

Here is the basic example of a thread that does some periodic work.

Threaded style:

Thread_for_D():
  loop forever:
    do work...
    wait(some time)

Event-driven style, where we just repost an event each time we are called:

Time_callback():
  do work...
  post event(some time, Time_callback)

Another advantage of event-driven models is that such a paradigm makes it clear that you need to be able to accept any call into the model at any time. This makes for more robust code, since it is quite easy to (intentionally or by mistake) encode an expectation on the sequence of activity in a threaded that might not be what actually happens at run-time. In particular, the state of any protocol being acted on will need to be explicitly rather than implicitly represented.

There is much more to be said on how to code in this style, but there are long papers out there to read on this.

High-Performance Event-Driven Simulation

Note that in high-performance virtual platform-style simulation, processors will usually be a special case in both threaded and event-driven styles. That is since the flow of instructions that they execute constitute very many very small actions that cannot affort a context switch between each. Here, the advantage of the event-driven model is even clearer, given some special-casing of processors. This is another long story that I will not reiterate here, but basically, most events as discussed above will be memory accesses from a processor to read and write device registers, and each such memory access can be handled in a single simulation step. No need to switch context or do anything but handle a simple function call. By not having a wait() call to deal with, this mechanism can be kept simple and cheap — which is essentially using an SC_METHOD in SystemC. But in the complete absence of SC_THREADs and their ilk, many other things can be optimized even better.

The End

What I wanted to provide in this almost-article-length post was an idea for the problems that I see threads cause as a modeling paradigm for hardware models, and the advantages offered by a reactive event-driven style. For some reason, this is misunderstood in the modeling community at large, probably because most operating systems and simulation systems in common use today present various forms of threads as the way to model concurrent behavior. However, threads as a prominent user-level programming model are known to be bad in many ways… and modeling is no exception to this rule.

Note that I realize that threads are needed at some level in order to take advantage of multicore hardware, but I think they are best hidden inside a simpler framework that presents a simpler understandable semantics to the user.

The panel had a clear consensus, which nobody really challenged, that virtual platforms for software development are different in kind from virtual platforms for hardware development. Indeed, a the taxonomy of “hardware virtual platforms” versus “software virtual platforms” was used frequently and proved quite appropriate.

A software virtual platform has to be fast and its timing can be fairly approximate. It main value, in this context, is that can be created quickly and is useful for early software development and debug. Opinions differed, however, on how to produce them and where to go with them.

• Markus Willems from Synopsys had the position that they are produced in some appropriate way as a separate task from hardware development. SystemC was his language of choice.
• Peter Flake proposed a methodology where you start by developing the software virtual platform and then refine it down towards more detailed models and finally hardware. He brought up Virtutech DML and SystemRDL, as examples of languages pointing in this direction.
• Loic Le Toumelin considered the software virtual platform as a something that is generated from a common design entry point, using some form of synthesis that can also generate the hardware and the hardware virtual platform.
• I think my realistic position right now is that a software virtual platform is created as a separate item, but that we want to make this work as short and easy as possible and that in the future, the vision is similar to Peter Flake’s: start with a software virtual platform to define the hardware-software interface.

It was also interesting in how different the opinion was when we got to the detailed hardware-oriented virtual platforms. The ones that tend to be clock-cycle level and attempt to be cycle-accurate (CA) in many cases.

• Markus said that the only good way to build a CA model was to take the RTL and convert it, or run it in an FPGA prototype. He echoed the sentiments I wrote about in July, that ARM is getting out of cycle-accurate models and the general difficulty of creating such a model by hand.
• Peter pointed out that you can have CA models before RTL, as a design tool. I strongly agree with this model of working, it is common in industry and definitely one way to go. However, for existing hardware, I agree that RTL-to-CA seems reasonable, even if the resulting models are painfully slow.
• Loic wanted the CA to come from the same source as the software VP, and was very keen on their being in complete agreement on semantics of the hardware.

The third major discussion was about the required accuracy and fidelity-to-hardware of a virtual platform. With a consensus that a software virtual platform has to be fast and with timing approximated, it is still clear that many people are uncomfortable about this idea of not being “exactly like the hardware”.

For some purposes, you do need complete fidelity to the hardware timing in a CA model. Loic definitely could not accept anything less when giving a customer a virtual platform, and some people in the audience echoed the same sentiment. Most, however, agreed that most software work can be done with simple timing, and that it does not matter all that much if there are some functionality bugs or omissions in the virtual platform. It is still far better than no platform at all!

What is clearly needed, at least for virtual platforms close to a hardware design process, is a way to check the software virtual platform and hardware virtual platform against the functionality and maybe timing of the final RTL. In the cases that you have the RTL, which is far from always in my world.

There were some other questions about software development tools support (of course you use the same debugger and compiler as with a physical platform) and other issues where the panel was mostly in agreement. I guess some of this also indicates that virtual platforms are not yet universally understood and that most people have not really had any experience with them.

Overall, this was a fun panel, and I hope the audience enjoyed it too and learnt something in the process.

Gary Stringham’s lists a number of tips on how to create hardware-software interfaces. Some of them are echoed on his monthly newsletter, which is worth a read (even if it is a bit short on detail). Unfortunately, there seems to be no publicly available version of the text. Gary has definitely kept lecturing on the topic since, at venues like the ESC and DVCon it seems, but more recent lecture notes that I have found in the ESC proceedings are pretty sparse. I guess being a consultant teaching people to do these things for a fee makes you a bit hesitant to share all your knowledge freely with the world… I can understand that position.

Anyway. Some of the comments in the text indicate to me the great value that virtual platforms can bring to the actual design of hardware up front, not just as an execution vehicle for the final design, used by a software engineer who has to take whatever is given.

In particular, the issue of getting ASIC and Firmware designers to collaborate on the same thing at the same time. Quote:

The key to designing an ASIC is to get the firmware engineers involved early. They are the customers that will be using the ASIC. Unfortunately, getting them involved early is often difficult to do because the ASIC design has to start several months before the firmware engineers will get parts

And

When the parts do arrive, the roles are reversed. The firmware engineers are trying to work with
it while the ASIC engineers have mainly forgotten it and have moved on to new projects.

The proposed solution to this problem is to involve the firmware people in the hardware design review process, which is a good idea.

It would be even better, however, if the firmware people could have the hardware interface to try as a live thing rather than just reading the documents. This is exactly what virtual platforms offer: quickly build a fast simple prototype of the interface, and hand it over to the software engineers to try.

This is something that I am currently exploring in some detail with Simics, and that I wrote a piece about in Chip Design earlier this year. Fundamentally, I think this is feasible, provided that hardware designers do not fret too much about timing details and the precise performance of the final implementation, and focus more on the programming interface design first — and then later go on and make sure the timing and performance is right.

It is just like software development is supposed to be done: start by designing a useful interface to a piece of functionality, and then add in the details and optimize performance within the boundary of that interface. Of course, the interface might need some adjustments to support certain optimizations, but it is quick and easy to provide a new virtual platform with a new behavior to the software engineer. Much faster than providing a new piece of silicon or even new documentation.

Register design tools like SystemRDL (now being standardized by SPIRIT), Spectareg, Semifore, etc. all touch on this, but all seem to be lacking the ability to actually describe what a device does beyond some simple basics like software and hardware read/write properties. You really need a full expressive language to write a truly executable model of the hardware (and I like DML for this).

The following appears to be to sequence of events:

• Cadence press release, in September, about their “Palladium Incisive Palladium Dynamic Power Analysis and Cadence InCyte Chip Estimator“, quoting Ran:

Cadence Incisive Palladium Dynamic Power Analysis enables SoC designers, architects and validation engineers to quickly estimate the power consumption of their system during the design phase, analyzing the effects of running various real software stacks and other real-world stimuli. The new offerings also include the Cadence InCyte Chip Estimator, which can now provide what-if power analysis through exploration of different low-power techniques. The InCyte Chip Estimator also generates automatically the Si2 Common Power Format (CPF), which helps drive architectural power specification and intent into implementation and verification.

• Frank Schirrmeister blogged “On Chameleons, Low Power and the Marketing Power of Copy Editing“, basically saying that what Cadence was selling was something that was bound to the RTL level and thus arriving with estimates pretty late in the design process, after most important architecture decisions had been made. Instead, he proposed a flow using virtual prototypes that contained a sequence of successively better estimates, from the usual initial spreadsheet to estimates actually derived from RTL later in the process (or for IP blocks that already exist). Synopsys is not alone in this, Neosera and ChipVision are after similar ideas. I think this approach makes excellent sense, following the idea that getting some kind of approximate feedback from a complete system early in the process is better than getting lots of details from a small part of a system late in the process.
• Ran Avinun then blogged a reply to Frank, at “The Power of Cadence System Power Flow vs. Viewing from the Top“. His contention there is that virtual prototypes have their uses, but that real designers will be using hardware accelerators, as that provides the key accuracy needed to do real power work. Also, he sees the creation of a virtual platform as a big problem, and cites a number of cases where running the actual semi-final RTL with power simulation was key to project success. Also, Ran sees the time needed to create a virtual platform as a big obstacle.
• Frank then replied to the reply, at “Hammers, Nails and the Spirits That I Called …“… where he points out that Ran has some misconceptions about virtual platforms, admits that the Cadence flow works well, but that it does miss the point of early power estimation before the design is too frozen to be much changed. There is a pretty but hard-to-read diagram in the post, from a 2005 article he wrote while at ChipVision in Germany, pointing out the need to evaluate designs with actual test data from the real world.

What do I make of all of this?

Ran’s Points

I must admit that I think the Palladium hardware simulation accelerator boxes are very cool pieces of hardware, which at least used to be based on custom logic systems that use several cycles of a fixed sized hardware to simulate multiples of the hardware’s based emulation capacity (so 10M capacity system can use 10 cycles per target cycle to simulate 100M, for example). However, I do agree with Frank that these are dependent on having actual RTL in place to be of much use.

Another issue with hardware emulators is their overall availability: compared to the number of PCs available in an organization, they are going to be very limited. As discussed in many different forums, a key advantage of a pure virtual platform is that it can turn any programmer’s PC into a target system running the real target software. Without having to book time on a limited set of physical target machines, and hardware accelerators are such limited-in-supply hardware machines. So a virtual platform is much more available to people within, and especially outside, a design organization. Also, unless you are happy to release RTL for your design to people outside your organization, hardware acceleration is going to do little to help your end users get the most out of your design, pre-silicon.

My final gripe with hardware emulators is their limited scope. They tend to max out at a the borders of a single chip, or less. A virtual platform, on the other hand, has much more room to scale, to include multiple chips, multiple boards, or even complete racks and networks of networks. You cannot really do that in any hardware simulation, as it involves too many billions of gates running too many billions on instructions. The general rule of simulation still applies with hardware acceleration: you need to increase the level of abstraction to handle larger systems.

As to the problem faced by Ran’s customers, having RTL but no virtual platform: what were they thinking of? Seriously, if you want to do design today a virtual platform should be your starting point, not an afterthought. Time and again, we see examples today where using virtual platforms gets chips to customers ahead of time and provides the ability to test ideas before committing to final RTL. It seems that Ran agrees with this need, but his means are different:

“As was stated above, big reason our customers use RTL emulation platforms is for accuracy, and while virtual platforms can offer certain performance, eventually the need to accuracy becomes critical and can not be overlooked, even for initial performance and power estimation analysis. Frank seems to forget in his statement above that the average bring-up time of new virtual platforms takes 6-12 months while the average bring-up time of many emulated designs takes days.”

The time to create a virtual platform is actually pretty short, if you do it at a sufficently abstract level of detail and don’t worry too much about cycle accuracy. Also, that bringing up of an emulation depends on having a detailed RTL-level description to start with… which is not necessarily the case. I must say that the cited six to 12 months for a VP (for a single SoC as discussed here) sounds reasonable to me — if you are building a cycle-level model that tries to emulate the final timing (which might not be really feasible at at all). If you work at a higher level of abstraction like loosely-timed TLM, that time shrinks by a factor of ten or so. I agree that in the end, accuracy is critical – but before you get there, the approximations used by the VP will have gotten you pretty far in terms of software development and architecture testing.

Ran is also afraid of the lack of accuracy:

Now, even if you build this platform successfully 9-12 months in advance, how do you know that your virtual platform representing your real design? How do you connect it to your verification and implementation environment and realistic power information? Frank seems to overlook these things. Looking at the analogy of the story described at the blog above, using a system-level platform that is not targeting the actual hardware for performance analysis and power trade-offs guarantees that the Chamelon will become a snake and you will get bitten.

As with all simulations, virtual platforms needs to be used with care and understanding. It might also once again be a matter of system scale: for RTL simulation, you are looking inside a single chip, and the detailed design to save power there. With a VP, you might be looking at whether a particular OS kernel does even care to try to turn off unused hardware at all… and that might be just as important in the end as being accurate in how functional units turn on and off inside an accelerator.

In today’s software-driven systems that mostly consist of existing off-the-shelf hardware, not any particular SoC that is being designed right now, the large-scale behavior and smarts of the software in a setting containing lots of chips and functions is far more important than optimizations inside a chip.

Frank’s Points

Since Frank is a virtual platform supporter just like me, I instinctively agree with his points about VPs being pretty fast to develop and available long in advance of actual silicon. I like the way he deals with power in the ARM DevCon presentation cited (do have a look at it), but still there are some lingering doubts and issues…

What I have a hard time understanding is just how detailed the virtual platforms need to be. The use of SystemC TLM-2.0 LT is sensible for speed, but it seems from the DevCon presentation that the main emphasis is on AT-level (and therefore pretty slow) timing-accurate simulations that look at power cycle by cycle in the target. If that is the case, I think we could almost just as well go get ourselves a hardware accelerator, as cycle-level models (even if transaction-driven)

However, Frank also says this which I cannot but agree with: you should not always run around with a hammer and look at everything like a nail — any reasonable chip design process needs both virtual platforms and hardware accelerators, one cannot really replace the other:

When discussing this matter with a friend, he pointed out rightfully so that both Ran’s and my post suffer from “Hammer and Nail-itis”. In fact, he pointed out, the combination of Cadence’s estimators (InCyte), C based synthesis, Palladium, and Synopsys virtual would be pretty powerful! It’s a good thing then that we acquired Synplicity which brought us Synplify high-level synthesis and Confirma FPGA Prototyping to Synopsys, and of course, that we have existing interfaces between our Virtual Platforms and Eve’s solutions.

Conclusion

To me, the lesson from this discussion is clear: A virtual platform should be the starting point of a new design, but once you get down to RTL, hardware acceleration is really pretty useful. You need both, and VP should come first, not second. It is not an either-or issue, rather I expect system and chip designers to use both tools, and the only question is what should come first, which I think is naturally the simulation in the form of a virtual platform. That also allows the chip to be set into a system context, which is otherwise pretty hard before silicon arrives, and something that large system integrators are screaming for.