The area of virtualization is one that I would definitely have looked at as an opportunity had I started out as a PhD student today. The work of his group is a good example of how Simics is being used for research and teaching in universities around the world.

Going one level up in abstraction, I note that this is probably the first time I have published an actual interview. I have been active in writing things since high school, but it has pretty much always been direct writing, not interviewing.  However, I really hope that this is not the last. Having a series of user interviews on the Wind River blog could be really neat, as a way to dive deeply into some particular areas of technology. Will be interesting to see if any other university user is interested in being featured.

FLOSS Weekly

I was a bit disappointed that they did not discuss paravirtualization, the concept that at least I think is the main contribution of Xen. Paravirtualization is the idea of modifying the hardware interface of an operating system just slightly to make it much more convenient and efficient to run it on top of a hypervisor. This is really not that different from the architecture in the classic heavy-duty server world, with the programming interface of the Sun Niagara, IBM POWER, and IBM zSeries machines all being not bare-metal but rather a more structured API to a layer partially implemented in software.

Xen is shaping up nicely to be a lot of things to a lot of people. I liked their work on live migration (done in the same way as I think I read IBM doing it in POWER). The discussion on snapshotting was also good, even if I felt he introduced it as something completely new, which it is not. VMWare has had it for ages. And my favorite simulator, Simics, also has it in an even more powerful way. Another interesting aspect is the use of Xen on ARM-based mobile devices, even where no real hardware support exists for virtualization (incidentally, this was the starting point for Xen on the x86 back in 2001 and the reason I understand for going with a paravirtual solution rather than a VmWare-style binary rewrite solution). Busy field, with lots of contenders.

The open-source nature of Xen is also very important to the project (and the reason for Ian Pratt being interviewed by FLOSS Weekly, as that show is all about open-source projects). In addition to enabling the exploration of new additions to the core “product”, it also means that Xen is  a great research tool. With Xen, researchers can implement virtualization-related ideas in a real industrial-strength product, which makes it much easier to do applicable and realistic research. Indeed, one of the greatest thing with the current strong open-source trend in the world is that research institutions have a complete system and application software stack that can be modified at any point to try new ideas. From the hypervisor to operating systems, compilers, middleware, language virtual machines, and lots of server and desktop application software.

Pratt was not exactly overly humble, but I think he is correct in taking some credit for having put virtualization front-and-center in the x86 market. Before Xen, it was not at all as hot as a topic. Or it might just be coincidence. I was a bit annoyed at his lack of painting a bigger picture: I would have liked more credits and more comparisons to previous work, especially the IBM classic architectures and what VmWare and others have been doing on the PC. If you had no context and listened to this show, you could well come away with the impression that Xen invented virtualization, which is not really the case at all.

Note that FLOSS weekly is still one of my regular podcasts, despite the occasional episode that I take issue with.

However, like all host-compiled execution (which I also like to call API-level simulation) it suffered from some problems, and virtual platforms today might offer the speed of host-compiled simulation without all the problems.

The problems are these:

Most companies that are using host-code execution today use “explicit access”.  This means they require all places in the code that access the hardware to call read() and write() functions so every hardware access goes through a common set of functions and then they use #ifdef to change the hardware accesses to call the simulator if they are doing verification with host-code execution. If they are running on the target system, then pointer dereferences are used.

…

This is where implicit access came in. It provided a way to automatically trap pointer dereferences that were reading and writing to hardware locations and convert the load or store instruction into a simulated read or write. For reads it would put the result into the proper host CPU register and the user had no idea that a line of C code would magically turn into a bus transaction on a Verilog BFM

Yes, that is a right pain, and I have seen lots of solutions for it, none of which have the elegant simplicity of a processor simulation. The “implicit access” system is basically trying to trap memory accesses without overtly changing the source code of a program. I guess the best way to do this is binary instrumentation, but it is still very hard to get to work right and robustly. A simulator is simply much simpler in principle here.

Jason continues later on:

Given the hassle of host-code execution I would prefer to cross compile the software and run the target instruction set. Beyond the implicit or explicit access issue, this also eliminates issues with differences in data type sizes, data structure layout, byte order (endianess) and other differences between the host and target processor.

That is absolutely true! Jason does not mention the additional fun of what happens when the target is running an OS that is happily fielding interrupts, scheduling software tasks, etc. Also, that having to maintain a separate build target and maybe code variant is very expensive, process-wise. The expense that a good virtual platform incurs can be paid for pretty quickly once such reduced friction costs are factored in.

So I guess I pretty  much agree with all that Jason is saying, and thanks him for mentioning Simics. Thanks for the insights into what was done in the 1990s, it always interesting to get pointers to old fundamental and interesting work.

About how the virtual platforms actually work inside: it is not that complicated in principle (but pretty hairy to get it quite right and fast in practice). You have to simplify the timing of the target processor, you have to convert from target processor binaries to host binary format using some kind of just-in-time compilation technique (also called dynamic binary translation or code morphing), and you have to provide some kind of direct access to target memory for the target processor simulation (like the DMI feature in SystemC TLM-2.0, but usually the difficult bits are on the CPU side of that, not the memory side).  The most interesting bit is how to build the surroundign system model to not slow the CPU model down, and for this I can recommend a couple of pieces of writing:

• My ESC 2008 general intro to the subject of virtual prototypes (slides, paper)
• Virtutech white paper on system modeling

There were some interesting comments on how to define efficiency in a world of plentiful cores. The theme from my previous blog post called “Real-Time Control when Cores Become Free” came up several times during the talks, panels, and discussions. It seems that this year, everybody agreed that we are heading to 100s or 1000s of “self-respecting” cores on a single chip, and that with that kind of core count, it is not too important to keep them all busy at all times at any cost. As I stated earlier, cores and instructions are now free, while other aspects are limiting, turning the classic optimization imperatives of computing on its head. Operating systems will become more about space-sharing than time-sharing, and it might make sense to dedicate processing cores to the sole job of impersonating peripheral units or doing polling work. Operating systems can also be simplified when the job of time-sharing is taken away, even if communications and resource management might well bring in some new interesting issues.

So, what is efficiency in this kind of environment?

It was clear from both the panel discussion and discussions over lunch that programmer productivity and predictability are things that can be traded for absolute100% load on all cores. Just like making 100% use of main memory is not usually a design goal today, so making 100% use of all processor cores is not a reasonable the goal tomorrow. Some resources are so plentiful that it makes sense not to try to push usage to the limit.

With 100s of cores, it is quite likely that even for the most performance-demanding loads like doing LTE decoding, it is not worth the herculean effort to get all cores running at full speed all the time. Getting 80% to 90% of the cores working on a workload is probably a good tradeoff.

Another tradeoff you can make is to increase determinism and debuggability by assigning tasks and schedules in a more static and predictable way. Instead of trying to balance loads across the cores, tasks could be assigned in some static or semi-static manner, so that the execution of a system can be repeated with some chance of success. That should not be too hard if all cores run a static cyclic scheduler, for example, or even a single task on each core. Dynamic scheduling might well be a global suboptimization in a world with plenty of cores, as it just makes things more complex for a fairly small increase in actual efficiency. You could also imagine putting debug agents and code on certain cores just to help you get better insight into what the system is doing. A bit like I blogged about after last year’s Multicore Day, asking designers to put more silicon into debug functionality. Maybe in a 100s of core device, we allocate cores to debug as well (I do not think we can do without dedicated debug circuitry, as that is needed to effect things like stopping cores quickly and similar).,

When I heard this, my gut reaction was that “hey, that is not particularly environmental” — any kind of waste of resources is really an anathema to the ecologically friendly society we need to build over the next 10-20 years. But then someone pointed out that a key part of the efficiency equation is that you turn off the unused cores and accelerators so they do not use any power. And since the cores are a resource that keeps increasing in count from basically the same use of resources  (manufacturing a chip will cost about the same amount of energy and materials for each chip, but with finer geometries you pack double the number of cores in it), it should be fine. It should also be noted that multicore computing by itself allows for more efficient processing units, for a variety of reasons.

Robustness also tends to increase if you have some slack in your system. For example, most hard real-time systems insist on not being more than 80% loaded or so (on a single CPU) even at the worst of tested times. To have some margin for the inevitable unexpected situations. For a 100s of cores device, you might also want to spare some cores for the case that hardware faults crop up in certain parts of the chip. Then you can shift loads to other cores (which obviously requires a pretty resilient interconnect to make any sense).

This final point bring me to my final thought on this was of building computing systems: in some way, we get closer to physical engineering habits when cores are free. We do not build bridges with the minimum amount of concrete and steel to handle the load we expect. Instead, there is a margin of error of a factor of three or five or so, to make sure that even in the most unexpected of unknown circumstances, that bridge will still stand. In a similar way, we might be able to use lots of free cores to engineer software systems that have far more resilience in them than todays systems that keep trying to make maximum use of the resource of clock cycles and instruction processing count. I do not quite know how that kind of system would look, but the analogy is very interesting.

Both IE8 and Chrome have taken to running each tab of a multi-tab browser as its own protected process, to make it both parallel processing and to increase robustness. I think that is a very good idea, and I am waiting for Firefox to catch up.

Why does running a browser as a parallel program make sense? If you look at the tradition, when the web started, you would load a page, render it, and read it for a long time. With multiple tabs and windows, each such display was really also just a set of static prints of pages that you flipped between. No point in being parallel there. However, in recent years, the web page model is changing. Pages are becoming far more active, starting a long time ago with Java applets, Active-X controls, and similar, and today the main drivers seem to be Javascript/AJAX/Web 2.0 pages and media players like Flash and Silverlight.

Basically, we see another example of a domain change enabling parallel processing to be applied. The domain of web pages has changed from single-shot renderings of single pages at a time, which is essentially serial, to lots of active programs running at the same time.

I think we are going to see more of parallel processing being used to enable richer user experience. This is one way that the world is making use of the increase in computing power and communications bandwidth, just because it is there. It gives us a nice sea of threads to run in parallel — the only issue probably being IO bandwidth and cache restrictions of single chips.

The use of processes for robustness is kind of an application-level virutalization. The OS provides isolation between processes, just like virtualization provide isolation between operating systems.

Living in a decently old city does provoke a certain interest in history and how to access the past. Note that Uppsala, like everything in Sweden, is fairly recent on the grand scale of things. We have a cathedral from the 1200s, and some older churches going back to the 1000s, but before that there is not much to boast about. It is not like Rome or Egypt or China with thousands of years of civilization and with impressive buildings that still stand. But I digress…

The real problem with archives and digital technology is that it is really hard to preserve a large portion of the intellectual data that we produce today. When I was an archivist at Smålands Nation here at the university in the 1990s, it was real fun to browse through the old papers we had in the archive (only stuff from the past century, all the really old and valuable stuff was in professional hands at the national archives). Reading a menu from a party held in 1945, or the time plan for a large formal celebration in the 1950s, or a guest list from such an event was actually quite intruiging. You could see how tastes and traditions change, even as students all believe they are maintaining ages-old traditions faithfully… Browsing the member list in old handwritten ledgers was also great fun.

In another fifty year’s time, what will we have left from today’s activities? The party plans are now all just printouts from a throwaway word document, the member list a database on a PC, and unless you make a conscious effort, memories and printed papers will quickly fade (you do need to use the appropriate laser toner on low-acid archive-quality paper if you want the things to last even a decade, let alone a hundred years) and the students fifty years from now will have no idea on we ran our parties, who were in the nation, and other such facts of daily life. “Print it out” is easy to say but hard to do. Much of today’s digital data does not really print out in a useful way. A wiki, for example, can be printed as a snapshot. But the links and the edit history is gone.

But imagine if we could retain these digital documents accessible, in their living digital form. Having a proper database available for use and query is so much more interesting and powerful than a stack of papers. Being able to look at the audit history of a wiki, or the commit history of a program’s source code would offer a much deeper insight into the people and processes that produced the end results compared to just looking at a static snapshot.

What does it take to do that? A huge and known problem in digital archiving is the physical media access. All great archival institutions in the world have scores of different tape readers, disk readers, some new, some antique, and try to use these to pry the bits of equally antiquated storage media. This is bad, just like we have problems looking at old movies as the reams of film are literally falling apart. Clay tablets start to look positively sophisticated and a smart choice compared to our very brittle technologies.

Second, once we have the bits, what can we do with them? Now we have to tackle the file formats, and having pried them open, to find the fonts and character sets which are suitable for displaying them. Rendering a MacWrite document from 1985 on a modern machine will likely not quite give the same wonderful black-and-white 72-dpi view that you got back then… Or figuring out that for a while, we used 7-bit ASCII to write Swedish texts by replacing [, ], |, {, }, \ with å, ä, ö, Å, Ä, Ö (not in exactly that order — but I used to be fluent in reading and writing that!). Even worse, decoding a database file or purely binary CAD/CAM file is going to require some serious reverse engineering of the old programs that created them… unless you could run these same old programs in some way.

At least to me, an “obvious” solution is to use virtual platform technology to make keep the old software stacks running. Some archivists are apparently working on OS emulation of various kinds to keep the old software running, but I do think that it is much simpler and more general to just simulate the entire machine hardware. This will result in the smallest risk of error, and the greatest fidelity to the original software as oddities such as word lengths and character sets can be faithfully reproduced. Simply because the SW-HW interface in a machine is the best document and narrowest interface in the entire machine. Old tapes and disks should be turned into files stored on the mass storage of current machines, and these files can then be migrated to new machines as they come online. As long as you maintain the copies of the materials by reproducing them on new hard drives, they will not rot.

It will also have the property that the entire look and feel of the old systems remains accessible. Which is a blessing in that it maintains our digital heritage and lets computer historians also go back and see how old software looked and worked. It is also considered a curse by practising archivists who do not particularly like the idea of learning how to operate strange old operating systems with arcane command lines. I think this is not necessarily a big problem. Just like people today study and learn ancient languages to learn more about ancient cultures as embodied in their written records, I can see historians of the future learning to use old operating systems and programming languages to study the ancient culture of our time as embodied in our computerized records.

Creating these kinds of virtual platforms is quite different from the dominant virtual platform thinking in the commercial market place today which is focused on modeling new and future hardwares. Rather, we need to study an existing artefact and make a very good model of it. It is a different type of task, requiring slightly different types of tools.

Of particular importance is that the simulation platform totally insulates simulation models from the underlying machine. You need to be able to port the simulation to new operating systems, machines, and architectures without changing a line in the source code of the models. This means making sure to locate all host dependencies inside the simulation framework, and making sure that models are completely portable regardless of word lengths, endianness, and other aspects. You want to be able to take a model that you run today on an x86-64 on Linux and run it on some future 128-bit middle-endian platform with a completely different type of operating system. Basically, the simulation platform has to be a complete virtual machine in its own right. Probably, over time, we are going to add more layers of virtual platforms to get around really major shifts in computer system architecture. There is nothing saying that you can only virtualize once, already the IBM S/370 showed that you could virtualize recursively and to an arbitrary depth.

I do think that some tools that we have today are perfectly useful for this kind of undertaking.

The business aspects are going to be interesting, though. We would like to use the technologies already available in the sophisticated commercial virtual platform tools, that’s kind of given. But the archival market place is not the most lucractive… and that they would like to have very good insurance that the tools remain available. Including source-code escrow that would make the requirements of 25-years military projects look positively light.

Developing a completely new totally open-source solution sounds like a nice idea in theory, but is probably a bit too much work. It also takes no advantage of all the commercial technology available today. Maybe as the virtual platform market matures, the industry can come up with some way to provide this technology for the greater benefit of society. Basically, doing a bit of social work where we really have the tools to help.

I have no good solutions to that to offer right now.

But imagine how cool it would be to, in fifty years from now, gather the old team from Virtutech around some kind of display device and watch an ancient Simics 4.0 run on top of a virtual quadcore x86 with Linux 2.6… all running on top of Simics 29.0 (estimating one major version every two years) on some future I-have-no-idea-what-it-will-be hardware and software system. That really is something that would be cool to see happen.

If you happen to be at the FTF in Orlando, do come by and look at the demos!

I have been involved in this work for the past year, and it is wonderful to finally see it coming out and be able to talk about it.

The paper in question was published in 1969, and is titled “A program simulator by partial interpretation“. In the previous post, I took note of its use of direct execution of software plus trapping of privileged instructions, but that was not really the most interesting bits in there.

They lay out in quite simple terms most of the key ideas behind today’s fast virtual platforms. Here are the best parts:

• They note that simulation of a computer is often used to overcome debugging difficulties, in particular repeating failed runs and tracing all that is going on in the target machine.
• They are hunting down race conditions using the simulator.
• They use recorded input and output to drive a deterministic simulation even of workloads involving communication with the external world.
• They simulate multiple processors on top of a single physical processor by means of giving each processor a certain time slice to do its work before switching to the next processor. This is known as temporal decoupling or quantized simulation today, and is a key to the high speed of solutions such as Simics. They note the same tradeoffs as we see today, 40 years later, for doing this: shorter slices more accurately depict the parallelism, but also cost performance.
• The temporally decoupled simulation also includes timers and similar non-CPU-hardware. Just like we do it today for virtual platforms.
• In a temporally decoupled simulation, they optimize the simulation of the IDL, Idle, instruction. When it is encountered, they skip immediately to the end of the time slice. This is what we today call idle-loop optimization or hypersimulation, and which is absolutely key to achieving scalable simulation of large multiprocessor and multi-machine setups (since most parts of a system are not usually maximally loaded).
• They are debugging operating systems on the simulator, not just user-level code.

The computer in question is a Japanese System/360-compatible machine called the HITAC-8400. The work was reported in 1969, but actually carried out in 1967.

There are some differences in scale and kind compared to today’s virtual platforms, but none that detract from the underlying principles. The 1967 system is host-on-host, so it is not the kind of cross-environment that is most common in today’s virtual platforms (Power Arch on x86, ARM on x86, etc.). The IO system is much easier to simulate since it is part of the instruction set of the processor rather than being a set of complex memory-mapped peripherals.

So the 1970 rule strikes again. Not the IBM rule, this time, this was all done by Hitachi. There are traces of similar work at IBM in other papers, but I have not been able to locate actual copies of any publication.

The interesting bit is really the uses that this system is put to:

In promoting the ETSS project a program simulator based on an idea of partial interpretation has been constructed, and its principle and design are described in the paper. This new approach has been introduced to provide the simulator with such features as high speed and high accuracy in simulation and simplification in implementation. The essence of the idea of partial interpretation is using direct execution of instructions by hardware and simulation of them by an interpreter in combination, wherewith the hardware interrupt mechanism intermediates the two phases of the whole simulation. An interruption takes place when executing a “privileged” instruction, which triggers the simulation of the instruction. The other type of instructions are normally rendered to direct execution by hardware. The simulation method for devices operating in parallel is also described with respect to the timing control and scheduling. A program simulator of this type provides a powerful tool for debugging “supervisor ” programs and opens a new approach to system expansion.

Note that last part. This is essentially a virtual machine used for operating-system debug. So far, the earliest mention of this idea that I have found. There are similar ideas in a classic 1972 IBM paper. If anyone has seen anything older, please comment and tell me!

It is also fun reading these old papers… they are usually scanned from a paper copy, and therefore really show how papers looked and felt forty years ago. Long before desktop publishing, or even TeX.

Their main important assumption is that the VMM cannot be tailored to avoid detection by any particular piece of software, but has to be sufficiently like the real thing to fool something the first time it appears. They discuss from the perspective of virtualization solutions like VmWare that aim at high performance before all else. The virtual PCs generated by VmWare, Parallels, KQemu, and others are all compatible with physical PCs — run the same software — but are not at all identical in detail. So they are not transparent in the words of the paper. This means that they are quite easy to spot.

There are some holes in functional differences that VMMs can quite easily plug. The paper shows how you can get a different-sized TLB (compared to the physical hardware), for example, from interference from the VMM. This can obviously be fixed in the VMM, at a cost in performance. The reason such differences are there is that VMMs are optimized for performance at almost any cost. As long as the requisite operating systems run as they should, the VMM is fine even if it is does actually correspond to any particular existing physical machine. This is a testament to the tolerance of modern operating systems towards their hardware. Basically, any OS that probes hardware and discovers what is there will work fine as long as the (virtual) hardware exposes devices that it can recognize. This is quite different from the 1970s or 1980s where an OS would definitely expect a very particular hardware setup with very peculiar timing to run at all. Thus, making a VMM totally identical to some physical machine is a waste of effort and performance.

Paravirtual approaches like Xen and what Sun has with Niagara and IBM on their Power servers, where the OS is rewritten by having drivers for a purely virtual hardware/software interface is an obvious generalization from the VmWare compatibility approach. Compatible versus transparent/invisible virtualization is really only an issue in the x86 PC world, since all other datacenter architectures are virtual by definition and all operating systems work towards a standard virtual layer. In such an environment, I have hard time seeing that the question posed in the paper does even make sense. You are always virtualized, period.

Embedded Virtual Platforms

Anyhow, back to the main thread. There is still a large set of targets where transparency and compatibility are of interest. x86 PCs is one such target, it is an interesting question for older architectures (Alpha, Vax, Sun and IBM in older generations). In particular, it is an important topic for embedded systems where you want to use virtual or simulated approaches to develop and test software. As part of that software development process on a virtual machine, you could potentially be examining malware of various kinds. A good not-too-hypothetical example are mobile phone viruses.

If we look at embedded system virtual platforms, the functionality of the simulator is usually more complete and more like a particular physical machine than what a VmWare-style datacenter VMM. This is partially due to embedded software stacks tending to be a bit pickier about what they run on, and partially due to the simple fact that the goal really IS to expose the hardware/software interface of a particular piece of hardware as closely as possible. Also, since this is usually cross-targets (Power Arch on x86, for example), there is no performance gain from using features of the host directly. So items like TLB counts, memory layout, memory content, flash memory programming, etc. are all going to be functionally identical to the physical machine.

Timing is Key

Thus, just like for a patched VmWare-style VMM as discussed in the article, the main attack vector remains timing.

The best way, according to the authors, to spot a VMM is to look for timing differences compared to the behavior on normal hardware. Despite the inherent variability of typical hardware, there are cases where VMMs by necessity vary detectable amounts. I would say this means a factor five or more over many tests of a case.

The authors discuss whether tools like Virtutech Simics could be used to overcome this problem in the context of x86 PCs. I think the main argument for something like Simics for this purpose is that by simulating the entire hardware platform and providing all timing measurements from a strong virtual time base, you do not see the types of time differences that can be used to detect a “normal” VMM. However, since the paper considers Simics and SimNow (from AMD) to be about ten times slower than native hardware, you can always detect them using a non-local time source. That is likely true. But it less obviously true for an embedded target where the simulator running on a fast PC might well be just as fast as the target.

The Multicore Timing Attack

A more intriguing aspect of embedded virtual platforms that could be used to detect virtual platforms is how simulation of multicore machines is handled. For performance reasons, simulators use temporal decoupling, where each virtual processors is run for a “long” time slice before switching to the next. We discussed the effect of this in a recent presentation at the multicore expo (link to previous blog post), and some of that data is worth repeating.

Here is a slide explaining how temporal decoupling works:

So what does this mean in practice for detecting that you are running in a virtual machine?

It means that the communication latency between parallel threads is proportional to the size of the time slicing. If you have two threads progressing in parallel doing spinlocks, on a real machine they will be stealing the lock from each other all the time. On a temporally decoupled simulator, you will rather see a behavior where you can take the lock and then recapture it a few times before missing it. This effect was captured by a simple test program that we wrote, and the data is shown in the slide below:

The program here is running two threads in parallel, updating a shared variable, with three types of locking for the accesses:

• No locking at all
• A local lock to each thread being used (“fake locking”)
• A proper lock

The interesting behavior is the execution time of the program for each of these locking styles. Obviously, running with no lock is the fastest, and with proper locking the slowest. The relative speed of these is the factor to consider. On real hardware, this program observes a very steep increase in execution time when using proper locking. On the simulator, as seen above, the difference in execution time between fake locking and proper locking is significantly smaller when using a long time slice compared to when using a short time slice. The behavior on physical machines is much more like that observed at time slice lengths of ten than that at time slices of 10000.

Normally, a multiprocessor simulator with any ambition to be fast has to use a time slice of 1000 or more. Thus, detecting that you are running inside a simulator is quite simple. If the outside world time seems right, check if you can see strange timing behavior when using locks. Since high speed requires a long time slice, you cannot have both correct real-world timing and a large performance difference. And on the other hand, if the behavior with locking seems reasonable, you should check the real-world time — as a simulator with a short time slice will be way slower than the real world.

The paper authors note a similar aspect in desktop/server x86 VMM detection. They discuss “performance cliffs” that appear when doing “unusual” things. For example, VmWare is engineered assuming a minimum use of self-modifying code. Performance is much worse if you use it extensively, and this can be used to detect VmWare quite effectively. This effect is quite similar to the time slice effect in embedded virtual platforms.

Hope you enjoyed this fairly long rant. And we have not even begun exhausting the contents of this topic… luckily, these discrepancies only very rarely impact the usefulness of virtual platforms. Since most software even on an embedded system does not care about detailed timing like this. In the example above, we still see the lock contention. So we know that we are getting an increase in execution time from the lock. Only not a complete picture of what it means in absolute terms. We will still find missing locks and overused locks.

In a virtualization-based setup, you use a single hypervisor program that then controls several single-processor operating systems running on the machine. That hypervisor also takes care of allocating shared resources to the operating systems, sometimes by sharing a single physical resources, sometimes by only letting one operating system access a certain resource. So in this case, you can actually reuse existing OS images on a new multicore chip and transparently transform an existing system.

Too bad there is still no embedded processor with strong support for heavy-duty virtualization like this.

On the other hand, it might be a passing need. The transition of old applications to new hardware will always involve some rewrite and retouch, and if that means doing a bit of change in the OS setup to handle an AMP case nicely, it is probably not too expensive (compared to redoing applications on top of the OS to be truly SMP). And for virtualization, this means that you can use a Xen-style paravirtual approach where the OS is modified to run on top of a simple hypervisor.

Running and booting an unmodified binary install of an OS is likely more of a server/desktop problem than one for embedded applications. We are going to see virtualization support in hardware to help light-weight approaches be even more efficient, and also to tackle the security issues of rogue code getting into some OS image. Hardware support is needed to contain an OS that has been taken over by bad guys, no amount of cooperation between OSes in an AMP setting is going to prevent that.

The basic premise is definitely one that makes sense. We have seen this at Virtutech, working with stripped-down Linux images on various embedded boards. Turning on and off individual services often has a significant impact on both execution speed and memory consumption of a particular target. The gut reaction when setting up a new target is to think about what can be stripped out rather than on putting in everything that could be useful. Since that will result in a bloated image that will consume resources with little additional value.

The difference is even more obvious for Windows machines, where the speed of the same virtual machine running NT, 2000, XP, or Vista is incredibly different. With Vista and XP in particular, there is much going on in the background eating up memory and processor time. Some simple tuning tricks like turning off animations in the GUI or background indexing can have a very large impact on speed. Same thing goes for graphical desktop Linux distributions, where turning off eye candy results in significant speed increases.

So I believe that Fastscale can offer what they promise, it is a nice simple idea. But as always with a simple idea that makes for a powerful product, there is something more than just the idea. There is a somewhat tricky piece of execution.

The trick that Fastscale brings is to automate the process of creating a minimal but sufficient substrate for a particular application, given the app, target OS, and server hardware. Sounds like dependency checking once you have the data on what needs what, but finding out the particular dependence chains in any particular OS is hard work. Doing it manually is pretty painful, and the Linux kernel configurator is not always up to the task.

It is also a nice example of a case where open-source opens up for new innovation. Doing this kind of configuration to Windows is much harder in practice, if nothing else since you do not get to recompile the kernel yourself.

Also, Linux in particular has rather good support for removing non-essential components, I believe to some extent thanks to its extensive use in embedded software. Embedded people have a tradition of application-specific configurations, and here is a nice case of when that results in a better desktop/server product in the end. Who said embedded requirements were just for a niche market?

I presented a short summary of a paper I and a colleague at Virtutech wrote last year together with Ericsson and TietoEnator, on the Simics-based simulator for the Ericsson CPP system (see the publications page for 2006 and soon for 2007). I also presented the Simics tool and demoed it in the demo session. Overall, nice to be talking to the mixed academic-industrial audience.

The papers presented really indicate current trends in embedded and real-time systems development. Focus is on topics like model-driven architecture and component-based software development, and on how to make these abstract ways to develop code create code that can actually run well on resource-limited processors. Which leads into scheduling analysis and the analysis of properties of collections of objects from the properties of the component objects. For example, what is the worst-case execution time of a certain piece of software, given the context of a particular system configuration. The industry cases gravitated towards the automotive domain, which is not surprising considering the make-up of Swedish industry. Mobile phones were notably absent, but they are not very real-time systems anyway.

The keynotes by Edward Lee in Berkeley (on his ideas for programming parallel processor machines) and by Erik Hagersten (on why multicore is different from multiprocessors) were well-received and well-presented. Nothing particularly new, but nice overviews but good speakers. I really like the ideas of Lee on how to program parallel machines using sequential programs coordinated by a higher-level coordination layer. It is a nice way to separate concerns.

Overall, I appreciate the effort of the organizers and really regret that I could stay for the conference dinner.