After a long break, this is another blog post in the series of “how to do modeling for virtual platforms”. The previous installments dealt with checkpointing and determinism.
This post is about the use of indirection in a model to increase its flexibility and ease of use, at the cost of a bit more work for the first model to be created.In particular, indirection in the sense of having explicit objects in a simulation to represent things like networks and cables connecting virtual machines.
There is a well-known saying (by David Wheeler) that “any problem in computer science can be solved with another layer of indirection”. Among computer architects, this is often used with addition “…or a cache”. I think this is true, most of the time. The number of times that adding some indirection to an architecture for a program has simplified it — or made it feasible at all — are too many to count. It is at the very core of object-oriented programming, and the number of times you end up passing around function pointers is innumerable.
In the world of virtual platforms, there is one particular area where I see a pretty useful layer of indirection missing. Networks. Many virtual platform solutions offer various ways of connecting a virtual platform to a physical world, for interfaces like USB, Ethernet, or serial. Most virtual platforms achieve this by making the virtual hardware directly connect to the outside world.
Here is an illustration for Ethernet, where I have included a PHY in the picture. Quite often, you don’t even get that, just an Ethernet device that includes its PHY and connects out to a physical network. That’s what Qemu tends to do, for example.
This approach is the simplest when your thinking is that you will model a single device, and simulate one virtual machine at a time, connecting to the physical network to receive stimulus. For USB, this means the useful feature of connecting a camera or USB disk on your PC to the virtual machine. And as a bonus, you can connect multiple machines together using some form of cross-connection on the PC (such as TAP network interface).
However, there is a much better structure that is employed in some simulators. It is based on making each network an explicit object in the simulation, and have all virtual devices talk to the virtual network. Connections to the physical world are then handled by the virtual network, or, even better, by another device attached to the same virtual network.What you also get is the ability to connect multiple virtual devices to each other over the virtual network, and to easily write simulation modules that inspect or do fault-injection on the network traffic.
The picture below illustrates the idea for Ethernet:
The cost of this architecture is that you have to create the virtual network object, and invent the interface between devices and the network. This increases the cost for the first network device you create, and if all you are tasked with is that single device, I can see why some simulation designers took the direct route. However, if you think about the task of creating tens of devices connecting to the same type of network, the “cost” of creating a virtual network is actually negative. Using an indirect approach like this makes creating each device simpler, and each device immediately gets the benefit of all the services that have been added to the virtual network. As long as a device can connect to the virtual network, it can connect to the physical network without any extra coding or cost.
Encapsulating entire networks with multiple virtual machines within a single simulation session is also very beneficial for control, inspection, and determinism. Relying on a physical connection between virtual machines makes all packets pass the unreliable and random real world on their way between machines, destroying any determinism or control you might have hoped to incur.
In the world of SystemC simulation, an indirect approach like this is also a way to overcome some silly language limitations. Unbelievable as it might sound to the uninitiated, in SystemC you set up a simulation once into a single static setup (in something called the elaboration phase), and then that is what you simulate. There is no option to setup connection between modules or even add new modules to the simulation after the initial setup. Here, you can use a layer of indirection as a work-around. At the start of simulation, connect all devices that might at some point in time be connected to a particular network to that network. During simulation, configure and reconfigure the network module to only allow traffic from and to certain modules, essentially creating a useful illusion that they are connected and disconnected from the network.
I hope I have convinced you: if you ever build a virtual platform, make sure to make all connections indirect.