There are times when working with virtual hardware and not real hardware feels very liberating and efficient (not to mention safe). Bringing up, modifying, and extending operating systems is one obvious such case. Recently, I have been preparing an open-source-based demonstration and education systems based on embedded PowerPC machines, and teaching myself how to do Linux device drivers in the process. This really brought out the best in virtual platform use.
The final result of my efforts will be more public early next year, when the students I have put to work on my Linux-based setup come back and show me what they accomplished (or not). Until then, here are some small tidbits on how easy it is to work with kernel-level code in a virtual machine. Actually, if I had been working on real hardware, I am not that certain that I would have had anything but a bricked machine in front of me — to put it simply, flash reprogramming seems to hate me, and I have managed to fail or destroy a few embedded boards that have been unlucky enough to cross my path.
The virtual platform was really very helpful to diagnose all the mistakes I made while creating my driver and making it talk to my custom hardware.
First of all, it was dead easy to test a new version of the driver: start the simulation from a checkpoint of a booted and configured machine, load the driver into the target file-system using the Simicsfs backdoor (similar to the VmWare hostfs solution), and then insmod it. This was automated in a script that typed the needed commands on the target-command line with no manual intervention. Each iteration takes a few seconds, which is just as fast an convenient as testing a simple program directly on the host.
Diagnosing what went wrong was greatly facilitated by the simulator: did the driver access the device I had prepared for it? Were values read as expected? Obviously, there were a lot of such cases, I am not the most expert device driver programmer (yet).
Here is one particularly interesting example: I empirically learnt that the Linux kernel “readl” function is always reading data little-endian, even on a big-endian machine. You have to use “readl_be” to get the big-endian data from a big-endian device attached to a big-endian machine. I guess the behavior makes sense for reuse of drivers across architectures, but it sure confused me when my driver was reading the right register but complaining about bad contents.
The simulator showed the problem very plainly:
- “value read is 0xabcd0101 (BE)”. Ok that looks right.
- “register r3 contains 0x0101cdab”. Strange, looks like the wrong byte order. WHY I screamed to myself.
- Using reverse execution to step back one instruction showed that the load instruction used was a byte-swapping 32-bit access. Aha!.
- Go into Linux kernel headers (include/asm/io.h) to find that there were a bunch of other varieties available, and guess that readl_be() was the right solution.
- Change device driver code, recompile, and retest. Now it worked.
I would have assumed that the book I was using as my guide, the highly-recommended Linux Device Drivers, 3rd edition” would have told me this. But it did not, as it is annoyingly tied to the horrible standard PC. It could really do with some extra chapters on drivers for PowerPC, ARM, and MIPS (to name some of the most important non-x86 architectures out there).
On the other side of the fence, I am using Virtutech DML to do the actual device, and that is working out very well. In my setup right now, I can change the device driver and the hardware it drives, recompile both, and then run an automated test script that starts from a checkpoint, inserts the hardware model in target memory, loads the device driver, and tests it in about five seconds. Very handy, and all completely automatic. The ability to load and insert hardware models on the fly during simulation is really very convenient here — I would have to have to reboot the target Linux from scratch each time I wanted to add or remove things from the virtual platform hardware setup.
To sum things up, so far, I have learnt quite a lot about doing Linux device drivers and how to setup hardware in a Linux system, and I think it would have been much harder to learn and experiment like I have done had I been stuck with physical hardware (not to mention the plain impossiblity of just inserting a new piece of hardware in a simple way into a physical system).
It really shows that quite often, virtual hardware is “even better than the real thing”.
For fun, here is a screenshot of a complete test run of loading the device driver:
I might have mentioned it before, but Eclipse really shines when writing code for the Linux kernel. I’m quite amazed that it manages to lookup the proper definition of all those *very* nasty Linux macros, but it does. Recommended if you haven’t tried it.
Then there is the thing with Linux: There are typically multiple ways of doing the same thing, and not always clear which one is correct. For your readl example, I tend to use in_be32/in_le32 instead depending on what endianness I’m after. It’s explicit at least.
I also before did some horrible hack to get some files into sysfs (/sys). I now know how to do it properly, and the end result is beautiful, but again not something which immediately obvious.
Developing on real hardware, especially U-boot, is OK as long as you can reflash the board via a JTAG interface 🙂
I am now trying to use Eclipse… we’ll see how it goes. Followed Simon’s instructions at http://simonkagstrom.livejournal.com/31079.html?view=19559#t19559 , think I got things right.
Maybe that will help me figure out how IRQs get locked in the PowerPC architecture — seems to have to be registered for use in the device tree blob or something like that. And how to convince the kernel to accept an arbitrary number of cores, and not just two.
> I also before did some horrible hack to get some files into sysfs (/sys). I now know how to do it properly, and the end result is beautiful, but again not something which immediately obvious.
Would like to see this if you can share the code…
Sure, a good resource is for example lis302dl.c from the openmoko project:
http://git.openmoko.org/?p=kernel.git;a=blob;f=drivers/input/misc/lis302dl.c;h=f743a241d87817a15bd748f0f911f7ff0a746946;hb=stable
You would have something like
static ssize_t set_wakeup(struct device *dev, struct device_attribute *attr,
const char *buf, size_t count)
{
…
return count;
}
static ssize_t show_wakeup(struct device *dev,
struct device_attribute *attr, char *buf)
{
return snprintf(…);
}
static DEVICE_ATTR(wakeup, S_IRUGO | S_IWUSR, show_wakeup, set_wakeup);
static struct attribute *lis302dl_sysfs_entries[] = {
&dev_attr_wakeup.attr,
NULL
};
static struct attribute_group lis302dl_attr_group = {
.name = NULL,
.attrs = lis302dl_sysfs_entries,
};
whatever_probe(…)
{
…
rc = sysfs_create_group(&lis->dev->kobj, &lis302dl_attr_group);
…
}
Adding new sysfs files is now easy to do by adding them to the _entries array and adding a show and/or set function to implement it. This works for all common and simple cases. If you need to sleep on the sysfs file, I think you might need to use the lower-level interfaces.
// Simon