First of all, what makes register design such a special task? My guess is because it is a higher-level aspect of the design: it describes an interface, which can be implemented in many different ways in hardware. Regular HDLs do not help you reason about register layouts in a good way. The register specification should also be used in other places than the hardware: in address definitions for device drivers, in manuals, and other documentation. All of this indicates that the information needs to be explicit, high-level, and in a format that facilitates automatic processing by tools. It basically screams for a domain-specific language (DSL) at some level of ambition (read my old post about how languages are grown from problems for more background on just how simple a DSL can be).

If you look at the register design tool vendors that Gary interviewed, you can essentially see two different approaches to how to support the task of register design. In particular, the input language differs quite radically between the tools.

There is domain-specific programming language approach. A register-design DSL makes it easy to work with register designs, and pretty hard to do anything else (imagine writing a sorting algorithm in a register-design language – I don’t think it can be done, and if it can be done, the result has to be absolutely contorted). The DSL approach assumes a willingness  on the part of a user to learn a new language, but it seems that for this domain, the languages are simple enough that learning them saves you time in the end. Especially if you maintain large register maps that keep changing or is updated and extended across generations of chips (indeed, ease of maintenance is an undersold aspect of most DSLs in my experience).

The underlying assumption of the DSL approach is that the entry language is not too important, as long as you can export data into other formats. To me, this makes sense. From a single source with sufficient descriptive power, you can ideally generate both HDL code to implement the decoder in hardware, as well as documentation, header files, and maybe even skeletons for virtual platform models. And standard formats like IP-XACT to feed into any other tool.

It does not matter if there is a single tool using each input language, since the outputs are what matters. This leaves the vendors free to invent on the input side, and provide a really powerful tool.

The second approach is transformation-based. Its idea is to not use a dedicated input language, but rather powerful import functions to read whatever specifications already exist in text files, Microsoft Word documents, Microsoft Excel spreadsheets, Framemaker documentation files, IP-XACT, or whatever you can come up with.  The assumption is that the tool is really about transforming data and not about compiling a language. Register designs kind of already exist, and since there is no standard language to compile, you just import whatever there is. It makes the assumption that the user base is not willing to learn a new language to handle register design. A funky side-effect is that we might end up actually doing programming work in Word docs.

Still, the transformation approach ends up generating the same outputs as the DSL approach. In  both cases, some of the outputs can be considered “industry standards”, like IP-XACT, while others are decidedly ad-hoc, like Excel sheets. To me, this demonstrates that the true value in standards is to enable tool interoperability, and not so much as a way to provide inputs.

Indeed, there is a clear difference between good input languages and good exchange formats. An input language tends to drive for expressive power and human readability, and it is OK to have a heavy compilation process associated with it. An exchange format like IP-XACT does not need to be human-readable, nor efficient to code in. It needs to be easy to parse and compile, so that as many tools as possible can work on it.

A typical example from the field of register design is dealing with repeated groups of registers (such as banks of registers for a DMA channel) and parametrized register designs. The input languages used by the tools that Gary Stringham covers all allow this – while the standard for register design exchange, IP-XACT, only uses a flat compiled map. It just lists all registers with sizes and offsets, not how these offsets were arrived at or if there is some logical grouping or iterated structure. Quite OK for a tool to work on, but not very useful as a repository for the actual information.

This distinction is worth to keep in mind.

I definitely side with the DSL idea, as that fits my idea of a good tool, but I can see why many people find the transformation from other formats attractive. In all cases, the goal is to have an original design specification that is as clear, succinct, and flexible as possible.

I have seen the “write 1 clears” solution before in real hardware, but I was not aware of the other two variants. The idea of having a “write mask” in one half of a 32-bit word is really clever.

However, this got me thinking about what the fundamental issue here really is.

As I see it, it is the fact that the processor cannot address small enough units atomically. The read-modify-write that was used to start the discussion in the Embedded Bridge #37 was needed in order to get the current state of a configuration register, change some setting that only occupied a few bits in it, and write back the result to the register. The way most configuration registers that I have seen in practice works.

But if each setting could be given its own register, the problem would go away. Each operation would target a unique address, achieving the same effect as the bit-wise masks or write-1 solutions proposed. The core problem is that hardware tends to share settings into registers, as it has been considered too expensive to put information that might cover a range as small as [0,1] into a 32-bit register. Probably, since there is a lack of addresses for registers, you cannot have 1000 settings cause each simple device to use up 1000 words of physical addresses.

But is that really an issue, if we look forward?

It seems to me that, as 64-bit instruction sets and addressing systems penetrate down into more and more embedded systems, a simple solution would be to throw address space at the problem. I don’t think it is uneconomical to allocate huge chunks of memory space to each device, giving each setting its own register, when you have 64 bit virtual addresses to work with. There is no way you can fill up a physical memory system (guess that will some day come back to haunt me)… even the highest-end machines today only use something like 40 bits for actually addressing physical memories.

The software would be simpler and more robust, with virtually no cost.

Another solution that I have also seen starting to appear is to dispense with register settings altogether, and rather define a command API that the processor “calls” by putting in command packets into some memory area. This does require quite a bit of silicon for a decoder, but it provides for a much higher level of interaction with devices. As hardware devices get defined in successively higher-level languages (C, C++, UML, MatLab, …), and their programming interfaces and associated drivers get autogenerated, this solution makes eminent sense.

As I dig deeper into operating systems and the hardware-software interface layer (mostly with the help of virtual platforms), I have come to appreciate just how hard and interesting that part of the computing stack is. I guess it is partially because that is the level where most of the nice thick layers of middleware and API software we use these days (and which to be frank I find fairly boring) just break down and have to start dealing with the real world. For some reasons, web servers and their programming feels barren and boring compared to dealing with interrupts, memory maps, and bit twiddling.

Several things I have read and heard about recently touch on this subject in various ways. All of them point to the fact that hardware-software interface design is important, and that there is a lot of right and wrong ways of doing it… which are rarely taught in universities and rarely approached in computing literature.

First, Brian Cantrill of Sun wrote a blog post blasting transactional memory in November of 2008, which I recently reread and got a bit of a epiphany from in this paragraph:

… Even if one assumes that writing a transaction is conceptually easier than acquiring a lock, and even if one further assumes that transaction-based pathologies like livelock are easier on the brain than lock-based pathologies like deadlock, there remains a fatal flaw with transactional memory: much system software can never be in a transaction because it does not merely operate on memory. That is, system software frequently takes action outside of its own memory, requesting services from software or hardware operating on a disjoint memory (the operating system kernel, an I/O device, a hypervisor, firmware, another process — or any of these on a remote machine). In much system software, the in-memory state that corresponds to these services is protected by a lock — and the manipulation of such state will never be representable in a transaction. So for me at least, transactional memory is an unacceptable solution to a non-problem.

In the same style, Keith Adams at VMWare picked up on the above and applied it to the microkernel idea:

It’s interesting to me that, as with microkernels, one of the principle reasons TM will fail is the messy, messy reality of peripheral devices. One of the claims made by microkernel proponents is that, since microkernel drivers are “just user-level processes”, they’ll survive driver failures. And this is almost true, for some definition of “survive.” Suppose you’re a microkernel, and you restart a failed user-level driver; the new driver instance has no way of knowing what state the borked-out driver left the actual, physical hardware in. Sometimes, a blind reset procedure can safely be carried out, but sometimes it can’t. Also, the devices being driven are DMA masters, so they might very well have done something horrible to the kernel even though the buggy driver was “just a user-level app.” And if there were I/Os in flight at failure time, have they happened, or not? Remember, they might not be idempotent… I’m not saying that some best-effort way of dealing with many of these problems is impossible, just that it’s unclear that moving the driver into userspace has helped the situation at all.

So what this shows is that the hardware-software interface is where the really hard and interesting problems start to pop up. I am big fan of abstraction and layers of indirection as programming methodologies, I am not a Steve Gibson who feels that programs are best written in assembly… but the abstractions do have to allow for the truth that is underneath the system. Bad abstractions or too simple abstractions make things more complex, rather than less.

Moving on from the software side of things to the hardware design side,Gary Stringham is running a nice series of tips for hardware design. Here, there are lots of interesting issues to confront as well to make hardware easy or worthwhile to use. He recently ran a link to a 2004 Microsoft article on how hardware should be designed, based on the experience of the Windows driver team at Microsoft.

If every hardware engineer just understood that write-only registers make debugging almost impossible, our job would be a lot easier. Many products are designed with registers that can be written, but not read. This makes the hardware design easier, but it means there is no way to snapshot the current state of the hardware, or do a debug dump of the registers, or do read-modify-write operations. Now that virtually all hardware design is done in Verilog or VHDL, it takes only a tiny bit of additional effort to make the registers readable.

Another typical hardware trick is registers that automatically clear themselves when written. Although this is sometimes useful, it also makes debugging difficult when overused.

I guess it is kind of sad that even five years later, this same issues do seem to crop up in new products and merit volumes of venom from driver developers… On the other hand, some companies do seem to be getting it. To me, the Freescale designs of recent years do seem to be fairly easy to configure and debug, and not feature write-only bits in any large number.

The article about hardware acceleration for TCP/IP by Mike Odell that I discussed in a previous blog post is also relevant: when do the complexity of hardware interfacing negate any performance benefit from an accelerator?

(for some reason, the initial posting of this post had an incomplete last paragraph, something weird in WordPress updates happened)

To sum up, I think the interaction of hardware and software in the context of full opreating systems and device driver stacks is a really interesting topic that seems to have not gotten very much academic coverage. I hope to be able to help remedy some of this, once I get the Simics setup used in my experiments with hardware accelerators packaged and available for academia. Full-system virtual platforms make for a very good experimental system, especially those where you use some third-party or standard operating system rather than just your own controlled code.