The complexity of modern processors – even a decade ago – was such that predictability was very difficult to achieve in practice. We used to joke that a complex enough processor would be like a random number generator.

Funnily enough, it turns out that someone has been using processors just like that.  Guess that proves the point, in some way.

I was recently introduced to the concept of the HAVEGE project – HArdware Volatile Entropy Gathering and Expansion, run at IRISA in Rennes in France from what seems to be around 2002 to 2006.  The main author, Andre Seznec, has also published in the WCET field. Today, the same idea is found nicely packaged in the HAVEGED code base for Linux, found at http://www.issihosts.com/haveged.

The idea behind HAVEGE is to run a piece of code that is designed to incur cache misses, confuse branch predictors, and generally strain the prediction mechanisms of a processor. In this way, the timing of the code will fluctuate even though it is basically straight-line code with no decision-making. These timing variations can be captured by reading a high-resolution timer such as the x86 processor’s TSC (Time Stamp Counter), or some other source that can report the execution time of a piece of code.

The key advantage of such a source of randomness is that it is easy to quickly acquire lots of randomness (or entropy in crypto language), and it is also impossible to predict the results. For cryptographic applications, this unpredictability from the perspective of an outside observer is very important, as it makes random numbers generated based on this much stronger in the face of an attack.

I think HAVEGE offers a good example of how to make lemonade from lemons.  If we conclude that processor timing cannot be predicted, consider that fact as a feature for cryptography rather than as a problem for WCET.

The first paper on HAVEGE is called “Hardware Volatile Entropy Gathering and Expansion: Generating unpredictable random numbers at user level“, IRISA internal report 1492, October 2002. It presents the core idea a little differently from later papers.  In it, they measure the cache and TLB effects on randomness, assuming the key to randomness being the effects of interrupts where OS code affect the cache and TLB entries used by the program.  An underlying assumption is that if you just run a program in isolation, the caching and speculation mechanism will converge to a good state for the program, with no or little timing variation as a result.

I wonder if that is still true on a modern machine. Their measurements were performed on a mid-1990s UltraSPARC II, which is in-order and much simpler than current Intel Core processors. Even an ARM Cortex-class processor is much more complex.  I would really like to see measurements about the inherent randomness in today’s processors, without any recourse to interrupts and hardware actions to disturb the picture.  I wonder if you would still see variations in the execution time of a body of code due to the different periods of various hardware mechanisms, or if it all converges to maximum throughput and minimal hardware latencies for all parts of the pipeline. For some reason, I have my doubts that the hardware would be that ideal in practice.

What makes the randomness of the actual hardware hard to evalutate  is that the available codebase is the HAVEGE code, which is an “expansion” of the basic HAVEG idea. The expansion being to couple a PRNG to the collection of entropy from the hardware, in order to produce much more random noise (in terms of random bits per second) than just the hardware would provide. While very practical, this also serves to obscure the fundamental randomness of the hardware from direct measurement.

Essentially, HAVEGE generates a ton of random data that appears to be of high quality in the tests provided.  But that data mixes three factors into a single measurement:

• Hardware low-level random fluctuations (cache, pipeline, branch predictor)
• Hardware coarse-grained variation (interrupt timing, the time taken
  to perform OS actions in response to interrupts)
• The effectiveness of the PRNG code

Picking these three apart would be interesting, and it is a shame that there seems to be no recent evaluation of HAVEGE.

I made some cocky comments about just how stupid the current implementations of this idea are a few years ago, but in my discussions with Tennessee I have realized that things are not that simple. Essentially, you are caught between the structures and strictures of the certification agencies, and the complexity of the hardware with its many shared resources making predictability and programming very difficult.

It is a field which touches to my old work on WCET, but with a target system that is much less amenable to analysis. I still think it is a good idea to try to use seas of simple cores and separate work physically rather than virtually, but that might be a battle that will never be won. If nothing else, even on a massive spatially-divided multicore device, there will be some shared resources that make life difficult when very low levels of jitter are tolerated.

Read the interview, and read Tennessee’s own blog – he has done some pretty cool things both in hardware, software, and virtual hardware.

But is that really always going to be true? Is it reasonably to think of CPU cores are being free but other resources as expensive? And what happens to program and system design then?

This idea was brought up again in an interview in edition 33 of TimeSys LinuxLink Radio podcast with the creators of the Propeller chip, Parallax. In this chip, they abandon interrupts and instead dedicate entire processors to various IO tasks. The cores stay in very low power mode until something interesting happen on IO pins, and then wake up and process. No interrupts, no interrupt handler, almost zero latency. Very good for real-time systems. The intuition you have just screams “why not share the IO load on a single processor”? Which is old wisdom, processors are expensive. And having a bunch of them just waiting in deep sleep for something to happen seems absolutely nonsensical. Especially as the current propeller chip only has eight cores, it seems a bit limiting and just screaming for timesharing and/or virtualization to handle a few more tasks than eight…

But is it really?

Consider the alternative design. A much more powerful processor running an operating system with all its associated overhead. For hard real-time control, what you do there is either use a static time-driven scheduler or use event-driven programs with an analysis that makes sure that you can always meet all deadlines. And these solutions have a hard time reaching 100 percent CPU usage and usually involve a fairly high interrupt latency. For really short real-time latencies, you end up vastly overprovisioning the processor in order to make sure that the cycles that an interrupt takes to process in overhead pass as quickly as possible. With caches, you either have to lock them or once again vastly overprovision the processor to make sure to meet guaranteed latencies.

The interesting question that I do not have really good numbers for is for a given problem of say ten critical real-time tasks, using ten simple cores doing one thing each or sharing ten critical tasks on a single core is the least expensive solution hardware-wise. Since it seems that in general, to make a processor ten times faster you need either ten times the clock frequency or a significantly larger die, it is not clear-cut. Especially not with a tough latency constraint.

If you factor in the cost of software development and validation of system functionality, I am pretty sure that the multiple simple cores approach is very competitive. Instead of analyzing a complex concurrent software system full of potential nasty shared resources and unknown software paths, you can handle ten small programs running on simple cores with simple timing, and with shared resources that are at least obvious from the hardware design.

There will be shared resources, be sure of that. If nothing else, the memory controller will be an issue. The propeller chip solves this by strictly temporally sharing memory access between the cores in a fixed static schedule. Predictability before all!

Note that the EU STREP project called MERASA is looking into something that could be related to this idea, with 2 to 16 cores per chip with lots of predictability piled on.

Marketing it

Selling such a chip as a standard part would seem to make sense as well, as most control systems are fairly low-volume. The propeller chip is very small, eight cores only, but if you look at what Ambric and Picochip are doing, it is easy to envision a controller part with twenty or a hundred little cores on. That could be really cool

If you have a high-volume part, I guess you could build that control ASIC yourself today, around something like a Tensilica pile of cores, which have been proven to scale to hundreds of cores even today in custom ASICs.

A half-way design in this style is the Freescale 5514-15-16 parts which combine a compute core with a smaller core to handle all the interrupts. If nothing else, this validates the fundamental idea of interrupts being an issue. Putting all interrupts on a secondary core could be seen as a bandaid on a bandaid… but it sure makes sense if the cores are not considered free.

Will be interesting to see what the future brings.

You can find the article at the ACM portal, or at the MRTC publications data base in Västerås.

I just got my personal copy, but my first impression of the book overall is very positive. The contents seems quite practical to a large extent, not as academic as one might have feared. Do check it out if you are into the field. It is not a collection of research paper, rather instructive chapters informed by solid research but with applications in mind.