Let’s start from the top.

When Steve said that computers are deterministic, I jumped. To me, a computer is anything but deterministic. The idea that rerunning a program does the same thing is an ideal state that you can rarely reach, and having an infrastructure like Simics that helps you achieve this is huge win for debugging.

Listening closely, what I think Steve really said is that an algorithm like a random number generator is deterministic. If you know its initial state, it will always compute the same result. That is indeed true for code that just converts an input into an output, and does no communication and is not dependent on time or timing. My experience in random and nondeterministic behavior comes from programs that feature multiple threads and often multiple processes, and plenty of asynchronous activity going on. So, same word, different contexts.

However, Steve also several times talk about computers as being deterministic predictable machines. I think that characterizing today’s computers as being deterministic is untrue. I would rather say that with multiple cores and multiple chips and timing variations all over the place, a computer has become fundamentally nondeterministic and non-repeatable, since there are so many little things going on where a nanosecond difference in time can cause behavior to diverge incredibly quickly. There is a nice paper from 2003 about the divergent behavior from minor differences, “Variability in Architectural Simulations of Multi-threaded Workloads“, by Alaa R. Alameldeen and David A. Wood.

The HAVEGE program I wrote about a while back is essentially an attempt to harness the fundamental unpredictability of modern hardware timing. Nice idea, which at least in theory fulfills the more important property for security of being unobservable. Security doesn’t really need “real” randomness, all you need is something that an attacker cannot predict or observe. The classic Netscape SSL lack-of-randomness in the random seed issue from 1996 is the best illustration of this. Certain things about a target can be inferred or observed, but the low-level hardware timing is not one of them, at least not for an x86 or high-end ARM class machine.

The solution that Steve prefers are the Yarrow and Fortuna algorithms that collect randomness from the environment of a computer and uses that as a seed to a normal random number generator, creating lots of useful random data from a fairly small seed. This is the same idea as HAVEGE, but with a different entropy source. In both cases the basic idea seems sound and reasonable, but I kind of hoped that Steve would know of some way to evaluate the quality of the entropy pool generated from hardware events.

Steve mentioned the NIST randomness test that was used to test HAVEGE. It is certainly an aggressive test, but as my testing showed, it only demonstrates that a random number generator is random in the data produced. It does not show that it is unpredictable, and it does not measure the benefit gained from using unobservable local events in hardware as the source of entropy. You need something else, like comparing repeated collections of randomness over time from the same system, to build confidence in unobservable and unpredictable randomness.

With a computer, you do have such a thing as repeatable, deterministic, and thus predictable randomness. In a modern desktop or server computer, you also have tons of totally unpredictable non-repeatable non-usefully-observable randomness in the low-level hardware timing and concurrent behavior of independent hardware units. Too bad it seems hard to prove this by measurement.

For yet more randomness discussion, especially randomness in embedded systems, I recommend the blog post by Joachim Strömbergsson. (it is in Swedish).

I got HAVEGE up on a Simics x86 target model with Linux pretty quickly, and ran the two provided tests. Ent, which is a quick entropy test, and nist which supposedly much more thorough.

To my surprise, they both said the randomness we got was totally acceptable. This would seem to invalidate the fundamental assumption of HAVEGE – that it needs to collect randomness from hardware in order to produce good-quality randomness. To try to understand a bit more of what was going on, I took at look at the execution using Simics Analyzer (the dredd.motherboard.processor lines are the processors, and the orange part is the HAVEGE program, yellow is the kernel):

Zooming in a bit:

We can see that the program is regularly interrupted by the OS, which could be  a reason for random timing variations. The instructions run by the OS should vary in count, which would disturb the time stamp counter values read by the HAVEGE program. That could be sufficient to cause random variations, essentially showing that HAVEGE really works well just from OS noise – even in an otherwise idle machine.

However, at this point I started to have my doubts. Something did not feel right.

So I tried to remove all variations from the HAVEGE program. I replaced the “HARDTICKS” macro in HAVEGE with the constant 0 (zero) rather than reading the time stamp counter of the processor. This immediately failed the randomness test.

However, when I used the constant 1 (one) instead, the ent test passed. And even nist almost passed with only a single missed test out of the 426 tests executed.

Thus, the conclusion is that we do not know how well HAVEGE ‘s collection of hardware randomness works, since the evaluation software is too weak. In essence, we do not know if the collection of hardware randomness matters or not, as the proposed measurement hides the randomness behind a pretty good PRNG algorithm.

Ideally, we would need a measurement that would evaluate the predictability of the randomness generated. Or at least one that can correctly estimate the impact of the variation of low-level hardware timing on the quality of the final random numbers. Unfortunately, that is not the case here, throwing the entire idea into doubt.

The complexity of modern processors – even a decade ago – was such that predictability was very difficult to achieve in practice. We used to joke that a complex enough processor would be like a random number generator.

Funnily enough, it turns out that someone has been using processors just like that.  Guess that proves the point, in some way.

I was recently introduced to the concept of the HAVEGE project – HArdware Volatile Entropy Gathering and Expansion, run at IRISA in Rennes in France from what seems to be around 2002 to 2006.  The main author, Andre Seznec, has also published in the WCET field. Today, the same idea is found nicely packaged in the HAVEGED code base for Linux, found at http://www.issihosts.com/haveged.

The idea behind HAVEGE is to run a piece of code that is designed to incur cache misses, confuse branch predictors, and generally strain the prediction mechanisms of a processor. In this way, the timing of the code will fluctuate even though it is basically straight-line code with no decision-making. These timing variations can be captured by reading a high-resolution timer such as the x86 processor’s TSC (Time Stamp Counter), or some other source that can report the execution time of a piece of code.

The key advantage of such a source of randomness is that it is easy to quickly acquire lots of randomness (or entropy in crypto language), and it is also impossible to predict the results. For cryptographic applications, this unpredictability from the perspective of an outside observer is very important, as it makes random numbers generated based on this much stronger in the face of an attack.

I think HAVEGE offers a good example of how to make lemonade from lemons.  If we conclude that processor timing cannot be predicted, consider that fact as a feature for cryptography rather than as a problem for WCET.

The first paper on HAVEGE is called “Hardware Volatile Entropy Gathering and Expansion: Generating unpredictable random numbers at user level“, IRISA internal report 1492, October 2002. It presents the core idea a little differently from later papers.  In it, they measure the cache and TLB effects on randomness, assuming the key to randomness being the effects of interrupts where OS code affect the cache and TLB entries used by the program.  An underlying assumption is that if you just run a program in isolation, the caching and speculation mechanism will converge to a good state for the program, with no or little timing variation as a result.

I wonder if that is still true on a modern machine. Their measurements were performed on a mid-1990s UltraSPARC II, which is in-order and much simpler than current Intel Core processors. Even an ARM Cortex-class processor is much more complex.  I would really like to see measurements about the inherent randomness in today’s processors, without any recourse to interrupts and hardware actions to disturb the picture.  I wonder if you would still see variations in the execution time of a body of code due to the different periods of various hardware mechanisms, or if it all converges to maximum throughput and minimal hardware latencies for all parts of the pipeline. For some reason, I have my doubts that the hardware would be that ideal in practice.

What makes the randomness of the actual hardware hard to evalutate  is that the available codebase is the HAVEGE code, which is an “expansion” of the basic HAVEG idea. The expansion being to couple a PRNG to the collection of entropy from the hardware, in order to produce much more random noise (in terms of random bits per second) than just the hardware would provide. While very practical, this also serves to obscure the fundamental randomness of the hardware from direct measurement.

Essentially, HAVEGE generates a ton of random data that appears to be of high quality in the tests provided.  But that data mixes three factors into a single measurement:

• Hardware low-level random fluctuations (cache, pipeline, branch predictor)
• Hardware coarse-grained variation (interrupt timing, the time taken
  to perform OS actions in response to interrupts)
• The effectiveness of the PRNG code

Picking these three apart would be interesting, and it is a shame that there seems to be no recent evaluation of HAVEGE.

S4D Talks, Themes, and Topics

More is available in “S4D part 2“.

Tracing and Instrumentation

The papers presented covered a wide variety of topics from a variety of angles. Still, everybody felt that two topics kept coming back in various forms in a majority of the papers and discussions: tracing and instrumentation.

Code instrumentation is not a dirty word anymore. The traditional judgment that inserting probes into your software is plain bad does not apply anymore, at least not in the minds of the people at S4D. Instrumentation was applied to drivers, OS kernels, and regular user-level software. I think the key insight is that there is clear value in having the developers that write a piece of software also mark points of interest in the code. When analyzing a trace of an execution, that means that the information in the trace becomes meaningful to the software developers, as it is on the right level of abstraction. Instrumentation naturally produces traces, which can be fed out using shared memory, networks, special-purpose hardware, and more.

One of the instrumentation trace solutions presented (the SVEN system from Intel Digital Home presented by Pat Brouillette), actually leaves the instrumentation in place in the shipping customer systems. In this way, you cannot really claim that instrumentation is intrusive – it is just part of the software, always. Customers can even activate the tracing in deployed systems, and ship the traces back to the developers for analysis of bugs found in the field. It is another approach to record and replay that touches on my paper on transporting bugs with checkpoints.

The increased interest in instrumentation probably has something to do with the nature of the systems that are being addressed. For systems using shared memory multicore hardware and general-purpose operating systems, the cost of instrumentation is easier to take than for very small constrained embedded systems. Essentially, as systems get more complex, instrumentation becomes more tractable.

Instrumentation can interact with hardware trace and debug functions is a neat way to build a system which is more powerful than a hardware or software system would be on its own. Especially for software stacks involving hypervisors and multiple complex operating systems, that is likely necessary.

Once we have a trace, just like last year, we need to have tools for analyzing the tons of data you get from tracing a modern system. ST talked about a tracing system that generated 100s of gigabytes of data.

One trace aspect that kept coming up was the need for time stamps on trace data. To reconcile multiple traces and understand how different concurrent units talk to each other, a global time stamping mechanism is crucial. There seems to be work on hardware to support this.

Security, Secrecy, and Debug

I moderated a panel on hardware support for debug, and posed the question on how to balance security and the need to debug. This generated a number of interesting answers from the panel and the audience.

The conflict between debuggability and secrecy is there. Even from the same customer you first get “you have to make the internal state of the controller inaccessible and hidden to avoid customers modifying their engines”… and then when a problem appears in the field, they ask for a way to analyze and trace that very same system. Hard to support both requirements in a reasonable way.

A sophisticated solution to debug security from companies like ARM, Infineon, and ST is debug that can be enabled using key exchange. The chips are built with a “locked door” in place, but the keys to the door are kept well-guarded. In this way the same chip can be used in development and in the field.

To support debug of systems involving secure modes like ARM TrustZone, ARM has defined several levels of access in their CoreSight hardware modules. This makes it possible for a debugger to be restricted to just debugging user-level code, just OS and user-level code, or all of the software stack. To me, this sounds like it could allow mobile phone manufacturers to “securely” let their application developers use hardware-based debug, without compromising operating systems or secure boot modes.

The classic technique of using fuses to turn off functions is also relevant, at least for systems with moderate levels of security. This can certainly be overcome using special tools to peel off the top of chips and reconnect the fuses, but the panel seemed to think that that level of attack was in general not worth protecting against. However, the audience pointed out that  this was actually being done to automotive engine controllers and there are people making a good living from such antics.

ESCUG Meeting

The ESCUG meeting was a mix of fairly slick commercial presentations from OVP/IMperas chief Simon Davidmann and SystemC guru John Aynsley, and research presentations of varying quality.

One thing that struck me was that the academics spent a significant time in all presentations about how their approaches were compatible with the existing SystemC structure, where they host their open-source efforts, etc. I guess that is good in that they show a certain concern for reality – but it is also a bit sad that they did not get time to actually talk that much about the core ideas they were bringing forward. I am personally much more interested in new ideas than infrastructure and project management. It does not bode well for European research if this is what people are forced to produce, in lieu of real innovation.

Thorsten Grötker’s Keynote

On Wednesday morning, Thorsten from Synopsys did a look back over the history of SystemC, free from product pitching. He only mentioned Synopsys in his introduction, where the high-level message was that the embedded software is really the key problem for industry today. I cannot disagree with that.

During the SystemC parts of his talk he did say a few things that I did not quite agree with… in particular that TLM was unknown prior to 1999. It was not called that, but it certainly existed in the field of full-system simulation. The main problem is that Thorsten only sees the EDA history of modeling, not the computer architecture and software-driven work that did simulations as far back as 1950 (the famous Gill paper), and fast simulation since at least 1967.

He also claims that with SystemC you have a single language for both detailed and TLM models. That is true… but you still need multiple models, one at each level of abstraction. So yes, one language, multiple models. However, that gluability really comes with a performance and complexity cost. It makes it too easy to slip into bad modeling even in TLM.

An interesting theme that Thorsten picked up from John’s talk at ESCUG is the use of SystemC to model software and RTOS, using the upcoming process control extensions. If you stretch that into the area of software synthesis, it means that SystemC is going to collide with the field of model-driven software development. Will you use SystemC, coming from the hardware world, or UML/MATLAB/Domain-specific languages coming from the software world?  Thorsten makes the interesting point that in order to integrate with that world, SystemC will require some concepts from that world (like pins and clocks enable interaction with RTL). I am not sure that is true, necessarily, I think you can just as well create point adaptors to the same effect.

Getting to Southampton

The University of Southampton hosted the event, and it took place in the university lecture halls. That means that we got free very fast WiFi (unlike any commercial conference venue I have ever seen). The university campus was full of services (unlike the desolate place that last year’s FDL/S4D choose). Housing in the Glen Eyre residential halls was a bit spartan but functional. Felt like being back in my days as a student living in student housing.

The instructions from the conference about how to get to the conference was a bit confusing and incomplete. In practice, it is very easy to get to Southampton from both Gatwick (direct train) and Heathrow (NationalExpess bus 203). At Heathrow, I had a bit of luck with the bus to Southampton. The instructions from the NationalExpress website had me believe that I had to get from Terminal 5 where we landed to the central bus station and then catch the bus at 15.00. As we landed 40 minutes late (14.40), this looked very hopeless… until I found the NationalExpress counter in the arrivals hall at Terminal 5 and they told me the bus would leave at 15.30. Nice, no stress. The bus to Southampton even had free Wifi on board!

Once in Southampton, you then had to take the bus U1A out to the university campus, and finding a bus stop for that was the most difficult part of the journey, actually. Some of the buses from Heathrow stop at Southampton university.

See also “S4D Part 2” for a few more tidbits from S4D.

There are two important aspects to this worm, as I see it.

First, it only works due to the fact that the software in question is running on regular Windows PCs.  An attack on a real embedded OS like Wind River VxWorks or Enea OSE would be more interesting, and much scarier since that would mean a much more devoted enemy. In this case, the attackers are opportunistic, using the window of vulnerability of a new Windows flaw to attack Windows-based plant control systems. They also use signed Windows drivers, which is apparently a new development in malware. All quite interesting in its own right, and worth reading about for those interested in security.

Second, the malware spreads using physical movement of USB memory sticks rather than attacks over the Internet or other networks. This makes the very important point that even if systems are not connected to the Internet, they can still be attacked if something crosses the “air gap” that separates them from the outside world. In this case, a plant would be infected by using social engineering to make some employee carry an infected USB stick into the plant and putting it into some internal PC. Once the infection is inside the plant, it might spread over networks or by USB sticks moving around inside the presumably protected perimeter.

The lesson I think we can draw from this is that using general-purpose desktop operating systems for critical systems is a bad idea. Using a more obscure real-time OS (or even Linux) would probably reduce the number of vulnerabilities – but more importantly, it would make it much more difficult to make an infection hop from computer to computer until it reaches its target.

In this particular case, all Windows machines are potential bearers and spreaders of the infection. An attacker can rely on that fact to seed the Windows ecosystems at some place, hoping to get the infection hopping from machine to machine until it reaches something interesting. There is no real need to seed the infection directly into the target plant. If the target systems had been running a different OS, the attackers would have had to get really close to the target, making them easier to stop.

Overall, embedded systems security is something that we need to take much more seriously going forward. As we rely on embedded control systems to run much of the modern infrastructure and economy, we really need to be concerned about how to secure these systems. Security needs to be part of the architecture of embedded systems, including their operating systems (please make them as robust as possible), application designs, and networking systems. Unfortunately, current embedded technology tends to be designed to work, with little care for how they could be broken by an intentional attack. One scary example of this was provided in the SecurityNow podcast episode #251, where a listener shows how easy it could be to take remote control over a car due to carelessly designed fleet management units.

Updated information

I found some more in-depth information on the issue at Infoworld. It notes that the software that is attacked is vulnerable since all installations use the same password to login – changing it is likely to break it. That is totally ridiculous as a security solution, period.

Ever since Unix became the role model for operating systems, C has been the “base” language. In desktop operating systems like Windows and Linux the platform API is expressed as C function calls, and the ABI (the binary calling conventions) for linking code from different compilation units and binary distribution units is also expressed in terms of the code that a C compiler would generate. A C compiler is the first thing you need to get the platform going, and C is a language that allows arbitrary applications to be developed and run on the platform. The semantics of dynamically loadable and shared objects are expressed in C, not C++, as the C++ ABI is too variable.

This ubiquitousness of C has also proven to  be a key enabler for higher-level languages. C is the language of choice to implement the basic virtual machines used by languages and programming systems like Perl, Python, Erlang, and Java. C (or C++) is also used as the target language of many modeling tools with code generation, such as MatLab/Simulink, Rational Rose, Rhapsody, and Labview. A single generator of C code can be reused to target many platforms. Sometimes C++ is the generated language (in particular for UML-based object-oriented tools), but C++ essentially falls back on the ubiquitousness of C to be able to call platform APIs and connect to other tools.

That is the state of things as we know them today. What seems to be happening is that this is slowly being deprecated… platforms are coming out where user code might not be possible to write in C at all, or where C programs cannot access the real platform API. For these platforms, it is quite difficult  port in an existing C/C++ portable program.

The first example is Google’s mobile phone operating system Android,where  Java was the only supported language at launch. Google have since made it possible to use C and C++ for some parts of an application, but it does not seem to be a full platform API allowing a whole program to be written only in C or C++:

“The NDK will not benefit most applications. As a developer, you will need to balance its benefits against its drawbacks; notably, using native code does not result in an automatic performance increase, but does always increase application complexity,” the documentation says. “Typical good candidates for the NDK are self-contained, CPU-intensive operations that don’t allocate much memory, such as signal processing, physics simulation, and so on.”

To port an application written in C/C++ such as Firefox to the Android platform, the app has to be modified to work as a backend to the Java interface.  ArsTechnica has a write-up on how Firefox was brought to Android through just such a modification. Note that this does mean that ports are not as straight-forward as they would be to other platforms with a directly accesible C API. Interestingly, the Android approach essentially inverts the traditional relationship between C and other languages, where it was common to have a C adapter layer around other languages (like Java) in order to access the platform.

Note that the NDK quote does not mention language run-times. One of my favorite languages, Python, has had to be completely reimplented to run on Android. Either using a “Jython” approach of compiling Python to Java byte codes, or using the Android Scripting Environment.

Other languages that you would ordinarily just port using a simple recompile of the its C code base are not helped by this at all. One interesting example is the Erlang runtime, which is basis for CouchDB. According to an interview on FLOSS weekly show 99, about Ubuntu One, this fact prevents Ubuntu One from synching data from your desktop to your phone.  This demonstrates that the assumption that you can run a C program on “any Unix-like system” is no longer true for a large numbers of smartphones… and that is already affecting how you have to develop products.

Microsoft is also moving in the “no C for you” direction with Windows Mobile 7, where C# is the default language. This also prevents easy reuse of existing C programs on smartphones. Ars Technica notes how this killed Firefox on Microsoft-based mobiles.

Finally, we have Apples downright weird approach to languages and programming. Their recent banning of anything except Objective C and their own compilers for iPhones (and iPods and iPads) is downright bizarre. They explicitly forbid code-generating tools to be used with their platform, as well as kicking out any alternative language runtimes (which is a move aimed at Adobe Flash that also hits Erlang, Python, et al.). This might make some sense from a security perspective, as it prevents programs from loading executable code at run-time on the phone… but it also makes for a much more restricted set of programming tools.

The only mobile platform that seems honest to old traditions is really Nokia’s Maemo. And Symbian. Suddenly, what used to be considered “closed” platforms have become the most open and most desktop-like of all the mobile operating systems. Really, that is a very important reason to get a Nokia N900 rather than an Android or Apple device.

I think this points to a somewhat more complicated future, where mobile applications will cannot be cross-platform, as you have to use Android-Java, Windows-C#, iPhone-ObjC, and Maemo-C/C++/anything to code. It could also point to a move even in the desktop and server space away from C and to more sandboxed, controlled, and not-as-common programming platforms. That would be really bad from a programmer productivity and language innovation perspective, as so much of the innovations today are actually based on the ubiquitousness of C and the use of C as a good implementation language and code-generation target language.

Updates and clarifications

Given the comments below, it seems that I need to make some things clearer…

First, the iPhone. It is really a special case, in that you do have C/objective-C access to the API. So in principle, you could port any C program including language virtual machines to the iPhone. Firefox, for example, would work. However, Apple for commercial (and maybe security) reasons does not allow programs that implement virtual machines to be distributed across their controlled application store. That you can implement a VM there does not really help you, as it is the first openly programmable platform that I have seen where a control body actually forbids certain classes of programs. The Apple approach is apparently even more idiotic than I first believed.  According to some more reports I read and heard, they do not enforce a technical limitation on the final program (such as running in a JVM or .net VM), but rather require all software to be originally and directly written in C, C#, or C++.  All just a swipe at Adobe and flash… so even Flash compiled to native code is disallowed.  And with it goes all other higher-level languages.

Second, on security. I do think that running programs on top of a VM like done with Java and .net does have security benefits. It gives you a level of indirection which can be used to check what applications does. Obviously not perfect since there will always be bugs and mistakes, but still it is a better architecture than raw access to the underlying machine. The  iPhone controlled mode of distribution could also be beneficial here. The SecurityNow podcast episode 245 discusses this topic.

Third, on appropriate programming languages for different tasks. I totally agree with some comments that C is not a very nice language for GUI programming. No doubt about that. However, that was not really the point I was trying to make. I want to use higher level languages! But not having C available might make that harder and with limited choice… leading up to point four.

Fourth, my core point. Having platform-level access in C is a basic technology used today to implement the higher-level environments. Languages like Python, Perl, Erlang, Lua, and even the basic Java and .net virtual machines, all depend on having C available to bootstrap the process of getting the core virtual machine going. If we take this away, we limit choice in language and might stifle innovation, as well as the attractiveness of cross-platform environments.

Fifth, I agree with the comments C is definitely not going anywhere in terms of being used to develop operating systems and embedded systems (that’s where I spend most of my time, by the way). My observation is about what is happening to user-level programming for certain systems, not the systems programming which is intentionally made separate from user-level programming.

Another Update:

The Inquirer just pointed out that Apple is explicitly forbidding interpreters to run on their phones, unless it is an interpreter they created or one explicitly allowed. That’s making the above very clear, Apple is consciously denying iPhone users all modern programming languages. And that’s just to make sure Adobe can’t weasel Flash in there. Politics sometimes make no sense at all. The Inq quote is worth quoting:

Famously, when Apple released its iPhone SDK in spring of 2008, the end user licensing agreement barred applications from downloading and running any interpreted code. “No interpreted code may be downloaded or used in an Application except for code that is interpreted and run by Apple’s Documented APIs and built-in interpreter(s),” it said.

As you can see from the webmail screenshot below, the probable folder contained almost 100 thousand spam email. These were collected since May of 2008, or in a space of roughly 21 months.

If I guesstimate that the other folder has about the same average size, that adds in another 400 thousand spam. Bringing the total caught by these filters to about half a million overall. If you divide it down to days, it is “only” about 630 per day, plus some more that secondary spam filters catch, plus the ones that get through and I manually have to delete. But these other ones won’t add up to much more than a few tens of thousands in the same time spamn.

It is amazing just how voluminous this infestation is…

Legit business emails can’t be much more than fifty per day, plus various general mailing lists. Still, that means that probably no more than 75% of all email I receive is spam. Maybe I should consider myself lucky, I have seen analysis talking about 99% of all email on the Internet being spam…

Interesting it was.

However, that small inconvenience is not really something to be bothered by. Any hardware-based login solution will have that, and the yubikey’s fitting into something you have anyway for getting in to places (you keys) makes it very logical. What could have made it even better had been if you would have activated it using a key-like turn rather than the somewhat bland keypress that does not produce any kind of haptic feedback. But I can understand that a twisting design like that would be an order of magnitude more expensive to produce, and probably another order of magnitude less durable…

It is also striking how well this system works compared the incredibly clunky login and signing facilities used by ICA Banken. There, you have a piece of hardware which is way larger than a key, into which you slide your credit card. Then, to log into the bank and effect a payment of a bill, you do:

• Type in customer number
• Type in PIN code
• Put your card into the signing device
• Press “login”
• Copy 8 digit code from web page to device
• Type card PIN code into device
• Copy 9 digit code from device to web page
• … enter data for bills …
• Bring out the signing device again
• Insert card
• Press “sign”
• Copy 8 digit code from web page to device
• Type card PIN code into device
• Copy 9 digit code from device to web page

I really think the “sign” step adds no security in practice, and most other bank systems I use seem to agree with this: once past login, no need for additional confirmation. I think that makes sense, and that the sign stage is here more as a warm fuzzy feeling kind of thing.

If it wasn’t for the possible constraint that the ICA solution has to work on public computers where you have no access to USB ports, I think a yubikey-based solution would make all of the above so much easier. The genius of the yubikey is really that it removes the “type in numbers from hardware device” from the login steps, which really is something that there is little value to having each user do every time they effect some kind of secure operation. If all banks used a yubikey, I think the world would save many thousands of people hours that could be used to have fun, be with the family, and other more beneficial uses.

The immediate impression is that is impressively small!  Compared to a standard USB memory stick, it is significantly smaller, and most importantly, very thin. This means that they can be sent in a regular envelope in the mail, since it is about as think as some folded papers. It also helps when you put it on your key chain, I guess. I don’t know if I dare do that yet, since my pocket tends to be fairly crowded with heavy sharp keys that could well scratch the innocent little  Yubikey. Here is a picture of the key along side a Sandisk Cruzer USB memory stick:

When it is attached to a computer, the little button ring lights up. When the computer is asleep, it pulsates nicely too.

Driver installation was automatic on my Vista machine, showing up as a human-interface device with no particular characteristics. That is the very idea of the Yubikey: it is a USB keyboard as far as the computer is concerned, which is amazingly simple and clever. Here is what Vista says about it:

In use, the Yubikey is still a bit of challenge to me, for one simple reason: the button feels hard to get pressed in the right way. It seems that I have to push pretty hard and for a long time to activate, and then I want to hold the key with my other hand too so that I do not break it at the point where it is connected to the computer.

Apart from that, it is a beautiful device, and compared to the security solutions I have with my various internet-based banks, it is much easier to use. No codes to type in, no 9-digit numbers to type into online forms (that is what ICA Banken currently requires you to do, copy a nine-digit number from a security device into which you insert your bank card…).

So overall, I really like the Yubikey, and it will be interesting to see how it lasts, physically, as I start taking it with me everywhere.

Update: The Yubikey does work to have on a key chain, I have started doing that and so far it works in the sense that it does not affect the physical size of the chain much. There is also some logic to the use mode of inserting the key from the key chain into my computer to “unlock” secure functions.

Finding information on this card was very hard, and here is the best that I could find:

With front and back LEDs and a removable system configuration card, the Sun Fire V120 server maximizes system availability by allowing system administrators to concentrate on scheduled service through easy installation and management. The removable system configuration card allows you to store a system’s host ID, MAC address, and NVRAM settings to another server while you perform routine maintenance. As a result, system downtime is minimized.

Why I find this interesting is that it is also a nod to commercial software companies relying on hostids for licensing. In this way, you can maintain the same hostid even when a server has issues, and without compromising the integrity of licensing. Sun’s hostids are unusually safe and reliable, unlike the common x86 anchors like Ethernet MAC addresses (which are easy to change) and disk IDs (which are not available on Linux typically).

Making the ID physical in this way is usually the best way to handle identity in general. A GSM/UMTS SIM card is another example of a physically represented identity, which is way preferable to virtual identies that are just software. Much easier to handle, and safer for all involved.

It is mostly in Swedish, though, but they have some English-language information at http://www.crimemedicine.com/lib/html/english.html.

I think this is pretty important stuff, way too many people are buying counterfeit and illegal and usually dangerous medical stuff over the Internet. That is not how you deal with your health: if you have a problem, go to a doctor or at least a physical pharmacy and talk to someone with proper credentials. Never ever self-medicate with stuff you find of the net. Or even sometimes buy in physical pharmacy in less well-policied countries. Real medications can be expensive — but most of the time, you are getting what you pay for. As always.

What was quite striking this year was the greater difference of opinion between the speakers. I guess that in 2007, most of the discussion was on the level of “ouch, here comes multicore and what are we going to do about it”. This year, we got a bit deeper and with one more year of experience and massive research work, the collective world of multicore have made some progress and gained insights. And that’s when the differences start to show up; the fact that we have differences of opinion tells us that we are starting to dig into details and turning up different answers due to different viewpoints and user experiences.

So where were the differences this time?

• Heterogeneous vs homogeneous cores (on a single chip). Kunle Olukotun clearly supported the heterogeneous style (which is what you with Sun’s Niagara that he designed the basis for). Erik Hagersten was more interested in the difference between thin and fat cores of the same basic ISA, and Anant Agarwal was strongly in favor of completely homogeneous systems (which is what they build at Tilera). In my biased view, I think the argument for heterogeneous in pure energy efficiency is always going to prevail. See some of my previous blog posts on this topic, for some background:
  • DNS Hardware Acceleration.
  • Interview with Kunle Olukotun at the Register.
  • Homogeneous vs heterogenous.
  • Homogeneous vs heterogeneous, continued.
  • IBM Z6 accelerators.
  • Montalvo and heterogeneous x86.
• Domain-specific vs general-purpose programming languages. The same sides here, with Kunle advocating domain-specific languages, and Anant and David Padua more in the general-purpose camp. I like domain-specific better, it seems to rhyme more with what I see people actually doing today to increase programming productivity overall.
• Memory bottleneck or not? The most interesting discussion came when memory bandwidth and cache sizes were discussed. One quite common school of thought over the past few years teach that caches per core will shrink, and bandwidth to get data into and out of a chip is going to be a severe restriction on what can be done. Not all in the panel agreed with this, there was the idea (mostly from Kunle) that in some way the massive bandwidths and low latencies achievable within a chip (compared to between chip in a classic discrete-processors multiprocessor) could make this less of a problem. Personally, I think this is going to be some kind of problem, but maybe not as much as passing data around faster might reduce the need to store it temporarily. Despite the need for more bandwidth, nobody really agreed with Erik’s thought that maybe it makes sense to build chips that do not max out on the number of cores they contain, but rather try to balance core count with achievable IO bandwidth. That idea has some merit.
• Core counts. Moore’s law tells us there are going to be thousands of cores on a chip fairly soon… but if we do not manage to make good use of them, maybe the growth in core counts will slow soon. Putting four or six or eight cores into a general-purpose system makes sense today, but more than that might turn out to be a waste for the vast majority of users that do not have problems to solve and programs to run that can make of more than that. In the same sense, maybe it is better with slightly fewer more powerful cores than a maximum amount of minimalistic cores, considering the state of software available today. So it sounds like a fairly divergent future here.
• Shared memory or local memories? Most of the seemed to be in the camp proposing that shared memory is too convenient not to have, even when it really is bad for you. Several bad jokes comparing shared memory to alcohol, and the moderator of the panel suggesting that a good way to avoid the hangover of shared memory is to stay drunk… whatever that means in practice.

Somethings were generally agreed upon, though.

• Programming is an issue, shared-memory or local-memory or whatever. the idea for the solution varied, however, as discussed above.
• Cores will still be plentiful and that operating-systems focusing on sharing time on a single very valuable core is an idea of the past. The keyword for the future is spatial sharing and reducing the overhead of management (I have some previous blog posts on this topic, especially on the subject of IMA and real-time control when cores are free).
• Virtualization and isolating partitions of a multicore chip from each are necessary mechanisms. Running multiple different operating systems on a single chip will be quite normal, probably under the control of some global hypervisor.

Any comments on this from my small audience? I think the topics under discussion are quite fascinating and the kind of issues on which the success of major chip design projects will be decided. A good architecture with a good programming model has a great chance of success (as long as it looks like a continuation of something existing ).

Both IE8 and Chrome have taken to running each tab of a multi-tab browser as its own protected process, to make it both parallel processing and to increase robustness. I think that is a very good idea, and I am waiting for Firefox to catch up.

Why does running a browser as a parallel program make sense? If you look at the tradition, when the web started, you would load a page, render it, and read it for a long time. With multiple tabs and windows, each such display was really also just a set of static prints of pages that you flipped between. No point in being parallel there. However, in recent years, the web page model is changing. Pages are becoming far more active, starting a long time ago with Java applets, Active-X controls, and similar, and today the main drivers seem to be Javascript/AJAX/Web 2.0 pages and media players like Flash and Silverlight.

Basically, we see another example of a domain change enabling parallel processing to be applied. The domain of web pages has changed from single-shot renderings of single pages at a time, which is essentially serial, to lots of active programs running at the same time.

I think we are going to see more of parallel processing being used to enable richer user experience. This is one way that the world is making use of the increase in computing power and communications bandwidth, just because it is there. It gives us a nice sea of threads to run in parallel — the only issue probably being IO bandwidth and cache restrictions of single chips.

The use of processes for robustness is kind of an application-level virutalization. The OS provides isolation between processes, just like virtualization provide isolation between operating systems.

Since Steve is a general PC guy, he seems to have a hard time acknowledging that you need anything but an x86 processor (or a few). However, in this episode he did note that this would greatly benefit from special-purpose acceleration hardware for PKC. So here is a clear-cut case where the addition of specialized accelerators make sense even in what is considered “general” computing. This is a favorite theme of mine, see previous blog posts like the Kunle Olukotun Interview, IBM z10 accelerators, and my Niagara 2 writeup.

So here we have it: special-purpose acceleration will save the Internet, and the only architecture missing processors with good crypto accelerators seems to be x86. SPARC, Power Arch, and zSeries all have chips with accelerators on them. One would presume that either AMD or Intel — maybe more likely AMD who are now working hard on integrating things like GPUs on their chips — will soon release an x86 with this kind of support. It is also a case where general multicore use does not really make much sense, as using an additional general-purpose core is going to have much worse performance per energy or per area than a dedicated accelerator.

The future is heterogeneous and full of accelerators, I still believe that is the case.

Their main important assumption is that the VMM cannot be tailored to avoid detection by any particular piece of software, but has to be sufficiently like the real thing to fool something the first time it appears. They discuss from the perspective of virtualization solutions like VmWare that aim at high performance before all else. The virtual PCs generated by VmWare, Parallels, KQemu, and others are all compatible with physical PCs — run the same software — but are not at all identical in detail. So they are not transparent in the words of the paper. This means that they are quite easy to spot.

There are some holes in functional differences that VMMs can quite easily plug. The paper shows how you can get a different-sized TLB (compared to the physical hardware), for example, from interference from the VMM. This can obviously be fixed in the VMM, at a cost in performance. The reason such differences are there is that VMMs are optimized for performance at almost any cost. As long as the requisite operating systems run as they should, the VMM is fine even if it is does actually correspond to any particular existing physical machine. This is a testament to the tolerance of modern operating systems towards their hardware. Basically, any OS that probes hardware and discovers what is there will work fine as long as the (virtual) hardware exposes devices that it can recognize. This is quite different from the 1970s or 1980s where an OS would definitely expect a very particular hardware setup with very peculiar timing to run at all. Thus, making a VMM totally identical to some physical machine is a waste of effort and performance.

Paravirtual approaches like Xen and what Sun has with Niagara and IBM on their Power servers, where the OS is rewritten by having drivers for a purely virtual hardware/software interface is an obvious generalization from the VmWare compatibility approach. Compatible versus transparent/invisible virtualization is really only an issue in the x86 PC world, since all other datacenter architectures are virtual by definition and all operating systems work towards a standard virtual layer. In such an environment, I have hard time seeing that the question posed in the paper does even make sense. You are always virtualized, period.

Embedded Virtual Platforms

Anyhow, back to the main thread. There is still a large set of targets where transparency and compatibility are of interest. x86 PCs is one such target, it is an interesting question for older architectures (Alpha, Vax, Sun and IBM in older generations). In particular, it is an important topic for embedded systems where you want to use virtual or simulated approaches to develop and test software. As part of that software development process on a virtual machine, you could potentially be examining malware of various kinds. A good not-too-hypothetical example are mobile phone viruses.

If we look at embedded system virtual platforms, the functionality of the simulator is usually more complete and more like a particular physical machine than what a VmWare-style datacenter VMM. This is partially due to embedded software stacks tending to be a bit pickier about what they run on, and partially due to the simple fact that the goal really IS to expose the hardware/software interface of a particular piece of hardware as closely as possible. Also, since this is usually cross-targets (Power Arch on x86, for example), there is no performance gain from using features of the host directly. So items like TLB counts, memory layout, memory content, flash memory programming, etc. are all going to be functionally identical to the physical machine.

Timing is Key

Thus, just like for a patched VmWare-style VMM as discussed in the article, the main attack vector remains timing.

The best way, according to the authors, to spot a VMM is to look for timing differences compared to the behavior on normal hardware. Despite the inherent variability of typical hardware, there are cases where VMMs by necessity vary detectable amounts. I would say this means a factor five or more over many tests of a case.

The authors discuss whether tools like Virtutech Simics could be used to overcome this problem in the context of x86 PCs. I think the main argument for something like Simics for this purpose is that by simulating the entire hardware platform and providing all timing measurements from a strong virtual time base, you do not see the types of time differences that can be used to detect a “normal” VMM. However, since the paper considers Simics and SimNow (from AMD) to be about ten times slower than native hardware, you can always detect them using a non-local time source. That is likely true. But it less obviously true for an embedded target where the simulator running on a fast PC might well be just as fast as the target.

The Multicore Timing Attack

A more intriguing aspect of embedded virtual platforms that could be used to detect virtual platforms is how simulation of multicore machines is handled. For performance reasons, simulators use temporal decoupling, where each virtual processors is run for a “long” time slice before switching to the next. We discussed the effect of this in a recent presentation at the multicore expo (link to previous blog post), and some of that data is worth repeating.

Here is a slide explaining how temporal decoupling works:

So what does this mean in practice for detecting that you are running in a virtual machine?

It means that the communication latency between parallel threads is proportional to the size of the time slicing. If you have two threads progressing in parallel doing spinlocks, on a real machine they will be stealing the lock from each other all the time. On a temporally decoupled simulator, you will rather see a behavior where you can take the lock and then recapture it a few times before missing it. This effect was captured by a simple test program that we wrote, and the data is shown in the slide below:

The program here is running two threads in parallel, updating a shared variable, with three types of locking for the accesses:

• No locking at all
• A local lock to each thread being used (“fake locking”)
• A proper lock

The interesting behavior is the execution time of the program for each of these locking styles. Obviously, running with no lock is the fastest, and with proper locking the slowest. The relative speed of these is the factor to consider. On real hardware, this program observes a very steep increase in execution time when using proper locking. On the simulator, as seen above, the difference in execution time between fake locking and proper locking is significantly smaller when using a long time slice compared to when using a short time slice. The behavior on physical machines is much more like that observed at time slice lengths of ten than that at time slices of 10000.

Normally, a multiprocessor simulator with any ambition to be fast has to use a time slice of 1000 or more. Thus, detecting that you are running inside a simulator is quite simple. If the outside world time seems right, check if you can see strange timing behavior when using locks. Since high speed requires a long time slice, you cannot have both correct real-world timing and a large performance difference. And on the other hand, if the behavior with locking seems reasonable, you should check the real-world time — as a simulator with a short time slice will be way slower than the real world.

The paper authors note a similar aspect in desktop/server x86 VMM detection. They discuss “performance cliffs” that appear when doing “unusual” things. For example, VmWare is engineered assuming a minimum use of self-modifying code. Performance is much worse if you use it extensively, and this can be used to detect VmWare quite effectively. This effect is quite similar to the time slice effect in embedded virtual platforms.

Hope you enjoyed this fairly long rant. And we have not even begun exhausting the contents of this topic… luckily, these discrepancies only very rarely impact the usefulness of virtual platforms. Since most software even on an embedded system does not care about detailed timing like this. In the example above, we still see the lock contention. So we know that we are getting an increase in execution time from the lock. Only not a complete picture of what it means in absolute terms. We will still find missing locks and overused locks.

But for some reason, they do not enforce the use of security keys on their customers. Even when you have obtained a security key (which is optional, weirdly enough) and said you are using it, you can still login without it doing some additional security questions. For the reason of convenience! Which basically reduces the security added to nothing, since you can still login in a traditional fashion.

I am all for listening to the needs of customers, but sometimes you have to assume that you know better than your customer. And security for financial institutions is one area where the financial institution does know better than their customers. The very idea of letting someone get around two-factor authentication for convenience is just amazing to me. Even more amazing is the Bank-of-America login that apparently (from Leos comments in the podcast) do not even use any kind of hardware token for login. This is akin to having safety deposit boxes put in the waiting area in a bank and asking customers to just put their own padlock on them.

Every Internet-based bank where I have been a customer have done SOMETHING more than just a password. There have been little crypto dongles where you enter a challenge number and get a response, a card with one-time passwords, or a smart card reader that gets a one-time number from the chip on the smart card itself. Or a one-time password sent over SMS to register a certificate on a computer. Not all perfect solutions, but in all cases security has at least been considered and not just customer convenience.

For banks, you do not want access to be too simple. You want your money to be safe. And it is OK to make access a bit more complex than just user name and password.

I hope the argument is not cost-based. The cost of giving out hardware tokens should be minor compared to the cost of lost customer money. It is just part of what it means to be in business as a bank, you do have to pay for offices (or at least server rooms for an internet-only bank) and customer service.

I guess this is one more thing that falls in the category of “the US is a strange land”. Because I hear an undercurrent of “convenience is more important than anything” and a fear of losing customers if login is too complex. Which in this case has to be considered the wrong priority.