The paper describes a simulation for a network-on-chip based homogeneous system containing a “ARM-subset” ISS instances with local instruction and data caches, some local RAM, and also some shared RAM. Each core runs its own local software load, there is no SMP operating system. All communication between cores is over shared memory, using explicit operations across the NoC. All cores run a single cycle before they check communications from their neighbors.

This last point is crucial to understanding why this is feasible at all – in general, simulating a general shared-memory multiprocessor machine on a shared-memory multiprocessor falls down on the synchronization overhead. If your simulation semantics dictate that you synchronize every cycle anyway, and you do not try to optimize each core simulator, there is clearly decent room for parallel execution. By including the cache, they increase scalability, since there is more work per target cycle that can be run in isolation.

After reading the article, I am impressed by their work – just getting this to work is pretty good work. But there are quite a few questions which are not really answered in the article and which are crucial to understanding just how well GPGPUs could be used for this kind of ISS work.

• The targeted level of abstraction is a bit confusing. The authors claim it is “instruction accurate and not cycle accurate”, but still simulate caches and cycle-based communications across the NoC. If I read the paper right, communications will take a varying number of cycles depending on the distance for messages to travel. This is more detailed than a typical “instruction accurate” simulator.
• The target system does not run an OS – that might (but I do not know) be an advantage for their approach, since it probably implies less variation in the instruction flow in cores, potentially enhancing the amount of time that all ISSes in a thread group in the GPU can execute the same instruction. This would seem crucial, as if each ISS was running a totally different program, the instruction execution part of the code would be running serialized.
• They should really try to run the same kind of simulation on a high-end x86 CPU like an Intel Sandy Bridge with 8 or more hardware threads. I wonder if their scaling might not work just as well there – and with a much faster serial execution engine. This should give  a much more relevant point of comparison for GPU vs CPU execution of the simulator than…
• the comparison object they use right now, a JIT-accelerated multicore simulation using OVP seems pretty irrelevant since it is not doing the same thing at all. That simulator does not simulate the caches or NoC, just a large number of isolated processors. They also do not run a parallel program on OVP, but rather a large number of single-core fibonacci and dhrystone programs. Thus, the fact that OVP uses a large temporal decoupling time slice does not matter for semantics. It just does not seem like a very relevant comparison point. OVP and their simulator try to solve different problems – fast execution of general code vs. performance profiling of massively parallel machines.
• As I understand it, the given “S-MIPS” numbers in the evaluation tell us the total number of MIPS that we get out across all target cores. That seems to peak around 2000 – which isn’t necessarily that fantastic if we compare to high-performance ISS work in general where a few GIPS is definitely achievable. It is pretty good considering the level of detail here, though, where i would expect a normal ISS + cache simulator to produce at most a few MIPS. Once again, the authors need to be a bit more precise as to what they compare to what.
• Not having an MMU and not implementing any interrupts or exceptions in the target machines avoids a large part of the complexity of a real ISS. That complexity might well be too much for the quite rigid execution environment of a GPGPU.
• They missed that Simics, unique among instruction-accurate mainstream simulators, is parallel since version 4.0.

So, overall, this paper does not really tell us much whether a GPGPU can be used for instruction-set simulation in general. It does tell us that it might be doable, but there are many crucial complications which are not addressed.

He showed a very interesting discussion from a few years ago on the x86 memory model and the implementation of spinlocks in the Linux kernel. Various experts went back and forth over whether the final MOV that sets a lock variable to 1 needed to be prefixed by LOCK or not. The discussion ended when Linus Torvalds said “I know that it is needed”. Only to see an Intel architect finally intervene and say “you know, really, it isn’t needed”. This was followed by a series of releases of Intel manuals documenting the x86 memory model, with increasing precision in each release. Intel also actually changed the published rules along the road, withdrawing some optimizations as they realized that they would break existing software.

Note that such a description of a memory model must both describe existing hardware, and serve as the guideline for future hardware. Therefore, there are optimizations that are not implemented today but which are possible given the rules. Such optimization opportunities can be removed from the rulebook as long as they have never been part of shipping hardware, so it is not as crazy as it might sound.

Anyway, the point that Francesco made was both to tell an interesting story from history, and making the point that describing and understanding memory models is hard. I certainly agree with that. I recall an ISCA many years ago when some computer architecture professors all agreed that very few people really understand consistency and weak memory models.

To make life easier for programmers, Francesco and Peter Sewell (in Cambridge) has defined their own set of rules for x86 memory consistency. This is not an architecture spec, but a rule set for regular programmers. It is found at http://www.cl.cam.ac.uk/~pes20/weakmemory/. Essentially, the conclusion is that x86 in practice implements the old SPARC TSO memory model.

They have also attempted to formalize the Power Architecture memory model. Both the actual memory model and their model of it can only be described as very complex. The programmer’s model is expressed in terms of store queues, speculative instruction execution, and commits of instructions. Not something you easily keep in your head. It is interesting to note that ARM MPCore essentially copied the Power Architecture.

He showed an interactive simulation of the Power memory model, and the way that you need to think about it in terms of propagating information between threads and committing them. It is possible to propagate values and then another propagation overrides a value before the thread commits… Fun. Or a headache.

The big take-away from the talk for me is that it confirms the observation made may times before that SPARC TSO seems to be the optimal memory model. It is sufficiently understandable that programmers can write correct code without having barriers everywhere. It is sufficiently weak that you can build fast hardware implementation that can scale to big machines.

Maybe TSO does not theoretically scale in the same insane way as Power or Alpha does/did. But the cost of that theoretical scalability is that programmers might have to litter their code with sync operations just to get it to run correctly. With too many sync operations, the code will run very slowly negating any advantage on the hardware level. Note that sync operations can be very expensive. Doug Lea, in the audience, pointed out that a sync can cost up to 300 cycles on a POWER5.

Cary’s articles deal mostly with database tuning in the Oracle ecosystem, but most of his observations apply to any kind of programming with a performance requirement. It is worth a read. It was good to see him dissect performance, including obvious – but not really obvious – concepts like the difference in usefulness between average and worst-case response times from a user perspective.  In essence, you need to watch the spread of response times, and try to keep the worst times from getting too bad, rather than just look at an average that might conceal extremes that frustrate users.

Cary also made the comment noted in the title of this post. In his opinion, the performance instrumentation built into Oracle has an overhead of -10% – or even -20% or -30%, since it enables optimizations that would otherwise have been impossible to do. This is something worth noting in general – overhead that looks bad when considered as a local cost might be a net benefit in the grand scale of things, by enabling measurements and insight that let a program run much faster.

The ACM case study on Photoshop can be found online as a resource at the ACM Queue, with what seems to be mostly the same content. It was written by Clem Cole, at Intel, who interviews Russell Williams of the Photoshop team. It is very instructive to see how the Photoshop team has built an application that works well with 2 to 4 and maybe 8 cores, but that really needs to reconsider parts of its architecture to scale beyond 8.

Clem from Intel pushes Russell by bringing up various examples of next-generation architectures, in particular the fact that clusters-on-a-chip and NUMA memories look inevitable. The Photoshop people seem to take a wait-and-see approach to this: they first want to see some architecture have real traction in the market before they commit and rearchitect their software to make use of it.

The problems of debugging parallel software are also brought up. There used to be a simple bug in the asynchronous I/O system in Photoshop that took ten years to uncover!  Essentially, the programmers had not considered atomicity properly in the presence of multiple threads. With that kind of example, it is not surprising that the Photoshop programmers are very careful when planning and performing parallelizations.

The target domain of Photoshop is to some extent naturally parallel, but not as much as I would have thought. Since a user might operate on any part of an image, large or small, and maybe start and then abort an operation, it is not just a matter of splitting a image evenly across threads or cores. There is a significant amount of variation in just how parallel things can be in Photoshop.

Photoshop has had an easy-to-use parallelization system in place since around 1994, which lets programmers write simple serial computational kernels which are automatically applied to parts of an image in parallel. The Photoshop program itself takes care of the synchronization between kernels, and the kernels can be simple and robust and without any parallel code inside. This is a pattern that has been seen before, and which does make a lot of sense – if it can be applied successfully. Apparently, this is not necessarily the easiest thing to scale beyond four cores.

The main performance limitation for Photoshop performance keeps being memory bandwidth, rather than raw compute performance. This also limits the need to aggressively scale to higher levels of parallelism: as long as multiple threads do not give more bandwidth, it has proven hard to use more than two or three threads on any multicore processor as that is sufficient to saturate the memory system. Apparently, this is different on the Nehalem (Core i7/i5/i3) generation of Intel multicore processors, where each core has a dedicated non-stealable slice of the memory bandwidth.

For the near future, it seems that the big step for Photoshop is going the route of using GPUs for acceleration, rather than 10+ core main processors.

I made some cocky comments about just how stupid the current implementations of this idea are a few years ago, but in my discussions with Tennessee I have realized that things are not that simple. Essentially, you are caught between the structures and strictures of the certification agencies, and the complexity of the hardware with its many shared resources making predictability and programming very difficult.

It is a field which touches to my old work on WCET, but with a target system that is much less amenable to analysis. I still think it is a good idea to try to use seas of simple cores and separate work physically rather than virtually, but that might be a battle that will never be won. If nothing else, even on a massive spatially-divided multicore device, there will be some shared resources that make life difficult when very low levels of jitter are tolerated.

Read the interview, and read Tennessee’s own blog – he has done some pretty cool things both in hardware, software, and virtual hardware.

VirtualBox is able to give a guest OS up to 32 virtual processors for its use. The system virtualizes cores so that you can allocate more cores than you have on your host. You don’t want more active cores than you have physically, or performance might suffer badly (there is a good blog post hosted by blogs.sun.com about VirtualBox 3 and its SMP support, read it before Oracle decides to kill off all the old content…).

Second, the development of that SMP support only took some six months. But getting it to work right took 18 months. Sounds like a familiar story for parallelizing software of this kind. The interviewees made it sound like the code for this was utterly complex, and I can believe that too.

VirtualBox makes extensive use of hardware virtualization support on x86 hosts to enhance performance (Intel VT-X and AMD AMD-V). As they see it, all other alternatives are inferior, including the VmWare tradition of binary translation and patching. They claimed that VmWare actually interpreted all of the code in a guest, but I find that a bit hard to believe.

It is interesting to compare their approach to something like Simics, which was made parallel in the Spring of 2008 with Simics 4.0 Acclerator. The advantage an IT run-time virtual machine like VirtualBox has over a virtual platform like Simics in creating a parallel system is that they can make use of the cache coherence on the host to propagate information between the cores. A virtual platform cannot assume that the host and target are of the same type, which is necessary for this to make sense. It also makes VirtualBox entirely nondeterministic, but that is just plain normal for a physical computer system. One interesting intermediate form here is the IBM CECsim system, where a z-series mainframe is used to simulate a slightly different z-series mainframe, including running multiple simulated processors in parallel. CECsim also makes use of the hardware cache coherence and is nondeterministic.

David Patterson’s article is very similar in its message to what we used to hear five years ago as everyone woke up and panicked when single-core computing ran out of steam. Chris Nicols is extrapolating from the current state in desktop PCs, and asking how the programs we run today will work when you have scores of cores rather than two or four.

The main message in both articles is that software needs to adapt to multicore, and that this is not happening as quickly as it needs to. It is clear that automatic parallelization of existing code is a no-starter, and the Computer World article proposes the use of domain-specific languages (which I agree is a very good way to go). David Patterson is less clear on what he thinks is a good programming model for multicore, leaving that as an open problem. Patterson does point out that we have quite a few success stories in parallel computing. I think it is important to not underestimate the set of problems which are amenable to parallelization. In particular, in the embedded field, we have many naturally parallel problem domains. For example, networking offers abundant parallelism as many clients, servers, and data streams are active.

However, both articles also miss some of the things which are happening to make multicore easier to use, especially in embedded. My favorite example of a technology that nobody seems to talk about outside of the embedded field is the hypervisor. With an hypervisor (such as the Wind River Hypervisor), you can take existing distributed systems and consolidate them onto a single multicore device, continuing a long tradition of multiple-processor programming in embedded. Also, the hardware-based debug tools which are available for embedded systems – including multicore – do not seem to register at all with the mainstream researchers. It is really a shame, and hopefully we can start to change that with conferences like the S4D where we start to bring  industrial debugging experience to the academic community.

Another aspect which is missing from both articles is any discussion about the debug and test of parallel software. Coding a parallel program is one thing, making sure it works is quite another. I think this is an interesting problem in its own right, as exposing parallel bugs is often just as hard as writing the buggy code to begin with. Wind River Simics is a tool that can be used to really stress multicore software, thanks to its ability to vary configuration parameters and inject some extra delays into the target system. Simics is also a very good tool for debugging multicore software thanks to its controlled deterministic execution environment. I have already discussed this in a Wind River blog post, and I will not repeat the argument here.

Post scriptum: setting some facts straight

I want to quickly correct some factual mistakes in the Computer World article: the processor power wall at 130W that he discusses is often much lower in embedded systems – a mobile phone drawing 130W would not be very popular, and in networking infrastructure, the magic limit today seems to be around 30W. So it all depends, and the Itanium example is totally irrelevant. In certain mainframe computers, the limit is higher than that. The same is true for the claimed upper possible clock speed limit at 4 GHz. This has been surpassed by several companies, most notably by IBM who clocked the Power6 to 4.7 GHz a few years ago. However, he is right that we will see cores multiplying in almost all systems as single cores run out of speed increases.

As Jack Ganssle said, us embedded folks get no respect.

This was the first multicore event that I have been to where we did not have a keynote speaker or technical paper from a hardware company. So there was really nothing here directly about how to build multicore chips. Rather, the workshop tended to be about how to program, use, measure performance on, verify software for, and generally work with multicore chips. From the perspective of software people, rather than hardware designers.

Obviously, hardware aspects enter into such talks, but it is the perspective of a user, not a designer. For example, a hardware designer could explain how an atomic compare-and-swap is optimized in a multicore device. But here, we saw measurements on the actual operation latencies observed on real machines using such operations. Quite refreshing, and closer to my personal interests.

The keynote by Andras Vajda of Ericsson was quite interesting. The slides are not online, but the main points that I picked up and that I might not have considered before:

• Software development costs can mean that the cheapest, fastest, most efficient hardware is not necessarily the most economic. Too hard to code for means the software development time and effort removes the advantage. Obvious, but worth reiterating. Software is king.
• The workload on a cellular basestation can sometimes be highly linear and single-threaded. For example, serving a single terminal with a very high bandwidth LTE connection. And suddenly shift to a massively parallel workload as a crowd of a thousand all suddenly appear and start doing data downloads. And then go back to serial again. This means that the age-old argument that signal processing naturally “conveniently concurrent” (and here) is not always true. Nice point!
• Thus, we need adaptable architectures that can trade serial and parallel performance over time, and rebalance quite quickly. In the same chip.
• He is a firm believer that homogeneous systems will win out in the end, I still hold on to a belief in accelerators and offload engines and DSPs. This is partially because of an admitted focus on servers and services processors, and not on the baseband and signalling side. Makes sense.
• Domain-specific languages (DSL) are the future of efficient programming. Agree.

On the topic of DSLs, there was a question about the cost to support them. To me, that is a non-issue. In the organizations that I have worked, it seems that maintaining a useful DSL requires at most one engineer. Developing one, a few good computer scientists for a fairly limited time. In any case, they tend to appear organically when good programmers generalize repeated tasks.

I gave a keynote about how multicore has impacted virtual platforms (in particular, Virtutech Simics) with the following main points:

• Multicore targets increase the performance pressure on a virtual platform, as more processors will have to be simulated.
• Multicore hosts means that sequential performance of the host is going down compared to the aggregate parallel performance demands from the targets.
• To handle large target systems, the virtual platform itself has to run multithreaded on a multicore host. Getting this in place is a major, interesting, and sometimes painful process.
• Once you have a parallel virtual platform, multicore hosts provide a very nice boost in scalability and the manageable system sizes. A single multithreaded virtual platform process is also a bit easier to manage from a user perspective.
• All features in the virtual platform have to be multicore and multimachine-aware… meaning that they often get a bit harder to use initially, as there is no “default processor” you can fall back to for debugging setups etc. Everything has to be explicitly targeted.
• Multicore targets have proven to  be a great sales driver for virtual platforms, as debugging software on a physical multicore, multichip, multiboard system is just too painful.

Overall, this was a fun event, looking forward to next year at Chalmers!

What was striking was his basic attitude to the target machines: he assumed “manycore” like 100-way Tilera, 320-way ATI/AMD graphics processors, and similar devices. With these many cores, the goal is to implement an algorithm using a few cores as possible to save power. He assumes that there are always cores available, and wastes no time reducing the core count. One kernel on each core, including replicated kernels and synthesized buffering kernels.

This was the best example I have seen so far on an actual implementation of idea that cores are free. For previous writing on this topic, see “what is efficiency when cores are free” and “real-time control when cores are free“.

I guess it is Murphy’s law — if you really set out to want a bug to show up in your code, your code will stubbornly be perfect and refuse to break. If you set out to build a perfect piece of software, it will never work…

So I was actually quite happy a few weeks ago when I started to get random freezes in a test program I wrote to show multicore scaling. It was the perfect bug! It broke some demos that I wanted to have working, but fixing the code to make the other demos work was a very instructive lesson in multicore debug that would make for a nice demo in its own right. In the end, it managed to nicely illustrate some common wisdom about multicore software. It was not a trivial problem, fortunately.

First, some notes about the program. It is a producer-consumer system using pthreads, with a single producer thread feeding a variable number of compute threads with data, over a shared queue structure (a simple one that uses a single lock to protect it, making it not very scalable for small data messages and lots of workers).

The queue contains a circular buffer, managed using a standard set of full/empty/tail/head kinds of variables. There is also a flag “done” which is set once we are out of data, to tell the compute threads to shut down and terminate the program. As this program is used to demonstrate and test scaling, it is actually something that terminates. The main program spawns off all the threads, and then waits for all threads to finish before it terminates itself.

This program and the queue subsystem had worked perfectly for a long time for me, running on an MPC8641 machine with a Linux 2.6.23 kernel, with 1 to 8 cores and 1 to 16 threads. Regardless of settings like thread counts, data sizes, number of packets to compute, it always ran smoothly and terminated.

However, the other week, I moved the program, the exact same binary even, over to a new software stack built on a Linux 2.6.27 kernel. Still on the same MPC8641 machine. Suddenly, I started to see occasional freezes where the program would never terminate. I added some more diagnostic printouts to the program, and saw that the main program would simply freeze waiting for the other threads to terminate and report in. The freezes had no real relationship to input variables. Maybe they were a bit more common with short packets, but no real pattern emerged. They also happened randomly, running the program with the same parameters for a few times in a row would sometimes result in a freeze. Using control-C to quit it and restart would keep the new instance of program running well. Doing some other demo work, I found the same effect on a P4080 machine with 8 cores and a 2.6.30 Linux kernel.

This is a common pattern for parallelism bugs: they only manifest themselves as actual visible crashes or freezes or bad computation results once something in the software stack has changed, even though the fundamental issues have been there all the time. In this case, I think it was the Linux scheduler, but it is really hard to tell. Just because a program runs fine today it does not have to run fine tomorrow.

After deciding to finally sit down and turn this lemon into lemonade, I had to reproduce the error. Thankfully, that is easy when you have a simulator. The first few times I had to run the target program 20 times or so before hitting the issue, but with some parameter and timing variations I managed to create a script that would open a checkpoint, and run the program a few times under script control, triggering the bug on the fourth run (every time, thanks to determinism).

To diagnose the problem I wrote some Simics script code that I actually felt was fairly cool. I guessed that the problem had something to do with the queue and its handling of “done”, since that is what told the threads to terminate.

The first problem was that the queue was not a global variable. Instead, it was dynamically allocated on the heap by a function, and a pointer passed around, but never stored in a global variable (a good computer science graduate never uses a global variable other than as the means of last resort). Finally, my script set a breakpoint on the line in the setup function that came after the allocation. With the program stopped at that point, I could read the local variable pointing to the queue, and find and store the addresses of all the interesting members of the structure.

The code looked like this (Simics CLI), for the record:

 $mbp = ($ctx.break ($st.pos (rule30_threaded.c:222)))
 $cpu = (wait-for-breakpoint $mbp)
 $pq_addr  = ($cpu.sym "pq")
 $pq_tail  = ($cpu.sym "&(pq->tail)")
 $pq_empty = ($cpu.sym "&(pq->empty)")
 $pq_full  = ($cpu.sym "&(pq->full)")
 $pq_head  = ($cpu.sym "&(pq->head)")
 $pq_done  = ($cpu.sym "&(pq->done)")

Next, I set breakpoints on all writes to empty, full, and done. This was the most expedient route to catch actual puts and gets to the queue. Breakpoints on the queue_put() and queue_get() functions are not really showing the true flow, as these functions start by contending for the lock. Looking at writes to the actual queue members gave me the point where the tasks had grabbed the lock.

The script that caught all writes to done, full, and empty, and on each write, it dumped the state of the queue including computing out the number of elements in the circular buffer (without having to run any code on the target). To get an idea for who was active, it also used OS awareness to find the currently executing thread ID, and scripted debugging to convert the current program counter into a position in the program source code (actually, the important issue was the name of the function we were executing in).

This trace of activity showed quite an interesting pair of patterns. When the program ran well, the queue was mostly full, and it looked like the producer task always got some kind of priority to fill it before consumers could get in and drain it. When the program froze, the queue was seldom more than a few elements deep. This was the same program, on the same kernel, just run a few milliseconds later.

Clearly, the Linux kernel can exhibit quite variable behavior even for a program this simple. I guess that’s why this is called “soft real time”… Another parallelism lesson here: the scheduler is very important, and a smart adaptive scheduler can wreak havoc with software that was accidentally tuned for a different scheduler.

In the end, the crucial hint was that whenever the program froze, the “done” flag was set with a queue that was empty or contained just a few elements. I was sure that I had handled this case in my code, checking specifically for that and making sure to wake up the other threads with a signal that “the queue is not empty any more, please come check for more work”… but looking closely at the code, it turned out the code only woke up a single thread. Thus, the froze resulted from the producer setting “done” with an empty queue, waking up a single compute thread, and then having the other threads wait forever for more data to be put into the queue. The fix was easy: use a broadcast signal rather than a single signal.

In retrospect, it seems really strange that this ever worked reliably… it almost that I suspect the old Linux kernel of having a flawed pthreads implementation where signals always wake up all waiting threads, and not just a single one like the documentation says. But that will wait for another day to be investigated.

Here is the code, for reference:

void rule30_packet_queue_signal_done(rule30_packet_queue_t *q) {
 //
 // Grab lock, set the done signal atomically
 //
 pthread_mutex_lock (&(q->mutex));
 q->done = 1;
 pthread_mutex_unlock (&(q->mutex));
 // Signal any threads waiting for data to wake up
 // and discover that we are indeed done
 //
 // This is the bug:
 // - It only wakes up one thread...
 pthread_cond_signal (&(q->notEmpty));
 // To be correct:
 // pthread_cond_broadcast (&(q->notEmpty));
}

Updated analysis:

My initial analysis was that when things worked, the “done” flag was set with enough data left in the queue that all threads had a chance to pull in data and come in and see the done flag being set.

However, today I went back and wrote a deeper analysis script that also checked for reads from the done flag (turning this check on only after the write to ‘done’ to reduce the noise). I expected there to be a single reader when the freeze happened… but that was not the case. In my current test case, three out of five threads actually got in to read the done flag and terminate.  The crucial code for the compute threads looks like this:

 // Grab mutex,
 //   Check if the queue is empty, if so wait for someone
 //   to push something onto the queue, or signal done.
 //   both of which are done by setting the not_empty conditional variable
 pthread_mutex_lock (&(queue->mutex));
 while ((queue->empty) && !(queue->done)) {
   pthread_cond_wait (&(queue->notEmpty), &(queue->mutex));
 }

To freeze, a thread actually has to be doing the conditional wait here. There are plenty of other places threads can be as the program is finishing. For example, they can be waiting to grab the initial mutex lock, or actually doing compute work. That explains why some threads actually still terminate even with the buggy version. It certainly also illustrates just how chaotic concurrent programs can be. More so that you can ever imagine, really.

Quotes from the blog post:

Sequential languages hiding parallelism:

Sequential imperative languages are notoriously good at hiding or at least obfuscating inherent parallelism in the applications – which may be good news for compiler providers who spend – and charge – big bucks for building reverse engineering technologies that can detect parallelism from sequential code; as well as for highly skilled programmers capable of building parallel code using sequential tools; however, it weights heavily on the total cost of developing software, not to mention the recurring cost of porting to newer HW with e.g. more cores.

Applying DSLs to signal processing:

During the work on the domain-specific language for DSP programming we have seen some interesting results; for example, several pages of optimized C code were re-written to one PowerPoint slide worth of DSL code, while the DSL to C compiler was able to output efficient code comparable in size to the original code.

I agree fully with the final note:

The big question is not anymore if DSLs will take off – it’s more if your pain level is higher than the cost of accepting an alternative and different approach; as the cost for taking in a new language tends to stay constant while the complexity of developing software is increasing, odds are that there will be an uptake of domain-specific languages.

Go and read the full article!

I think that for next year, the organizers need to find keynote speakers that are not from the general computing multicore world. The Microsoft talk this year was a step in that direction, as it rather came from multicore programming than multicore hardware. Richard and Anders gave very interesting and good talks, no doubt about it. But it would have been nice with someone from ARM or Freescale or Tensilica or TI or ST or Ericsson or Cisco talking about the kinds of multicore embedded hardware that is being developed and used today. For example, the “next new thing” touted by the keynotes this year was GPGPU. Interesting for HPC and desktops, certainly. But pretty irrelevant for most of the people that I know. GPUs are huge, expensive, and power hungry.

GPGPU was one part of the theme this year. It is definitely catching on as the way to do number crunching in the desktop, server, and HPC world. It is not the universal panacea for any kind of parallelism, however, as Hazim and I noted in the panel discussion that ended the day. There are applications (such as parallel Simics…) that scale well on general-purpose cores, but that will never ever work on GPUs. In general, the class of problems that work on GPUs is pretty limited to massive data-parallel problems like image and video manipulation.

In the eternal homogeneous vs heterogeneous debate (follow the tags in my blog for more posts on this topic), GPGPU was grudingly accepted as a good candidate for something that will not be homogeneized with the main processors. Additionally, Richard Kaufmann gave some hints that Intel or AMD are coming out with new chips with more accelerators on board… I guess it will be security, as is already done by Sun and IBM. When I brought up the topic of more accelerators like pattern matching, compression, and the other things we see in chips from Freescale, Cavium, and others, the response was very “can only be economical for very high volume applications”.

It is striking how the GPGPU idea is bringing the classic telecommunications DSP-data plane/CPU-control plane division into the desktop and server space. Without any recognition being paid or any experience being reused from the 40 years that that has been done in telecoms and consumer electronics… as Jack Ganssle often says, us embedded folks get no respect.

In terms of programming, this year was all about general programming languages. Hazim from Microsoft talked about (and demoed) the quite pervasive addition of parallelism to both native C/C++ and managed .net code in Visual Studio 2010. Microsoft is dead serious about parallel programming, and are bringing out a whole set of different libraries and support structures to allow easier expression of parallel code. In the “LINQ” data query language subset of C#, you could add some easy modifiers to “foreach” statements to make them parallel, for example. Having a language that is your own and which you can extend at will certainly pays off in terms of innovation here. C++ moves far slower than C#, that is becoming clearer and clearer. C# and its cousins in the .net system seem to be sneaking in lots of powerful language design ideas from places like Python, and also results from Microsoft’s powerful group of language researchers.

When I tried to bring up the idea of using domain-specific languages to program parallel applications, Hazim had the wonderful comment that “that might be applicable in certain domains…” — yes, that is the idea. By being narrow in terms of target domains, you gain expressive power and semantic insight that helps move programming from “how” towards “what”. But it sounds like domain-specific is a foul word inside of Microsoft — when the audience asked whether LINQ was not a exactly a domain-specific language for data access, Hazim was a pains to point out that it is Turing-complete and that someone had managed to write a Raytracer using it… interesting. This feels more political than market-based. I guess Micro

Richard Kaufmann had some interesting notes on throughput vs TTC (time-to-completion) jobs in servers. In the “cloud computing” era, throughput is much easier to scale: just add more servers. Classic HPC is more oriented towards TTC, as you do want your results within a reasonable time. Quite often, you can most work into a throughput-oriented style by simply running lots of jobs in parallel rather than pushing through a series of jobs sequentially. Note however that we have the entire field of real-time control, real-time communications, etc., that do not work like this. But that is not the market that HP is building servers for, or that Intel and AMD are servicing.

Outside the keynotes, Per Holmberg of Ericsson gave an interesting presentation on the adoption of multicore in the control plane of the Ericsson CPP platform. The core of his talk was the observation that in these kinds of systems, multicore is not such a big revolution.

They have been distributed since the beginning. Thus, scaling by adding more processors (with local memories) is easy and multicore is only a packaging change from that. Also, most performance-intense operations are already offloaded onto DSP groups, network processors, ASICs, or FPGAs. There is not much parallelism left for the control plane to exploit. Essentially, only functions that unexpectedly become performance bottlenecks due to changes in traffic patterns are likely candidates for parallellization. Interesting point, and might be why the EETimes noted that multicore is slow to catch on in communications (the article is a bit flawed).

Patrik Nyblom from Ericsson held a talk about how the Erlang runtime engine was parallelized. From a practical perspective, the most interesting aspect was that this made applications parallel without changing a single line of code in the applications. Of course, applications had to be threaded to start with, but that is the most natural way in Erlang. He mentioned systems containing up to a quarter of a million threads — hard to do that in anything except Erlang.

He described how they had evolved from a simple implementation that worked well on synthetic benchmarks to a truly industrial-strength implementation. The difference was quite radical, as real codes feature more complex communications patterns, and make heavy use of device drivers and network stacks. This process forced the use of more and finer locks, and rethinking the balance between shared and separate heaps for threads.

They also had the opportunity to test their solution on a Tilera 64-core machines. This mercilessly exposed any scalability limitations in their system, and proved the conventional wisdom that going beyond 10+ cores is quite different from scaling from 1 to 8… The two key lessons they learned was that no shared lock goes unpunished, and data has to be distributed as well as code. Very interesting to hear this story from real software developers solving real problems.

The next multicore event taking place around here is the Second Swedish WOrkshop on Multicore Computing (MCC 2009), in Uppsala, November 26-27.

Update: note that the presentations from the event are available via http://www.multicore.se/.

Øredev

Anyhow, I looked at this year’s program to see how they developed it. In particular, it was striking that their main set of topics (see front page) did not include parallel programming or multicore.

However, there is a session called Meanwhile, which includes five talks on parallel programming.

Interesting, I guess this is indicative of how little impact parallel programming has yet had on the general world of IT programming.

It is one of the more comprehensive introductions to how to think about and use multicore architectures in the high-end embedded space. It is free to download and print, but if you want a printed copy, such can be ordered at a price of (I am told) 15 USD (did not try it myself).

The PDF is at http://www.freescale.com/files/32bit/doc/ref_manual/EMBMCRM.pdf .

It will also be linked from the “Documentation” section for most Freescale multicore chips’ information pages.

An interesting point made is that the light-weight user-level threading of the virtual machine in Erlang optimizes for massively threaded performance. The key property is that the context for each thread is very small compared to an OS-level application thread (like pthreads, for example), and this means that the context switch cost is dramatically smaller thanks to less cache and TLB contents needing to be swapped in and out. Thus, for lots of threads, Erlang tends to get more work done per time unit, as there is less execution time lost to friction in the memory system. I am not sure you can emulate this in C using a user-level package. The very small initial stack and heap size of the Erlang VM is partially achieved by the very fact that in a VM, you have more insight into and control over when memory allocation happens, and thus you can more easily do stack and heap grow operations in small units.

Another interesting aspect of Erlang as opposed to C/C++ brought out in the interview is how to do error handling. In Erlang, this is part of the language, while in C/C++, writing code to handle all cases (and handle them correctly) quickly gets painful and overwhelming. Instead in Erlang, you have a system policy to kill any thread that does something bad and restart it. With that simple strategy imposed on you, the code gets much simpler.

The podcast also brought up a StackOverflow question about CouchDB that resulted in a good explanation of the concurrency model (optimistic concurrency on entire documents, an nothing smaller or larger than that). Damien Katz came in with some more insights on transactions and CouchDB, in a discussion on how to solve the classic bank account problem: moving money from one account to another. The “ACID” solution is to make sure that changes to two accounts are always both done or none done. The CouchDB solution is to put in a record of the account-to-account money transfer (I won’t use the word “transaction” as that is overloaded in this context) in the database, and just go through all records pertaining to a particular account to arrive at its current balance. That does feel more like proper bookkeeping practice, rather than having a single unauditable  balance in an account record…

Overall, worth its time to listen to.

Last year in a blog post on video encoding for the iPod Nano, I complained about the lack of performance on my old Athlon. A bit later, I noted that (obviously) video encoding is a good example of an application that can take advantage of parallelism. Yesterday I put these two topics together in a practical test. And it worked nicely enough.

My new Core i7 920-based machine was very well utilized by the Nero 8 suite’s Nero Recode 3 application when converting some children’s movies for use on my Nano. Here is a screenshot of the CPU load at one point in the computation:

It was much higher than this at times, but capturing that using the snipping tool was harder than expected.

Essentially, the article is a report on a talk by Linley Gwennap at the Multicore Expo last week. The most interesting point are that simple single-core processors are taking over from the traditional heterogeneous processor + big accelerator pattern examplified by the Freescale PowerQUICC series. And that even Freescale themselves are “replacing” PowerQUICC chips based on the venerable CPM with “simpler dual-core chips”, which has to mean the MPC8572 currently and probably the QorIQ P2000-series chips later on.

The main point is that people are moving away from the “complexities” of the CPM-style heterogeneous hardware design, to symmetric multiprocessing designs that are simpler in one way. But harder to program when you want to have a regular old program use more than one core, as we all well know. It is a good question whether this is actually the case: I am not too sure that correctly writing a parallel threaded program for a shared-memory multiprocessor is easier than calling a hardware accelerator API or using a heterogeneous architecture… more familiar to general-purpose programmers, sure. But easier? Not necessarily.

What I do take some issue with is the implication that the quad-core and dual-core processors expanding into the market, in Gwennaps opinion, do not have these “complex hardware accelerator APIs”… all the hardware I have seen for the comms field certain feature very powerful offload and acceleration engines for tasks like network interface, TCP/IP processing, regular expression matching, security computations, etc.

Look at the feature sets of chips like the Freescale QorIQ P4080 or the Cavium Octeon Plus CN58xx family: their core acceleration engines look every bit as complex as the old CPM to me. The programming might be a bit different, their presence spun as accelerators rather than as a processor in the marketing talk, but still they are complex acceleration blocks that definitely have a lot of power. They also seem quite intent on staying and proliferating, and not going away. I see no sign that the future of computing is anything but lots of programmable cores augmented by lots of accelerators. The benefits of heterogeneous architectures in terms of power, throughput, and chip size are simply too compelling.

What is interesting in the article is also both the claimed poor state of software that is slowing the adoption of multicore, and that this means that the software stacks actually get some more time than could be expected to adapt to multicore and truly parallel hardware.

It has a nice discussion about AMP and SMP, and OS scheduling policies. It is particularly interesting to see how OSE tries to combine the two. Unfortunately, the article is in Swedish, but I would expect the CMP network that Elektronik i Norden is part of will place this article in English into EETimes or some other publication of theirs.

The article discusses some Freescale targets, such as my favorite the MPC8641D, the MPC8572E dual-core, and the upcoming QorIQ P4080.

The beginning of the argument was found in an EDA Design Line article titled “Unified Verification for Hardware and Embedded Software Developers” by Lauro Ritazzi of Eve USA. In it, he makes the following claim:

While some may have achieved the scope of jump-starting software development, they only address application programs that do not require an accurate representation of the underling hardware design. They fall short when testing the interaction of the embedded software with hardware, such as firmware, device drivers, operating systems and diagnostics. For this testing, embedded software developers need an accurate model of the hardware to validate their code, while hardware designers need fairly complete software to fully validate their application specific integrated circuit (ASIC) or SoC.

The interesting part here is really that jump-start is just for applications, and that OS and drivers require more details than a fast virtual platform can supply. I do not quite agree with this. But let’s first see what Frank Schirrmeister said:

the majority of the software development on virtual platforms is spent on firmware, device drivers, operating system porting and diagnostics. And that is not – as one could assume – on cycle accurate models, but on functionally accurate models with only essential timing, the type of models called loosely timed (LT) in SystemC.

I totally agree with this. As is evident from many different public use cases, OS, BSP, and driver development is a big use of virtual platforms. For example, last summer, Freescale announced the QorIQ P4080 with pretty good software support in terms of Linux and VxWorks operating systems, as well as some middleware stacks. All developed on Simics using an even more timing-abstracted model of the hardware.

However, Frank then makes the following claim that I have a harder time with:

In contrast, application software is developed more often than not using completely hardware independent techniques, including cross compilation from the host development machine using development kits like Apple’s iPhone development kit.

This is to some extent true, but as time goes on, I think this type of development environment is going to be less useful. Traditionally, OS vendors have had tools like VxSim and OSE SoftKernel in place to help customers “run code on their desktop”, while using the API of the operating system of choice. However, such solutions have lots of problems in how close they can get to the target.

• If you have any kind of third-party binary-only application, or want to use an existing binary component without lots of complex recompilation, you need a virtual platform running the underlying OS. You cannot squeeze that into a host-compiled API simulator.
• You are not using the same compiler and code-generation settings and build settings as you are for your actual target, and this can (read: will) introduce nasty compiler version issues.
• It forces you to maintain an additional build variant for your code, which can be pretty expensive for a complex build.
• You are not using the real OS scheduler, device drivers, and interrupt structure found on the target system. This can have a huge impact, especially for multithreaded multiprocessor systems.
• The API simulator needs to be kept in synch with the real software stack, and customized in the same way for any particular target. This is hard to get right (even though it has been done).
• The API simulator does not handle heterogeneous systems very well, such as chips or boards or racks mixing two or more different OS kernels in the same system (like a DSP and a main processor OS).
• API simulation completely falls apart when the OS is no longer the lowest level of the software stack, but you also have a hypervisor layer underneath the OSes on your target system. An API simulator simply cannot represent this kind of case.
• Using a virtual platform and the real target binaires also fits with the very important “fly what you test, test what you fly” principle of embedded software development.

For various subsets of these reasons, I see many users picking up virtual platforms as a way to streamline application development. For example, NASA recently selected a virtual platform based on Simics to develop the software for the new Orion spacecraft. That is going to be a complete software stack, not just OS and drivers, which tend to to be fairly off-the-shelf component for these kinds of systems. Most of the effort is on the application level, and the platform used is a virtual platform.

However, note that there are cases where a fast virtual platform like we are discussing here is not sufficient to validate all aspects of the code. I think the main reason we see different viewpoints on this, is that we are looking at very different types of software-hardware integration.

In a blog post I wrote last year on the dead-ness of cycle-accurate simulation, Grant Martin of Tensilica pointed out that some software desperately needs cycle-accuracy as it is intimately dependent on the timing of the hardware. This is certainly true for some aspects of drivers, and more so for the really early boot code.

Here, FPGA-based hardware-accelerated simulation of the actual design in VHDL or Verilog makes eminent sense as a way to get the details perfectly right. But that is only one part of a much greater system development puzzle, and it really only applies to very small subsystems as  it is kind of hard to fit much more than a single chip inside a hardware acceleration unit. Just as Frank Schirrmeister says, hardware accelerated simulation is very important. The nice article on the IBM z10 development that I blogged about earlier says exactly that: for some parts of the validation, there is no way around using the actual hardware RTL design.

And in the end, you have to test the timing and analogue aspects of a design on physical hardware anyway. There should not be too many suprises at this stage, if you have used all of the cool current tools right. But there surely will be some — even a VHDL simulation is a simulation, and not reality, after all.