What was striking was his basic attitude to the target machines: he assumed “manycore” like 100-way Tilera, 320-way ATI/AMD graphics processors, and similar devices. With these many cores, the goal is to implement an algorithm using a few cores as possible to save power. He assumes that there are always cores available, and wastes no time reducing the core count. One kernel on each core, including replicated kernels and synthesized buffering kernels.

This was the best example I have seen so far on an actual implementation of idea that cores are free. For previous writing on this topic, see “what is efficiency when cores are free” and “real-time control when cores are free“.

Now a study in IEEE Spectrum shows that even for the classic embarrassingly parallel problems like weather simulations multicore offers little benefit. The curve in that article is priceless. As the number of cores grow from two to 64 performance plummets by a factor of five. Additional processors nullify each other.

Call it the Nulticore Effect.

I think that Jack misunderstood some of the article. What it really says, as far as I can tell, is that certain types of applications will have problems with the lower external memory bandwidth per core afforded by a 16-way or 32-way multicore based on traditional processor architectures.

As I read it, regular classic “embarrassingly parallel” (or as Grant Martin would say, “proudly parallel”) problems can be handled by managing data location and computation location carefully to colocate data and code, which lends itself to on-chip caching and probably also local-memory architectures.

When other problems that are less regular are going to run into the memory bandwidth wall:

But an increasing number of important science and engineering problems—not to mention national security problems—are of a different sort. These fall under the general category of informatics and include calculating what happens to a transportation network during a natural disaster and searching for patterns that predict terrorist attacks or nuclear proliferation failures. These operations often require sifting through enormous databases of information.

So while I think the Sandia people have a very good point to make, it is not the end of the usefulness of multicore. It is only the case for bandwidth-intense irregular algorithms, while many systems today make good use of hundreds of cores without a problem. Also, the research in IEEE Spectrum proposes a solution in the form of stacked memory — so what we really have is a bit of PR for a particular kind of architecture…

There were some interesting comments on how to define efficiency in a world of plentiful cores. The theme from my previous blog post called “Real-Time Control when Cores Become Free” came up several times during the talks, panels, and discussions. It seems that this year, everybody agreed that we are heading to 100s or 1000s of “self-respecting” cores on a single chip, and that with that kind of core count, it is not too important to keep them all busy at all times at any cost. As I stated earlier, cores and instructions are now free, while other aspects are limiting, turning the classic optimization imperatives of computing on its head. Operating systems will become more about space-sharing than time-sharing, and it might make sense to dedicate processing cores to the sole job of impersonating peripheral units or doing polling work. Operating systems can also be simplified when the job of time-sharing is taken away, even if communications and resource management might well bring in some new interesting issues.

So, what is efficiency in this kind of environment?

It was clear from both the panel discussion and discussions over lunch that programmer productivity and predictability are things that can be traded for absolute100% load on all cores. Just like making 100% use of main memory is not usually a design goal today, so making 100% use of all processor cores is not a reasonable the goal tomorrow. Some resources are so plentiful that it makes sense not to try to push usage to the limit.

With 100s of cores, it is quite likely that even for the most performance-demanding loads like doing LTE decoding, it is not worth the herculean effort to get all cores running at full speed all the time. Getting 80% to 90% of the cores working on a workload is probably a good tradeoff.

Another tradeoff you can make is to increase determinism and debuggability by assigning tasks and schedules in a more static and predictable way. Instead of trying to balance loads across the cores, tasks could be assigned in some static or semi-static manner, so that the execution of a system can be repeated with some chance of success. That should not be too hard if all cores run a static cyclic scheduler, for example, or even a single task on each core. Dynamic scheduling might well be a global suboptimization in a world with plenty of cores, as it just makes things more complex for a fairly small increase in actual efficiency. You could also imagine putting debug agents and code on certain cores just to help you get better insight into what the system is doing. A bit like I blogged about after last year’s Multicore Day, asking designers to put more silicon into debug functionality. Maybe in a 100s of core device, we allocate cores to debug as well (I do not think we can do without dedicated debug circuitry, as that is needed to effect things like stopping cores quickly and similar).,

When I heard this, my gut reaction was that “hey, that is not particularly environmental” — any kind of waste of resources is really an anathema to the ecologically friendly society we need to build over the next 10-20 years. But then someone pointed out that a key part of the efficiency equation is that you turn off the unused cores and accelerators so they do not use any power. And since the cores are a resource that keeps increasing in count from basically the same use of resources  (manufacturing a chip will cost about the same amount of energy and materials for each chip, but with finer geometries you pack double the number of cores in it), it should be fine. It should also be noted that multicore computing by itself allows for more efficient processing units, for a variety of reasons.

Robustness also tends to increase if you have some slack in your system. For example, most hard real-time systems insist on not being more than 80% loaded or so (on a single CPU) even at the worst of tested times. To have some margin for the inevitable unexpected situations. For a 100s of cores device, you might also want to spare some cores for the case that hardware faults crop up in certain parts of the chip. Then you can shift loads to other cores (which obviously requires a pretty resilient interconnect to make any sense).

This final point bring me to my final thought on this was of building computing systems: in some way, we get closer to physical engineering habits when cores are free. We do not build bridges with the minimum amount of concrete and steel to handle the load we expect. Instead, there is a margin of error of a factor of three or five or so, to make sure that even in the most unexpected of unknown circumstances, that bridge will still stand. In a similar way, we might be able to use lots of free cores to engineer software systems that have far more resilience in them than todays systems that keep trying to make maximum use of the resource of clock cycles and instruction processing count. I do not quite know how that kind of system would look, but the analogy is very interesting.

But is that really always going to be true? Is it reasonably to think of CPU cores are being free but other resources as expensive? And what happens to program and system design then?

This idea was brought up again in an interview in edition 33 of TimeSys LinuxLink Radio podcast with the creators of the Propeller chip, Parallax. In this chip, they abandon interrupts and instead dedicate entire processors to various IO tasks. The cores stay in very low power mode until something interesting happen on IO pins, and then wake up and process. No interrupts, no interrupt handler, almost zero latency. Very good for real-time systems. The intuition you have just screams “why not share the IO load on a single processor”? Which is old wisdom, processors are expensive. And having a bunch of them just waiting in deep sleep for something to happen seems absolutely nonsensical. Especially as the current propeller chip only has eight cores, it seems a bit limiting and just screaming for timesharing and/or virtualization to handle a few more tasks than eight…

But is it really?

Consider the alternative design. A much more powerful processor running an operating system with all its associated overhead. For hard real-time control, what you do there is either use a static time-driven scheduler or use event-driven programs with an analysis that makes sure that you can always meet all deadlines. And these solutions have a hard time reaching 100 percent CPU usage and usually involve a fairly high interrupt latency. For really short real-time latencies, you end up vastly overprovisioning the processor in order to make sure that the cycles that an interrupt takes to process in overhead pass as quickly as possible. With caches, you either have to lock them or once again vastly overprovision the processor to make sure to meet guaranteed latencies.

The interesting question that I do not have really good numbers for is for a given problem of say ten critical real-time tasks, using ten simple cores doing one thing each or sharing ten critical tasks on a single core is the least expensive solution hardware-wise. Since it seems that in general, to make a processor ten times faster you need either ten times the clock frequency or a significantly larger die, it is not clear-cut. Especially not with a tough latency constraint.

If you factor in the cost of software development and validation of system functionality, I am pretty sure that the multiple simple cores approach is very competitive. Instead of analyzing a complex concurrent software system full of potential nasty shared resources and unknown software paths, you can handle ten small programs running on simple cores with simple timing, and with shared resources that are at least obvious from the hardware design.

There will be shared resources, be sure of that. If nothing else, the memory controller will be an issue. The propeller chip solves this by strictly temporally sharing memory access between the cores in a fixed static schedule. Predictability before all!

Note that the EU STREP project called MERASA is looking into something that could be related to this idea, with 2 to 16 cores per chip with lots of predictability piled on.

Marketing it

Selling such a chip as a standard part would seem to make sense as well, as most control systems are fairly low-volume. The propeller chip is very small, eight cores only, but if you look at what Ambric and Picochip are doing, it is easy to envision a controller part with twenty or a hundred little cores on. That could be really cool

If you have a high-volume part, I guess you could build that control ASIC yourself today, around something like a Tensilica pile of cores, which have been proven to scale to hundreds of cores even today in custom ASICs.

A half-way design in this style is the Freescale 5514-15-16 parts which combine a compute core with a smaller core to handle all the interrupts. If nothing else, this validates the fundamental idea of interrupts being an issue. Putting all interrupts on a secondary core could be seen as a bandaid on a bandaid… but it sure makes sense if the cores are not considered free.

Will be interesting to see what the future brings.