Essentially, as the (embedded) software systems we are developing get more and more complex, we need to update our debuggers to keep up with the times. This means moving beyond looking at single processors, threads, programs, or even nodes – and making debuggers that support working with multiple contexts at once. For example, debugging a couple of different boards in a rack from a single debug session. This would mean supporting multiple processor architectures, execution modes, and operating systems inside a single debugger session. Debuggers need to orient themselves towards handling software that is already running in a target, rather than starting the target software. The debugger becomes more of a passive observer of the target system than an active component.

Take a look!

It is pretty very nice demo in which you first start a period program A, which prints the value of an incrementing counter every target second.  You then run a supposedly unrelated program B, resulting in the values that program A prints to become corrupted.  Perfect to show off reverse execution and data breakpoints in reverse as you go from the point where the corrupted value is printed to the piece of code that overwrote the variable.

But then I ported the demo to a new platform… and the bug didn’t work anymore. My bug had caught a bug and was now not working, or at least not they way I expected it to. What had happened?

Very simple. I changed the compiler I used. Since my bug relied on an unspecified behavior in C, the change was totally valid and really expected.  Still, it was interesting to see how things played out… in the end, we got a different bug from the same code thanks to the change.

The code is essentially the following, with some simplifications that make it easier to read for those not familiar with VxWorks, and ignoring all the code to start tasks initially.

// Global variables
int     iDataArray[100];
int     myWdISRcount;
WDOG_ID myWatchDogId;

// Periodic task - program A
void myWdISR(void)
{
  /* Increment ISR invocation count */
  myWdISRcount = myWdISRcount+1;
  printf("wd Fired %d times\n",myWdISRcount);

  /* Start off next invocation */
  wdStart (
    myWatchDogId,
    WD_INTERVAL,
    (FUNCPTR) myWdISR,
    (int) NULL
  );   
}

// Overwrite code - program B
int myCompletelySafeRoutine(void)
{
  uint32_t *a,i;
  a = iDataArray;   
  // This loop writes one word beyond the
  // limit of the iDataArray
  for (i=0; i<=100; i++) {
     a[i] = 0x7fffffff;
   }   
   return OK;
}

In the original setup, compiled for a Power Architecture target, iDataArray ended up right before myWdISRcount in memory.  Thus, the buffer overflow changed the value of the counter from something like 10 to 0x7fffffff.  Very noticeable in the printouts from the periodic task.

When I changed to an x86 target (using a compiler from the same family, but obviously with a different code generator since the target was different), the variable order in memory changed and it seems that we got iDataArray placed last.  Suddenly, the effect of running the safe code was that nothing happened at all.  A bit annoying for a demo.  Some small source-code changes and a recompile later, the effect was instead to crash the target with a triple fault (page fault inside a page fault handler). Seems the program now managed to corrupt some kernel state. While impressive as a bug, it was not quite what I was looking for.

I then changed the compiler type to compiler 2, and the data layout changed once more.  This produced a very useful bug, but it took me a while to actually understand this.  Now, when program B was run, program A stopped.  This looked like a bug in my program, and I actually started trying to fix this – until I realized that this was the bug I was looking for.  Running program B kills program A is just as good a bug as corrupting a counter value, after all.  In this case, the array overrun hits the myWatchDogId variable, and when that gets corrupted, the wdStart call ignores the request since it does not recognize the ID it gets.

So, in the end, I got a bug that was just as good as the first, and arguably a bit more intruiging. It is still obviously a contrived example – but I think that a good demo or lab exercise can be artificial as long as it gets the point across.  Judging from how people who have done the lab reacts, the goal seems to have been
achieved.

The moral of the story is really that compilers are free to change things which are explicitly implementation-defined or not specified at all in the C standard. That is a good thing as it gives the compiler freedom to optimize the code. If you want to control how variables are laid out in memory, I guess you have to resort to linker scripts or similar – but that was too much pain for me in this case. Just changing things around until I got a good bug, and then freezing the binary (and not recompiling it ever again) is a sufficient strategy.

Software is Concrete. Once poured it becomes extremely difficult and very expensive to change.

It comes from a blog post by Robert Howe, CEO of Verum, a company selling formal-methods-based and model-based programming tools. It does capture something of the phenomenon we all know: that software can be pretty darn hard to change, once it has shipped and is in use. It fits well with the fact that the later bugs are found, the more expensive they are to fix.

But it also provoked quite a bit of opposition when I put the quote up on Facebook, and I have to agree that maybe not all is as simple as that blog makes it out to be.

The main problem as I see it is that it compares the development of software to the construction phase of a bridge (or other civil engineering structure). When software development is really much more like the architecture and exploration work that goes on long before the bridge is starting to be built. The blog post recognizes this in a way, I admit that. But you cannot escape the feeling that the fundamental thinking behind it all is waterfall-based. It never says so, but it feels like that.

Which is a shame – I cannot see that there is any fundamental dichotomy between being iterative, agile, and changing things as requirements clarify, and using model-based development and formal methods.  The idea of model-based development is to program at a higher level of abstraction to get more efficiency. And programming at a higher level of abstraction fundamentally supports agile and iterative development, since the amount of “code” you must change is smaller. Model-based development ought to support drastic and quick changes to software just like modern dynamic programming languages and powerful frameworks support it. Add in some model checking to find important classes of bugs (like deadlocks and dead-end states) automatically, and you have a very agile programming environment.

In software development, I don’t think there needs to be a great step from design and architecture to implementation. It seems to be more gradual, you build code as you go along and ideas get more and more detailed, and slowly you get to something fairly concrete and indeed fixed. The trick is to make this process gradual and find issues early, so that they are cheap to fix.

Overall, it seems that software is more like clay than concrete. It slowly goes from soft to hard… but you have a chance to make changes during the process.

I also have to apologize for the slow blogging in January of 2011. There was too much going on at work and quite a few days taking care of sick kids. Hopefully, the pace can improve going forward.

I made some cocky comments about just how stupid the current implementations of this idea are a few years ago, but in my discussions with Tennessee I have realized that things are not that simple. Essentially, you are caught between the structures and strictures of the certification agencies, and the complexity of the hardware with its many shared resources making predictability and programming very difficult.

It is a field which touches to my old work on WCET, but with a target system that is much less amenable to analysis. I still think it is a good idea to try to use seas of simple cores and separate work physically rather than virtually, but that might be a battle that will never be won. If nothing else, even on a massive spatially-divided multicore device, there will be some shared resources that make life difficult when very low levels of jitter are tolerated.

Read the interview, and read Tennessee’s own blog – he has done some pretty cool things both in hardware, software, and virtual hardware.

Barry Lock

Barry lock gave a very entertaining keynote, from his viewpoint as essentially the champion of physical hardware debug. Lauterbach is clearly focused on debugging using hardware assists in real systems, with not much to do with high-level programming or virtual platforms. Barry has been working with computers longer than I have lived, and have seen both the semiconductor and software side of things.

His main message was that you have to take debuggability into account when buying chips for your embedded project. Saving a few cents by buying a chip with no or limited debug features will come back to haunt you, many times, in many nightmares. He had had grown men crying over the phone, asking for a miracle to save their projects after debugging had utterly failed for many months. He had seen startup companies go under, burning all their money chasing the last bug… and claimed that 75% of all product starts never got to market, blaming debug problems for a large proportion of that.

The most important debug feature is trace - which follows the theme of this being the S4D of trace. After trace, you want hardware breakpoints. Apparently, you need at least two to breakpoints to debug a system with a virtual memory RTOS. One to keep a look at the MMU, and one to actually debug code. More are better, but it is rare to see silicon vendors include many more breakpoints.

Barry gave  a number of examples of projects which had failed by not buying the right hardware. He put the blame both on the buyers chasing a few cents of costs in the end product, but also on the poor quality of silicon salespeople who rather took the price route than the quality route when selling chips.

He also noted that there seemed to be a positive correlation between industry leaders and buying debuggable hardware. Companies like Bosch, Ericsson, and Nokia always spend the extra money to get hardware that can be debugged, and have results to show for it.

Philosophy of Debug

During our two panel discussions on debug, there were two ways to look at debugging that stood out from the crowd.

The first was the observation that debugging today is very much a craft. When things go really bad, you go to the proven expert. Debugging is a craft you learn by apprenticeship with a master, and master debuggers are incredibly valuable for their organizations. This reliance on masters indicate that general programming education to a large extent overlooks debugging as a crucial skill for programmers. It also means that, in the words of one member of the audience, debug cannot scale. As problems become more complex, we still rely on single individuals, which reduces our ability to tackle problems.

The second observation was to liken debugging to the diagnosis of human diseases. As systems become more complex, their behavior gets so complicated and rich that it is hard to even precisely identify what a bug is. A simple crash or illegal operation is clear-cut – but when results of programs are just a bit off? When control loops don’t quite do the right thing, but almost? When the quality of a picture of a television just feels wrong? In those cases, we might be looking at composite measurements of many different parameters and factors in a system, and making a diagnosis of error based on the whole picture rather than each factor in isolation.

Based on these observations, I can envision a somewhat weird future where we train computer doctors (as in medical doctor, not PhD)  to diagnose computer problems based on holistic, systematic, approaches. Such education could be separate from the training of programmers and testers, as their specialty would be the diagnosis of system outputs against the expected outcomes at a high level, rather than the details of code.

There are two important aspects to this worm, as I see it.

First, it only works due to the fact that the software in question is running on regular Windows PCs.  An attack on a real embedded OS like Wind River VxWorks or Enea OSE would be more interesting, and much scarier since that would mean a much more devoted enemy. In this case, the attackers are opportunistic, using the window of vulnerability of a new Windows flaw to attack Windows-based plant control systems. They also use signed Windows drivers, which is apparently a new development in malware. All quite interesting in its own right, and worth reading about for those interested in security.

Second, the malware spreads using physical movement of USB memory sticks rather than attacks over the Internet or other networks. This makes the very important point that even if systems are not connected to the Internet, they can still be attacked if something crosses the “air gap” that separates them from the outside world. In this case, a plant would be infected by using social engineering to make some employee carry an infected USB stick into the plant and putting it into some internal PC. Once the infection is inside the plant, it might spread over networks or by USB sticks moving around inside the presumably protected perimeter.

The lesson I think we can draw from this is that using general-purpose desktop operating systems for critical systems is a bad idea. Using a more obscure real-time OS (or even Linux) would probably reduce the number of vulnerabilities – but more importantly, it would make it much more difficult to make an infection hop from computer to computer until it reaches its target.

In this particular case, all Windows machines are potential bearers and spreaders of the infection. An attacker can rely on that fact to seed the Windows ecosystems at some place, hoping to get the infection hopping from machine to machine until it reaches something interesting. There is no real need to seed the infection directly into the target plant. If the target systems had been running a different OS, the attackers would have had to get really close to the target, making them easier to stop.

Overall, embedded systems security is something that we need to take much more seriously going forward. As we rely on embedded control systems to run much of the modern infrastructure and economy, we really need to be concerned about how to secure these systems. Security needs to be part of the architecture of embedded systems, including their operating systems (please make them as robust as possible), application designs, and networking systems. Unfortunately, current embedded technology tends to be designed to work, with little care for how they could be broken by an intentional attack. One scary example of this was provided in the SecurityNow podcast episode #251, where a listener shows how easy it could be to take remote control over a car due to carelessly designed fleet management units.

Updated information

I found some more in-depth information on the issue at Infoworld. It notes that the software that is attacked is vulnerable since all installations use the same password to login – changing it is likely to break it. That is totally ridiculous as a security solution, period.

And almost inevitably, as you play around with a complex modern piece of software (which is what most of the phone is, after all), you find some obvious things which are just plain broken. You wonder, “why didn’t they think of this”, and “how could this ever escape testing?” My current best example is that the built-in web browser does not render the pages from Blackberry’s own support knowledgebase.

This is something that they have control over from both sides, so the only reasonable explanation is that they just have not thought to test that site with the browser. It seems strange that  a new user would uncover such holes after a few hours, when presumably thousands of planned and spontaneous testing hours are being spent at Blackberry on each device they release.

To be fair, this is not a showstopper for me, it is just a good example of how users tend to poke holes in all modern software products. Blackberry is not unique in this regard, and I fully admit that the software that I have worked on over the years did not always fulfill the requirement of being bulletproof. There is a classic saying that if you think something is foolproof, you just haven’t found the right fool yet.

So what is this telling us about software development? I think the main lesson is that you need people who just use your product and do so creatively, as part of the development process. Software developers usually lack the destructive imagination necessary to really poke holes in what they have created, and are usually not given time to develop elaborate test cases.

The moment you put code into the hands of users, you can expect them to find “obvious” issues in your product – simply because their “normal” use is something you yourself never dreamt could happen. In my example, I presume that Blackberry assumed that people will use the support forum when their phones don’t work – and therefore not surf to them from a phone. Sounds reasonable, until you press on a bit and ask if someone might not look at a friend’s broken phone out in the field and need to search support… or check for solutions to non-fatal errors.

I guess the key lesson for me, one which I apply in my daily work, is that product management, marketing, or testing, has to work through use cases with the software they create. Really work them through, and when sidetrack ideas appear, follow through on them, not ignore them to focus on the main task at hand. We have to combine features, and do more than what’s in typical tutorials.

Feature combination is especially powerful, I still recall the sense of “aha” I got when as a PhD student I talked to some colleagues researching the field of feature interaction. It sounds like quite an interesting field, even though I am not sure how you could formally capture something that will generate the test case “read our support site using the phone browser”…

This post used to end in “I guess this counts as a software development rant”. Now, I classify this as a “tester’s lament”…

I have seen the “write 1 clears” solution before in real hardware, but I was not aware of the other two variants. The idea of having a “write mask” in one half of a 32-bit word is really clever.

However, this got me thinking about what the fundamental issue here really is.

As I see it, it is the fact that the processor cannot address small enough units atomically. The read-modify-write that was used to start the discussion in the Embedded Bridge #37 was needed in order to get the current state of a configuration register, change some setting that only occupied a few bits in it, and write back the result to the register. The way most configuration registers that I have seen in practice works.

But if each setting could be given its own register, the problem would go away. Each operation would target a unique address, achieving the same effect as the bit-wise masks or write-1 solutions proposed. The core problem is that hardware tends to share settings into registers, as it has been considered too expensive to put information that might cover a range as small as [0,1] into a 32-bit register. Probably, since there is a lack of addresses for registers, you cannot have 1000 settings cause each simple device to use up 1000 words of physical addresses.

But is that really an issue, if we look forward?

It seems to me that, as 64-bit instruction sets and addressing systems penetrate down into more and more embedded systems, a simple solution would be to throw address space at the problem. I don’t think it is uneconomical to allocate huge chunks of memory space to each device, giving each setting its own register, when you have 64 bit virtual addresses to work with. There is no way you can fill up a physical memory system (guess that will some day come back to haunt me)… even the highest-end machines today only use something like 40 bits for actually addressing physical memories.

The software would be simpler and more robust, with virtually no cost.

Another solution that I have also seen starting to appear is to dispense with register settings altogether, and rather define a command API that the processor “calls” by putting in command packets into some memory area. This does require quite a bit of silicon for a decoder, but it provides for a much higher level of interaction with devices. As hardware devices get defined in successively higher-level languages (C, C++, UML, MatLab, …), and their programming interfaces and associated drivers get autogenerated, this solution makes eminent sense.

When this data was brought to the attention of the gcc developers on the gcc mailinglist, a very interesting and elucidating discussion followed. See http://gcc.gnu.org/ml/gcc/2009-12/msg00176.html and follow all the subthreads from there.

There is a key issue in the gcc discussions, about whether code that relies on uninitialized variables and similar make any sense. I personally think that any code that triggers warnings on one or more compilers should be disqualified, as only really portable code should be reasonable to compare in a shoot-out like this. Code that is “undefined” by the C standard really does not provide any interesting information.

Personally, I would also like to see some comparisons between compilers aiming for size, for targets where size really matters. For example, ARM Cortex-R or Cortex-M CPUs, with compilers from commercial players like ARM, IAR, GreenHills, etc. I understand that such a shoot-out is very hard to arrange in practice, however .

Another interesting measure would be the evenness or robustness of a compiler. Is a compiler consistently generating decent code, or does it swing wildly from very good to very bad? I guess you could approximate this if you compared the size for a function generated by a particular compiler to the average of all other compilers’ code for that function. Could give an interesting graph for each compiler.

The code sources are quite interesting: it is isolated functions collected from a large spread of open-source projects. Quite often it looks like the extremes of differences from one compiler to another corresponds to extreme functions. I liked this one, where clang created a file 20x the size of llvm. I can see how a naive switch generator gets big here, while an optimized switch code generator can make some very simple comparison statements out of it (this is a bit from Qemu, from a PCI device model, which is close to my interest of virtual platforms, we get this kind of code a lot).

sh_pci_reg_write (void *p, target_phys_addr_t addr, uint32_t val)
{
uint32_t *__tmp__282;

switch ((int) addr)
{
case 0:
case 1:
case 2:
case 3:
case 4:
case 5:
case 6:
case 7:
case 8:
case 9:
case 10:
case 11:
case 12:
case 13:
case 14:
case 15:
case 16:
case 17:
case 18:
case 19:
case 20:
case 21:
case 22:
case 23:
case 24:
case 25:
case 26:
case 27:
case 28:
case 29:
case 30:
case 31:
case 32:
case 33:
case 34:
case 35:
case 36:
case 37:
case 38:
case 39:
case 40:
case 41:
case 42:
case 43:
case 44:
case 45:
case 46:
case 47:
case 48:
case 49:
case 50:
case 51:
case 52:
case 53:
case 54:
case 55:
case 56:
case 57:
case 58:
case 59:
case 60:
case 61:
case 62:
case 63:
case 64:
case 65:
case 66:
case 67:
case 68:
case 69:
case 70:
case 71:
case 72:
case 73:
case 74:
case 75:
case 76:
case 77:
case 78:
case 79:
case 80:
case 81:
case 82:
case 83:
case 84:
case 85:
case 86:
case 87:
case 88:
case 89:
case 90:
case 91:
case 92:
case 93:
case 94:
case 95:
case 96:
case 97:
case 98:
case 99:
case 100:
case 101:
case 102:
case 103:
case 104:
case 105:
case 106:
case 107:
case 108:
case 109:
case 110:
case 111:
case 112:
case 113:
case 114:
case 115:
case 116:
case 117:
case 118:
case 119:
case 120:
case 121:
case 122:
case 123:
case 124:
case 125:
case 126:
case 127:
case 128:
case 129:
case 130:
case 131:
case 132:
case 133:
case 134:
case 135:
case 136:
case 137:
case 138:
case 139:
case 140:
case 141:
case 142:
case 143:
case 144:
case 145:
case 146:
case 147:
case 148:
case 149:
case 150:
case 151:
case 152:
case 153:
case 154:
case 155:
case 156:
case 157:
case 158:
case 159:
case 160:
case 161:
case 162:
case 163:
case 164:
case 165:
case 166:
case 167:
case 168:
case 169:
case 170:
case 171:
case 172:
case 173:
case 174:
case 175:
case 176:
case 177:
case 178:
case 179:
case 180:
case 181:
case 182:
case 183:
case 184:
case 185:
case 186:
case 187:
case 188:
case 189:
case 190:
case 191:
case 192:
case 193:
case 194:
case 195:
case 196:
case 197:
case 198:
case 199:
case 200:
case 201:
case 202:
case 203:
case 204:
case 205:
case 206:
case 207:
case 208:
case 209:
case 210:
case 211:
case 212:
case 213:
case 214:
case 215:
case 216:
case 217:
case 218:
case 219:
case 220:
case 221:
case 222:
case 223:
case 224:
case 225:
case 226:
case 227:
case 228:
case 229:
case 230:
case 231:
case 232:
case 233:
case 234:
case 235:
case 236:
case 237:
case 238:
case 239:
case 240:
case 241:
case 242:
case 243:
case 244:
case 245:
case 246:
case 247:
case 248:
case 249:
case 250:
case 251:
case 252:;
__tmp__282 = (uint32_t *) ((((SHPCIC *) p)->dev)->config + addr);
*__tmp__282 = val;
break;
case 448:;
((SHPCIC *) p)->par = val;
break;
case 452:;
((SHPCIC *) p)->mbr = val;
break;
case 456:;
((SHPCIC *) p)->iobr = val;
break;
case 544:;
pci_data_write (((SHPCIC *) p)->bus, ((SHPCIC *) p)->par, val, 4);
break;
}
return;
}

This was the first multicore event that I have been to where we did not have a keynote speaker or technical paper from a hardware company. So there was really nothing here directly about how to build multicore chips. Rather, the workshop tended to be about how to program, use, measure performance on, verify software for, and generally work with multicore chips. From the perspective of software people, rather than hardware designers.

Obviously, hardware aspects enter into such talks, but it is the perspective of a user, not a designer. For example, a hardware designer could explain how an atomic compare-and-swap is optimized in a multicore device. But here, we saw measurements on the actual operation latencies observed on real machines using such operations. Quite refreshing, and closer to my personal interests.

The keynote by Andras Vajda of Ericsson was quite interesting. The slides are not online, but the main points that I picked up and that I might not have considered before:

• Software development costs can mean that the cheapest, fastest, most efficient hardware is not necessarily the most economic. Too hard to code for means the software development time and effort removes the advantage. Obvious, but worth reiterating. Software is king.
• The workload on a cellular basestation can sometimes be highly linear and single-threaded. For example, serving a single terminal with a very high bandwidth LTE connection. And suddenly shift to a massively parallel workload as a crowd of a thousand all suddenly appear and start doing data downloads. And then go back to serial again. This means that the age-old argument that signal processing naturally “conveniently concurrent” (and here) is not always true. Nice point!
• Thus, we need adaptable architectures that can trade serial and parallel performance over time, and rebalance quite quickly. In the same chip.
• He is a firm believer that homogeneous systems will win out in the end, I still hold on to a belief in accelerators and offload engines and DSPs. This is partially because of an admitted focus on servers and services processors, and not on the baseband and signalling side. Makes sense.
• Domain-specific languages (DSL) are the future of efficient programming. Agree.

On the topic of DSLs, there was a question about the cost to support them. To me, that is a non-issue. In the organizations that I have worked, it seems that maintaining a useful DSL requires at most one engineer. Developing one, a few good computer scientists for a fairly limited time. In any case, they tend to appear organically when good programmers generalize repeated tasks.

I gave a keynote about how multicore has impacted virtual platforms (in particular, Virtutech Simics) with the following main points:

• Multicore targets increase the performance pressure on a virtual platform, as more processors will have to be simulated.
• Multicore hosts means that sequential performance of the host is going down compared to the aggregate parallel performance demands from the targets.
• To handle large target systems, the virtual platform itself has to run multithreaded on a multicore host. Getting this in place is a major, interesting, and sometimes painful process.
• Once you have a parallel virtual platform, multicore hosts provide a very nice boost in scalability and the manageable system sizes. A single multithreaded virtual platform process is also a bit easier to manage from a user perspective.
• All features in the virtual platform have to be multicore and multimachine-aware… meaning that they often get a bit harder to use initially, as there is no “default processor” you can fall back to for debugging setups etc. Everything has to be explicitly targeted.
• Multicore targets have proven to  be a great sales driver for virtual platforms, as debugging software on a physical multicore, multichip, multiboard system is just too painful.

Overall, this was a fun event, looking forward to next year at Chalmers!

The idea is to not only use model-based development as a way to create code to run, but also to have models which are live at run-time, reflecting on the software which is being run. The core idea is very much like reflection as found in languages like C# and Python.With the key difference that there is a mandatory user-interface aspect, allowing end users to really look at the software (if they have the tools, one would assume).

If you take this one step further, you are entering even more interesting land. If you let a programmer modify the model of their entire software system during run time, we are really in the same area as classic LISP machines and Forth systems. Essentially, you could imagine a “UML machine” where you redraw diagrams and change the machine interactively, kind of like an interpreted model-driven programming system. Replace an interactive prompt with a model editor, and you get something really interesting.

So it is a feature worth reading about, unfortunately it either costs money or requires you to be an IEEE Computer Society member.

I guess it is Murphy’s law — if you really set out to want a bug to show up in your code, your code will stubbornly be perfect and refuse to break. If you set out to build a perfect piece of software, it will never work…

So I was actually quite happy a few weeks ago when I started to get random freezes in a test program I wrote to show multicore scaling. It was the perfect bug! It broke some demos that I wanted to have working, but fixing the code to make the other demos work was a very instructive lesson in multicore debug that would make for a nice demo in its own right. In the end, it managed to nicely illustrate some common wisdom about multicore software. It was not a trivial problem, fortunately.

First, some notes about the program. It is a producer-consumer system using pthreads, with a single producer thread feeding a variable number of compute threads with data, over a shared queue structure (a simple one that uses a single lock to protect it, making it not very scalable for small data messages and lots of workers).

The queue contains a circular buffer, managed using a standard set of full/empty/tail/head kinds of variables. There is also a flag “done” which is set once we are out of data, to tell the compute threads to shut down and terminate the program. As this program is used to demonstrate and test scaling, it is actually something that terminates. The main program spawns off all the threads, and then waits for all threads to finish before it terminates itself.

This program and the queue subsystem had worked perfectly for a long time for me, running on an MPC8641 machine with a Linux 2.6.23 kernel, with 1 to 8 cores and 1 to 16 threads. Regardless of settings like thread counts, data sizes, number of packets to compute, it always ran smoothly and terminated.

However, the other week, I moved the program, the exact same binary even, over to a new software stack built on a Linux 2.6.27 kernel. Still on the same MPC8641 machine. Suddenly, I started to see occasional freezes where the program would never terminate. I added some more diagnostic printouts to the program, and saw that the main program would simply freeze waiting for the other threads to terminate and report in. The freezes had no real relationship to input variables. Maybe they were a bit more common with short packets, but no real pattern emerged. They also happened randomly, running the program with the same parameters for a few times in a row would sometimes result in a freeze. Using control-C to quit it and restart would keep the new instance of program running well. Doing some other demo work, I found the same effect on a P4080 machine with 8 cores and a 2.6.30 Linux kernel.

This is a common pattern for parallelism bugs: they only manifest themselves as actual visible crashes or freezes or bad computation results once something in the software stack has changed, even though the fundamental issues have been there all the time. In this case, I think it was the Linux scheduler, but it is really hard to tell. Just because a program runs fine today it does not have to run fine tomorrow.

After deciding to finally sit down and turn this lemon into lemonade, I had to reproduce the error. Thankfully, that is easy when you have a simulator. The first few times I had to run the target program 20 times or so before hitting the issue, but with some parameter and timing variations I managed to create a script that would open a checkpoint, and run the program a few times under script control, triggering the bug on the fourth run (every time, thanks to determinism).

To diagnose the problem I wrote some Simics script code that I actually felt was fairly cool. I guessed that the problem had something to do with the queue and its handling of “done”, since that is what told the threads to terminate.

The first problem was that the queue was not a global variable. Instead, it was dynamically allocated on the heap by a function, and a pointer passed around, but never stored in a global variable (a good computer science graduate never uses a global variable other than as the means of last resort). Finally, my script set a breakpoint on the line in the setup function that came after the allocation. With the program stopped at that point, I could read the local variable pointing to the queue, and find and store the addresses of all the interesting members of the structure.

The code looked like this (Simics CLI), for the record:

 $mbp = ($ctx.break ($st.pos (rule30_threaded.c:222)))
 $cpu = (wait-for-breakpoint $mbp)
 $pq_addr  = ($cpu.sym "pq")
 $pq_tail  = ($cpu.sym "&(pq->tail)")
 $pq_empty = ($cpu.sym "&(pq->empty)")
 $pq_full  = ($cpu.sym "&(pq->full)")
 $pq_head  = ($cpu.sym "&(pq->head)")
 $pq_done  = ($cpu.sym "&(pq->done)")

Next, I set breakpoints on all writes to empty, full, and done. This was the most expedient route to catch actual puts and gets to the queue. Breakpoints on the queue_put() and queue_get() functions are not really showing the true flow, as these functions start by contending for the lock. Looking at writes to the actual queue members gave me the point where the tasks had grabbed the lock.

The script that caught all writes to done, full, and empty, and on each write, it dumped the state of the queue including computing out the number of elements in the circular buffer (without having to run any code on the target). To get an idea for who was active, it also used OS awareness to find the currently executing thread ID, and scripted debugging to convert the current program counter into a position in the program source code (actually, the important issue was the name of the function we were executing in).

This trace of activity showed quite an interesting pair of patterns. When the program ran well, the queue was mostly full, and it looked like the producer task always got some kind of priority to fill it before consumers could get in and drain it. When the program froze, the queue was seldom more than a few elements deep. This was the same program, on the same kernel, just run a few milliseconds later.

Clearly, the Linux kernel can exhibit quite variable behavior even for a program this simple. I guess that’s why this is called “soft real time”… Another parallelism lesson here: the scheduler is very important, and a smart adaptive scheduler can wreak havoc with software that was accidentally tuned for a different scheduler.

In the end, the crucial hint was that whenever the program froze, the “done” flag was set with a queue that was empty or contained just a few elements. I was sure that I had handled this case in my code, checking specifically for that and making sure to wake up the other threads with a signal that “the queue is not empty any more, please come check for more work”… but looking closely at the code, it turned out the code only woke up a single thread. Thus, the froze resulted from the producer setting “done” with an empty queue, waking up a single compute thread, and then having the other threads wait forever for more data to be put into the queue. The fix was easy: use a broadcast signal rather than a single signal.

In retrospect, it seems really strange that this ever worked reliably… it almost that I suspect the old Linux kernel of having a flawed pthreads implementation where signals always wake up all waiting threads, and not just a single one like the documentation says. But that will wait for another day to be investigated.

Here is the code, for reference:

void rule30_packet_queue_signal_done(rule30_packet_queue_t *q) {
 //
 // Grab lock, set the done signal atomically
 //
 pthread_mutex_lock (&(q->mutex));
 q->done = 1;
 pthread_mutex_unlock (&(q->mutex));
 // Signal any threads waiting for data to wake up
 // and discover that we are indeed done
 //
 // This is the bug:
 // - It only wakes up one thread...
 pthread_cond_signal (&(q->notEmpty));
 // To be correct:
 // pthread_cond_broadcast (&(q->notEmpty));
}

Updated analysis:

My initial analysis was that when things worked, the “done” flag was set with enough data left in the queue that all threads had a chance to pull in data and come in and see the done flag being set.

However, today I went back and wrote a deeper analysis script that also checked for reads from the done flag (turning this check on only after the write to ‘done’ to reduce the noise). I expected there to be a single reader when the freeze happened… but that was not the case. In my current test case, three out of five threads actually got in to read the done flag and terminate.  The crucial code for the compute threads looks like this:

 // Grab mutex,
 //   Check if the queue is empty, if so wait for someone
 //   to push something onto the queue, or signal done.
 //   both of which are done by setting the not_empty conditional variable
 pthread_mutex_lock (&(queue->mutex));
 while ((queue->empty) && !(queue->done)) {
   pthread_cond_wait (&(queue->notEmpty), &(queue->mutex));
 }

To freeze, a thread actually has to be doing the conditional wait here. There are plenty of other places threads can be as the program is finishing. For example, they can be waiting to grab the initial mutex lock, or actually doing compute work. That explains why some threads actually still terminate even with the buggy version. It certainly also illustrates just how chaotic concurrent programs can be. More so that you can ever imagine, really.

Quotes from the blog post:

Sequential languages hiding parallelism:

Sequential imperative languages are notoriously good at hiding or at least obfuscating inherent parallelism in the applications – which may be good news for compiler providers who spend – and charge – big bucks for building reverse engineering technologies that can detect parallelism from sequential code; as well as for highly skilled programmers capable of building parallel code using sequential tools; however, it weights heavily on the total cost of developing software, not to mention the recurring cost of porting to newer HW with e.g. more cores.

Applying DSLs to signal processing:

During the work on the domain-specific language for DSP programming we have seen some interesting results; for example, several pages of optimized C code were re-written to one PowerPoint slide worth of DSL code, while the DSL to C compiler was able to output efficient code comparable in size to the original code.

I agree fully with the final note:

The big question is not anymore if DSLs will take off – it’s more if your pain level is higher than the cost of accepting an alternative and different approach; as the cost for taking in a new language tends to stay constant while the complexity of developing software is increasing, odds are that there will be an uptake of domain-specific languages.

Go and read the full article!

There were several topic areas, such as automotive, consumer, and networking. Networking was mostly focused on the issues of multicore hardware and software.

Of particular interest to me was to see a Freescale QorIQ P4080 8-core networking/control-plane processor live for the first time. This chip was announced in the Summer of 2008, with a full ecosystem of software support thanks to Virtutech Simics. Now, when the silicon is here, software is indeed running on it thanks to the long headstart development got with the virtual platform. Note that several demos at the event used the Simics simulator to show the software support for the P4080, as there was only a single chip to go around.

I would have loved to have a meaningful picture of the first P4080 in Europe, but a chip is not really very photogenic – the P4080 processor was in an open computer case, but covered with a 10 cm-high heat sink which made it fairly hard to actually see. That’s the challenge with infrastructure things: they are not designed to be seen… just to do their job well. If you have a new consumer electronics processor, you can at least drive a screen quickly or something. But watching 28 Gbps of Ethernet traffic is not as easy

Jonas Svennebring of Freescale gave a good talk about how the process of bringup on the P4080 had worked out. It was a total validation of the methodology of using virtual platforms, at different levels of abstraction, and slipping in a bit of hardware emulation as well.

Freescale started software development on the functional fast model, and when clock-cycle-level detailed models of subsystems became available, they started using them as well for performance validation for small pieces of code. Any discrepancies in behavior between the two models was then used to correct the models and documentation. Finally, as the RTL for the silicon began to become available, they used a few emulation setups to run parts of the actual RTL (the emulator could only handle a subset of the entire chip), and validate the performance numbers in the detailed model and the behavior of both models. In the end, when the first silicon became available, Linux was up in a very short time (I cannot give the exact number, but it was a matter of days rather than weeks).

This is the typical iterative process that all chip designers are implementing today: using virtual platforms you can get a head start on development of software, and then as more details become available, you tune models and update both designs, models, and software, iterating towards a hardware/software combination that just works once the silicon realization of the hardware comes around.

So that was all cool.

Jonas also showed a die photo of the QorIQ, and that confirmed by opinion from the SiCS Multicore Day: embedded multicore is not just about processor cores and cache, it is very much about accelerators to help offload repetitive work from the processing cores. More than half the chip was such acceleration logic! To me, this is a clear confirmation that heterogeneity is the future of hardware design, and a useful way to spend hundreds of millions of transistors to boost SoC performance.

The same was true for most other Freescale hardware showcased at the event. For example, there was the MPC5606S dashboard processor, running an LCD display with lots of dynamic graphics with 0.2% CPU load on a 60 MHz e200 Power Architecture processor. All the work was done by its display driver and accelerator. It is hard to argue with that kind of efficiency. That chip did not need a heatsink, either. It was just mounted on the back of an example board with no need for any external logic chips. Apparently, it could also have moved some physical gauges and blinked LEDs, but that demo was considered too distracting for this particular setting.

I also gave a talk at the DWF, about debugging software on multicore using virtual platforms. That was fun, as always. Need to get out more on the road and talk in conferences, I think

However, in fact, most the embedded industry and the “virtual platform” industry rely on C and C++ to get our daily jobs done. Question is, how much longer can we expect to do that? An interesting post at Embedded.com by Michael Barr brought back my argument that modeling needs to move up in levels of abstraction just like mainstream programming.

Michael Barr wrote the column “Real Programmers Program in C“, where he points out that knowledge of C is declining among computer science graduates. It is simply not efficient enough for simple mainstream work like creating web services and custom IT applications.

Clever though he is, the young man admitted he wasn’t making that quote up on the spot. That “real men program in C” is part of a lingo he and his fellow computer science students developed while categorizing the usefulness of the various programming languages available to them. Exploring a bit, I learned the quiche-like phrase assigns both a high difficulty factor to the C language and a certain age group to C programmers. Put simply, C was too hard for programmers of their generation to bother mastering.

Obviously, if you take this argument to the extreme, you end up with the Monthy Python sketch where a bunch of old men are trying to trumph each other with the tough childhoods they had. In the end, they claim to have eaten just a handful of cold gravel for breakfast, walked 50 km to school, and having to clean the road each day… and kids these days, they just don’t understand…

But apart from the fact that kids in the western world today are very lazy and can’t stomach running 15km to school each day and therefore lack the toughness to match Kenyans in marathons there is a real issue here.

The bottom line is that embedded programmers aren’t going to stop using C anytime soon. There are several reasons for this. First, C compilers are available for the vast majority of 8-, 16-, and 32-bit CPUs. Second, C offers just the right mix of low-level and high-level language features for programming at the processor and driver level. Until the use of C starts to turn down in future such surveys, C programming skills will remain important.

The issue is that universities are moving up in the efficiency scale of languages, teaching students good things rather than hard things. Not all universities do (and I am trying my best to lobby for keeping assembly language and device driver programming in the core computer science curriculum whenever I can), but it is clear that the market for “general IT stuff” is so much bigger that it will attract more students to “easy” languages like Ruby and VisualBasic.

So we need to move both embedded programming and virtual platform technology much more in this direction to maintain  a steady influx of smart people into the field. High-level synthesis of hardware and virtual platform models from a VisualBasic form? Sounds like a stretch…

We also need to jump into the education system and create the courses and motivate professors to teach lower-level languages. Not all are that familiar with actual practices in industry, unfortunately.