It turned out that trying to show a group of ten impatient kids that the robot read instruction codes and moved accordingly failed. Using a chaser program, where the robot drove around waiting for something to show up close in front of it and then drove full speed forward to try to hit it, was much easier to appreciate.  Classics like driving around avoiding to hit walls was also just as much fun as ever. Like my previous attempts, the kids piled up on the robot and tried to make it turn the way they wanted.

Clearly, physical exercise is easier to sell than cerebral exercise.

That said, I did manage to get some more thoughtful kids interested in getting the robot to go particular places. The setup I used was to put the robot on a starting square marked on the floor, and then putting a red bucket out on the floor some distance away. The kids were asked to help me program the robot to either hit the bucket or drive around it. I explained what the robot did as taking steps forward and turning, and actually the approximately 50 cm long runs it did for each blue programming tile was fairly similar to a serious step by a five-year-old (some kids instead started tip-toeing forward, which did not quite provide the correct stride).

As a safety device and to retain a semblance on intelligence on the part of the robot, I equipped it with a bumper in front. As soon as the bumper is hit, the robot is supposed to say “stop” and stop all activities.

The problems we tried were:

• Put the bucket down in front and try to hit it, needing three forward to get there
• Drive around said bucket, which required a program like blue, green, blue, yellow, blue, blue, yellow, blue, green (forward, right, forward, left, forward, forward, left, forward, right).
• Leave the base and drive to the back instead of front.

I sat down with a group of three kids who really got into this part (even with ten other kids running around playing cops and robbers at a very high level of intensity). They did get the idea and did propose steps towards the solution. However, it is clear that at least initially; one cannot expect them to plan out an entire program in their heads.

Rather, we ended up in a style quite familiar to anyone learning a new programming language or taking their first programming course. Try one step, see where it takes us, add another step or two to the program, and try again. Very much how I myself solve programming and scripting problems where I interact with a complex and possibly unpredictable environment. In this way, we did manage to solve some problems, and I do think the kids understood what was going on.

To me, I feel the idea of the Lego-programmable robot as a preschool-level programming course is at least partially proven. As in most other educational ventures, what is needed is a good overall teaching environment. Next time, I will have to try this with a small group of kids, just after they have had a snack so that they are alert and fresh. I will also make sure to break up the hard work with some simple play, since that is needed to keep the kids from being bored.

Back in the late 1980s, in what seems to have been a golden era of software innovation, there were a number of companies trying to sell graphical programming environments for the graphics-oriented Macintosh computer. Makes sense, right? I remember that I did not think so at the time, and the reason was that things tended to look more complex in a visual system than when expressed as plain text. I think one of the examples I saw was Prograph, where unwieldy flow charts replaced nice and dense if-then statements.

Today, twenty years later, it is hard to dispute the success of model-driven and graphical environments like MatLab, LabView, and others. Dataflow is very good paradigm for many things in signal processing, and it fits very nicely with graphical representations. UML statecharts do express state machines in a pretty readable manner which is far superior to any particular code encoding of the same information.

However, I think the key here is that the languages are domain-specific to a large extent. General-purpose programming is still not really feasible in a graphical way, and I doubt it will ever be. You also need to “go with the flow” in these systems to get full benefit from them. If you start to think in imperative coding terms, you tend to quickly get lost… Here is a personal example that a friend (who will remain anonymous to protect the guilty) and I cooked up in NXT, trying to code an algorithm to guide a robot along a line. The picture both contains the graphical representation and the approximately equivalent imperative-style code.

All the little red boxes are value reads, writes, and manipulations. It is kind of elegant that you can pass values on from one block to the next, but overall the graphical representation must be said to be totally unreadable. The logic was pretty good, actually, but it did not do things like you are supposed to do in Mindstorms. Rather than a reactive “act wait act” cycle, we tried to code complex loops.

I guess this is an example of the old saying that a good (or determined) programmer can write FORTRAN in any programming language. As an aside, I recall a Russian-schooled professor I had as an undergraduate who managed to abuse Prolog to create a deterministic loop from its backtracking system. Ugly indeed, but also quite an impressive feat of semantic wrestling.

The Mindstorm-style solution to following a line was found by a bit of Googling, and for the robot that I built a fairly (not entirely, though) robust program looked like this. Credit to http://www.nxtprograms.com/line_follower/steps.html for the original program that I had to modify just a bit to work with Mindstorms NXT 2.0.

In a sense, this program is bad since it obviously relies on non-functional properties of the NXT environment which are not semantically specified (i.e., the speed of sensing and how to drive the motors the fastest). However, the sense-act loop with no state saving is the right way.

Obviously, other graphical environments will have other strengths and weaknesses, but I think that this is a nice illustration that the typical constructs of general-purpose programming languages are poor fits for graphical visualization. Variables are simply not very visual in their nature…

As discussed in my previous blog post about Kindergarten robots, I wanted to see if I can teach kids the core idea of programming. This project has now progressed to the point that I have a working prototype of a programmable robot.

Essentially, the robot is programmed by putting colored Lego bricks in a sequence on top of the robot. This should be accessible and direct enough to work with kids — and with no computer needed, just direct physical interaction with the system. For some reason, I think the extra level of abstraction from a screen to a robot is just an unnecessary obstacle at this level.

For an example run, see the Youtube video linked below:

The robot is shown in the picture below. Since I use the same base as a fast chaser robot, it uses the large wheels from a Technic kit I happened to have around, and a decent swiveling rear wheel. Using treads for the drive would probably have given me a bit more precision, but this is faster and simply feels more fun.

The inspiration for the programming method came from the boardgame Roborally, where you are dealt a set of card with meanings like “forward”, “back”, “turn right”, “turn around”, etc., and have to make your robot navigate a hazardous landscape to get to a destination. In my kindergarten robot, I simplified the programming to just forward, left, and right. And there are no conveyor belts, lasers, pits, or other hazards on the field.

The program steps available are the following (the color sensor can only distinguish a few primary colors):

• Black – start bit
• Blue – move forward
• Green – turn right
• Yellow – turn left
• Red – honk the horn (beep)
• White – stop bit

As a programming language, this is definitely not Turing-complete and nothing that you could program anything useful with. In particular, the programming model lacks decision statements, loops, and variables. However, it conveys what I feel is at the heart of programming: obtaining a goal by putting together a sequence of very small and simple steps. If you cannot grasp the idea of many small steps, you will never be able to program in any meaningful way.

User Testing

I have not tested this with a group of kids yet, but only with myself and my five-year-old son. So far, it seems to work. After some initial confusion, he is starting to get the idea of putting steps in order and having the robot execute them.

Here are some very simple missions that can be used to get the programming thinking going:

• Turn around a corner (of the fridge in this case)
• Move to hit a balloon lying on the floor some distance away
• Move in a square pattern to get back where it started

I am sure I will think of more as I go along. I will blog some more once I have more experience with the robot.

Details of the Program Memory

The major problem that had to be solved to build this robot was creating a working program memory. I have decided to call the program memory MEPROM, for “Mechanically Erasable Programmable Read-Only Memory”. It is a ROM, since the robot itself cannot change it.  Since you can both remove program steps and put in new steps using mechanical means (human hands, typically), it is definitely an M-EPROM.

The idea for the memory is simple: use Lego bricks to represent moves. It took a while to come up with the current form, as there are many ways to move colored bricks past a sensor.

Two of the program steps can be seen below: I used 3×1 standard lego bricks (which I borrowed from my son’s Legos, since  it seems very hard to find good kits containing this particular type of brick):

To move these programming blocks past the color sensor in order to read them, I used some classic Lego treads (which seems to have been in use since the late 1970s!). Today, these can be best obtained by buying the Mini Bulldozer kit (8259). The program steps are put on alternate tread elements:

The Mindstorms color sensor only sees six colors, as discussed above. Worse, it tends to see black in dark shadows. For this reason, it needs to read the bricks head-on to be reliable. I have tried in vain to make the system self-adjusting, so currently, the program start has to be manually aligning the start bit (black brick) in front of  the sensor. For the rest of the run, turning the driving motor 90 degrees reliably puts the next brick in place in front of the sensor.

The program turns out to be quite simple in the NXT programming environment. Turn program motor 90 degrees, sample the color sensor, and act accordingly using a “switch statement”:

On one hand, we have the task of supporting the design of new hardware bits, for the purpose of creating it. On the other hand, we have the task of describing a particular design for the purpose of simulating it. These two are not necessarily the same.

To use an analogy with building a house, a design language helps the architect create the house (piece of hardware). Since the architect relies on craftsmen and experts (compilers) to do detailed design (how to put in windows, where to put light switches, etc.), the high-level description does not contain all the details of the house. However, if you are trying to simulate the house (piece of hardware) so that its inhabitants (software) don’t see the difference to the real thing, the details are sometimes what matters most. For example, the precise way to operate the stove in the house is very important for familiarity, but is a detail most likely left out of the architect’s initial drawings.

A design language can leave many things unspecified to be filled in by a compiler, but these things can be absolutely core to a description language. In particular, programming register maps tend to be created as a not-too-important side activity in hardware design. They do not really need to be visible in higher-level ESL languages, as they can obviously be filled in later by a tool or a human. But for a description language, they are absolutely core.

A description language can also leave out many parts of the hardware. If the software being used or written does not use certain modes or functions of a piece of hardware, those pieces can be ignored and implemented as dummies. That means that support for dummies is very important in description languages. But dummies make little sense in a design language, as you are unlikely to design a chip with lots of area spent on dummy functions that do nothing.

A description language can also ignore crucial aspects like power constraints and synthesis constraints. These are guidelines for a compilation step that has no bearing on the description of the hardware — the description language should describe what ended up happening, not the if, please, what, and buts that guided how we got there.

For virtual platform creation, you seem to need a bit of both. I maintain that most of a VP is based on old hardware that exists, which calls for languages with strong description abilities. That’s the space that Simics DML was designed for. For the small part of the hardware that is novel would be nice to have some way to convert from a design language to a virtual platform. Here, I don’t really see any usable current tools or languages — SystemC is really more a design language, but if you want a virtual platform model, you have to use it as a description language. There is no automagic getting to a fast abstract model from a design-oriented description. That’s why we need new, higher level systems, that can push out decent descriptions from a design.

When I was a PhD student in Uppsala back around 2000, we bought a pile of the Lego Mindstorms RCX kits, for use in real-time courses. Obviously, the students loved the opportunity to play with Lego (including the few females). What was less obvious and much more interesting was what happened when we brought in a bunch of children from a local kindergarten to visit — they really took a liking to our little yellow robots running around a classroom. They treated the robots as little animals, wondering what they were doing and why…

With that in mind, I decided to try to reprise this myself with my own son and his kindergarten friends. Last week, I took my robot kit with me and went to meet the kids.

My top-level goal with exercise was really to get the kids interested in technology as a future field of study and venture… you have to start early to counteract the prevalent tendency of people to want to be famous and work in media or something… Only time will tell if this had any effect at all (I doubt it).

In addition to the top-level goal, I wanted to communicate some about how things work. Make technology more understandable and less magical, and more accessible. In particular:

• Autonomy: the robot is not under remote control, it acts autonomously based on its programming.
• Sense-compute-actuate: the robot perceives the world and makes decisions that get sent to the motors.
• Limited sensors: the robot does not see like we do, it uses far simpler sensing.
• Programmability: the same physical setup can do a lot of different things just by switching to a different program.
• Concurrency: the robot can do several unrelated things at once.
• Stupidity: the robot is really dumb and just reacts very predictably to its environment.

I think these points can be brought across using an indirect approach. You cannot tell a four-year-old about programmability. But you can show that if you go to the robot’s control unit and press some buttons it does something different.

Note that some of the points above are artificial, in that you could build much smarter programs with more complex behaviors and less direct reactions. But that would make things more magical and “human-like”, which is not what I wanted to communicate.

Anyway, in the end I used two configurations of a driving base, with a few different programs.

The first configuration is shown below, using a color sensor pointing downwards and an ultrasound sensor pointing forward.It also had a shooter pointing forward (in this picture, the organic styling of the Lego Bionicle Zamor-derived shooter comes across very nicely, as well as the apparently quite special silver- and gold-colored balls I picked up on a sale).

First, I had this robot follow a black line on the floor. Putting the line down was fun, watching the robot follow the line did not work out as intended. Too hard to explain the algorithm for following the contrast between black and not-black. Adding the behavior to stop when the robot hit a red line was also too subtle.

Next, I tried to put the robot on a table with the program “stay within black lines”. Or rather “reverse and turn if you hit a black line”. Problem was that the table had a spotted pattern that included small specks of black, and the robot just kept doing avoidance maneuvers… so that failed. I still want to do a table-dancing robot, but I guess relying on an ultrasound sensor pointing downwards to detect the chasm at the edge will work better.

What was fun though was when I activated the ultrasound sensor and the shooter: the robot would crawl along the line, and if something got in the way, it would fire a ball. That was huge fun! All the kids crowded around to get the robot to shoot and collect the balls. Note for future attempts: dramatic actions are great!

It was even better when I set the robot to rove freely, using the ultrasound sensor to turn when something got in its way. This was an easy-to-understand behavior that the kids appreciated, putting feet and hands in the way of the robot to make it turn. The poor robot was often the center of a pile of kids that were all trying to make it turn.

I also tried a configuration using bumper sensors:

Here, the robot only had one program, to move forward until it hit something, and based on the side that hit, back up and turn to avoid hitting the same obstacle again. This was a very successful configuration. The kids started to play “don’t touch the robot” with it, having it drive between their legs or bending over the robot to make a bridge – but not touching it and making it turn.

Note that I did not bring any computer with me, all programs were preloaded on the NXT brick. Very handy, actually.

In summary, I had a great time doing this, and I will be back with new configurations and programs. Creating the programs was very quick and easy in the Mindstorms environment, proving the value of domain-specific programming.

The kids have hopefully learned that you can play with and control technology, and that you should not be too respectful. Maybe someone also picked up some of the ideas I presented at the start of this blog post.

Gender-Theoretical Notes

I do have to end with a slightly sad note.

It was surprising to me just how differently the boys and the girls reacted and behaved. I had assumed that kids would just be kids at this age (three to five years old), with no gender-related differences. It was very striking that this was not the case in reality. The boys just ran in and started playing, and very quite hard to get to listen to anything I had to say. The girls walked in and quietly waited for instructions, and took some warming-up before they would interact with the robot.

I cannot blame our kindergarten for this, they are definitely trying to avoid gender-based stereotypes.I guess it shows that fighting societal norms is just as hard as everyone says it is.

It was also interesting that the kids were surprised to see a parent build with Lego and have a great time. For some reason, that was not expected behavior from a dad. Sad too.

As you can see from the webmail screenshot below, the probable folder contained almost 100 thousand spam email. These were collected since May of 2008, or in a space of roughly 21 months.

If I guesstimate that the other folder has about the same average size, that adds in another 400 thousand spam. Bringing the total caught by these filters to about half a million overall. If you divide it down to days, it is “only” about 630 per day, plus some more that secondary spam filters catch, plus the ones that get through and I manually have to delete. But these other ones won’t add up to much more than a few tens of thousands in the same time spamn.

It is amazing just how voluminous this infestation is…

Legit business emails can’t be much more than fifty per day, plus various general mailing lists. Still, that means that probably no more than 75% of all email I receive is spam. Maybe I should consider myself lucky, I have seen analysis talking about 99% of all email on the Internet being spam…

Interesting it was.

I have seen the “write 1 clears” solution before in real hardware, but I was not aware of the other two variants. The idea of having a “write mask” in one half of a 32-bit word is really clever.

However, this got me thinking about what the fundamental issue here really is.

As I see it, it is the fact that the processor cannot address small enough units atomically. The read-modify-write that was used to start the discussion in the Embedded Bridge #37 was needed in order to get the current state of a configuration register, change some setting that only occupied a few bits in it, and write back the result to the register. The way most configuration registers that I have seen in practice works.

But if each setting could be given its own register, the problem would go away. Each operation would target a unique address, achieving the same effect as the bit-wise masks or write-1 solutions proposed. The core problem is that hardware tends to share settings into registers, as it has been considered too expensive to put information that might cover a range as small as [0,1] into a 32-bit register. Probably, since there is a lack of addresses for registers, you cannot have 1000 settings cause each simple device to use up 1000 words of physical addresses.

But is that really an issue, if we look forward?

It seems to me that, as 64-bit instruction sets and addressing systems penetrate down into more and more embedded systems, a simple solution would be to throw address space at the problem. I don’t think it is uneconomical to allocate huge chunks of memory space to each device, giving each setting its own register, when you have 64 bit virtual addresses to work with. There is no way you can fill up a physical memory system (guess that will some day come back to haunt me)… even the highest-end machines today only use something like 40 bits for actually addressing physical memories.

The software would be simpler and more robust, with virtually no cost.

Another solution that I have also seen starting to appear is to dispense with register settings altogether, and rather define a command API that the processor “calls” by putting in command packets into some memory area. This does require quite a bit of silicon for a decoder, but it provides for a much higher level of interaction with devices. As hardware devices get defined in successively higher-level languages (C, C++, UML, MatLab, …), and their programming interfaces and associated drivers get autogenerated, this solution makes eminent sense.

When this data was brought to the attention of the gcc developers on the gcc mailinglist, a very interesting and elucidating discussion followed. See http://gcc.gnu.org/ml/gcc/2009-12/msg00176.html and follow all the subthreads from there.

There is a key issue in the gcc discussions, about whether code that relies on uninitialized variables and similar make any sense. I personally think that any code that triggers warnings on one or more compilers should be disqualified, as only really portable code should be reasonable to compare in a shoot-out like this. Code that is “undefined” by the C standard really does not provide any interesting information.

Personally, I would also like to see some comparisons between compilers aiming for size, for targets where size really matters. For example, ARM Cortex-R or Cortex-M CPUs, with compilers from commercial players like ARM, IAR, GreenHills, etc. I understand that such a shoot-out is very hard to arrange in practice, however .

Another interesting measure would be the evenness or robustness of a compiler. Is a compiler consistently generating decent code, or does it swing wildly from very good to very bad? I guess you could approximate this if you compared the size for a function generated by a particular compiler to the average of all other compilers’ code for that function. Could give an interesting graph for each compiler.

The code sources are quite interesting: it is isolated functions collected from a large spread of open-source projects. Quite often it looks like the extremes of differences from one compiler to another corresponds to extreme functions. I liked this one, where clang created a file 20x the size of llvm. I can see how a naive switch generator gets big here, while an optimized switch code generator can make some very simple comparison statements out of it (this is a bit from Qemu, from a PCI device model, which is close to my interest of virtual platforms, we get this kind of code a lot).

sh_pci_reg_write (void *p, target_phys_addr_t addr, uint32_t val)
{
uint32_t *__tmp__282;

switch ((int) addr)
{
case 0:
case 1:
case 2:
case 3:
case 4:
case 5:
case 6:
case 7:
case 8:
case 9:
case 10:
case 11:
case 12:
case 13:
case 14:
case 15:
case 16:
case 17:
case 18:
case 19:
case 20:
case 21:
case 22:
case 23:
case 24:
case 25:
case 26:
case 27:
case 28:
case 29:
case 30:
case 31:
case 32:
case 33:
case 34:
case 35:
case 36:
case 37:
case 38:
case 39:
case 40:
case 41:
case 42:
case 43:
case 44:
case 45:
case 46:
case 47:
case 48:
case 49:
case 50:
case 51:
case 52:
case 53:
case 54:
case 55:
case 56:
case 57:
case 58:
case 59:
case 60:
case 61:
case 62:
case 63:
case 64:
case 65:
case 66:
case 67:
case 68:
case 69:
case 70:
case 71:
case 72:
case 73:
case 74:
case 75:
case 76:
case 77:
case 78:
case 79:
case 80:
case 81:
case 82:
case 83:
case 84:
case 85:
case 86:
case 87:
case 88:
case 89:
case 90:
case 91:
case 92:
case 93:
case 94:
case 95:
case 96:
case 97:
case 98:
case 99:
case 100:
case 101:
case 102:
case 103:
case 104:
case 105:
case 106:
case 107:
case 108:
case 109:
case 110:
case 111:
case 112:
case 113:
case 114:
case 115:
case 116:
case 117:
case 118:
case 119:
case 120:
case 121:
case 122:
case 123:
case 124:
case 125:
case 126:
case 127:
case 128:
case 129:
case 130:
case 131:
case 132:
case 133:
case 134:
case 135:
case 136:
case 137:
case 138:
case 139:
case 140:
case 141:
case 142:
case 143:
case 144:
case 145:
case 146:
case 147:
case 148:
case 149:
case 150:
case 151:
case 152:
case 153:
case 154:
case 155:
case 156:
case 157:
case 158:
case 159:
case 160:
case 161:
case 162:
case 163:
case 164:
case 165:
case 166:
case 167:
case 168:
case 169:
case 170:
case 171:
case 172:
case 173:
case 174:
case 175:
case 176:
case 177:
case 178:
case 179:
case 180:
case 181:
case 182:
case 183:
case 184:
case 185:
case 186:
case 187:
case 188:
case 189:
case 190:
case 191:
case 192:
case 193:
case 194:
case 195:
case 196:
case 197:
case 198:
case 199:
case 200:
case 201:
case 202:
case 203:
case 204:
case 205:
case 206:
case 207:
case 208:
case 209:
case 210:
case 211:
case 212:
case 213:
case 214:
case 215:
case 216:
case 217:
case 218:
case 219:
case 220:
case 221:
case 222:
case 223:
case 224:
case 225:
case 226:
case 227:
case 228:
case 229:
case 230:
case 231:
case 232:
case 233:
case 234:
case 235:
case 236:
case 237:
case 238:
case 239:
case 240:
case 241:
case 242:
case 243:
case 244:
case 245:
case 246:
case 247:
case 248:
case 249:
case 250:
case 251:
case 252:;
__tmp__282 = (uint32_t *) ((((SHPCIC *) p)->dev)->config + addr);
*__tmp__282 = val;
break;
case 448:;
((SHPCIC *) p)->par = val;
break;
case 452:;
((SHPCIC *) p)->mbr = val;
break;
case 456:;
((SHPCIC *) p)->iobr = val;
break;
case 544:;
pci_data_write (((SHPCIC *) p)->bus, ((SHPCIC *) p)->par, val, 4);
break;
}
return;
}

So what is the significance of this code?

Essentially, there are three main communications protocols or channels involved with GDB. There is the direct command-line interface familiar to most GDB users. There is the gdb-serial protocol for talking between a GDB and a GDB server on some remote machine.There is the GDB-MI, Machine Interface, protocol used to directly control a GDB debugger from another program. In particular, Eclipse uses GDB-MI to control GDB, when using GDB as the debugger back-end for Eclipse.

With GDB-serial, a remote machine can also be a simulator, hardware emulator, or anything else that presents the right interface to GDB. Indeed, using gdb-serial is the most common way to attach “anything” to be debugged by GDB. Simics, VMWare, and many other virtual platform solutions support gdb-serial as a way to get the power of the GDB debugger (source code lookup, scripting, and the smarts to deal with complex languages like C++ and Ada, and much more) for debugging code running on the virtual machine. The requirements on a gdb server are very simple, it really is just start/stop/step, and setting breakpoints and modifying memory.

When GDB 7.0 came out, support for reverse debugging was the really big piece of news. With the initial release of 7.0, you could access reverse debug from the GDB command-line interface, driving either local process record/replay targets, or remote gdb-serial targets. Simics was among the first simulators to support reverse debugging over gdb-remote, since the command set for doing reverse debug was essentially contributed by Virtutech with the initial release of reverse debugging in Simics back in 2005. Michael Snyder got that code into the reversible GDB development from the start, we can claim no credit for getting it into the mainline GDB.

However, the MI interface was missing reverse debugging commands, and I have been involved in getting that support into the GDB source tree. It has taken quite a while longer than I expected, and it has been quite an interesting journey.

When we started the work back in late 2008, we had a set of MI commands that was used in a Virtutech-specific patch to GDB 6.8. This set of commands and their supporting code was submitted to the GDB mailinglist in early 2009, but got rejected by the maintainers. In the end, we changed the commands according to the wishes of the maintainers, and added test cases and documentation (my bit). This patch was then resubmitted, reviewed, and updated in a few more iterations. The final update was to align the code with the latest code updates to GDB.

Along the way, Tomas and I had to sign some legal papers to assign copyright to the code we contributed to the FSF. For some reason, the first batch of papers got lost on their way across the Atlantic to us to get signed, forcing a delay of some months.

So, now, we have GDB with MI supporting reversible debugging. Finally, I hope it arrived in time to help the Eclipse people get reverse debugging into Eclipse as a standard feature.

For me, even though this process sometimes felt like it would never end, it has been very interesting.

• The process is frustrating in that if you disagree with a maintainer on some issues, things can stall until you realize the errors of your way and repent. We had such a discussion on the style of the GDB reverse commands, both in MI and the main command-line interface. The viewpoint that I supported lost, which I think is a shame, but at least the functionality is there.
• Related to that, the discussions on the gdb mailinglists have been quite interesting. If you have relevant points, people will tend to listen. The value of ideas is judged from how well they work, and usually not from set agendas or corporate grand strategies.
• Sometimes, you really miss the function of an overall product manager for GDB. Somewhere or someone that takes responsibility for the overall direction of the product. Today, we have a very productive anarchy, which certainly works, but is a bit unfamiliar to someone used to commercial and planned software development.
• Compared to some other standards work that I have been involved in, GDB is a much faster process. Collaboratively improving a shared body of code is a nice way to work, but it might be very hard to port over to company-driven future-pointing standards. For example, if no code exists, the model hardly works.
• The user-interface design coming out of the GDB community is clearly quite geeky and often not what I think a UI designer would come up with. The GDB reverse commands, with a “set direction” command is one example. Another interesting one starts here, with my input here.

Happy reversible debugging, all!

Turned out that the problem was not at all related to the names of the parameters to the VM. Xmx is the right way to grow the heap.This set of parameters does work when put into an eclipse.ini file (for Eclipse 3.5):

-vmargs
-Dosgi.requiredJavaVersion=1.5
-Xms40m
-Xmx1024m

The problem was that Eclipse actually never read its eclipse.ini file, because I had changed the name of the main Eclipse binary. I have several Eclipse installations with different versions active at the same time, and to bring some sanity to shortcuts and Windows start menu items, I started renaming the executables to include the version name. Thus, I had “eclipse-3.5.1.exe” rather than “eclipse.exe” in the problematic case.

It seems that eclipse when starting reads a file like “my-file-name”.ini, and thus all changes to the file “eclipse.ini” has no effect for “eclipse-3.5.1.exe”.

Why do I always find these bugs?

For my parental leave, I have just bought myself a Lego Mindstorm NXT 2.0 kit. It is not much fun for our youngest, who mostly gets a bit scared by a piece of Lego driving around making noises, but I hope to be able to use it to teach my older child (almost five) to program. Let’s see how that turns out. It looks hard to make the NXT environment provide the kind of Roborally-style programming blocks that I had hoped to create, as I cannot for some reason get a sufficiently custom icon onto custom blocks.

It also presented me with an opportunity to try some domain-specific high-level graphical programming. The programming environment provided for the NXT series of Mindstorms kits is based on LabView from National Instruments, and it really does seem to work. It even features parallel tasks, which I tried to use…

It turned out that it is not that easy to get it right. I was trying to have a few different tasks look at different sensors and steer the robot in different directions based on the sensor readings. However, this quickly turned into a literal deadlock: the robot just sat there, doing nothing, after the first time that my ultrasound sensor task tried to steer it. Failure.

The manual is quite silent on the tasking semantics, and there are no (or I have not yet found) shared variables, locks, or message-passing mechanisms to synchronize the tasks.

Looking for an answer on the web, I came across a nice tutorial on tasking:

• http://www.ortop.org/NXT_Tutorial/tasks.html

It is a video, and it ends with the note that the only reliable way to multitask in the NXT environment is to NOT try to control the same thing from multiple tasks. If I want to listen to multiple sensors, the only way is to create a loop with switching on inputs. I.e., manual polling. So much for using the natural language of tasks to handle naturally concurrent activities. At some point, I might figure out if I can construct it from tasks and data value transfers, but for now, I will just go with the flow and write (or rather draw) some polling loops.

Here is an example from the manual. Note that the top “beam” controls motor A, and the bottom controls motors B and C:

That last part about data wires is intriguing… but I will need to find some more reference materials.

Anyway, the way to express parallel tasks here is really quite neat. At least as long as they are dealing with completely separate aspects of the control of the robot.

If this had been a more hard-core environment, it would have been fun to put different tasks on the ARM7 and AVR processors that are inside the NXT 2.0 brick. Yes, it is a true multiprocessor, if very limited in capabilities. At http://mindstorms.lego.com/en-us/whatisnxt/default.aspx you have the specs. ARM7, 256 kB of FLASH, 64 kB of RAM, and the AVR has 512 bytes of RAM and 4 kB of FLASH. A nice real embedded machine!

For the next few months, I will be on parental leave, so there is likely to be less blogging about technical subjects (and less blogging overall). There is simply less inspiration about virtual platforms, and a bit more about toys.

In the picture below, we have the “Knight’s Castle” kit from 1984 (set #6073). I remember this as being a major kit at the time, one of the dream things to buy, and one of the biggest ones around. Compare it to the new Police Command Center (set #7743) from 2008. The small police station accompanying the large trailer is as high as the old castle, and the trailer as long as the castle is wide. It has about 25% more pieces that the old castle.

Also, note the old small vehicles in the front. Four Lego notches wide, unlike the six-wide police trailer. I remember the highway truck as being one of the magical models of my childhood.

To me, the most interesting aspect of comparing Lego sets is that it is a very good illustration of the economic growth that we have seen over the past 30 years. In relative terms, Lego sets have become far more affordable. Average income has certainly increased faster than the price of Lego, and my kids have quite a few more pieces to play with than I did at the same age. The nice thing with comparing Lego buying power with my childhood is that it is a product that is mostly “the same”, unlike comparisons involving electronics…

The approach taken for their checkpointing system is to save the entire state of  the running simulation program (i.e., an entire host operating system process), and later recreate the process in the same state. This gets around the need program all simulation models to explicitly support checkpointing, but it also limits the applicability of checkpointing severely. Of the checkpointing operations described in my previous post, it only supports the bringing up of a checkpoint into the same model, on the same machine (or a machine that runs a completely identical software stack down to the exact versions of all libraries, drivers, etc., which is not very likely to happen). This is honestly admitted in the paper, I appreciate that.

Process-based checkpointing in this way is also used by Cadence, and just like Cadence’s solution, the problem that appears when implementing it in practice is how to handle all the OS resources opened by a process. The process itself is not enough. The resources are typically files open for input and output, connections to debuggers, and other things that reach out of the virtual world of the simulation into the real world of the host machine. The solution is just like Cadence’s solution: provide callbacks for the SystemC side of such connections that can save the state of the connection in some way, and restart the connection when a checkpoint is opened. Doing this right does require some care in what to do in which order, and the paper nicely explains this.

For example, you need to close down all connections before taking a checkpoint, and then restore them after taking it. This is a fairly destructive operation, compared to the Simics-style checkpointing where all you do is interrogate the state of all objects and save that. I had not thought of that before, but it makes perfect sense.

To me, this also points to an important architectural issue in SystemC simulations in general: what are these simulation modules doing opening files on their own anyway? In my opinion, a simulation model should never do such low-level thing, it should operate using only the defined simulation system API and simulation-level connections to other simulation modules. If you need to load test data, do that by putting it into some storage system managed by the simulator itself. In all my professional life, I have considered file I/O to be a fundamental service provided by the application framework I use, nothing that my actual payload code should have to deal with. For example, in Simics, we tend to solve the loading and checkpointing of test data vectors using a memory “image”, which supports checkpointing in itself. Then a script just loads a file into the memory image, and a device reads the memory and moves transactions into the simulated system. This means that file I/O is totally absent from the model, and also that the test data can be managed and inspected by existing infrastructure.

The cost of taking a checkpoint is quantified in the paper, and it is not as bad as one could fear. The size of  a checkpoint easily reaches 100s of MB for even small target systems, but saving and loading that today does not take all that long. Still, going to a system with only 16 target processors (8 ARM, 8 DSP, and 8 times 64 MB of target memory = 512 MB) generates a checkpoint taking 1.7 GB to store and 30 seconds to save.

Once again, it is interesting to compare this to the Simics-style approach, where only differences are saved for target memories, and only target memory that is in use needs to be saved at all. This makes for usually far more compact checkpoints, which save and open in a few seconds in most cases. A checkpoint taking 30 seconds to open in Simics is rare indeed, and basically requires a target containing many gigabytes of target memory that is all in use (not host memory).

The paper does raise a novel point for why checkpointing is good: it saves you the time to setup a debugger. It is certainly true that setting up a debug environment for a session does take time, but it is not made clear just how it is recreated. My impression is that really what is going on here is that the debugger remains alive and loaded, and the simulation goes back and forth in time using checkpoints. Which is perfectly valid and nice.

In Simics, we solve this problem in two ways: when it comes to setting up the internal debugger when opening a checkpoint, the solution is to use a script to configure it. On purpose and by design, Simics checkpoints do not store the simulation session state, only the target system state, as that is the only way to be portable over time and across widely different machines. it would be very strange to open a checkpoint a few years after it was created, by some random user, start a simulation, and have it stop just because there was some breakpoint still in place for some long-forgotten reason. In any case, the simulator-side of the debugger has to be adopted in all systems to accept checkpointing.

Another strange point in the paper that I would have liked to ask the authors about is the idea that you can always change the target software in the middle of a session. But what is the value of checkpointing then? Since the idea is saving the cost of booting a machine and setting up a debug session, it is hard to see what is gained by booting a machine, keeping the hardware state, but replacing the software state. The expensive operation is presumably setting up the software state? At least it is in my experience.

I commend the paper for actually running something more than the archetypal “ARM+DSP”, by running eight copies of an ARM+DSP subsystem. That is at least starting to look like something interesting to simulate.

Reviewer Notes for the Paper

Finally, I have some small critiques on the academic paper itself, of the kind that I tend to provide to paper authors when active as a reviewer for conferences.

The SoC conference paper itself is well-written, but I have point out that the authors for some reason ignore the history of checkpointing in full-system simulation. The seminal work here was the checkpointing system used for changing the level of abstraction from fast functional simulation to cycle-accurate simulation in the SimOS system developed at Stanford already in the early 1990s. This has later been continued in both IBM Mambo and Virtutech Simics, and remains the most powerful way of doing checkpointing until this day. I don’t think the authors were completely ignorant of this work, even though finding good references can be a bit difficult. Even so, here are some to add to future versions of that paper:

• SimOS: A Fast Operating System Simulation Environment, Stanford CS Technical Report CSL-TR-94-631, 1994. The earliest mention of using checkpointing in a full-system simulator (for switching from fast to detailed simulation).
• “Design and validation of a performance and power simulator for PowerPC systems“, from the IBM Journal of Research and Development in 2003. Mentions that IBM Mambo does checkpointing like SimOS.
• SIMFLEX: A Fast, Accurate, Flexible Full-System Simulation Framework for Performance Evaluation of Server Architecture, ACM SIGMETRICS Performance Evaluation Review, 2004. Also see the Simflex homepage. Shows an ambitious use of checkpointing in computer architecture simulations.

It is also a bit disingenuous to dismiss the issue of just how the process checkpointing is performed by a reference to an early requirements paper from the BLCR project.  All that says is that there are lots of possible variants of implementation, nothing on what was done in this particular instance. It would have been nice to have had some more details: does the approach use a kernel module or not?

The main theme of the interview is the need to shift business thinking from small details to entire systems. From operations research where you spend lots of time understanding some process or department in great detail, to a system-level thinking where you focus on what an entire enterprise is doing.

For me, this struck a chord in my system-level heart… in my world of computer systems and virtual platforms, system-level is what it is so hard to get engineers to. Far too much time is spent (in my opinion) understanding, modeling, and tweaking subsystems. Far too little effort is spent on understanding the whole, how things fit together in practice, taking software, hardware, and software system evolution over time into account. The analogy is not perfect, but there are more things that are alike than are not.

The most interesting analysis that Russell Ackoff fires off from his perspective is that of comparing companies and architecture. An architect knows the whole of a building, but does not entirely go into details on just how it is to be built. He/she trusts the carpenters, bricklayers, and other workers to know how best to solve their local problems. Basically, applying hierarchical abstraction to the task of constructing an actual building.

This got me thinking some of why this is the case. I think it could be because building things (castles, cathedrals, houses, walls, pyramids, canals, …) must have been among the most complex tasks undertaken for a very long time in human history. Thanks to this long history, we have perfected the abstraction and division of labor in construction. Buildings are built in a certain way, by a certain set of crafts, since that method has been proven to work well for a very long time. So just like in the case of the design patterns craze in the late 1990’s, architecture might have something to teach us about how to build hardware/software systems too.

Note that for some reason, I cannot find a link to the podcast on the BBC homepage. But if you subscribe in iTunes or similar, I think you will find it. Something is not as user-friendly as it could be.

If you look at the facts, SAAB has very rarely made a profit. Apparently, the 1980’s were decent, but since GM took over in 1990, maybe there have been two profitable years. With that kind of history, it is hard to motivate why they should have been allowed to continue for this long in a market economy.

That said, the reason that SAAB is closing now is squarely and simply GM. General Motors has to be one of the worst-run companies in the business, and they have badly mauled and mistreated all their car brands except maybe Opel. SAAB was bought as European luxury brand for GM, which could have worked very well. If only the company had been given solid funding and a strategic direction, it could well have grown into a Lexus-Toyota pairing with Opel. Now, SAAB was never given the room to innovate and develop new products, instead it turned a steady draining loss each year. GM managed to do the worst possible: it didn’t invest in the brand to give it a fighting chance, but it also kept it going as a loss-making unit for almost twenty years. Overall, this long slow decline must have cost more money than a solid up-front investment back in 1990…

I guess in five years time, we will be seeing the real analysis of what happened, and this might be taught in business schools as a classic example of how not to manage car companies — or any brand, for that matter.

Anyway, here is a screenshot of the manual:

It is one of four thread-sleep-state control instructions in the 64-bit server variant of the Power ISA. Essentially, it is an IBM extension for their POWER series machines, as well as the Cell and Xenon CPUs I guess. See the Power ISA tutorial from the Power Architecture Developer’s Conference 2007 for some more on this.

I like this kind of whimsicalness in technical systems. It makes them human and more approachable. Sometimes, big companies (and small companies) once they are mature end up trying a bit too hard to sound business-wise and “professional”… ending up being plain boring and stone-faced and cold. There is no contraction between a chuckle and a professional system for most people.

Some people would put the Power Architecture “eieio” instruction in the same category of slightly funny. However, the limit for all assembly languages I have ever encountered seems to be the natural name for an instruction to Sign EXtend something. It is never called what it “should” be.

Note that this instruction is not new, it has been around since 2005 at least, probably longer. There are no history notes in the manual, and I have no intention of reading through lots of old manuals to find the first when this one did not appear.

This was the first multicore event that I have been to where we did not have a keynote speaker or technical paper from a hardware company. So there was really nothing here directly about how to build multicore chips. Rather, the workshop tended to be about how to program, use, measure performance on, verify software for, and generally work with multicore chips. From the perspective of software people, rather than hardware designers.

Obviously, hardware aspects enter into such talks, but it is the perspective of a user, not a designer. For example, a hardware designer could explain how an atomic compare-and-swap is optimized in a multicore device. But here, we saw measurements on the actual operation latencies observed on real machines using such operations. Quite refreshing, and closer to my personal interests.

The keynote by Andras Vajda of Ericsson was quite interesting. The slides are not online, but the main points that I picked up and that I might not have considered before:

• Software development costs can mean that the cheapest, fastest, most efficient hardware is not necessarily the most economic. Too hard to code for means the software development time and effort removes the advantage. Obvious, but worth reiterating. Software is king.
• The workload on a cellular basestation can sometimes be highly linear and single-threaded. For example, serving a single terminal with a very high bandwidth LTE connection. And suddenly shift to a massively parallel workload as a crowd of a thousand all suddenly appear and start doing data downloads. And then go back to serial again. This means that the age-old argument that signal processing naturally “conveniently concurrent” (and here) is not always true. Nice point!
• Thus, we need adaptable architectures that can trade serial and parallel performance over time, and rebalance quite quickly. In the same chip.
• He is a firm believer that homogeneous systems will win out in the end, I still hold on to a belief in accelerators and offload engines and DSPs. This is partially because of an admitted focus on servers and services processors, and not on the baseband and signalling side. Makes sense.
• Domain-specific languages (DSL) are the future of efficient programming. Agree.

On the topic of DSLs, there was a question about the cost to support them. To me, that is a non-issue. In the organizations that I have worked, it seems that maintaining a useful DSL requires at most one engineer. Developing one, a few good computer scientists for a fairly limited time. In any case, they tend to appear organically when good programmers generalize repeated tasks.

I gave a keynote about how multicore has impacted virtual platforms (in particular, Virtutech Simics) with the following main points:

• Multicore targets increase the performance pressure on a virtual platform, as more processors will have to be simulated.
• Multicore hosts means that sequential performance of the host is going down compared to the aggregate parallel performance demands from the targets.
• To handle large target systems, the virtual platform itself has to run multithreaded on a multicore host. Getting this in place is a major, interesting, and sometimes painful process.
• Once you have a parallel virtual platform, multicore hosts provide a very nice boost in scalability and the manageable system sizes. A single multithreaded virtual platform process is also a bit easier to manage from a user perspective.
• All features in the virtual platform have to be multicore and multimachine-aware… meaning that they often get a bit harder to use initially, as there is no “default processor” you can fall back to for debugging setups etc. Everything has to be explicitly targeted.
• Multicore targets have proven to  be a great sales driver for virtual platforms, as debugging software on a physical multicore, multichip, multiboard system is just too painful.

Overall, this was a fun event, looking forward to next year at Chalmers!

Walking to and from the conference from my hotel, I listened through a FLOSS Weekly interview with David Heinemeier Hanson, the creator of Ruby on Rails. His approach to programming and languages was quite unlike that exposed at FDL. In his world, anything that is repeated in code should be put into the language or library. In Ruby, that is easier than in many other cases, as the language can be extended arbitrarily without recompiling the VM. His focus on programmer productivity and convenience is in stark contrast to the FDL discussions which mostly dealt with how to simulate things in a single language, SystemC. Quite boring from a programming language perspective.

Another podcast that triggered thoughts on programming and how to improve it using languages was Stackoverflow Episode 73. In the listener questions section, the topic of language evolution came up. Joel and Jeff pointed out that C# is a glowing example of a language that quickly evolves and adds useful features, including things from the field of dynamic languages. Quite interesting. They made the crucial point that backwards compatibility in a language is not really needed, as long as you can link code compiled from the old and the new languages together. So, if C# 3.0 won’t compile all C# 2.0 code, it is no big deal, as you can still have the old C# 2.0 compiler around, and then link with the new C# 3.0 code.

The key is linkability between modules, not the standard of the input language. Here, Microsoft’s .net system is starting to make a very impressive showing, I think. C#, VB, F#, Python, Ruby — a ton of languages all share the same common language runtime and the basic libraries of .net. After hearing a talk by Tim Harris of Microsoft UK at MCC 2009, I am even more impressed by what .net can do.

.net was also the topic of the FLOSS Weekly interview with the team behind IronPython. IronPython is Python on top of the .net framework, and the interview went into a lot of interesting details on how that has played out. The short answer is: very impressive, very smart, and very much the way things should be.

Note that even if the perspective is that “ESL languages describe a single hardware chip configuration, which is fixed”, having a language which is more dynamic still helps.Remember that modeling is programming, and anything that makes programming more efficient is a good. All you need to do is to have a “freeze” operation that says that “this particular set of things is my design”. But you might get there by interactively adding and removing things at a command-line interface.

Working in OSCI CCI WG, I have come to realize just how useful reflection in languages like Python is (or as we implement it in Simics). When all you have is a static C++ compiled binary, you cannot easily do things like ask objects for their type and other metadata like documentation. Since it just is not there. While in Python, you can do such inspection, and also extend things at run-time, which is very useful. If you want to add configuration hooks to a class, Python makes it dead easy, while C++ makes it major painful.

The Dream ESL Language

Overall, what that I get from all of this that a sound design for an “ESL” language, had we started today, would be:

• Basic semantics given by a virtual machine, not an input language.
• Opportunities for several different input languages of potentially very different styles to be used, all compiling and linking into the same VM. That would open up for real innovation.
• Extensive reflection and introspection features.
• Dynamic reconfiguration during run-time, optionally frozen if the goal is to actually describe some hardware design for synthesis. But such synthesis would be from VM code, not some input language.

Essentially, taking the approach of providing a stable interoperability layer between languages in the form of a VM, and allowing languages to be anything anyone could care to invent.