The houses at Lalandia are all very new, the block where we stayed (the one furthest away from the central building) cannot be more than two years old. They have not finished the roads yet, nor are the lawns completely grown up.

Everything feels fresh and new, and you cannot complain about the standard of the houses.You have a fully equipped kitchen, as well as a dishwasher and a washing machine – families with children is clearly the target audience. You do need to bring some basic household supplies with you from home, there are no spices, washing powder, or dishwasher tables available in the houses. You have to pay for electricity and water in addition to the basic rent, which feels unnecessarily stingy as the basic rent is pretty high and the cost for utilities ends up being about one or two percent of the cost. I guess it is a way to steer people towards not wasting resources, though.

Lalandia is intended mainly for visitors who come by car. Each house has two parking spaces in front of it, and the idea is clearly that several families can share a house. The smallest houses are sufficient for four adults, we rented a six-person house (which worked well with five adults and two children), and there are also eight-person houses available in a few different styles. Since the place is next door to Billund Airport, you could also fly in, distances are short enough that you don’t really need a car during your stay anyway (you can literally walk from the airport terminal to Lalandia and Legoland). The airport does cause the occasional burst of noise as aircraft take off, but is was not really bothersome while we where there.

Since you are likely to cook most of your food yourself, you also need to go buy it somewhere. There is a somewhat pricey “supermarket” in the main building, which is almost impossible to avoid thanks to its convenient location and good opening hours. Shopping in downtown Billund is about ten minutes walk away, but this being a small town, shop hours are pretty poor, 9-18 or 9-19, shorter on Saturdays and closed (!) on Sundays. This was  a surprise for someone used to Swedish shop hours which tend to be 10-22 all days all weeks. Try to arrive on a weekday, or bring lots of supplies for the first day.

It also seems that Denmark is not quite as credit-card-friendly as other countries. If you use a foreign card, they charge you, the customer, the 2.5 to 3% charge that the credit card company charges the store. This is something I have only seen in very small shops living on tight margins in Sweden, and in the Netherlands. In everyday life, this never happens in Sweden or the US, or in most other countries. The fact that Denmark is outside the Eurozone makes some of these charges possible, since the tougher EU rules against card charges currently only applies to transactions in Euros. To be fair, Lalandia actually has lower credit card charges than other places in Billund.

I read a number of online reviews of Lalandia, and they tended to complain that Denmark is expensive. I can agree, it feels about 20-25% more expensive than Sweden, but it is not astronomical or any worse than downtown München or Paris or London.

Lalandia itself is really not about the houses or the food or credit card charges, though. It is a waterpark, and the hangar-like main building contains the Aquadome, biggest waterpark I have ever seen. It also holds a fairly camp indoor fake Mediterranean town square (with a roof painted like a blue sky), an indoors playground, a gym, and several restaurants (which we never tried). It reminds me of the plastic fake milieus in Disneyland Florida.

The Aquadome is just fantastic! It is claimed to be the biggest water park in Scandinavia, and I can certainly believe that. It is all indoors, except a small section on the outside that you can swim out to from the inside. It has the best water slides I have ever tried, all of which require you to use some form of craft to go down. They also all leave the building and come back in, making for a very dramatic ride. See the picture below for how the water slides protrude from the building. The biggest tube is a water slide where you ride four people at once, on a four-person rubber raft!

There were pools with activities suitable for children as small as one year, and all the way up to those who rather 40 than 4 years old.  Very impressive. You need several visits to explore all the options, and fortunately unlimited access is part of the package with the holiday homes.

It is annoying that they do not allow you to bring your own food inside. Since we have some allergies in the family, it is not that easy to find food that is safe, and we would much prefer to bring our own. Overall, you do get the feeling that they are trying a bit too hard to make you part with your money; it is a bit too aggressive to feel entirely right.

Legoland Denmark is right across the street from Lalandia’s main building, and I will write a separate blog post about that.

A final note is that the ownership structure of Lalandia is interesting. Each house is actually privately owned, and the Lalandia company is just a broker organizing the renting-out of the houses. The are still selling houses, and claim that the yearly cost with mortgages is no higher than the cost of renting for a week. It does look like quite a few Norwegians have taken up on the offer, judging from the large number of expensive cars from Norway we saw parked in front of something like half the houses.

]]> http://jakob.engbloms.se/archives/1162/feed 0 http://jakob.engbloms.se/archives/1156?&owa_from=feed&owa_sid= http://jakob.engbloms.se/archives/1156#comments Sat, 26 Jun 2010 18:34:55 +0000 Jakob http://jakob.engbloms.se/?p=1156

The call for papers for the third annual Swedish Workshop on Multicore Computing (MCC10) is now out! MCC is a nice venue for multicore research about everything from computer design to software and debug. It is fairly informal, but still attracts some good papers and provides good discussions. It is not restricted to Swedish submissions, in 2009 there were several international participants. I gave a keynote talk about multicore and Simics at last year’s MCC, and for this year, I am on the program committee and looking forward to many great submissions to review!

So what is the significance of this code?

Essentially, there are three main communications protocols or channels involved with GDB. There is the direct command-line interface familiar to most GDB users. There is the gdb-serial protocol for talking between a GDB and a GDB server on some remote machine.There is the GDB-MI, Machine Interface, protocol used to directly control a GDB debugger from another program. In particular, Eclipse uses GDB-MI to control GDB, when using GDB as the debugger back-end for Eclipse.

With GDB-serial, a remote machine can also be a simulator, hardware emulator, or anything else that presents the right interface to GDB. Indeed, using gdb-serial is the most common way to attach “anything” to be debugged by GDB. Simics, VMWare, and many other virtual platform solutions support gdb-serial as a way to get the power of the GDB debugger (source code lookup, scripting, and the smarts to deal with complex languages like C++ and Ada, and much more) for debugging code running on the virtual machine. The requirements on a gdb server are very simple, it really is just start/stop/step, and setting breakpoints and modifying memory.

When GDB 7.0 came out, support for reverse debugging was the really big piece of news. With the initial release of 7.0, you could access reverse debug from the GDB command-line interface, driving either local process record/replay targets, or remote gdb-serial targets. Simics was among the first simulators to support reverse debugging over gdb-remote, since the command set for doing reverse debug was essentially contributed by Virtutech with the initial release of reverse debugging in Simics back in 2005. Michael Snyder got that code into the reversible GDB development from the start, we can claim no credit for getting it into the mainline GDB.

However, the MI interface was missing reverse debugging commands, and I have been involved in getting that support into the GDB source tree. It has taken quite a while longer than I expected, and it has been quite an interesting journey.

When we started the work back in late 2008, we had a set of MI commands that was used in a Virtutech-specific patch to GDB 6.8. This set of commands and their supporting code was submitted to the GDB mailinglist in early 2009, but got rejected by the maintainers. In the end, we changed the commands according to the wishes of the maintainers, and added test cases and documentation (my bit). This patch was then resubmitted, reviewed, and updated in a few more iterations. The final update was to align the code with the latest code updates to GDB.

Along the way, Tomas and I had to sign some legal papers to assign copyright to the code we contributed to the FSF. For some reason, the first batch of papers got lost on their way across the Atlantic to us to get signed, forcing a delay of some months.

So, now, we have GDB with MI supporting reversible debugging. Finally, I hope it arrived in time to help the Eclipse people get reverse debugging into Eclipse as a standard feature.

For me, even though this process sometimes felt like it would never end, it has been very interesting.

• The process is frustrating in that if you disagree with a maintainer on some issues, things can stall until you realize the errors of your way and repent. We had such a discussion on the style of the GDB reverse commands, both in MI and the main command-line interface. The viewpoint that I supported lost, which I think is a shame, but at least the functionality is there.
• Related to that, the discussions on the gdb mailinglists have been quite interesting. If you have relevant points, people will tend to listen. The value of ideas is judged from how well they work, and usually not from set agendas or corporate grand strategies.
• Sometimes, you really miss the function of an overall product manager for GDB. Somewhere or someone that takes responsibility for the overall direction of the product. Today, we have a very productive anarchy, which certainly works, but is a bit unfamiliar to someone used to commercial and planned software development.
• Compared to some other standards work that I have been involved in, GDB is a much faster process. Collaboratively improving a shared body of code is a nice way to work, but it might be very hard to port over to company-driven future-pointing standards. For example, if no code exists, the model hardly works.
• The user-interface design coming out of the GDB community is clearly quite geeky and often not what I think a UI designer would come up with. The GDB reverse commands, with a “set direction” command is one example. Another interesting one starts here, with my input here.

Happy reversible debugging, all!

The approach taken for their checkpointing system is to save the entire state of  the running simulation program (i.e., an entire host operating system process), and later recreate the process in the same state. This gets around the need program all simulation models to explicitly support checkpointing, but it also limits the applicability of checkpointing severely. Of the checkpointing operations described in my previous post, it only supports the bringing up of a checkpoint into the same model, on the same machine (or a machine that runs a completely identical software stack down to the exact versions of all libraries, drivers, etc., which is not very likely to happen). This is honestly admitted in the paper, I appreciate that.

Process-based checkpointing in this way is also used by Cadence, and just like Cadence’s solution, the problem that appears when implementing it in practice is how to handle all the OS resources opened by a process. The process itself is not enough. The resources are typically files open for input and output, connections to debuggers, and other things that reach out of the virtual world of the simulation into the real world of the host machine. The solution is just like Cadence’s solution: provide callbacks for the SystemC side of such connections that can save the state of the connection in some way, and restart the connection when a checkpoint is opened. Doing this right does require some care in what to do in which order, and the paper nicely explains this.

For example, you need to close down all connections before taking a checkpoint, and then restore them after taking it. This is a fairly destructive operation, compared to the Simics-style checkpointing where all you do is interrogate the state of all objects and save that. I had not thought of that before, but it makes perfect sense.

To me, this also points to an important architectural issue in SystemC simulations in general: what are these simulation modules doing opening files on their own anyway? In my opinion, a simulation model should never do such low-level thing, it should operate using only the defined simulation system API and simulation-level connections to other simulation modules. If you need to load test data, do that by putting it into some storage system managed by the simulator itself. In all my professional life, I have considered file I/O to be a fundamental service provided by the application framework I use, nothing that my actual payload code should have to deal with. For example, in Simics, we tend to solve the loading and checkpointing of test data vectors using a memory “image”, which supports checkpointing in itself. Then a script just loads a file into the memory image, and a device reads the memory and moves transactions into the simulated system. This means that file I/O is totally absent from the model, and also that the test data can be managed and inspected by existing infrastructure.

The cost of taking a checkpoint is quantified in the paper, and it is not as bad as one could fear. The size of  a checkpoint easily reaches 100s of MB for even small target systems, but saving and loading that today does not take all that long. Still, going to a system with only 16 target processors (8 ARM, 8 DSP, and 8 times 64 MB of target memory = 512 MB) generates a checkpoint taking 1.7 GB to store and 30 seconds to save.

Once again, it is interesting to compare this to the Simics-style approach, where only differences are saved for target memories, and only target memory that is in use needs to be saved at all. This makes for usually far more compact checkpoints, which save and open in a few seconds in most cases. A checkpoint taking 30 seconds to open in Simics is rare indeed, and basically requires a target containing many gigabytes of target memory that is all in use (not host memory).

The paper does raise a novel point for why checkpointing is good: it saves you the time to setup a debugger. It is certainly true that setting up a debug environment for a session does take time, but it is not made clear just how it is recreated. My impression is that really what is going on here is that the debugger remains alive and loaded, and the simulation goes back and forth in time using checkpoints. Which is perfectly valid and nice.

In Simics, we solve this problem in two ways: when it comes to setting up the internal debugger when opening a checkpoint, the solution is to use a script to configure it. On purpose and by design, Simics checkpoints do not store the simulation session state, only the target system state, as that is the only way to be portable over time and across widely different machines. it would be very strange to open a checkpoint a few years after it was created, by some random user, start a simulation, and have it stop just because there was some breakpoint still in place for some long-forgotten reason. In any case, the simulator-side of the debugger has to be adopted in all systems to accept checkpointing.

Another strange point in the paper that I would have liked to ask the authors about is the idea that you can always change the target software in the middle of a session. But what is the value of checkpointing then? Since the idea is saving the cost of booting a machine and setting up a debug session, it is hard to see what is gained by booting a machine, keeping the hardware state, but replacing the software state. The expensive operation is presumably setting up the software state? At least it is in my experience.

I commend the paper for actually running something more than the archetypal “ARM+DSP”, by running eight copies of an ARM+DSP subsystem. That is at least starting to look like something interesting to simulate.

Reviewer Notes for the Paper

Finally, I have some small critiques on the academic paper itself, of the kind that I tend to provide to paper authors when active as a reviewer for conferences.

The SoC conference paper itself is well-written, but I have point out that the authors for some reason ignore the history of checkpointing in full-system simulation. The seminal work here was the checkpointing system used for changing the level of abstraction from fast functional simulation to cycle-accurate simulation in the SimOS system developed at Stanford already in the early 1990s. This has later been continued in both IBM Mambo and Virtutech Simics, and remains the most powerful way of doing checkpointing until this day. I don’t think the authors were completely ignorant of this work, even though finding good references can be a bit difficult. Even so, here are some to add to future versions of that paper:

• SimOS: A Fast Operating System Simulation Environment, Stanford CS Technical Report CSL-TR-94-631, 1994. The earliest mention of using checkpointing in a full-system simulator (for switching from fast to detailed simulation).
• “Design and validation of a performance and power simulator for PowerPC systems“, from the IBM Journal of Research and Development in 2003. Mentions that IBM Mambo does checkpointing like SimOS.
• SIMFLEX: A Fast, Accurate, Flexible Full-System Simulation Framework for Performance Evaluation of Server Architecture, ACM SIGMETRICS Performance Evaluation Review, 2004. Also see the Simflex homepage. Shows an ambitious use of checkpointing in computer architecture simulations.

It is also a bit disingenuous to dismiss the issue of just how the process checkpointing is performed by a reference to an early requirements paper from the BLCR project.  All that says is that there are lots of possible variants of implementation, nothing on what was done in this particular instance. It would have been nice to have had some more details: does the approach use a kernel module or not?

Thanks to the community transcripts wiki:

Spolsky: Quite probably. I mean C is a car, it’s very dangerous – it doesn’t have seatbelts, but it’s very powerful because it goes very fast. In fact, I’d go so far as to say, unless some people are gonna punch me about this one, but you cannot write multi-threaded code in the C programming language. You just can’t. Although technically all the capabilities are there, it is beyond the capability of mortal humans – and I know, some of you out there are very smart and you think that you have the capability of writing multi-threaded code in the C programming language, because you’re hot shit. Well, let me tell you, it is beyond the capability of humans on this planet, for their brains are just not adequate to the task of writing multi-threaded code in most languages, least of all low-level languages like C. It’s just not gonna work, it’s just not gonna make you happy. So there.

The only problem with this statement is that it means we are all doomed: since all operating systems are written in C, and all low-level libraries, it means that we have no way to build the first level of abstraction needed to provide the higher-level primitives for C# and Python and everything else higher up the stack.

But in general, I think this is a fairly true statement for volume programming.

Thanks to Simon Kågströms post (and the even better second-generation with screenshots) about using Eclipse for the Linux kernel, I have a much nicer work environment now for my ongoing work in learning Linux device drivers on PowerPC, which has helped me work my way through several hard-to-figure-out system calls. Here is a screenshot that I found pretty cool… the tool has found the definition and comments for the IRQ registration function:

2009-Feb-01:

I had to rebuild my indexing from scratch in the past weekend, and as a result, I have a word of warning: you have to create a “C project” in Eclipse, if you accidentally create a “Project”, the Eclipse workspace file will have the wrong name (.project instead of .cproject), and the autoconf-to-eclipse script will not work.

Anyhow, what we set out to do this year was to replace (most of) the twenty-year-old wooden decking on the backside of our small row house with a new one. This was quite an adventure, as we discovered all kinds of interesting designs and problems with the old decking structure. Problems, which do reflect on the realities of computer programming and simulation.

Preserving legacy structures

For the simple reason that we do not want to spend the entire four weeks of vacation building things, we decided to keep the original shed that is located at one end of the decking. Technically, this meant taking a saw, cutting the decking close to the shed, and rebuilding the rest of it. It also meant that we had to constrain the rest of the construction to preserve the original height of the decking, and the thickness of the boards used to construct it. Had we rebuilt everything, we would have done things differently and reconstructed the entire supporting structure of beams.

Now, we were forced to preserve the existing design, which is not quite optimal or fully modern. Very much like how it is when updating a large piece of software. Most of the time, you have to preserve old APIs, pieces of code, and design decisions that are not really optimal or modern anymore. Fixing it means starting from scratch, and quite often it is not considered worth the investment in time and effort.

You also tend to find amazingly poorly done items in old software, and so it was with our decking. A large part of the old deck had sunk some 10cm below its original level. When digging out to find out why, it became apparent that the beams had just been put on a bed of sand with no concrete or metal supports on the part closest to the house. That bed of sand had then been hollowed out by a misconstructed fix to a drain pipe that let the water spill out around the proper drain. Sigh. Not the first instance of poor workmanship around this house… and we had to do a workaround involving bricks, more beams, and a very large load of sand (I have shoveled some 800kg of dirt and sand during this project).

Reflection on simulator construction

The next observation is based on how the decking turned out in terms of the number and size of boards needed to cover the area. I had done some measurements beforehand indicating the width and length of the area to be covered. It looked like a number of 95mm standard boards with 3mm spacing would do perfectly to cover the area and provide a nice even end.

In the end, we wound up buying a small number of 120mm boards for the final row. For some reason, one side was 10mm wider than the other side. Also, the 95mm boards turned out not to actually cover the area quite as well as expected. Due to minor variations between the boards (in practice, they are 94 to 96mm) as well as imperfect spacing and the general plasticity of wood (it is a living material, not lego bricks laid out peg by peg) made our plans fall just short forcing a last-minute change to 120mm boards for the end.

As a simulation expert, you could say that this shows the shortcoming of the analytical approach. By making a single measurement of the width of the field, and assuming a single number for the width of a board, I set myself up for surprises when confronting a variable reality. If I had built a better model of the deck, measuring it at multiple places and measuring my boards and working out on paper – simulating – how they were to be put together, I could have predicted this and bought the 120mm boards in the original load instead of as an afterthought. In principle, this might be true.

But on the other hand, doing that simulation would have been tantamount to actually building the thing. Understanding the spacing between the boards and how their variations interact would have necessitated constructing in my mind, computer, or paper a perfect model of the deck. I doubt that it would have been saving me any time compared to just going out and putting the thing together, especially as the act of putting it together taught my just how variations occurred and how they worked out in aggregate.

I think the same often applies to computer simulation. Yes, you need to have more details and more variation than a simple analytical model. However, you also need to keep the model simpler than reality in order to make it feasible to construct, maintain, and execute at decent speed. Any model will have imperfections where it diverges from reality, and as a simulator constructor the game is not to be perfect but to provide a useful tool to your end users. If it is a bit imperfect, so what, as long as it is imperfect in unimportant ways and mostly provide good guidance. For more on the philosophy of simulation, I would suggest reading a blog post from 2007 about golf games and computer simulation.

So what would a useful model have been for my decking? I guess measuring a few more points would have helped, as well as measuring some set of boards and the device we used for spacing. That might have helped us spot the issue with the too-narrow final board ahead of time. But it was more fun this way, and this is vacation time and hobbyist work and not professional carpentry.

Conclusions

What conclusions can be drawn from this?

1. I guess I am a hopeless nerd who can never stop thinking about technology.
2. I have a very nice deck now, looking forward to some decent days in the next few weeks to make use of it.
3. Building physical things is quite a satisfying experience, even if they do not turn out perfect in all details.

See you there!

Highly recommended.

On x86, the virtualization uses the virtualization extensions in recent editions of Intel and AMD processors. There is no such support on the C64+ DSP series, so they fall back to paravirtualization, modifying the operating systems to play nice with the virtualization layer. Nothing particularly magical about this.

The interesting part is really that people are considering using Linux which is usually very hard to tune for non-standard hardware platforms to run code on a real high-power DSP. In general, that is considered a bad idea since what you want is a thin layer of software to let you do compute programs that make maximum use of the processor. Doing interrupts and other typical OS work on a DSP is a bad idea since it breaks program flow. Some instructions sequences on a C64-type DSP even require you to turn off interrupts in order to work correctly! So I guess the performance of Linux programs are going to be pretty poor… However, the idea here seems to be to move some control-plane functions onto the DSPs in the system, which is going to be poorly performing code anyway.  I guess it makes sense if you have spare cycles on the DSP to be able to do away with the additional control-plane processor (which is usually a small slow processor anyway on the kind of base station software being targeted here).

In a virtualization-based setup, you use a single hypervisor program that then controls several single-processor operating systems running on the machine. That hypervisor also takes care of allocating shared resources to the operating systems, sometimes by sharing a single physical resources, sometimes by only letting one operating system access a certain resource. So in this case, you can actually reuse existing OS images on a new multicore chip and transparently transform an existing system.

Too bad there is still no embedded processor with strong support for heavy-duty virtualization like this.

On the other hand, it might be a passing need. The transition of old applications to new hardware will always involve some rewrite and retouch, and if that means doing a bit of change in the OS setup to handle an AMP case nicely, it is probably not too expensive (compared to redoing applications on top of the OS to be truly SMP). And for virtualization, this means that you can use a Xen-style paravirtual approach where the OS is modified to run on top of a simple hypervisor.

Running and booting an unmodified binary install of an OS is likely more of a server/desktop problem than one for embedded applications. We are going to see virtualization support in hardware to help light-weight approaches be even more efficient, and also to tackle the security issues of rogue code getting into some OS image. Hardware support is needed to contain an OS that has been taken over by bad guys, no amount of cooperation between OSes in an AMP setting is going to prevent that.

The Rock is a very interesting piece of work. It tries to be both a throughput-oriented design like the Niagara/Ultrasparc T machines, and a single-thread high-performance design. Even though on balance, it is more skewed towards the throughput computing aspect. What is very cool is how they use additional threads to help boost the performance of a main thread using “scout threads” (a concept I saw presented back at ISCA 2004). This makes it possible to use threads to either boost single-thread performance OR do throughput, creating a more flexible design than is usually the case. It is also the first commercial implementation of transactional memory. And 16-way. And due for next year.

So far, Rock seems like a very successful and very visionary project that is trying in yet another way to gain momentum by pure hardware innovation. Just like the UltraSparc T line, Sun is trying to out-invent IBM and Intel/AMD. Who seem to be mostly progressing by just piling on more of the same old features. I really hope this play goes well, if we were down to just IBM/PPC & System Z and Intel-AMD/x86-64 on the server and desktop side, the world would just be too boring.

The Intel and IBM talks on programming were both grounded in the idea that to make people accept a new programming language/API, it has to be an evolution of what the programmers already know. Which pretty much ties us down to C/C++/Java/C# with extensions and modified semantics.

X10 is basically Java with some nicely considered features to support local and global memories and programs that can scale to BlueGene-style massively clustered machines. Tim basically tells everyone to stop inventing new languages and focus on improving existing frameworks like MPI and OpenMP in collaboration with industry. Presented in a very funny style, Tim is a great presenter, and tries hard to get the audience to react. In this crowd, most people agreed. Except the Erlang people, who feel that they do have a better solution to multithreading and multicore than any patched-up language in the C-Java family. I must agree with them, and I do feel that Erlang today is mature enough to serve that purpose.

The panel session at the end was very entertaining, where some people (including myself and Joe Armstrong) tried to ask tough questions to the keynote speakers (and Ulf Wiger of Ericsson). Quite engaging and a rare chance to directly engage with some industry heavyweights who otherwise tend to sit on the other side of the Atlantic.

I think the prize for coolest tech of the day goes to QuviQ, a spin-off from Chalmers doing automated testing tools that really work well for parallel and distributed systems. Their method of minimizing the trace of a failed test case is really interesting, and finds things that no human tester would ever find.

I also presented a talk on “Debugging Multicore Software using Virtual Hardware”, in the breakout sessions. I guess our Tools track was the least visited of the three tracks, but the audience asked some good questions. And there were some good discussions afterwards.

However, to summarize the day, I am a bit disappointed that not more is being done on the hardware side to help people debug their multicore and multiprocessor parallel programs. Transactional memory is all nice and dandy and can help simplify low-level locking primitives for threaded programs. But I would like to see much more in terms of smart tracing, hardware breakpoints and triggers, massive synchronized stops, and similar features. And instructions and features that make parallel expressions simpler. Here, the embedded folks doing things like ARM CoreSight seems to have been much more successful than the server-class designers at Sun, Intel, and IBM. But even ARM do not spend more than 10-15% of the chip area on debug support.

I think it would be interesting to see what would happen if you could spend 25-30% of the chip on some seriously powerful debug features. Full support for remote control of all cores at the same time, lots of bandwidth for debug data and commands, and fat traces of all traffic on and off the chip. Performance and event counters everywhere. That would make the peak performance of chip likely less than a competing chip not spending as much space on debug support — but it would make achieving a high utilization much easier, and that might actually make the debug-intense chip more economical. Would be interesting to try. But I guess nobody would dare to buy such a design.