A new record, replay, and reverse debugger has appeared, and I just had to take a look at what they do and how they do it. “rr” has been developed by the Firefox developers at Mozilla Corporation, initially for the purpose of debugging Firefox itself. Starting at a debugger from the angle of attacking a particular program does let you get things going quickly, but the resulting tool is clearly generally useful, at least for Linux user-land programs on x86. Since I have tried to keep up with the developments in this field, a write-up seems to be called for.
A long time ago, when I was a PhD student at Uppsala University, I supervised a few Master’s students at the company CC-Systems, in some topics related to the simulation of real-time distributed computer systems for the purpose of software testing. One of the students, Magnus Nilsson, worked on a concept called “Time-Accurate Simulation”, where we annotated the source code of a program with the time it would take to execute (roughly) on the its eventual hardware platform. It was a workable idea at the time that we used for the simulation of distributed CAN systems. So, I was surprised and intrigued when I saw the same idea pop up in a paper written last year – only taken to the next level (or two) and used for detailed hardware design!
Continue reading “Time-Accurate Simulation Revisited – 15 years later”
Intel is a big Simics user, but most of the time Intel internal use of Simics is kept internal. However, we recently had the chance to interview Karthik Kumar and Thomas Willhalm of Intel about how they used Simics to interact with external companies and improve Intel hardware designs. The interview is found on the Wind River blog network.
It is also my last blog post written at Wind River; since January 18, I am working at Intel. I am working on ways to keep publishing texts about Simics and simulation, but the details are not yet clear.
I just posted a short blog post on the Wind River blog, introducing a video demo of the Web API to Wind River Helix Lab Cloud. In the post and video, I show how the Lab Cloud Web API works. For someone familiar with REST-style APIs, this is probably baby-level, but for me and probably most of our user base, it is something new and a rather interesting style for an API. Thus, doing a video that shows the first few steps of authentication and getting things going seems like a good idea.
I have read some recent IBM articles about the POWER8 processor and its hardware debug and trace facilities. They are very impressive, and quite interesting to compare to what is usually found in the embedded world. Instead of being designed to help with software debug, it seems the hardware mechanisms in the Power8 are rather focused on silicon bringup and performance analysis and verification in IBM’s own labs. As well as supporting virtual machines and JIT-based systems!
I have a long-standing interested in debugging in general and reverse debugging in particular and the related idea of record-replay debug (see a series of blog posts I did a few years ago on the topic: history 1, history 2, history 3, S4D report, updates, Simics reverse execution, and then Lab Cloud record/replay). Recently, I found out that Undo Software, one of the pioneers in the field, had released a product called “Live Recorder“. So I went to check it out by reading their materials and comparing it to what we have seen before.
The recent news that a hacked version of Apple Xcode has been used to insert bad code into quite a few programs for Apple iOS was both a bad surprise and an example of something that has been hypothesized for a very long time. For the news, I recommend the coverage on ArsTechnica of the XCodeGhost issue. It is very interesting to see this actually being pulled off for real – I recall seeing this discussed as a scenario back in the 1990s, going back to Ken Thompson’s 1983 ACM Turing Award lecture.
I just added a new blog post on the Wind River blog, about how you do fault injection with Simics. This blog post covers the new fault injection framework we added in Simics 5, and the interesting things you can do when you add record and replay capabilities to spontaneous interactive work with Simics. There is also a Youtube demo video of the system in action.
I have read a few news items and blog posts recently about how various types of software running on top of virtual machines and emulators have managed to either break the emulators or at least detect their presence and self-destruct. This is a fascinating topic, as it touches on the deep principles of computing: just because a piece of software can be Turing-equivalent to a piece of hardware does not mean that software that goes looking for the differences won’t find any or won’t be able to behave differently on a simulator and on the real thing.
The Security Now Podcast number 497 dealt with the topic of Vehicle Hacking. It was fairly interesting, if a bit too light on the really interesting thing which is what actually went on in the vechicle hack that was apparently demonstrated on US national television at some point earlier this year (I guess this CBS News transcript fits the description). It was still good to hear the guys from the Galois consulting firm (Lee Pike and Pat Hickey) talking about what they did. Sobering to realize just how little even a smart guy like Steve Gibson really knows about embedded systems and the reality of their programming. Embedded software really is pretty invisible in both a good way and a bad way.
I have been thinking about the role and prestige of testing for the past several years. Many things I have read and things companies have done indicate that “testing” is something that is considered a bit passe and old-school. Testers are dead weight that get into the way of releases, and they are unproductive barnacles that slow development down. Testers can all be replaced by automatic testing put in place by brilliant developers. The creative developer types are the guys with the status anyway. I might be exaggerating, but there is an issue here. I think we need to be acknowledge that testers are a critical part of the software quality puzzle, and that testing is not just something developers can do with one hand tied behind their back.
In a dusty bookshelf at work I found an ancient tome of wisdom, long abandoned by its previous owner. I was pointed to it by a fellow explorer of the dark arts of computer system design as something that you really should read. The book was “Fortress Rochester”, written by Frank Soltis, and published in 2001.
Last year, I concluded a programming project at work that clearly demonstrated that real programming tasks tend to involve multiple languages. I once made a remark to a journalist that there is a zoo of languages inside all real products, and my little project provided a very clear example of this. The project, as discussed previously, was to build an automated integration between a simple Simics target system and the Simulink processor-in-the-loop code testing system. In the course of this project, I used six or seven languages (depending on how you count), three C compilers, and three tools. Eight different compilers were involved in total.
I am going to be speaking at the 2015 Embedded World Conference in Nürnberg, Germany. My talk is about Continuous Integration for embedded systems, and in particular how to enable it using simulation technology such as Simics.
My talk is at 16.00 to 16.30, in session 03/II, Software Quality I – Design & Verification Methods.
There is a new post at my Wind River blog, about how you can use Simics to enable the automatic testing of pretty much any computer system (as long as we can put it inside a simulator). This is a natural follow-up to the earlier post about continuous integration with Simics and Simics-Simulink integrations — automated test runs is a mandatory and necessary part of all modern software development.
I just found and read an old text in the computer systems field, “Why Do Computers Fail and What Can Be Done About It?” , written by Jim Gray at Tandem Computers in 1985. It is a really nice overview of the issues that Tandem had encountered in their customer based, back in the early 1980s. The report is really a classic in the computer systems field, but I did not read it until now. Tandem was an early manufacturer of explicitly fault tolerant and highly reliable and available computers. In this technical report Jim Gray describes the basic principles of fault tolerance, and what kinds of faults happen in the field and that need to be tolerated.
At the ISCA 2014 conference (the biggest event in computer architecture research), a group of researchers from Microsoft Research presented a paper on their Catapult system. The full title of the paper is “A Reconfigurable Fabric for Accelerating Large-Scale Datacenter Services“, and it is about using FPGAs to accelerate search engine queries at datacenter scale. It has 23 authors, which is probably the most I have ever seen on an interesting paper. There are many things to be learnt from and discussed about this paper, and here are my thoughts on it.
The Mill is a new general-purpose high-performance processor design from out-of-the-box computing (http://ootbcomp.com/). They claim to beat typical high-end out-of-order (OOO) designs like the Intel Haswell generation by crazy factors, such as being 2.3x faster while using 2.3x less power compared to a Haswell. All the while costing less. Ignoring the cost aspect, the power and performance numbers are truly impressive – especially for general code. How can they do something so much better than what we have today? For general code? That requires some serious innovation. With that perspective, I ask myself where the Mill is really significantly different from what we have seen before.
I recently made my first acquaintance with Windows 8, having bought a new Sony ultrabook for the family. Including a touch screen. The combination of the touch-based interface and the phone-like look of Windows 8 even on a PC has led me to think about the (unconscious) expectations that I have come to have on how systems behave and how services are accessed, from how smart phones and tablets have come to work in the past few years. In particular, where are web-based services going?
Apple just released their new iPhone 5s, where the biggest news is really the 64-bit processor core inside the new A7 SoC. Sixty four bits in a phone is a first, and it immediately raises the old question of just what 64 bits gives you. We saw this when AMD launched the Opteron and 64-bit x86 PC computing back in the early 2000’s, and in a less public market the same question was asked as 64-bit MIPS took huge chunks out of the networking processor market in the mid-2000s. It was never questioned in servers, however.
I have a silly demo program that I have been using for a few years to demonstrate the Simics Analyzer ability to track software programs as they are executing and plot which threads run where and when. This demo involves using that plot window to virtually draw text, in a way akin to how I used to make my old ZX Spectrum “draw” things in the border. But when I brought it up in a new setting it failed to run properly and actually starting hanging on me. Strange, but also quite funny when I realized that I had originally foreseen this very problem and consciously decided not to put in a fix for it… which now came back to bite me in a pretty spectacular way. But at least I did get an interesting bug to write about.
Debugging – the 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems by David Agans was published in 2002, based on several decades of practical experience in debugging embedded systems. Compared to the other debugging book I read this Summer, Debugging is much more a book for the active professional currently working on embedded products. It is more of a guidebook for the practitioner than a textbook for students that need to learn the basics.
Via the EETimes, I found a very interesting talk by Bristol professor David May, presented at the 4th Annual Bristol Multicore Challenge, in June of 2013. The talk can be found as a Youtube movie here, and the slides are available here. The EETimes focused on the idea to cut down ARM to be really RISC, but I think the more interesting part is Professor May’s observations on multicore computing in general, and the case for and against heterogeneity in (parallel) computers.
There is a new post at my Wind River blog, about how Simics sessions are started and the mechanics of system setups in Simics. It also has a link to a Youtube video demonstrating various ways of starting Simics simulation sessions.
Nudge theory is an intriguing fairly new take on how to manage societies and modulate the behavior of people to make them behave in a “better” way. My layman’s interpretation of the idea is that instead of threating punishment and setting up rules, you try to encourage desired behavior with rewards and discourage undesired behavior by making it require an effort. When working with the new Test Runner view in the Simics 4.8 GUI, I realized that we had invented exactly such a nudge.
Last year, I did a Simics webinar which included a two-part demo of how to use Simics to debug an endianness bug in a networked system as it migrates from big-endian to a little-endian system. Along the way, I also showed off various Simics features like reverse execution and checkpointing and scripted execution.
The demo is now online at the Wind River Youtube channel, and the setup is explained in a blog post at the Wind River company blog which is worth reading before watching the video.
LEGOs seem to be a favorite analogy for people bemoaning the state of software development today. “If only it would be as simple as putting Legos together” is a common enough statement, along with various proposals to make software that is Lego-like. Sometime, I wonder if people making these statements have actually tried to build anything non-trivial from Lego recently. Here, I will look a bit closer at the Lego-programming analogy. There is indeed quite a lot to it, but it is not all about child-level simplicity. I think there are some good lessons that can be learnt from analogizing Lego and programming.
There is a new post at my Wind River blog, telling the story of how some of the Simics developers used Simics itself to debug an intermittent Simics program crash caused by a timing-sensitive race condition.
Running Simics on itself is pretty cool, and shows the power of the simulator and its applicability even to really complex software.
Logging as as debug method is not new, and I have been writing about it to and from over the past few years myself. At the S4D conference, tracing and logging keeps coming up as a topic (see my reports from 2009, 2010 and 2012 ). I recently found an interesting piece on logging from the IT world in the ACM Queue (“Advances and Challenges in Log Analysis“, Adam Oliner, ACM Queue December 2011). Here, I want to address some of my current thoughts on the topic.
After some discussions at the S4D conference last week, I have some additional updates to the history and technologies of reverse execution. I have found one new commercial product at a much earlier point in time, and an interesting note on memory consistency.