I have a two-part series (one, two) on testing posted on my Software Evangelist blog on the Intel Developer Zone. This is a long piece where I get back to the interesting question of how you test things and the fact that testing is not just the same as development. I call the posts Mindset and Toolset
It is really sad that the European Space Agency (ESA) lost their Schiaparelli lander last year, as we will miss out on a lot of Mars science. From a software engineering and testing perspective, the story of why the landing failed rather instructive, though. It gets down to how software can be written and tested to deal with unexpected inputs in unexpected circumstances. I wrote a piece about this on my blog at the Intel Developer Zone.
The SiCS Multicore Day took place last week, for the tenth year in a row! It is still a very good event to learn about multicore and computer architecture, and meet with a broad selection of industry and academic people interested in multicore in various ways. While multicore is not bright shiny new thing it once was, it is still an exciting area of research – even if much of the innovation is moving away from the traditional field of making a bunch of processor cores work together, towards system-level optimizations. For the past few years, SiCS has had to good taste to publish all the lectures online, so you can go to their Youtube playlist and see all the talks for free, right now!
UndoDB is an old player in the reverse debugging market, and have kept at it for ten years. Last year, they released the Live Recorder record-replay function. Most recently, they have showed an integration between the recorder function and Jenkins, where the idea is that you record failing runs in your CI system and replay them on the developer’s machine. Demo video is found on Youtube, see https://www.youtube.com/watch?v=ap8552P5vss.
A long time ago, when I was a PhD student at Uppsala University, I supervised a few Master’s students at the company CC-Systems, in some topics related to the simulation of real-time distributed computer systems for the purpose of software testing. One of the students, Magnus Nilsson, worked on a concept called “Time-Accurate Simulation”, where we annotated the source code of a program with the time it would take to execute (roughly) on the its eventual hardware platform. It was a workable idea at the time that we used for the simulation of distributed CAN systems. So, I was surprised and intrigued when I saw the same idea pop up in a paper written last year – only taken to the next level (or two) and used for detailed hardware design!
Continue reading “Time-Accurate Simulation Revisited – 15 years later”
Thanks to the good folks at Vector Software, I was pointed to a conference recording on Youtube, from the Google Test Automation Conference (GTAC) 2015 (Youtube video). The recording covers quite a few talks, but at around 4 hours 38 minutes, Brian Gogan describes the testing used for the Chromecast product. This offers a very cool insight into how networked consumer systems are being tested at Google. Brian labels the Chromecast as an “Internet of Things” device*, and pitches his talk as being about IoT testing. While I might disagree about his definition of IoT, he is definitely right that the techniques presented are applicable to IoT systems, or at least individual devices.
In a blog post at Wind River, I describe how the Wind River Helix Lab Cloud system can be used to communicate hardware design to software developers. The idea is that you upload a virtual platform to the cloud-based system, and then share it to the software developers. In this way, there is no need to install or build a virtual platform locally, and the sender has perfect control over access and updates. It is a realization of the hardware communication principles I presented in an earlier blog post on use cases for Lab Cloud.
But the past part is that the targets I talk about in the blog post and use in the video are available for anyone! Just register on Lab Cloud, and you can try your own threaded software and check how it scales on a simulated 8-core ARM!
I have a long-standing interested in debugging in general and reverse debugging in particular and the related idea of record-replay debug (see a series of blog posts I did a few years ago on the topic: history 1, history 2, history 3, S4D report, updates, Simics reverse execution, and then Lab Cloud record/replay). Recently, I found out that Undo Software, one of the pioneers in the field, had released a product called “Live Recorder“. So I went to check it out by reading their materials and comparing it to what we have seen before.
I have been thinking about the role and prestige of testing for the past several years. Many things I have read and things companies have done indicate that “testing” is something that is considered a bit passe and old-school. Testers are dead weight that get into the way of releases, and they are unproductive barnacles that slow development down. Testers can all be replaced by automatic testing put in place by brilliant developers. The creative developer types are the guys with the status anyway. I might be exaggerating, but there is an issue here. I think we need to be acknowledge that testers are a critical part of the software quality puzzle, and that testing is not just something developers can do with one hand tied behind their back.
There is a new post at my Wind River blog, about the Trinity of Simulation – the computer, the system, and the world. It discusses how you build a really complete system model using not just a virtual platform like Simics, but you also integrate it with a model of the system the computer sits in, as well as the world around it. Like this:
Read more about it in the blog post, and all the older blog posts it links to!
Last year, I concluded a programming project at work that clearly demonstrated that real programming tasks tend to involve multiple languages. I once made a remark to a journalist that there is a zoo of languages inside all real products, and my little project provided a very clear example of this. The project, as discussed previously, was to build an automated integration between a simple Simics target system and the Simulink processor-in-the-loop code testing system. In the course of this project, I used six or seven languages (depending on how you count), three C compilers, and three tools. Eight different compilers were involved in total.
There is a new post at my Wind River blog, an interview with Andreas Buchwieser from the Wind River office in München. It discusses how Simics can be applied to the field of safety-critical systems, including helping test the software to get it certified. Really interesting, and in particular it is worth noting that qualifying tools in the IEC 61508 and ISO 26262 context is much easier than in DO-178B/C. The industrial family of safety standards have been created to allow for tools to help validate an application without forcing incredibly high demands on the development of those tools.
There is a new post at my Wind River blog, about how Simics is used to simulate large wireless networks for IoT (Internet-of-Things) applications.
It is funny for me to be back at the IoT game. A decade ago (time flies, doesn’t it?), at Virtutech, I and Johan Runeson took part in an EU research project on exactly this topic. Unfortunately, we had to back out of that project due to economic circumstances and failing management commitment, but we still learnt a few things that were relevant now that we are back in the IoT game. In particular, how to simulate wireless networks in a reasonable way in a transaction-level simulator. Thus, payback for the investment took 10 years to arrive, but it did arrive. To me, that underscores the need to be a bit speculative, take some risk, and try to explore the future.
I am going to be speaking at the 2015 Embedded World Conference in Nürnberg, Germany. My talk is about Continuous Integration for embedded systems, and in particular how to enable it using simulation technology such as Simics.
My talk is at 16.00 to 16.30, in session 03/II, Software Quality I – Design & Verification Methods.
There is a new post at my Wind River blog, about how you can use Simics to enable the automatic testing of pretty much any computer system (as long as we can put it inside a simulator). This is a natural follow-up to the earlier post about continuous integration with Simics and Simics-Simulink integrations — automated test runs is a mandatory and necessary part of all modern software development.
The September 2013 issue of the Intel Technology Journal (which actually arrived in December) is all about Simics. Daniel Aarno of Intel and I served as the content architects for the issue, which meant that we managed to contributed articles from various sources, and wrote an introductory article about Simics and its usage in general. It has taken a while to get this journal issue out, and now that it is done it feels just great! I am very happy about the quality of all the ten contributed articles, and reading the final versions of them actually taught me some new things you could do with Simics! I already wrote about the issue in a Wind River blog post, so in this my personal blog I want to be a little bit more, well, personal.
As an old embedded systems and real-time guy, I have always worked with computer systems that are in some way tied to their environment. Simics has often been used to model such computer systems, inside of customer organizations. Which makes it a bit hard to show… however, recently I have cooked up a demo showing Simics simulating a computer system alongside a physical system.
I just put out a post on the Wind River blog, pointing to both a video of my own “water heater” demo and some other Youtube videos showing Simics integrated with simulations of the real world. A screenshot of my setup in action is shown on the side of this post.
I just read the EETimes coverage of the recently concluded court case in the US, where Toyota settled for 3 million USD in damages due to experts finding that the software in a 2005 Camry L4 could indeed cause “unintended acceleration”. In the particular case that was concluded, the accident resulting from the issue caused one driver to be injured and one driver to get killed. This feels like it could be the beginning of something really good, or just as well this could go really wrong.
Debugging – the 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems by David Agans was published in 2002, based on several decades of practical experience in debugging embedded systems. Compared to the other debugging book I read this Summer, Debugging is much more a book for the active professional currently working on embedded products. It is more of a guidebook for the practitioner than a textbook for students that need to learn the basics.
This blog post is a review of the book “If I Only Changed the Software, why is the Phone on Fire“, (see more information on Amazon, for example), by Lisa Simone. The book was released in 2007, on the Elsevier Newnes imprint. It is a book about debugging embedded systems, written in a murder-mystery style with a back story about the dynamics of an embedded development team. It sounds strange, but it works well.
Logging as as debug method is not new, and I have been writing about it to and from over the past few years myself. At the S4D conference, tracing and logging keeps coming up as a topic (see my reports from 2009, 2010 and 2012 ). I recently found an interesting piece on logging from the IT world in the ACM Queue (“Advances and Challenges in Log Analysis“, Adam Oliner, ACM Queue December 2011). Here, I want to address some of my current thoughts on the topic.
Last week, I attended my fourth System, Software, SoC and Silicon Degug conference (S4D) in a row. I think the silicon part is getting less attention these days, most of the papers were on how to debug software. Often with the help of hardware, and with an angle to how software runs in SoCs and systems. I presented a paper reviewing the technology and history of reverse debugging, which went down pretty well.
The 2012 edition of the SiCS Multicore Day was fun, like they have always been in the past. I missed it in 2010 and 2011, but could make it back this year. It was interesting to see that the points where keynote speakers disagreed was similar to previous years, albeit with some new twists. There was also a trend in architecture, moving crypto operations into the core processor ISA, that indicates another angle on the hardware accelerator space.
I am going to be talking about how to transport bugs with virtual platform checkpoints, in the Software Tools track at the Embedded Conference Scandinavia, on October 3, 2012, in Stockholm (Sweden). The ECS is a nice event, and there are several tracks to choose from both on October 2 and October 3. In addition to the tracks, Jan Bosch from Chalmers is going to present a keynote that I am sure will be very entertaining (see my notes from a presentation he did in Göteborg last year).
We just uploaded a short movie about reverse execution and reverse debugging to Youtube, to the Wind River official channel. In the short time available in this demo, we really only show reverse execution. Reverse debug, as I define it, is not used much at all, as explaining what goes on when you start to put breakpoints into a program and analyze its behavior takes a surprising amount of time.
Once upon a time, all programming was bare metal programming. You coded to the processor core, you took care of memory, and no operating system got in your way. Over time, as computer programmers, users, and designers got more sophisticated and as more clock cycles and memory bytes became available, more and more layers were added between the programmer and the computer. However, I have recently spotted what might seem like a trend away from ever-thicker software stacks, in the interest of performance and, in particular, latency.
Every once in a while I need to build demo setups to show debugging in action. As I have blogged before, finding a good bug when you need one isn’t always easy. The solution is to try to invent artificial bugs, and I was very happy when I managed to stage a buffer overrun in a VxWorks program.
It is pretty very nice demo in which you first start a period program A, which prints the value of an incrementing counter every target second. You then run a supposedly unrelated program B, resulting in the values that program A prints to become corrupted. Perfect to show off reverse execution and data breakpoints in reverse as you go from the point where the corrupted value is printed to the piece of code that overwrote the variable.
But then I ported the demo to a new platform… and the bug didn’t work anymore. My bug had caught a bug and was now not working, or at least not they way I expected it to. What had happened?
Continue reading “My Bug Doesn’t Work!”
Software is Concrete. Once poured it becomes extremely difficult and very expensive to change.
It comes from a blog post by Robert Howe, CEO of Verum, a company selling formal-methods-based and model-based programming tools. It does capture something of the phenomenon we all know: that software can be pretty darn hard to change, once it has shipped and is in use. It fits well with the fact that the later bugs are found, the more expensive they are to fix.
But it also provoked quite a bit of opposition when I put the quote up on Facebook, and I have to agree that maybe not all is as simple as that blog makes it out to be.
There is a new post at my Wind River blog, about how Simics was used to kick-start the development of the 64-bit version of VxWorks. It is an interesting example of how to use a virtual platform as a model of something much simpler and gentler than actual hardware systems.