Poking Holes in Products

I recently started using a new mobile phone, a Blackberry Bold 9700. I am a bit ambivalent on some of its design features, but it is certainly a very different device from the much more friendly SonyEricsson I had before. Like anybody would do, I have been playing around with it to see what it can do and what not (notable things not working: the “AppWorld” application store is not available in Sweden, YouTube videos do not play in any way that I can figure out).

And almost inevitably, as you play around with a complex modern piece of software (which is what most of the phone is, after all), you find some obvious things which are just plain broken. You wonder, “why didn’t they think of this”, and “how could this ever escape testing?” My current best example is that the built-in web browser does not render the pages from Blackberry’s own support knowledgebase.

This is something that they have control over from both sides, so the only reasonable explanation is that they just have not thought to test that site with the browser. It seems strange that  a new user would uncover such holes after a few hours, when presumably thousands of planned and spontaneous testing hours are being spent at Blackberry on each device they release.

To be fair, this is not a showstopper for me, it is just a good example of how users tend to poke holes in all modern software products. Blackberry is not unique in this regard, and I fully admit that the software that I have worked on over the years did not always fulfill the requirement of being bulletproof. There is a classic saying that if you think something is foolproof, you just haven’t found the right fool yet.

So what is this telling us about software development? I think the main lesson is that you need people who just use your product and do so creatively, as part of the development process. Software developers usually lack the destructive imagination necessary to really poke holes in what they have created, and are usually not given time to develop elaborate test cases.

The moment you put code into the hands of users, you can expect them to find “obvious” issues in your product – simply because their “normal” use is something you yourself never dreamt could happen. In my example, I presume that Blackberry assumed that people will use the support forum when their phones don’t work – and therefore not surf to them from a phone. Sounds reasonable, until you press on a bit and ask if someone might not look at a friend’s broken phone out in the field and need to search support… or check for solutions to non-fatal errors.

I guess the key lesson for me, one which I apply in my daily work, is that product management, marketing, or testing, has to work through use cases with the software they create. Really work them through, and when sidetrack ideas appear, follow through on them, not ignore them to focus on the main task at hand. We have to combine features, and do more than what’s in typical tutorials.

Feature combination is especially powerful, I still recall the sense of “aha” I got when as a PhD student I talked to some colleagues researching the field of feature interaction. It sounds like quite an interesting field, even though I am not sure how you could formally capture something that will generate the test case “read our support site using the phone browser”…

This post used to end in “I guess this counts as a software development rant”. Now, I classify this as a “tester’s lament”…