UndoDB Recording during Automatic Testing

undo-logoUndoDB is an old player in the reverse debugging market, and have kept at it for ten years.  Last year, they released the Live Recorder record-replay function.  Most recently, they have showed an integration between the  recorder function and Jenkins, where the idea is that you record failing runs in your CI system and replay them on the developer’s machine.  Demo video is found on Youtube, see https://www.youtube.com/watch?v=ap8552P5vss.

This is a fundamentally good idea, and something we have promoted for Simics for quite a few years now (check the flow diagram some distance down in the post in particular, replicated below). It is an obvious application for record-replay abilities no matter how you implement it:

ci-flow-feedback

However, there was something a bit odd with their implementation that makes the approach seem less practical.  In their video, they show the user interface of a Jenkins Blue Ocean instance… with the recording being its own step:

screenshot_20161103-195704

This kind of defeats the whole idea I must say. Record-replay should be used to reproduce the run that fails. The underlying assumption is that there is something variable or non-deterministic that is not entirely under control. It could be the precise timing of incoming network packets, the exact scheduling of threads in program across multiple cores, the results of random number generators, or the values read from IO devices.  Thus, the point of recording is to capture all the uncontrollable factors in a run so that that precise run can be reproduced again.

In this demo, what we see is a rerun of the failing tests using the recorder, in order to produce a recording to pass on.  This does not make sense, since it will not be the same run as the initial run that failed. If an error can be reproduced this easily in the test system, it would seem just as easy to reproduce it on the developer’s own machine.  The record-replay feature just does not add a lot of value.

There is some merit to the recording as a mechanism to encapsulate inputs and the program under investigation. You could also make the case that the environment on the test server and the developer’s machine will be different, and the recorder captures the truth at the point of origin.  But overall, it seems that this is an admission that the overhead of recording is a bit too high to use it on all test runs.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *