I just read the EETimes coverage of the recently concluded court case in the US, where Toyota settled for 3 million USD in damages due to experts finding that the software in a 2005 Camry L4 could indeed cause “unintended acceleration”. In the particular case that was concluded, the accident resulting from the issue caused one driver to be injured and one driver to get killed. This feels like it could be the beginning of something really good, or just as well this could go really wrong.
The facts of the case are quite interesting. Back in 2011, a team from NASA reported that they had found no clear evidence that the control software could cause unintended acceleration. Toyota was cleared by the US NHTSA. I blogged about the report on my Wind River blog, and noted that NASA reported not running all the code in simulation due to a lack of tooling. Now, the new panel of experts appears to have actually managed to simulate the system, and found ways to make it crash in the interaction between multiple tasks. The fact that, as EETimes report, a certain task crashing can cause acceleration to continue without control, is pretty indicative of issues arising in integration rather than unit testing. What have gone wrong appears to be not the basic control logic, but its implementation with multiple tasks, shared variables, and variable values not checked against corruption.
The EETimes quote Michael Barr:
Memory corruption as little as one bit flip can cause a task to die. This can happen by hardware single-event upsets — i.e., bit flip — or via one of the many software bugs, such as buffer overflows and race conditions, we identified in the code. There are tens of millions of combinations of untested task death, any of which could happen in any possible vehicle/software state. Too many to test them all. But vehicle tests we have done in 2005 and 2008 Camrys show that even just the death of Task X by itself can cause loss of throttle control by the driver
Scary, that the software is so sensitive even a single bit value change can cause it to completely go off. It does have to be a very (un)lucky bit change, however. But the buffer overruns and race conditions I can certainly believe in as likely causes of an intermittent crash. Would have been interesting to know if some particular environmental condition caused this, or if it was indeed totally random. Probably, nobody knows.
So, on balance, it would seem we have another case of software being implicated in deadly accidents. It has happened before, but I do think this is the first time for cars. Therefore, this is a landmark in the history of embedded software.
However, pulling this kind of case through the courts feels instinctively wrong, but in this case at least the experts hired by the plaintiff appear to have done a really good job. I cannot make myself like the outlook for millions or even billions of dollars being pulled out of Toyota for various damaged parties via the US court system – it just does not really help solve any kind of problem, and randomly benefits the people in the US that sued. I am not saying that those who have been hurt should not be compensated, but the US court system really tends to do this in a way that is overly costly and without reasonable limits. Maybe selling cars in the US is now turning into one of those business you just do not want to be in due to the risk of lawsuits.
On the good side, maybe better regulations can come out of this. The industry has been preparing for that for a long time, but the car industry is also dead scared of becoming like the airplane industry where stringent certification requirements make development projects take decades and make rapid changes and feature additions completely impossible. So far there have been no accidents reported for civilian airplanes that have been traced back to software, which is a good thing. But the weight and cost of the development seems very hard to bear for the car industry.
So regulation can be both good or bad.
We will see where this ends up.
I will write in English because many people use this crap language.
Seriously Jakob if you die or when you die your software will flip all its bits to 1? or to 0?
“and variable values not checked against corruption.” – any value from 0 to 4G on a 32 bit machine is a valid value.
@Timpuri Noi: Any possible number formed from a particular combination of the 32 bits is a valid value, that’s true. Let’s say, however, that the number holds your bank accounts balance of today. Would you still think, that -2147483648 is a valid number in that context? If yes, I’ll be happy to provide you with a place where you can send the positive difference amount to.
The reason for punitive damages against corporations is because corporations view those judgements as a loss. If the loss is smaller than the cost of fixing the problem, they will not fix the problem. Ford was caught doing this in the US. They knew that the Pinto had a problem where it would explode and kill occupants when impacted from the rear. But Ford did the math. It would cost more to fix the Pinto than it would to settle the lawsuits that were likely to arise.
So when you say that taking millions or billions of dollars out of Toyota does not help solve any problem, you’re wrong. It provides the only motivation Toyota is capable of perceiving to actually fix the problem. If the court case had only been for medical and legal fees for the injured driver and $1M for the killed driver (standard value of a human life in US courts), Toyota would not bother to exert any more effort. In fact, by making the act profitable, you would have guaranteed that Toyota would never take the financially destructive act of improving their development processes. To do so would be to actively harmful to themselves. Look at Goldman Sachs. They get sued repeatedly and fined by the SEC in the US. But the SEC always fines them less than the amount of profit made from breaking the law. That makes breaking the law standard operating procedure for Goldman Sachs.
The reason they’re called ‘punitive’ damages is because they are meant to provide punishment. Monetary damages are literally the only stimulus that a corporation can perceive.
My take on the size of the settlement is, how big does it have to get before companies start taking software correctness seriously in critical systems?
In addition to the problems you mention, either this or another investigation of the software found many examples of unsafe coding practices – not issues of style, but matters that either had a definite probability of causing a fault, or which made it impossible to have confidence that the system would work safely. What bugs me most about this issue is that, for a relatively small increase in cost (much closer to the amount spent on these studies, rather than the total cost of developing a new model), Toyota could have significantly reduced the risk it passed on to its customers. It is even possible that by tightening their development practices and employing more talented developers, they could have saved money, principally by reducing rework.
If Toyota is concerned about the negative impact of regulation, it should have been proactively more responsible. I think the downside of regulation is exaggerated anyway, given that car companies have made this complaint for decades, yet innovation has accelerated as regulation has grown. You can make a good argument that regulation has driven innovation.
To be fair, one can wonder why Toyota has had to admit to criminal acts, while the financial institutions that created the 2008 crash have not had to do so. This, however, is more an indication of ineffective financial regulation than it is of a witch-hunt against Toyota.
With havin so much content do you ever run into any issues of plagorism or copyright violation? My blog has a lot of completely unique content I’ve either written myself or
outsourced but it looks like a lot of it is popping it up all over the web without my permission. Do you know any ways to help prevent
content from being ripped off? I’d really appreciate it.
my blog post windows 8 loader (Antoine)