The Edge website has an article on the misuse of statistics in modeling. My experience as a model builder (pointing control systems on satellites) suggests that there are a lot of lessons to be learned from this Edge article.
The author of the Edge article has written the books "Fooled by Randomness" and "The Black Swan". One of the author's big points was that the financial system relied on models. Models based on statistical assumptions by people who didn't fully understand the intricacies of statistics. To make matters worse, the model builders are rarely the model users. Most model users know even less about the short comings of a given model than the model builders.
The primary lessons for engineers are
- The processes that underly system disturbances are not always random and uncorrelated. Almost all of our modeling techniques are based on the assumption that the processes are random and uncorrelated. How does this "color" the model output? How accurate can the output really be?
- Most specifications are based on 1 or 3 sigma (standard deviations) values based on an assumption that the distribution is Gaussian (or normal). A 1 or 3 sigma value in distributions other than Gaussian means entirely different things. Does the systems engineer passing that requirement realize that the distribution may not be Gaussian? Does the design engineer need to re-interpret the requirement in some way to make sure the system actually works?
My thoughts can be seen in full detail at
http://blogs.controltheorypro.com/2008/09/engineering-control-systems-and-statistics/
http://blogs.controltheorypro.com/2008/09/observations-on-control-system-modeling/
"Almost" Good Answers: