Wednesday, January 03, 2007

Some final thoughts on the design of experiments

In my previous two posts I described two different experiments in which variation was not properly considered.

The first experiment failed to consider variation altogether. The goal was to identify a material resistant to cavitation erosion. Only one sample of candidate material was run. The hidden, underlying assumption by the experimenters was that the coefficient of variation ( standard deviation/average ) was neglible. Without some background knowledge this is no a justified decision.

I talked once to a buyer whose responsibilities included certifying materials for nuclear use. One of the tests required to certify a lot (heat) of stainless steel for nuclear use is a corrosion test. He told me that different lots of identical grades of stainless steel from the same manufacturer can have vastly different corrosion resistance as measured by the standard corrosion test.

In the case of the cavitation erosion test the testing was expensive because of the nature of the testing and the requirement to decontaminate the samples prior to examination. Hence the decision to run only one sample of each material. The question arises though, was it worth conducting the experiment at all. Without some idea of the underlying statistical distribution, no, the experiment was not worthwhile. It does not provide sufficient data to make a sound decision.

The second experiment, the optical fiber radiation attenuation test attempted to consider variation. The method for measuring variation, however, was ill thought out. The two samples run for each fiber type were from the same lot. To accurately measure variation the samples should have been from multiple lots.



Anonymous John said...

A statistics professor once told me "statistics is about uncertainty." How on earth can engineers attain a reasonable level of confidence without running properly designed experiments? The sad news is it happens all too often.

Thanks for addressing this critical issue.

Saturday, January 13, 2007  

Post a Comment

<< Home