This article reviews that almost every engineer is involved in some way in the planning of experiments, whether they are laboratory experiments, field tests, or computer simulations. A set of techniques called Design of Experiments (DOE) has proved to be an extremely useful methodology for enhancing the effectiveness of that planning. Six Sigma quality programs have been a mechanism for promulgating DOE in industry, but new research shows that a complementary approach, based on a set of adaptive one-factor-at-a-time experiments, leads to better results under many conditions. The full-factorial DOE method, which was initially developed to study agriculture, sets up an experiment for each possible combination of the factors that need to be tested. For example, consider the development of a new clutch. One aspect of its performance is drag torque—the amount of torque the clutch transmits when it is disengaged. A performance goal is to minimize drag torque by appropriate selection of materials, geometry, and other parameters. DOE is a remarkably successful procedure, which has had a profound influence on the professional practice of engineering.
Almost every engineer is involved in some way in the planning of experiments, whether they are laboratory experiments, field tests, or computer simulations. A set of techniques called Design of Experimet1ts, or DOE, has proved to be an extremely useful methodology for enhancing the effectiveness of that planning.
Six Sigma quality programs have been a mechanism for promulgating DOE in industry, but new research shows that a complementary approach, based on a set of adaptive one-factor-at-a-time experiments, leads to better results under many conditions.
The full-factorial Design of Experiments method, which was initially developed to study agriculture, sets up an experiment for each possible combination of the factors that need to be tested. For example, consider the development of a new clutch. One aspect of its performance is drag torque-the amount of torque the clutch transmits when it is disengaged. A performance goal is to minimize drag torque by appropriate selection of materials, geometry, and other parameters.
The design team may decide to consider two factors surface texture (warned and smooth) and surface shape (flat and wavy). In a full factorial design of experiments, 22 experiments will test four clutches-one with a flat waffled surface, another with a flat smooth surface, a third with a wavy waffled surface, and yet another with a wavy smooth surface. The study of three factors would require eight experiments, or 23.
One-factor-at-a-time experiments succeed each other as a series in which, at each step, a single factor is changed while other factors remain constant.
In a one-factor-at-a-time series of experiments, drag torque would be determined for a baseline configuration of the design, perhaps using flat, smooth clutch plates. Then, the experimenter changes a single factor-for example, by introducing a wavy clutch plate-while all other factors remain constant. The drag torque is measured in the new configuration. Once the effect of the flatness is determined, the effect of another parameter, say surface texture, can be evaluated by changing from a smooth to a waffle pattern.
Engineering experiments are frequently done in series (often on a single apparatus). If that is the case, the sequential nature of the experiments creates an opportunity to adapt to the new results that emerge. One way to take advantage of this opportunity is with an adaptive variant of a one-factor-at-a-time experiment. As each factor is assessed, an alternative that brings improvement is retained in studying the effect of the next factor.
If the wavy plates prove effective in reducing drag torque, the waffle pattern is applied to wavy clutch plates rather than to flat ones. If the change in surface texture improves the design, the new texture is retained, but if the waffle pattern increases the drag torque, that change is reversed before proceeding to study other factors, such as plate material or plate spacing. The adaptive one-factor-at-a-time plan thus constantly exploits new information as it is discovered.
In the 1920s, Sir Ronald Fisher, motivated by problems of agricultural experimentation, developed the techniques of factorial Design of Experiments. In most agricultural experiments, uncertainty is high, because field conditions are difficult to control. In agricultural experimentation, it is also common to run a large number of trials in parallel since results cannot be obtained quickly. They usually require a full growing season.
Because a large number of trials are run in parallel, they must be planned at one time. Design of Experiments makes use of the ability to plan the experiments to reduce the effects of experimental uncertainty.
In the four tests of the clutch, there are two observations with flat clutch plates, two with wavy, two with warned, and two with smooth. Averaging the observations reduces the impact of uncertainty on the results.
The full factorial design also enables the assessment of interactions among experimental factors. An interaction is the effect that factors have when applied jointly. In the clutch example, the surface texture reduces drag torque when applied to flat plates, but increases drag torque when applied to wavy plates. A one-factor-at-a-time experiment does not allow the experimenter to determine such interactions.
A disadvantage of the full factorial design is that the minimum number of trials needed grows rapidly with the number of factors to be studied. Every time an additional factor is added, the minimum number of trials doubles at least. As a result, full factorial experiments are rarely conducted with a large number of factors.
To decrease the minimum number of required experimental units, fractional factorial designs are often used. For example, engineers can use just four trials to examine the effects of texture, flatness, and material on drag torque. A full factorial design with three factors would require eight experiments. The fractional factorial design is a selected subset of the full factorial experiment that preserves many of its useful properties.
A risk in using a fractional factorial approach is the possibility of confounding interaction effects (effects that two variables produce in concert) with main effects (effects produced by a factor independent of other factors). It has been shown, for example, that the flatness and texture of clutch plates exhibit such interaction. If texture, flatness, and material type are studied together, the interaction between texture and flatness may be mistakenly attributed to material type. The possible confounding of interactions with main effects is a principal drawback of fractional factorial designs.
Accounting for Error
In planning any experiment, it is important to consider the experimental error-the combined effects of many uncertainties or random variations, such as the repeatability of measuring instruments or small fluctuations in conditions like ambient temperature. One approach to dealing with experimental error is replication running the same experiment in repeated trials and then averaging the measurements. Replication is effective, but increases the cost of the experiment. The size and influence of experimental error is a key aspect in the choice of whether to use a one-at-a-time or a Design of Experiments approach.
The measure of drag torque in our clutch will vary from trial to trial because of the imperfect repeatability of the dynamometer, changes in oil viscosity over time, and differences in manufacture of the clutch plates. If these experimental errors cause large variations in the measurements, many replicates may be required to determine the drag torque with adequate precision. Suppose, for purposes of illustration, that two replications are needed to determine whether one configuration is better. The one-variable approach would need two replications for each of the four determinations, making a total of eight tests in all.
The Design of Experiments approach, which contains the desired two observations with each factor, would need only four trials.
How can engineers decide which alternative-one-factor-at-a-time or DOE-is preferable? Recent research by the authors addresses that question.
We examined data from 66 published full factorial experiments on a variety of different engineering components and systems. Each of the 66 data sets was used in simulations the effects of experimental error. We analyzed the data to infer the preferred settings of each design variable and recorded the tabulated drag torque for the combination of preferred levels. This process was also repeated 1,000 times for all 66 data sets.
A fractional factorial approach was also simulated. Eight different clutch designs (the same number as that used for the one-at-a-time approach) were selected according to a fractional factorial design. Each design was subjected to simulated tests by looking up the corresponding drag torque in a table and adding a random number to mimic the effects of experimental error. We analyzed the data to infer the preferred settings of each design variable and recorded the tabulated drag torque for the combination of preferred levels. This process was also repeated 1,000 times for all 66 data sets.
The information from the clutch simulations can be used to illustrate the overall trend of the results. The best clutch design found by Lloyd had a drag torque of 1.4 foot-pounds. When the experimental error was low, the simulation of the one-at-a-time approach led, on average, to 1.5 ft.-lbs. of drag torque. Thus, with only eight experiments, the one-at-a-time approach accomplished almost the same result as a full factorial approach requiring 128 experiments.
By comparison, the fractional factorial approach, also with eight experiments, led to 1.9 ft.-lbs. of drag torque, on average. As the experimental error increased, the results for both approaches were worse (the minimum torque increased), but the fractional factorial design was affected much less than the one-at-a-time approach.
Using the 66 data sets, we could define a generic pattern. The main finding was that the one-factor-at-a-time approach provides better results than a fractional factorial design, if either the experimental error is small or the interaction among factors is strong.
Specifically, if the variance due to simulated experimental error was less than 40 percent of the variance caused by the design changes, the adaptive approach led to better designs than the fractional factorial approach. The adaptive one-at-a-time approach was also superior, if interaction effects accounted for more than 25 percent of the observed variance in the system.
In addition, there is another important feature to an adaptive one-at-a-time approach. Sometimes, a series of experiments is terminated before the plan has been carried out perhaps because of funding, scheduling, or experimental difficulties. An incomplete one-at-a-time approach can determine, even early on, some changes that lead to improvement. This is especially significant when the development team is able, based on engineering insights, to prioritize their efforts and investigate the most important factors first.
While the results of this study appear to be at odds with current practices, support for the results is not without precedent.
Milton Friedman, the Nobel Prize-winning economist, and the statistician Leonard Savage published a paper in the 1940s, which presented arguments about why factorial design is not as effective as a one-at-a-time plan, when the primary goal is design optimization. Friedman and Savage noted that a factorial design spreads its observations across the design space, while a one-at-a-time plan, by its nature, concentrates observations in areas of the design space that appear promising. This concentration of effort is one explanation for the effectiveness of the one-at-a-time plan.
The statistician Cuthbert Daniel also recognized the benefits of a one-at-a-time approach, but for a different set of reasons. He noted that one-at-a-time approaches allow experimenters to "learn something from each run" and "react to data more rapidly." He also acknowledged the need for a cutoff in terms of degree of experimental error.
What's in it for Engineers
Design of Experiments is a remarkably successful procedure, which has had a profound influence on the professional practice of engineering. The impacts of Design of Experiments on industry are increasing, with more education and training activities, better software support, and more expert statistical assistance.
Although the role of one-factor-at-a-time experiments has diminished, our research suggests that this approach can provide a real benefit in engineering experimentation. Specifically, experimenting one-factor-at-a-time should be considered in engineering situations in which the experimental error is small, or the interactions between factors are strong, or there is a possibility that the planned set of experiments may not be fully completed.
The bottom line is that whatever methods are used in conducting experiments, it is important that design engineers use the full range of tools at their disposal, with knowledge of both their strengths and weaknesses.