Return to this week's Bulletin

14 December 2001

Design Your Experiments

by Kevin Kilty

Sheldon Greaves and Norman Stanley have discussed a series of articles for the SAS Bulletin regarding the design of experiments, or what some people call DoE. This is a good idea. Not only is there paltry literature available for amateurs regarding the subject--the extensive literature is tailored to very specific audiences and is verbose beyond reason--but also, the point of view changes quite a lot from one audience to another. Since Norman is busy with his excellent series on chemistry, I have decided to get things underway. However, I hope this eventually involves other people as well, even if this means only through their feedback on the topic de jure. Please feel free to reach me directly.

Let me take this first note in the series to outline a few issues. Then, over the course of many weeks, I'll tackle topics one at a time. What I hope to do is eventually equip an experimenter's toolkit. Some of the tools are mathematical, some are procedural, and some are philosophical.

What is experimental design?

Engineers, biologists, and physicists each have a slightly different take on experiments. I am in a good position to see this because my formal understanding of DoE comes from engineering, while my formal education is as a geophysicist. Engineering experiments often have a different motivation than scientific experiments. Engineers typically have an idea in working form; that is, they have a process or product that works, but they need to figure out what factors influence the process or product so they can make improvements. They design experiments to screen many factors and find the most important ones. Engineers also use experiments to make robust designs. When products or processes become available for use, they have many random influences which the engineer has no control over. The idea of a robust design is to make a product or process work well even in circumstances that the engineer could hardly have forseen in advance.

In chemistry and biology there are circumstances in which the experimenter would like to screen factors to decide which are the most important. In these situations the engineering model of DoE is useful. However, there are scientific issues where the goal is not to find out what factors most influence a process, but, rather, to decide between or among competing theories. The engineering model of experimental design is not so useful, here, and what is required is a much more philosphical perspective on experimental design. To take a logical positivist approach for a moment, the first step in experimental design is to decide which measurements separate one theory from another and allow us to reject theories that fail. Good theories make predictions and the essense of an experiment is to decide exactly which predictions to test. For example, quantum theory predicts that specific heat of an insulator crystal should vary as temperature to the 3^rd power at low temperature. Testing the adequacy of this theory means making measurements of heat capacity as we vary the temperature, and DoE in this instance is not concerned with screening factors, but in how to make measurements with the resolution needed to disprove the theory.

What do we need to begin designing an experiment?

Experiments work best if we design them in the light of everything we know about something. Learn all you can about a subject before embarking on experiments. It is perfectly alright to be motivated to study a subject through accidental, serendipitous observations of something unusual. This happens to me a lot. But it makes little sense to begin doing experiments in a state of total ignorance, especially when you can remove that ignorance easily through library research or talking with someone who has an interest in the topic. This is what makes the SAS forum such a valuable resource. The Forum lets you find people with particular interests. In fact, the more prepared you are about a subject, the more likely you are to notice unusual little things that lead to genuine discoveries. As Louis Pasteur said, "Chance favors the prepared mind."

Please do not interpret what I just said as meaning that there is no reason to repeat what other people have done. Many mistakes in science are discovered through repetition of a single experiment, and it pays to convince yourself that you can repeat other peoples' results before launching in some new direction.

Recently I read a fun little book entitled "How to get ideas." It is written by an advertising man, and he documented how it is that many creative people--advertising people included-- get ideas by becoming thoroughly immersed in a subject. This is an initial step in most research. Never overlook its importance.

Observation versus experiment

Some sciences, like chemistry, physics, and biology, lend themselves to carefully controlled experiments. Others, in contrast, have to make do with observations of the experiments that nature provides. This does not mean that design of experiments doesn't apply, but that the issues of design now focus on topics such as types of equipment to use, where to make observations, how often, and deciding on the needed resolution. These are not topics of formal DoE, per se, but I'll examine them in detail. One other interesting issue to examine is "When do simulations act as legitimate experiments?"

Resolution and propagation of error

Rarely is it possible for someone to take a direct reading of some parameter. Generally we have to transform a reading into a derived value that is of direct interest. This means that all sorts of factors, such as corrections, instrumental constants, and calibration are involved in the experiemnt. Each of these factors provides an avenue by which errors propagate into a result. The formal analysis of error propagation not only helps quantify the uncertainty in measurements, it also helps identify those factors which contribute most to uncertainty and helps us design better experiments as a result.

Inference and Inversion

When we obtain data from an experiment or observation, what shall we do with it? In some instances the result of the experiment is of interest itself. In such a case I would likely just report the result along with some assessment of its incertainty, and then summarize what I could infer from this. Statistical tools of various sorts are helpful, here. In other cases I will wish to obtain values for the parameters of some model from my observations, and what is called for is regression, or some other form of inverting the observed values into model parameters. This was a big topic of research in geophysics 25 years ago. In the meantime the idea of inversion has diffused throughout the sciences and engineering.

This brings my outline of issues full circle. Regression is probably the most direct means by which to interpret results of DoE in the formal engineering sense of the term. What engineers call a response surface, a scientist or statistician recognizes as a regression model. Every amateur scientist can benefit from an understanding of regression, and how to perform it with a computer spreadsheet application. In fact, some concepts in models and model building may be a good topic to begin with next time.