5. Interpret
5.2.1 The strength of inference
Let's think about the strength of inference in the following two examples. After the Exxon Valdez oil spill, scientists working for the government went out and observed the survival rate of pink salmon eggs incubating in streams that they had previously classified as either affected or not affected by oil. What they observed was that pink salmon embryos in streams classified as affected had lower survivals. This would seem to indicate that the presence of oil in the stream lowered the survival of the salmon. However, those streams classified as oiled were generally north facing. It turns out that the oil generally moved from north to south as storms caused the movement of oil within Prince William Sound. On closer inspection, these streams also tended to be shorter and to be more influenced by wind and wave action. Although these observations were an important part of understanding the effects of the oil spill, this kind of observational study produces a relatively weak inference because there are alternate explanations for the outcomes. In the second example, later controlled laboratory conditions enabled other scientists to assign pink salmon eggs to well-defined treatments (levels of oil exposure) and to control most sources variation in survival rate, such as temperature and amount of mechanical disturbance. Then by the use of random assignment of experimental units to treatments, by knowing exactly how much exposure each experimental unit had, and by the use of replication, these scientists were able to make very strong inference about the dose-response relationship of the oil (Rice et al. 2001).
In recognizing the importance of studies of uncontrolled events, Schwartz (1998) talks about an increase in the strength of inference that is possible as the analyst increases the control over variation, as shown in the second example above. He talks about a continuum, as studies moves from what he calls descriptive surveys (where some parameter like a mean is estimated without explanatory variables), to observational studies (phenomena are observed in one or more occurrences), to analytical surveys (phenomena are observed in selected example occurrences), and then to designed experiments (units are assigned to treatments).
References
Rice, S.D., R.E. Thomas, M.G. Carls, R.A. Heintz, A.C Wertheimer, M. Murphy, J.W. Short, A. Moles, Adam. 2001. Impacts to pink salmon following the Exxon Valdez Oil Spill: persistence, toxicity, sensitivity, and controversy. Reviews in Fishery Science Volume 9(3): 165-211.
Schwarz, C. 1998. Studies of uncontrolled events. Pages 19-39 in V. Sit, and B. Taylor, editors. Statistical Methods for Adaptive Management Studies, Land Management Handbook No. 42. British Columbia Ministry of Forests, Research Branch, Victoria, British Columbia.
