2.2 Mechanism Monitoring Design
2.2.2 Choosing an appropriate design for understanding mechanisms
Overview Choosing a spatial and temporal design Results and next steps
The questions on this page will help you determine the relative merits of various spatial and temporal designs of monitoring programs that aim to learn about causal mechanisms. If you are unsure how to answer any question, click on the "Help me decide" button below it.
As is described in Step 1 under "Mechanisms, Basic Questions", users of this web site need to be able to comparing data from different situations to estimate the relative contributions of different possible mechanisms to observed changes in salmon indicators. These situations must have contrasting levels of exposure to different potential causal mechanisms through before/after comparisons, spatial differences in those exposures, or ideally both. Thus, this current section on choosing a design helps users by examining designs that simultaneously use various combinations of spatial and temporal groupings.
The set of questions below will help steer you toward an appropriate set of potential monitoring designs; different users will likely face different situations. Some may have either directly observed or reconstructed historical data for both prior to and after a change in some potential driving factor that influences status of a salmon population, whereas others may only have the "after" data. Similarly, some situations will enable identification of appropriate "reference sites" or "reference periods" that can be compared with other times and places, whereas other situations cannot lead to such comparisons. Our questions will help you work through your situation.
Here is the logical branching tree created by your answers to the series of questions below:
You will encounter three terms that are synonymous: "covariate", "explanatory variable", and "independent variable". These terms all refer to a variable, such as habitat quality or quantity, that enters into a statistical model, such as a multiple regression, to help explain the observed variation in the dependent variable. Such explanatory variables are chosen to reflect potential mechanisms causing changes in one or more salmon indicators.
Questions
Your best monitoring design
Based on your answers above, the best monitoring option for your situation is:
Please answer the three questions above to have your result appear here.
Caution! This "best" design shown on the line above is only a starting point for comparison with other options. Each design has pros and cons that you must weigh. Furthermore, there are many logistical and financial constraints that may preclude using the above "best" design. Nevertheless, if you know what type of design is theoretically best for your situation, you will at least be primed to watch for ways in which you could modify the situation to permit use of a better design.
Discussion of the best monitoring design
There are eight possible outcomes from answering the three questions above. Each outcome is associated with one or more types of spatial and temporal monitoring designs (often, but not always, large-scale designs). Many researchers have explored the characteristics of this wide range of monitoring/sampling designs in ecological systems, and monitoring practitioners have also gained extensive field experience with these designs. This web site synthesizes that research and practical experience.
In the next section, inside the expanding text box that appears below a category of best possible type of design, we briefly summarize some of the fundamental features of the main types of designs that could be used to understand mechanisms that affect salmon. We also describe the main pros and cons of each category of design in those text boxes as well as in a figure at the bottom of this page. We use the following criteria to characterize the pros and cons of different designs:
- Degree of confidence in conclusions about mechanisms behind observed changes
- Extent of area across which conclusions can be extrapolated (universe of inference)
- Costs of monitoring
- Length of time before answers will be obtained
- Complexity of the design
- Number of opportunities or situations in which a design can be applied
- Chance of taking INeffective management actions to mitigate effects of causal mechanisms due to weak conclusions about those mechanisms.
We encourage users to seek more information from the following sources: Stewart-Oaten et al. (1986), Schmitt and Osenberg (1996a), Schwarz (1998), Downes et al. (2002), Roni (2005).
Category # Best possible type of design
1 Substitution of space for time
- Definition, pros, and cons
-
Category #1, Substitution of space for time: This situation results from answering "no" to Question 1 as well as "yes" to both questions 2 and 3. That is, you unfortunately do not have data on the before-impact situation for some mechanism that potentially affects salmon. However, it is known when and where that mechanism has already had its effect (and may still be doing so). Thus, in this situation, the effect of that mechanism could be inferred from the spatial and/or temporal pattern of salmon responses (known as substituting space for time). For instance, in the case of spatial patterns, comparison groups could be created from reference sites where the mechanism was known to operate and locations where it did not operate. Alternatively, there could be a set of locations with different intensities for that mechanism (e.g., low, medium, and high silt loads). In the case in which all locations have the same intensity of the mechanism and there are no appropriate reference sites, then temporal patterns of variation in salmon indicators and explanatory variables could be examined by fitting models that include explanatory variables to the time series data.
2 Baseline monitoring (passive, observational)
- Definition,
pros, and cons
-
Category #2, Baseline monitoring: This situation results from answering "no" to Question 1, "yes" to Question 2, and "no" to Question 3. That is, you unfortunately do not have data on the before-impact situation for some mechanism that potentially affects salmon. Even though it is known when and where that mechanism has already had its effect (and may still be doing so), there are no adequate reference sites or time-varying data on covariates related to mechanisms. Thus, you cannot create any comparison groups. Therefore, this situation cannot lead to any conclusion about the relative importance of a mechanism's effect on salmon. At best, data collected by the resulting passive baseline monitoring program may stimulate new hypotheses about mechanisms if comparison groups can be found in the future, or if fortuitous situations arise such as with the example of the collection of atmospheric CO2 data on Mauna Loa starting in the late 1950s, which contributed to the understanding of causes of global warming.
3 Baseline monitoring (passive, observational)
- Definition,
pros, and cons
-
Categories #3, 4, 7, and 8, Baseline monitoring: These situations result from answering "no" to Question 2. Without knowing where and when the mechanism will operate (or has already acted), then you cannot create any comparison groups that will identify its effects separately from the effects of any other process, either natural or human. Thus, this situation cannot lead to any conclusion about the relative importance of a mechanism's effect on salmon. At best, data collected by the resulting passive baseline monitoring program may stimulate new hypotheses about mechanisms if comparison groups can be found in the future, or if fortuitous situations arise such as with the example of the collection of atmospheric CO2 data on Mauna Loa starting in the late 1950s, which contributed to the understanding of causes of global warming.
4 Baseline monitoring (passive, observational)
- Definition,
pros, and cons
-
Categories #3, 4, 7, and 8, Baseline monitoring: These situations result from answering "no" to Question 2. Without knowing where and when the mechanism will operate (or has already acted), then you cannot create any comparison groups that will identify its effects separately from the effects of any other process, either natural or human. Thus, this situation cannot lead to any conclusion about the relative importance of a mechanism's effect on salmon. At best, data collected by the resulting passive baseline monitoring program may stimulate new hypotheses about mechanisms if comparison groups can be found in the future, or if fortuitous situations arise such as with the example of the collection of atmospheric CO2 data on Mauna Loa starting in the late 1950s, which contributed to the understanding of causes of global warming.
5 Staircase, but other designs are also possible, listed below
in order from the most preferred down to the least preferred:
- Definition,
pros, and cons
-
Category #5a, Staircase design: This situation results from answering "yes" to all three questions. In many senses, it is the most ideal situation for trying to understand causal mechanisms. One reason for its ideal nature is that the staircase design deals with a common cause of confounding. Specifically, it is surprising how often unexpected changes that are out of our control occur at about the same time as some management action or natural event of interest also changes, thereby creating a confounding of interpretation of the cause of any observed changes in salmon. For instance, think about all the times when a tantalizing signal of some response to a management action like stream restoration coincides with favorable (or unfavorable) changes in water flow, water temperature, disease conditions, harvest rate, invasion of predators, and so on. Such confounding precludes making conclusions about the effectiveness of your restoration actions. All designs that involve before-after comparisons are subject to this confounding to some extent.
To reduce this problem of time-treatment interactions, Walters et al. (1988) developed a staircase design. Its essential feature is that the multiple "treatment" sites in the study are split up into two or more groups and the time at which some treatment begins is different for each of those groups. This staggering across groups of the time that delineates the before-after comparisons severely reduces (but cannot completely eliminate) the chance that some confounding variable will emerge because it would have to change at the same time as each treatment started in each group of sites. This design also permits estimation of what the authors called the transient (i.e., time varying) response to disturbance or human activities.
- Multiple before-after-control-impact paired series (MBACIPS)
- Definition,
pros, and cons
-
Category #5b, Multiple before-after control-impact (MBACI) design: With multiple sites in space that are sampled more or less simultaneously, there will be one or more sampling units with which you can compare results with the unit(s) in which the mechanism is potentially operating at different levels. This type of design is called a multiple before-after control-impact (MBACI) design (Downes et al. 2002). Such designs can lead to a relatively strong conclusion about the relative importance of a mechanism's effect on salmon.
- Before-after-control-impact paired series (BACIPS)
- Definition,
pros, and cons
-
Category #5c, Before-after control-impact paired series (BACIPS) design: With multiple sampling sites in space as well as multiple sampling times at each site, there will be one or more sampling units with which you can compare results with the unit(s) in which the mechanism is potentially operating at different levels. This type of design with several time periods at a given site is called the before-after control-impact paired series (BACIPS) design (Underwood 1991). This is only one of several enhanced designs suggested by Underwood (1991, 1992) and Bence et al. (1996) to increase the power to distinguish among alternative mechanisms causing observed changes. Such enhanced designs focus on estimating the time series of mean differences between control and impact sites and comparing those differences before and after the mechanism starts operating. These enhanced BACI designs can lead to a relatively strong conclusion about the relative importance of a mechanism's effect on salmon.
- Before-after-control-impact (BACI)
- Definition,
pros, and cons
-
Category #5d, Before-After-Control-Impact (BACI): If in addition to having before-after data, you know where and when a postulated mechanism will operate, you can create a before-after, control-impact (BACI) design for your comparisons.
For example, if some regions are affected by a purported mechanism and others are not (e.g., logging), and you also know where and when logging will occur (to create a before/after, control/impact design for comparison -- BACI), then stronger conclusions are possible than with a simple before-after design. However, even here, you cannot conclude that the purported mechanism caused the observed change unless you can rule out the existence of all other factors that could have caused it. Just by chance, some inherent differences between "before" control and impact sites might lead to the wrong conclusion.
- Before-after (BA)
- Definition,
pros, and cons
-
Category #5e, Before-After (BA) design: You have the opportunity to either monitor before the mechanism begins operating or reconstruct data for that "before" situation. Thus, you can create a before-after (BA) design for making comparisons. In this case, you can only infer causal mechanisms from temporal differences that exist in the "before" and "after" periods. Such a BA design cannot lead to any clear conclusion about the purported mechanism causing the observed temporal change because other simultaneously operating factors could have caused that change instead. This classic situation of confounding effects is the main disadvantage of the before-after design.
6 Before-after (BA)
- Definition,
pros, and cons
-
-
Category #6, Before-After (BA) design: This situation results from answering "yes" to questions 1 and 2, but "no" to question 3. That is, either by direct observation or through reconstruction of past situations, you have data on the before-impact situation for some mechanism that potentially affects salmon. As well, the when and where of the operation of that mechanism are known. However, conclusions will be limited to a before-after (BA) comparison if there are no adequate reference sites or time-varying data on covariates related to mechanisms; such time variation must occur in both the "before" and "after" periods.
- In this case, you can only infer causal mechanisms from temporal
changes. That is, you have the opportunity to monitor before the
mechanism begins operating as well as after. If all monitoring sites are
equally exposed to the purported mechanism (e.g., from broad-scale
climatic drivers), then you only have a before-after design. Such a BA
design cannot lead to any clear conclusion about the purported mechanism
causing the observed temporal change because other simultaneously
operating factors could have caused it instead. A classic example of
this is the Salmonid Enhancement Program in British Columbia, which
dramatically increased the number of released juvenile salmon through
hatcheries and artificial spawning channels starting in 1976. However, this
perturbation coincided with the 1976/77 regime shift in the Northeastern
Pacific Ocean, which drastically increased the productivity of the
lower parts of the food chain. It is therefore not possible to say
anything about the success of the enhancement program from simply comparing
abundance of adult returns alone before and after 1976.
- In this case, you can only infer causal mechanisms from temporal
changes. That is, you have the opportunity to monitor before the
mechanism begins operating as well as after. If all monitoring sites are
equally exposed to the purported mechanism (e.g., from broad-scale
climatic drivers), then you only have a before-after design. Such a BA
design cannot lead to any clear conclusion about the purported mechanism
causing the observed temporal change because other simultaneously
operating factors could have caused it instead. A classic example of
this is the Salmonid Enhancement Program in British Columbia, which
dramatically increased the number of released juvenile salmon through
hatcheries and artificial spawning channels starting in 1976. However, this
perturbation coincided with the 1976/77 regime shift in the Northeastern
Pacific Ocean, which drastically increased the productivity of the
lower parts of the food chain. It is therefore not possible to say
anything about the success of the enhancement program from simply comparing
abundance of adult returns alone before and after 1976.
7 Baseline monitoring (passive, observational)
- Definition,
pros, and cons
-
Categories #3, 4, 7, and 8, Baseline monitoring: These situations result from answering "no" to Question 2. Without knowing where and when the mechanism will operate (or has already acted), then you cannot create any comparison groups that will identify its effects separately from the effects of any other process, either natural or human. Thus, this situation cannot lead to any conclusion about the relative importance of a mechanism's effect on salmon. At best, data collected by the resulting passive baseline monitoring program may stimulate new hypotheses about mechanisms if comparison groups can be found in the future, or if fortuitous situations arise such as with the example of the collection of atmospheric CO2 data on Mauna Loa starting in the late 1950s, which contributed to the understanding of causes of global warming.
8 Baseline monitoring (passive, observational)
- Definition,
pros, and cons
-
Categories #3, 4, 7, and 8, Baseline monitoring: These situations result from answering "no" to Question 2. Without knowing where and when the mechanism will operate (or has already acted), then you cannot create any comparison groups that will identify its effects separately from the effects of any other process, either natural or human. Thus, this situation cannot lead to any conclusion about the relative importance of a mechanism's effect on salmon. At best, data collected by the resulting passive baseline monitoring program may stimulate new hypotheses about mechanisms if comparison groups can be found in the future, or if fortuitous situations arise such as with the example of the collection of atmospheric CO2 data on Mauna Loa starting in the late 1950s, which contributed to the understanding of causes of global warming.
Graphical summary of pros and cons
Here is a summary of some key pros and cons of the designs above in the context of attempting to learn more about mechanisms that cause changes in salmon indicators.
Note that data gathered by applying any of these designs need to be interpreted carefully and according to appropriate statistical assumptions. This topic is covered in Step 6 (Interpretation and analysis of data).
Next: Results and next steps| Go Back
