Assigning quality scores for each refuge

Rationale

As you sort through the refuge data, you will find that some refuges clearly provide better data than others.  Some refuges have formal invasive species monitoring programs and may even know the precise locations of most invasive plant populations.  Other refuges have responded to the survey, but the invasive list they provide is based just on a handful of anecdotal observations.

Because datasets vary in quality, it is important to be able to weight the refuges differently in our analyses.  This is, we want to rely more on datasets that we believe to be good and less on datasets that seem to be poor.  So, we need to have a scheme for weighting data quality based on what we know about the data.

Short of going out and actually surveying plants at all the refuges, there is no foolproof way to measure data quality.  For example, a refuge may have entered very detailed and professional-looking data, but they may have entered these data incorrectly. Nevertheless, it’s important to try to come up with a metric that assesses the relative reliability of the different datasets based on what we know about them.

Below I’ve outlined the weighting scheme that will be used for the refuges.  Essentially, you just look through the information provided for each refuge in the online survey and assign points as dictated by this scheme.  An affirmative response (“true”) to each of these questions gets the number of points indicated in the parentheses.  Scores range from 0 to 24, though a refuge with 0 wouldn’t have any data to include in our analysis, and you’re unlikely to see any refuges with weights that approach 24.  Most refuges will good data will score between 12-18 and most with poor data will be in the 2-5 range.  A refuge with a score of 15 would effectively count 5 times as much as a refuge with a score of 3 in our analysis.  Please note that you take points for either #6 or #8 (but not both), and you can take points for either #9 or #10 (but not both). 

Weighting scheme

 (items 1-7 from the data in the invasive species survey)

1. Refuge has a terrestrial plant list: (1)
2. Refuge has an aquatic plant list (1)
3. Refuge has a vegetation map: (3)
4. Invasive Species source populations have been identified: (1)
5. Refuge maintains invasive species monitoring program (3)
6. (If refuge maintains a list of non-natives), Estimated completeness of this list: 0-33% (1), 33-66% (3), 67-100% (5)
7. The Refuge has conducted the following types of non-native plant species surveys: 

             observation: (0)

             base line inventory: (2)

             experiments: (1)

             systematic: (3)

             multiscale: (2)

             photograph: (1)

(items 8-10 from info sent by managers or on refuge website)

8. Manager sends you a list of problem invasive species (5), score counts instead of item #6 above

9.  Manger sends you a plant species list for the refuge (3)

10.  A plant species list is available on the website (4)

(points for 9 or 10 but not both)

 Max score: 24

Notes/extras

Note that most items just get one point each.  However, a few items are particularly important.  For example, if someone specifies that their refuge’s invasive species list is 80% complete, that’s worth 5 points alone.  If the refuge conducts systematic surveys, that’s 3 points.  One could quibble with the weighting scheme but the goal is really just to differentiate the good datasets from the poor ones.

If you don’t like adding here's an excel spreadsheet that will do the adding for you.