Topic 2: How can we account for the loss of information in indicator aggregation (i.e. losing the intent of the data from averaging)?

How can we account for the loss of information in indicator aggregation (i.e. losing the intent of the data from averaging)?

My initial reaction to this question is to suggest other mathematical and statistical tools that may help illuminate the shape of the data of an indicator within a region, population, or other boundary condition.  I am by no means an expert in this field, so my suggestions are limited to things like medians and modes that could, in the case of data whose average is not a strong representative of its shape and spread, help convey a better picture.  However, I am sure there are other formulas that may be more fitting.

However, this topic raises the issue of why having a more fully formed picture of the data is important.  From a broad perspective, averages are a quick and easily accessible tool for understanding overarching trends and bringing two indicators into conversation with each other (e.g. how two different indicators have changed in a similar or different way over time).  Yet averages only work to compound the problem of overlooking underrepresented groups, which often exist as outliers.  This issue is particularly relevant to discussions about sustainability, as the welfare of the worst-off must be given distinct importance if humans are to be at the heart of sustainability science, where this monograph places them.  Disregarding those whose with the lowest quality of life would do a grave disservice to the entire field, potentially leaving some of its conclusions unfounded and unproductive.  Thus examining data through more methods than simple averages seems to be a necessary process for sustainability scientists.

I would love to hear others’ thoughts on this matter, and am wondering if those more mathematically inclined could offer stronger tools for understanding the true shape of data.

Previously Christina Nolfo wrote:

How can we account for the loss of information in indicator aggregation (i.e. losing the intent of the data from averaging)?

I agree with the comment above that this question goes beyond statistical reporting methods to the very heart of the trade-offs between breadth and depth that are implicit in almost any kind of research we do. Time, resource and computational constraints rarely allow us to achieve both depth of analysis and breadth of comparison across regions of the world. Averaging across many quantitative indicators is a powerful tool for achieving the latter but leaves us lacking with respect to the former. While looking back at the shape of the data may help us to recover some of what is lost in the process, I think it’s important to note that we must have some sense of what we are looking for lest the sheer volume of data overwhelm us.

This is where theory comes into play. We must reflect on the specific causal mechanisms through which we'd expect certain policies and characteristics to affect the indicators (and also vice versa - how defining these indicators might affect institutional incentives at various levels). In my view, this kind of intuition is unlikely to come from looking at macro-level data sets. It will come instead from a complementary form of research -- for example, qualitative in-depth analysis of specific cases. What we learn from these contextual analyses may help us to cope more effectively with the loss of information from aggregation.

I was intrigued by the brief discussion in the Parris and Kates (2003) article about “top-down” versus “bottom-up” indicator creation and would have appreciated a lengthier discussion about it. First, I was very curious about the choice of terminology. As someone interested in community development, "bottom-up" carries with it connotations that would be more appropriately used to describe something like the Boston Indicators Project, which (if I’ve understood correctly) attempted to be inclusive of diverse stakeholder viewpoints and objectives. Here it is defined instead as an inductive approach to indicator construction, with the State Failure Task Force as the lone example.

Putting aside the semantics for a moment, this inductive approach (using more in-depth analysis of specific examples to inform the creation of predictive indicators) is analogous to the type of qualitative in-depth research to which I refer above. Since this approach incorporates in-depth knowledge of particular cases into a theory that actually predicts successes (or failures) in achieving sustainability, it seems to me that it attenuates somewhat the problem of information loss through aggregation.

As researchers, we should be careful that our work informs the creation of quantitative indicators at least as much as the indicators inform our work. To avoid methodological tunnel vision, I think that we as sustainability scientists need to engage constantly in this kind of dialogue between qualitative and quantitative, micro- and macro-level research. To me, this is equally as important as the cross-disciplinary dialogue that we discussed last week.

Previously Christina Nolfo wrote:

How can we account for the loss of information in indicator aggregation (i.e. losing the intent of the data from averaging)?