Benchmarks, not Rankings, Have Value in Policymaking

Everywhere you turn these days, some magazine, newspaper, or think tank is publishing the “Top 10” list of something. In the economic development community, these lists are both a source of bragging rights (if your state or community comes out on top) or an opportunity to try to explain away the results. Our politicians also seize on these rankings to make headlines or make points, often without really finding out what’s behind the headlines.

A case is in point are the recent Forbes rankings on Best State for Business. Here in Maine, we came in dead last, again. This isn’t a one-time deal, it has happened quite a few years in a row. Predictably, the Governor, who loved to quote Forbes during his campaign three years ago, now says that they don’t understand what’s happening in Maine.

From my point of view, these rankings are not worth the paper they are printed on, or the electrons that bring them to us over the web. A distinguished researcher named Dr. Peter Fisher recently studied these business climate rankings and concluded:

“When we scrutinized the business climate methodologies, we found profound and elementary errors. We found effects presented as causes. We found factors that have no empirically proven relationship to economic growth. Given these underlying flaws, it is no surprise that the rankings wildly contradict each other and fail to predict which states’ economies will thrive.”

To develop a ranking, one would normally chose a list of variables that you think are relevant to the objective, and decide how to weigh each variable and how to combine them, if at all. It turns out that each of these decisions is somewhat arbitrary, and can be used to change the outcome of the rankings. This is why many economists find that the rankings have no predictive value.

On the other hand, benchmark studies are different. These studies are usually designed to look at a single economic unit, like a state or a metropolitan area, and track the value of a variety of relevant indicators over time. Examples are, when looking at entrepreneurship, for instance, the number of new business starts, the amount of venture capital financing, and the number of initial public offerings. In some benchmark studies, other locations are used for comparison. If properly done, the other locations are matched along size, demographics, and economic dimensions to eliminate other explanations of any observed differences.

Examples are the Maine Innovation Index, the Portland (ME)’s Economic Scorecard, the Index of the Massachusetts Innovation Economy, the Rhode Island Innovation Index and the Silicon Valley Index (full disclosure – my friend and colleague, Jim Damicis, and I have collaborated on some of these.)

The benefits of benchmark studies are that they offer a framework for watching trends on key indicators in a state or metropolitan area over time, and an opportunity to see if policies are helping or hindering the desired outcome. They also allow a place to look at other, similar places, to set their aspirations higher, or to see how to improve.

Unlike rankings, benchmarks are actionable, and the results of actions can be observed over time. This makes them useful tools for policymakers who want to know if their investments and decisions are having the desired effect.

Posted in Uncategorized and tagged with , , , , . RSS 2.0 feed.

One Response to Benchmarks, not Rankings, Have Value in Policymaking

  1. Scott says:

    It seems to me, that we have done enough studies. This is not about benchmarks, it is about action. We have failed to act on the Brookings Institute studies, we glaze over the MDF Measures of Growth and make excuses why we can’t do better. Maine is driven by outside forces and until we can control some of these forces, we will be at the mercy of them.

    “From my point of view, these rankings are not worth the paper they are printed on” – my sentiments exactly.

    We need action, not more measurements.

    sorry for the rant.

Leave a Reply

Your email address will not be published. Required fields are marked *