By Kjell Christophersen, Tim Nadreau, and Aaron Olanie
EMSI economists
Download a PDF version of this article
1. Introduction
Economic impact studies are carried out for thousands of colleges, universities, workforce boards, and every other type of enterprise each year. The quality of many of these studies, however, is often questionable because so many lead to overstated impact measures.
The focus of this article is to highlight what constitutes a true impact study and what does not. Nobody wants to allocate millions of dollars to a project and then find out after the fact that the report they used to justify the investment was deeply flawed. There is considerable political risk associated with such activity, as documented in the Reinhart and Rogoff study analyzing debt, growth, and austerity measures. When economists make mistakes and policy is informed by those mistakes, it often means increased inefficiency and increased unemployment. While no model is perfect, we strive to stay on the cutting edge by following best practices. We owe this to our clients, and they, in turn, need to have a firm grasp of what they’re buying.
We have come a long way since we unveiled the first version of our college economic impact model in 2000. The Association of Community College Trustees (ACCT) contracted with us to build an initial version of the model for seven pilot colleges. This effort led to the launch of the first full-fledged EMSI economic impact model. Our scope of work from ACCT was to 1) build a generic model that could handle the inputs from any college and analyze it in a standardized fashion, 2) make that generic model as comprehensive as possible, capturing things like social metrics and increased productivity stemming from education, and 3) to include an investment analysis identifying the return government and students received as a result of the education provided by the college.
Although economic impact studies for colleges and universities have been conducted for decades, there still exist significant differences in the approaches taken.
Did we successfully capture this vision? Yes, but just as Ford’s Model-T captured a vision, compared to the cars on the road today, it certainly wasn’t perfect. When EMSI launched its first impact model, it was at the Model-T stage. The model was state-of-the-art then and it worked well. But the results generated were certainly not as well anchored in mainstream theory as we would have liked. In subsequent years we have rebuilt, redesigned, increased the accuracy and tolerances of our assumptions, and improved our methodology with up-to-date data, new literature, etc. All of this is to say that we have been conforming to best practices and have moved the frontier of best practices forward along the way.
Although economic impact studies for colleges and universities have been conducted for decades, there still exist significant differences in the approaches taken. There appears to be a great deal of confusion regarding what exactly constitutes an economic impact. Some studies measure true impacts, while others don’t. These studies often use similar terminology throughout, though the underlying methodology is distinct, which exacerbates the confusion.
Time and time again a new college will approach us about conducting an impact study and after receiving their draft reports they ask for clarification, telling us their similarly sized neighbor college across the way had an impact study done and their results were larger than ours. How can that be? Or perhaps they had a different firm conduct a study for them two years ago and they have grown substantially since then so how could the impact be smaller when measured by us? The answer is often that the other studies were not impact studies at all.
To clear the air, this article will walk through some of the various types of studies, discussing their strengths and weaknesses and why some studies are preferred to others. After the types of studies are defined, we will discuss and define the terminology used in these studies, specifically focusing on the counterfactuals that must be accounted for in a true impact analysis. This should serve as the “what to look for” checklist when deciding what type of study you need, or whether the study you received was actually what you paid for.
Our aim is to provide comfort for our higher education clients who regularly contract with us for economic impact studies of their institutions. We want them to fully understand what constitutes economic impacts – i.e., what the colleges add to the economic well-being of the regions they serve – as opposed to other measures billed as impact studies, but which in reality are not.
2. Types of studies
Broadly speaking, studies claiming to estimate economic impacts generally fall into three categories: studies estimating contributions, gross regional products, or true economic impacts.
Contributions
The most common type of economic activity study is a contributions analysis. Here, any and every dollar that touches a specific industry or firm is measured along with the associated multiplier effect. Many colleges and universities receive contributions studies labeled as economic impact studies. Two ramifications of this mislabeling warrant mention:
1) In any region there is significant overlap between colleges, other industries, and firms. If a contributions analysis were conducted for every firm in the region, the sum of these contributions would far exceed the true volume of dollars circulating in the regional economy. As such, a contributions study may be thought of as the interconnectedness of industries or firms within the region.
2) A contributions study says nothing about the opportunity cost or crowding out effects taking place as money is allocated between alternative and competing ends. For example, consider a contributions analysis of a college that receives public funding. The analysis says nothing of the fact that if the college did not exist, the public money may have been allocated to another source that generates its own contributions. Extending our example further, what if the public money had not been collected through taxes to begin with? Households and consumers would have increased disposable income, a portion of which would have been spent in the regional economy, in turn generating contributions. To spend money in one industry, possible spending in other industries must be sacrificed. Contribution studies do not consider the sacrificed alternative spending.
Taking stock, a contributions analysis labeled as an impact study does two things: 1) it vastly overstates the true impacts of a college or any industry; and 2) it doesn’t account for the lack of growth in other industries. A contributions analysis will not measure the growth in the region’s economic base.
Gross Regional Product (GRP)
GRP analysis examines the fraction of total value added in a region that stems from a particular industry or firm. One can think of value added as the increased value an industry or firm produces through its restructuring and mixing of the inputs used in its production process. In a bakery, flour, milk, and yeast have less value on their own than when they are mixed together and baked into bread. The additional value that the bread has over and above its intermediate inputs is the value added generated by the bakery.
Conducting a GRP analysis of every firm in the region would sum to the total GRP of the region. It is therefore a smaller subset of the broader contributions analysis. Notice, however, that this analysis still does not answer the question of how much of that value added is solely attributable to the bakery, or how much money would be lost to the economy if the bakery had never existed. This analysis still suffers from the failure to account for the opportunity cost and crowding-out effects, though perhaps not to the same extent that the contributions analysis does.
True Economic Impacts
Impact analysis is the narrowest of all economic activity analyses. A true impact as defined by Watson et al. is “the net change to the economic base of a region that would not otherwise be there without the industry or firm under analysis.”
The key word in this definition is net. An impact analysis needs to take account of: 1) only the moneys brought into the region by the college operations, or 2) the local dollars retained in the region that would otherwise be lost in absence of the college operations. These measures are referred to as export base and import substitution effects, respectively.
Local dollars received, and then spent, by the college are simply a reshuffling of local resources. Rearranging the furniture is not an impact. If a study includes local revenues in their study, the reader should be wary and ensure that the local dollars claimed are heavily discounted through the use of counterfactual measures (see below). Watson et al. define impacts as
“…the value added portion of revenues that are generated by an industry from sales outside the region (exports), the value added portion of sales to visitors that come to the region (tourist spending), and the value added from locals who would have spent their money on goods from outside the region had the industry [firm] not been present inside the region (import substitution).”
EMSI’s economic impact model claims an additional impact metric for educational institutions since there is a difference between college outputs and firms such as lumber mills, for example. Where a lumber mill produces a physical product or an accounting firm provides a particular service, educational institutions increases the productivity of workers, making the firms hiring those workers more efficient and enjoying higher output and profitability. As such, the impacts associated with education should rightly include the increased output of students.
More on this later, as there is significant discounting of the impacts that must be accounted for to ensure an accurate reflection of the productivity of students as they enter the workforce.
What Watson’s definition shows is that impacts are a subset of GRP. Thus the three types of economic activity studies are nested within one another. Contributions are the largest, with GRP being the next smallest, and impacts being the smallest as shown in the figure below.
The natural question arises, “When advocating for a college, wouldn’t you just want the biggest results?” Our answer is no. If all you want is the largest possible numbers, contract for an advocacy study that shows the college’s contributions. Beware, however, more often than not, these enlarged results are misrepresented and misunderstood, resulting in the exact opposite of the initial intent of the study as a communication and advocacy report. This is particularly true in these times with the increased financial scrutiny over the returns to education.
If, on the other hand, the college wants a solid and defensible study that will enhance rather than detract from the community engagement efforts, the answer is to contract for a true impact study. The numbers might be smaller, but the value is much higher!
In the next several sections we will discuss the counterfactuals necessary to accurately estimate the impact of a college or university. These are the adjustments made to the college revenues and expenses that ensure that the end results are expressed as true impacts, not contributions or GRP measures.
3. Counterfactuals
Economic impact analysis takes into account the opportunity costs or crowding-out effects not considered in a contributions or GRP analysis. These are considered using a number of counterfactuals. Three counterfactuals must be recognized: 1) the alternative education effect, 2) the substitution effect, and 3) the alternative use of funds.
Alternative education
How many students would be able to avail themselves of an alternative education if the college were to shut down? Obviously, absence of the college does not mean that education would effectively be barred for all students in the region. Some students would find a way to get a similar education regardless. The benefits that these students generate are not impacts. Recall an impact analysis measures what the economy would lose if the college were to close. Only the benefits generated by students who would otherwise be unable to procure an alternative education should be claimed. The alternative education variable measures this portion of students. This will vary considerably across colleges. For example, a college in an urban area with many peer institutions will have a higher alternative education variable than a rural college where the nearest peer institution is an hour away.
This counterfactual is almost never addressed in our competitors’ studies, even though it is strongly recommended in the 2006 working paper by economists at Vanderbilt University.
Substitution effects
In analyzing the students’ increased productivity in the region we must explicitly recognize the fact that if a college’s students were not receiving their education and entering the workforce with specific skills, the firms that need workers with these specific skills would ultimately recruit them from outside the service area. Thus, not all the productivity effects would have been lost to the region if the college were to close its doors.
A number of complexities determine the magnitude of the substitution effect. Does the region have the requisite amenities to be attractive to educated labor? Does the local firm have the resources to pay higher wages to induce a prospective employee to move? It would be far more difficult for a business in Detroit, now in bankruptcy, to attract new employees than a business in New York City. These types of questions could consume an economist’s career. Most studies side-step the issue by ignoring the effect all together.
Alternative use of funds
Most studies claim all the college spending and then use some predefined multiplier to calculate the total impact. Usually this looks something like “for every $1.00 spent by the college the economy grows by $1.45.” The problem with this structure is that large portions of the college’s revenues are derived from local sources. These dollars and the associated multiplier effects should not be claimed as impacts since those dollars already existed in the economy. If those local dollars had not been spent at the college, they would have been spent elsewhere in the economy. As mentioned before, the college is crowding out spending in other industries. We adjust for this in what we call the alternative use of funds.
The reality is that spending at the college, a service-based industry, will generate more income than the spending on say groceries. Taking the total of all college spending plus multipliers (as most studies do) and then subtracting the “alternative use of funds” yields the total local spending impact, which we will then reduce to local income.
All told, the real argument here is that a rearranging of the furniture does not actually add to the economy. The new dollars from outside the region are injections into the economy, i.e., new furniture. The import substitution argument is the “local” furniture that would disappear if the college closed. Standard media practice is to discuss this as “created or saved” usually in the context of jobs or businesses.
4. Sales vs. Income
In the EMSI model we express all of the impact results in terms of earnings, not sales, for good reason. Consider the following example:
Two visitors spend $50,000 each in the economic region. One visits a local auto dealer and purchases a new luxury automobile. The other undergoes a medical procedure at the local hospital. In terms of direct economic impact, both have spent $50,000 and the auto dealer and hospital have made sales worth $50,000 each. However, the sales have very different meanings to the local economy.
The kinds of economic activity occurring in any region matter. All industries will exert different impacts on the overall regional economy and models must capture these nuances to be labeled credible.
Of the $50,000 spent for the luxury automobile, perhaps $10,000 remains in the county as salesperson commissions and auto dealer income (part of the economic region’s overall earnings), while the remaining $40,000 leaves the area for Detroit or somewhere else as wholesale payment for the new automobile.
Contrast this to the hospital expenditure. Here perhaps $40,000 appears as physician, nurse, and assorted hospital employee wages (part of the county’s overall earnings), while only $10,000 leaves the area, to pay for hospital supplies, or to help amortize building and equipment loans. In terms of sales, both have the same impact, while in terms of earnings, the former has one-fourth the impact of the latter.
The kinds of economic activity occurring in any region matter. All industries will exert different impacts on the overall regional economy and models must capture these nuances to be labeled credible. In the example above, the sales and income figures are far closer in the medical profession than in the auto industry. Thus, it may be OK to use a sales approach if analyzing a service-based industry with relatively low capital requirements, e.g., law offices, accounting firms, hospitals, and schools. However, if the sales approach is being conducted in a legitimate fashion, the benefits of doing so will be small since the results will be almost identical to the income figures.
The sales numbers are more impressive, especially when applied to an industry where they must be taken with a grain of salt. If 80% of your sales numbers leak out of the economy, they cannot be counted as impacts attributable to the company. They are not impacts at all since those dollars are not even in the economy.
Also important to note with the sales vs. income distinction is that as the region becomes larger and larger the sales and income figures become closer and closer, though they will only ever equal each other at the global level. EMSI always reports on the income basis to avoid overstating the impacts. One could conduct an impact analysis at a national level on a sales basis and doing so would not raise any eyebrows among economists, but doing so at the zip-code or county level would be unjustifiable.
The fundamental reason we use income rather than sales is this: Sales may be bringing money into the economy, but until that money start rippling through the economy, it has no impact. The allocation of the sales dollars to employees in the form of income is the beginning of the ripple effects. The reader should be on guard if sales numbers are being reported. If the region is sufficiently large, such as a state or very large metropolitan area, and the industry of analysis is service-based, then the results will not be overstated to a degree that would render the study fraudulent.
5. Conclusion
Impacts are the narrowest measure of economic activity – i.e., the new economic activity solely attributable to the institution that would otherwise not exist. EMSI conducts impact studies. When the goal is to accurately articulate the economic value of a higher education institution, it is our belief that an impact study has advantages over contributions and GRP studies. The key difference is that impacts are net measures, whereas contributions and GRP are not. Unfortunately, the term impact is used loosely throughout the industry and contributions and GRP studies often claim to measure impacts. In our impact studies, we take into account a number of counterfactuals ignored by contributions and GRP studies. We encourage our readers to question the robustness of any study claiming to measure impacts when these counterfactuals are ignored. Additionally, we emphasize the importance that all impacts are measured in earnings or income, rather than sales.
Illustration by Monique Harby
If you have questions or would like to learn more, please contact us.