A massive fail on ROI calculations for community colleges

Written by: Donald Heller

Primary Source: The Dean’s Blog

Last spring, I posted about the challenge of calculating a return on investment (ROI) for individual colleges.  The problem is with obtaining accurate data on the earnings of the graduates of a given college or university.  I described there the problem with a ROI website created by the firm PayScale, which relies on self-reported data from individuals.  There are a number of issues with PayScale’s methodology, but the biggest is that there is just no way to know how representative the respondents to their surveys are of the totality of the graduates for any single institution.   There is also no way to judge the accuracy of the data reported.

Now there is another report out that uses the PayScale data to calculate ROI measures for community colleges. Last week, a study issued jointly by the Nexus Research and Policy Center and the American Institutes for Research (AIR) took on the challenge of calculating the returns to associate’s degrees not just for individuals, but also for taxpayers in the form of the government subsidy provided to community colleges.  There are a number of methodological problems with this study, which examined 579 community colleges around the country, but key among them is the study’s reliance on the PayScale data as the outcome measure for the ROI calculations.  AIR is also the author of the College Measures website which, as I described in last spring’s post, shares the same flaw in using the PayScale data for some of its analyses.

The Nexus/AIR report is not just an academic exercise; President Obama’s recent proposal for holding colleges and universities accountable for their use of federal funds talks about the use of similar measures, as I wrote about in August.  Proposals to tie eligibility for Title IV federal financial funds to how colleges’ graduates perform in labor markets is a serious threat to hold over the institutions.

After the Nexus/AIR report was issued, I was approached by The Chronicle of Higher Education to write an op-ed responding to the report.  You can read the full essay,  but here is a preview from the last two paragraphs:

To the authors’ credit, the report’s introduction notes the limitations of their study, stating that “the data needed to fully understand what, on average, a degree is worth to a graduate or what the tax benefit of that degree is to taxpayers, is not currently available.”

But then the report goes on to do precisely what the authors say cannot be done – so precisely that they calculate exact dollar amounts (graduates of Colorado Northwestern Community College, for example, are expected to have an average net financial return of $850,903 over their lifetimes). The precision of the numbers reported in this study, based on data of such unknown validity, is careless at best and grossly misleading at worst. And the notion that this study can be used to compare community colleges is preposterous.

The following two tabs change content below.
Donald Heller
Donald E. Heller is Dean of the College of Education and a professor in the Department of Educational Administration at Michigan State University. Prior to his appointment in January, 2012, he was Director of the Center for the Study of Higher Education and professor of education and senior scientist at The Pennsylvania State University. He also has held a faculty appointment at the University of Michigan. His teaching and research is in the areas of educational economics, public policy, and finance, with a primary focus on issues of college access and choice for low-income and minority students. He has consulted on higher education policy issues with university systems and policymaking organizations in California, Colorado, Kansas, Massachusetts, Michigan, New Hampshire, Tennessee, Washington, Washington DC, and West Virginia, and has testified in front of Congressional committees, state legislatures, and in federal court cases as an expert witness. Before his academic career, he spent a decade as an information technology manager at the Massachusetts Institute of Technology.
Donald Heller

Latest posts by Donald Heller (see all)