Written by: Donald Heller
College rankings have become big business. There are numerous media and other organizations that have jumped in to create their own rankings, each with a unique methodology. In 2013 President Obama announced that the federal government would get into the college ratings business as well. After almost two years of effort, and the release of a preliminary framework for such a system last December, the U.S. Department of Education last month announced that it was abandoning the effort in favor of simply providing more data to consumers.
So when Money Magazine announced that it was implementing a new methodology for its rankings, I was curious and took a look. The Money rankings consist of three major components, each contributing one-third to an institution’s final ranking: 1) quality of education; 2) affordability; and 3) outcomes. Each component has four or five individual measures that are weighted into a component grade.
In the outcomes component, Money uses data from PayScale to calculate three different measures of the earnings of the graduates of each institution; these three measures together sum to 75 percent of the outcomes component, or almost 25 percent of the entire ranking (1/3 X 75% = 25%). PayScale is a website that aggregates users’ reports of salary information to calculate average earnings for colleges and universities across the country. While this concept may sound appealing, I have previously written (here and here) about problems with the PayScale data. But the Money rankings provide another opportunity to look at how these data are used, and the problem with utilizing them to try to rank or rate colleges and universities.
Not surprisingly, because it uses different criteria than other rankings, Money comes up with a list that looks different from many of the others. For example, here are the top ten institutions as ranked by Money:
1. Stanford University
2. Babson College
3. Princeton University
5. Cal Tech
6. Harvey Mudd College
6. Harvard University
8. Maine Maritime Academy
9. Amherst College
9. U. of California, Berkeley
The Money rankings show the average earnings for recent graduates (five or fewer years since graduation) of each institution. The school with the highest early career earnings average in the top ten, Harvey Mudd College at $76,400, is focused primarily on engineering degrees, so it is not surprising that it has such a high figure. Harvey Mudd’s average exceeded even those of its more well-known competitors, Harvard, Princeton, and Stanford Universities.
But by looking at each of these institutions on the PayScale website, you can see exactly how many self-reports of the graduates went into the calculation of the average earnings for each institution. For example, Harvey Mudd – tied for sixth in the Money rankings – has a total of 48 salary reports on the PayScale website. Even though Harvey Mudd is a small school, with an enrollment of about 750 undergraduates, 48 self-reports of earning information is not very many. And only 20 of these are listed as being from individuals with less than four years of work experience.
In an email conversation with the senior writer who is responsible for the Money rankings, Kim Clark, she reported that the magazine asked PayScale to provide data from its last three iterations of its college earnings calculations. So this means that the ranking for Harvey Mudd was based on perhaps 150 self-reports from Harvey Mudd grads, maybe 60 of whom were recent graduates. And there is no way to tell if any of these people submitted their data more than once across the three years.
For Amherst College, tied for 9th in the Money rankings, the numbers are even smaller. The PayScale website shows 31 reports for this college, with 12 of these in the category of four or fewer years of experience. So the average across its three years’ of data would be about 36 or so individuals whose data were used to calculate the institution’s early career earnings of $55,700 as shown in the Money rankings.
The takeaway from this information is that any of the earnings data gleaned from the PayScale website, and the rankings on which those data are based, should be taken with a large grain of salt. The average earnings for any given institution may be based on a very small number of reports, and there is no way to verify whether those reports are truly representative of all of the graduates of that school. As the methodology section of the Money rankings admit, the PayScale data are, “the only game in town for anyone who wants to compare colleges’ outcomes nationwide.” But being the “only game in town” does not necessarily make the data accurate. – See more at: http://edwp.educ.msu.edu/dean/2015/rankings-galore/#sthash.Ffq1iqon.dpuf
Latest posts by Donald Heller (see all)
- A dean’s farewell - December 28, 2015
- Once again, the New York Times misrepresents student debt - November 30, 2015
- Duncan’s higher education legacy - October 6, 2015