Researchers Revisit TNTP’s “Widget Effect”, Teacher Evaluations

Written by: Dave Reid

Primary Source: Green & Write, March 28, 2016

Researchers Matthew A. Kraft and Allison F. Gilmour recently released a working paper which examined if new teacher evaluation systems have made any progress differentiating amongst teacher quality.

In short, they find many states now do a better job distinguishing great teachers from good teachers, but states continue to struggle between differentiating good, average, and below average teaching.

The results from this work have educational pundits arguing new evaluation measures have done very little to improve the reliability of teacher evaluation scores, while others argue the results of this work should be taken with optimism and new teacher evaluation systems have in fact made some difference.

Photo Courtesy of Lynell Burmark

Photo Courtesy of Lynell Burmark

TNTP’s Report

In 2009, The New Teacher Project (TNTP) released a report that found professional development for teachers was inadequate, novice teachers were often neglected and had low expectations for performance, and the poor performance of teachers often went unaddressed by administrators.

However, the most cited and talked about result from this report was the finding that almost all teachers are rated good or great. Specifically, less than one percent of teachers in the study received a rating of unsatisfactory. This was concerning not only because it seems unlikely that only one percent of teachers are unsatisfactory, but also because the authors of the report found 81% of administrators in the study and 57% of teachers in the study could identify at least one teacher in their school who was ineffective.

Why the Revisit?

Since the release of this report and since the Race to the Top initiative (also 2009), most states have made substantial changes to the way teachers are evaluated. These changes include using student assessment data in teacher evaluations and using more evidence-based observation rubrics when observing teacher instruction. Other aspects of the evaluation process, such as the number of observations administrators conduct with teachers and the amount and type of feedback teachers receive, have also changed.

Given all of the money, time, and other resources that states and school districts have poured into reforming their teacher evaluation systems, new research is needed to see if these systems are having their intended effects of better differentiating teacher quality.

Impact on Teacher Quality

While additional work is needed to see exactly how teacher evaluation reform has impacted teacher quality over the past seven years, this working paper sheds some light on a common theme found in education – street level bureaucrats impacting how policies look in practice.

As Kraft and Gilmour write, “Our exploratory case-study reveals how the failure to differentiate between teachers is a product of conscious choices by evaluators as they navigate implementation challenges, competing interests, unintended consequences, and perverse incentives. These findings exemplify Michael Lipsky’s (1980) seminal observation that policies are ultimately made by the “street-level bureaucrats” who implement them rather than the policymakers who design them.”

While the design of teacher evaluation policies has changed substantially since the release of TNTP’s report in 2009, individuals controlling how these policies ultimately play out in practice remains constant. The individuals charged with implementing new policies, no matter how well-designed, have to make decisions they believe are in the best interest of their local context, have to balance relationships with staff, and have to take into consideration everything that goes in to running a school.

While administrators may know close to 100% of their teaching staff is not effective or highly effective there are a host of other reasons (time, resources, a fear that they may lose and not be able to replace an ineffective teacher) why teacher evaluation policies alone will not be able to differentiate between teachers. Instead, things such as less formal evaluations, peer feedback, and professional development must be considered when trying to differentiate teacher quality.

Additionally, the individuals charged with implementing teacher evaluation policies must be incentivized to accurately evaluate teacher performance and evaluators must have confidence these important policies will ultimately improve their teaching staff and better serve students. Until then, little will change.

Contact Dave:

The following two tabs change content below.
Dave Reid
Dave is a doctoral student in Educational Policy and Teacher Education. His current research interests include how teachers make sense of and interact with the policies they are asked to implement. Prior to returning to graduate school, Dave was a 7th and 8th grade special education teacher and also served as an instructional coach for elementary and middle school teachers. Dave earned his bachelor’s degree in journalism and mass communications from Franklin Pierce College and an M.Ed. in education from Arizona State University. - See more at: