Randomized Controlled Trials of Public Policy

Written by: Nicolas Bichay

Primary Source:  Michigan Policy Wonk Blog, June 8, 2016

At IPPSR, we are underway building a comprehensive database full of policy-relevant research. A large focus of the database is to find randomized controlled trials (RCTs) of public policy experimentation. These studies enable causal inference by randomly assigning a policy intervention to some people or areas and comparing the results to a control group. So far, we are proud to have summarized roughly 60 published studies utilizing RCTs to examine the effects of policy implementation. These RCTs range in their focus, and can be categorized into 7 categories. In this post we briefly summarize each category, and end with a comprehensive list of all RCTs, complete with a brief summary of each.

Health

The biggest category, by far, involves health-related policy. Several studies utilize data from the famous Oregon Medicaid Experiment to assess changes in utilization, outcomes, and preventative screenings in Medicaid patients. Important findings from the Oregon experiment have shown that gaining access to Medicaid has important medical benefits (decreased depression and increased self-reported health), decreases financial hardship, and increases utilization of primary care visits and specialist referrals in community health centers. Other studies not related to the Oregon experiment have looked at methods of reducing childhood obesity, eliminating risky behaviors, and testing the efficacy of nurses’ home visitations.

Education

In terms of education policy, RCTs have proven to be quite popular. A popular subject focuses on testing the effectiveness of various pre-school and student tutoring programs, including the prominent Head Start Impact study. Many of these results paint a similar story: programs have early benefits that dissipate quickly once students move into standard classrooms. These studies have shown the necessity of devising new programs better able to maintain long-term benefits. Other studies have looked at methods of increasing university attendance and retention rates, relative effectiveness in differing approaches to sexual-education, and the cost-effectiveness of school vouchers.

Poverty, Welfare, and Social Services

Another burgeoning use of RCTs has focused on measuring the effectiveness of various social programs. Randomized experiments have been used to assess the success of homelessness prevention programs, welfare time-limits and employment restrictions, and job-training programs. Results have shown a great success in New York’s Homebase Community Prevention Program in terms of reducing homelessness. Further, analyses of Florida’s Family Transition Program (which added time-limits to welfare while providing job-search aide) concluded that the program did not increase the probability of recipients finding employment, and further increased mortality rates amongst recipients.

Crime and Criminal Justice

RCTs have become a useful tool in analyzing police and criminal policy. Through RCTs, different police policies including crime hot-spot patrolling, body-worn video cameras, and procedurally-just traffic stops have been assessed in terms of success and efficiency. Further, criminal justice policies such as ignition breathalyzer lock systems have been tested for crime prevention. These studies have shown that body-worn cameras successfully decrease allegations and complaints against police officers, while increasing police patrols in crime hot-spots successfully act as a deterrent to crime.

Business and the Economy

RCTs have been successful in different aspects of the economic and business world. Multiple studies have focused on methods for improving tax compliance, others have looked at mechanisms to increase retirement planning, and have tested the effectiveness of OSHA inspectors. Some results have shown that close supervision of business tax-compliance has a negative effect on on-time tax payments, while negative media exposure and increased perceptions of tax-complexity also decrease compliance rates.

The Environment

Though still moderately scarce, the environment poses a prime sector for RCTs in the future. Currently, our database includes two studies on the environment. One attempts to find the most productive method for enticing consumers to switch to an electricity rate-plan that incentivizes off-peak usage, eventually finding that these differing methods provided statistically zero difference. The other studies the impact of Delhi, India’s experiment to curb automobile pollution by mandating each car in the territory only be allowed to operate every other day. The results showed dramatic decreases in pollution, decreases in traffic congestion, and a significant increase in carpooling and public transportation usage.

Voting Behavior and Government Operations

Recently RCTs have been increasingly popular in this area with the sudden increase in voter restriction (see our post). Many studies focus on the effects voter ID, early voting, and absentee ballot laws have on turnout. Beyond voting, RCTs have been utilized to test mechanisms for increasing citizen engagement, the role of gender in policy-making, and the effect public opinion has on legislators’ votes. An important result for policy makers has shown that sending letters to constituents helps move public opinion towards the legislator’s position, while simultaneously avoiding a decrease in approval rating by expressing unpopular policy stances.

All-in-all, RCTs have proved to be an immensely popular and successful tool in policy evaluation across all policy-areas. In the future, their use is likely to increase, and we look forward to the knowledge they provide.

Below we list all RCTs in our database by category:

Education

Impact Evaluation of the U.S. Department of Education’s Student Mentoring Program


Evaluates the effects of the federal Student Mentoring Program

The Study of Mentoring in the Learning Environment (SMILE): A Randomized Evaluation of the Effectiveness
of School based Mentoring

Investigates the effect of school-based mentoring (SBM)

Mentoring in Schools: An Impact Study of Big Brothers Big Sisters School Based Mentoring

Tests the impact of the Big Brothers Big Sisters mentoring program on students

The Long-Term Effects of a Peer-Led Sex Education Programme (RIPPLE): A Cluster Randomised Trial in Schools in England

Tests the relative effectiveness of peer-led sexual education classes

The High Scope Perry Preschool Study Through Age 40

Examines the effect of preschool on children from impoverished families

Improving Numeracy and Literacy Evaluation Report and Executive Summary

Evaluates the Improving Numeracy and Literacy program regarding students’ mathematical and reading skills

Independent Evaluation of University of Exeters Grammar for Writing Programme: A Cluster Randomised Controlled Trial

This report summarizes an experimental intervention that aims to improve the writing and grammatical skills of young students

Summer Nudging: Can Personalized Text Messages and Peer Mentor Outreach Increase College Going Among Low Income High School Graduates?

Analyzes the usage of text messages to increase college attendance

Free Healthy Breakfasts in Primary Schools: a Cluster Randomised Controlled Trial of a Policy Intervention in Wales, UK

Studies the effects of offering free breakfast to schoolchildren in Wales

School Vouchers and Student Achievement: First year Evidence from the Louisiana Scholarship Program

Examines the academic difference between students who won a lottery for private school vouchers, and those who lost

A Randomized Control Trial of a Statewide Voluntary Prekindergarten Program on Children Skills and Behaviors through Third Grade

Evaluates the effect of Tennessee’s voluntary Pre-Kindergarten program (hereafter “pre-k”)

Head Start Impact Study Final Report

The Head Start Impact Study was a government run randomized experiment to measure the effect of the Head Start program

Parent-Child Information Frictions and Human Capital Investment: Evidence from a Field Experiment

Examines the effect of increased flows of information to parents on student success

Teachers, Race, and Student Achievement in a Randomized Experiment

Uses data from the Tennessee project STAR and tests the effect of minority students having a teacher from the same minority group

Efficacy of a Theory-Based Abstinence-Only Intervention Over 24 Months

Tests the effectiveness of abstinence-only education among youth

Health

Impacts of Child Development Accounts on maternal depressive symptoms: Evidence from a randomized statewide policy experiment

Analyzes the effect Child Development Accounts (long-term investment accounts) have on mother’s depression

School Gardens and Physical Activity: A Randomized Controlled Trial of Low income Elementary Schools

Assess the impact school gardens have on physical activity

The Impact of Screening and Advice on Inattentive, Hyperactive and Impulsive Children

Analyzes the effect of in-school ADHD screenings

Effectiveness of Home Visitation by Public health Nurses in Prevention of the Recurrence of Child Physical Abuse and Neglect: a Randomized Controlled Trial

Examines the effects of nurse’s home visitation on incidence of abuse and neglect

A Randomized Trial Testing the Efficacy of Modifications to the Nutrition Facts Table on Comprehension and use of Nutrition Information by Adolescents and Young Adults in Canada

Examines mechanisms for improving young consumers’ abilities to understand and compare food nutrition labels

What Does Medicaid Expansion Mean for Cancer Screening and Prevention? Results from a Randomized Trial on the Impacts of Acquiring Medicaid Coverage

Part of the Oregon Medicaid experiment, this study analyzes the effect Medicaid expansion has on the prevalence of cancer screenings

Effectiveness of a Nurse-led Intensive Home-visitation Programme for First time Teenage Mothers

Evaluates the effect of a home-visitation nurse program for first time mothers

Encouraging Primary Care Physicians to Help Smokers Quit: A Randomized, Controlled Trial

Evaluates various methods to encourage Physicians to convince patients to quit smoking

An Evaluation of the Effectiveness and Cost Effectiveness of the National Exercise Referral Scheme in Wales, UK

Evaluates the National Exercise Referral Scheme (NERS), a program set up in Wales to increase physical activity by proving exercise instruction, group exercise classes, and motivation and goal-keeping consultations with professionals

The Oregon Experiment – Effects of Medicaid on Clinical Outcomes

In 2008 Oregon expanded its Medicaid through a lottery, drawing 30,000 names out of 90,000 applicants. This was used as a randomized experiment to analyze multiple effects of Medicaid access

Community Health Center Use After Oregon Randomized Medicaid Experiment

In another study based on the Oregon randomized Medicaid experiment, this experiment analyzes community health center utilization by Medicaid patients

Promotion of Healthy Eating Through Public Policy: A Controlled Experiment

Analyzes the effectiveness of different methods of decreasing consumption of unhealthy food and beverages

A Family Intervention to Delay Nursing Home Placement of Patients with Alzheimer Disease

Evaluates a program providing counseling to spouses of Alzheimer’s patients

Health Insurance and the Demand for Medical Care: Evidence from a Randomized Experiment

Explores the effect of cost-sharing on the demand of medical care

Does Free Care Improve Adults’ Health? Results from a Randomized Controlled Trial

Measures the effect the provision of free healthcare has on health outcomes

Is Cost a Barrier to Screening Mammography for Low-Income Women Receiving Medicare Benefits?

Tests the effectiveness of offering free mammography screening to low income patients (specifically, older women)

Reducing Children’s Television Viewing to Prevent Obesity

Measures the effect a curriculum to decrease television viewing has on childhood obesity

Six-year Follow-up of the First Waterloo School Smoking Prevention Trial.

Measures the effect of an educational program designed to resist peer-pressure to smoke

Communities Mobilizing for Change on Alcohol: Lessons and Results from a 15-community Randomized Trial

Attempts to evaluate a policy intervention that was designed to alter “alcohol-related behaviors”

Poverty, Welfare, and Social Services

Banking the poor via savings accounts: Evidence from a field experiment

Estimates the demand and utility that providing banking institutions to poor households in Nepal brings to Nepal citizens and the country

Evaluation of the Homebase Community Prevention Program

Explores the effect of the Homebase Community Prevention Program in New York City

The Family Transition Program: Final Report on Florida’s Initial Time Limited Welfare Program

Assesses Florida’s Family Transition Program (FTP), the first attempt of a state to initiate time limits on welfare benefits

Welfare Programs That Target Workforce Participation May Negatively Affect Mortality

Analyzes results from the Florida Family Transition Program, an experiment looking at time limits on welfare benefits

The Effects of Exposure to Better Neighborhoods on Children: New Evidence from the Moving to Opportunity Experiment

Analyzes the effects of the “Moving to Opportunity” experiment, where families from highly impoverished housing projects were given vouchers to move into less impoverished neighborhoods

The benefits and costs of JTPA Title II-A programs: Key findings from the National Job Training Partnership Act Study

Evaluates employment training and services through the Job Training Partnership Act (JTPA)

Crime and Criminal Justice

General Deterrent Effects of Police Patrol in Crime Hot Spots: A Randomized, Controlled Trial

Investigates the effects of police-officer patrols in high crime “hot spots” in Minneapolis

Police, Camera, Evidence: Londons Cluster Randomised Controlled Trial of Body Worn Video

Tests the effects of police use of body-worn video cameras

Effects of Ignition Interlock License Restrictions on Drivers with Multiple Alcohol Offenses: A Randomized Trial in Maryland

Measures the effectiveness of a program that instituted breath-alcohol detectors connected to the ignitions of drinking-and-driving offenders for one year

Shaping Citizen Perceptions of Police Legitimacy: A Randomized Field Trial of Procedural Justice

Assesses the impact “procedurally just” traffic stops have on perceptions of the police

Business and the Economy

Effects of supervision on tax compliance: Evidence from a field Experiment in Austria

Examines the effect supervision has on the prevalence of tax compliance

Randomized Government Safety Inspections Reduce Worker Injuries with No Detectable Job Loss

Analyzes the effect of Occupational Health and Safety Administration (OSHA) inspectors

An Empirical Analysis of Taxpayers Perceptions of Tax Related Media Coverage and its Impact on Tax Compliance

Examines the effect of tax related media on consumers trust in government and on their perceived power of the government to enforce tax policy

The Role of Information and Social Interactions in Retirement Plan Decisions: Evidence from a Randomized Experiment

Assesses the impact of additional information on university employee enrollment in retirement savings plans

The Environment

Can Behavioral Economics Be Used to Encourage Consumers to Switch to Green Energy Tariffs?

Attempts to find the most productive method to entice consumers to switch to a rate plan incentivizing off-peak usage.

Delhi Odd Even Plan as a Public Policy Experiment

Evaluates Delhi, India’s experiment to to curb automobile pollution by mandating that cars with an even number registration would be allowed to drive on even dates, while cars with odd registration could drive on odd dates

Voting Behavior and Government Operations

The Effects of Voter ID Notification on Voter Turnout: Results from a Large Scale Field Experiment

Tests the effect of voter identification laws on voter turnout

Women as Policy Makers: Evidence from a Randomized Policy Experiment in India

Analyzes the effect of women policymakers on policymaking

Invitation Phone Calls Increase Attendance at Civic Meetings: Evidence from a Field Experiment

Evaluates differing methods to increase attendance at town hall meetings

Can Learning Constituency Opinion Affect How Legislators Vote? Results from a Field Experiment

Analyzes what effect learning public opinion has on a legislator’s stance on an issue

Get Out the Vote-by-Mail? A Randomized Field Experiment Testing the Effect of Mobilization in Traditional and Vote-by-Mail Precincts

Taking advantage of voting rules in San Diego, an experiment was devised to test door-to-door canvassing effects on traditional voting compared to precincts that solely use mail-in ballots

How Do Citizens React When Politicians Support Policies They Oppose? Field Experiments with Elite Communication

Analyzes the effect of state legislators expressing opinions to their constituents

How Not to Increase Participation in Local Government: The Advantages of Experiments When Testing Policy Interventions

Tests different mechanisms to increase public participation in municipal committees

The following two tabs change content below.
Nicolas Bichay
Nicolas Bichay is a Doctoral Student in the Department of Political Science. He holds a Bachelor of Science degree (magna cum laude) from Grand Valley State University, where he studied Political Science and Economics. Nicolas’s fields of study include comparative politics and research methodology. His specific area of research focuses on the Political Economy of Developed Countries. He currently serves as Research Assistant in the Institute for Public Policy and Social Research.
Nicolas Bichay

Latest posts by Nicolas Bichay (see all)