Collaborating Within State Agencies to Ensure the Effective Use of Evidence

This is the fourth post in our eight-part blog series on using research evidence under the Every Student Succeeds Act (ESSA). Check back every week for the latest installment. Also, be sure to sign up for our March 7 webinar, ESSA and Research Evidence: Opportunities and Challenges for States.

Carrie Conaway and Russell Johnston

As Massachusetts began to plan its strategy for implementing a new state law to intervene in some of the state’s lowest performing schools, we recognized that we would need to accompany our work with research to help us understand where our strategy was having the desired impact on student outcomes.

Our state’s collaboration on school turnaround research began on Martin Luther King Day in January 2010, when then- Governor Deval Patrick signed the Achievement Gap Act of 2010. This legislation gave those schools additional autonomies around collective bargaining, personnel assignment and scheduling, the length of the school day, funding, and other organizational and structural issues, and also created the possibility of state receivership if the schools did not dramatically improve their performance.

However, even after one year of implementation it was evident that some schools were improving while others were not. We were particularly interested in determining whether we could identify the differences between those schools, as we felt that answering that question could hold the key for improving results for all schools.

The state’s Office of School and District Turnaround worked with the Office of Planning and Research to commission two related research projects to answer these questions. The first was a quantitative analysis of student academic outcomes, comparing results for schools that received federal School Improvement Grant (SIG) funding to similar schools in the same districts that did not.

The second was an implementation study comparing the schools that improved to the schools that didn’t, using information systematically collected through our annual school monitoring site visits as the primary source of data.

What emerged from these analyses was nothing short of astonishing. Unlike the national SIG program, students enrolled in SIG schools in Massachusetts on average saw improvements equivalent to an extra year of learning after one year. Gains continued to accelerate, and by three years after funding, the typical SIG school in Massachusetts had seen student achievement improve by 0.4 to 0.5 standard deviations: roughly half of the achievement gap between African American and white students. Overall, 57 percent of our lowest performing schools have turned around in Massachusetts, well above the results in most states.

Not all schools made these large improvements, though, and comparing improvers to non-improvers proved to be the most revealing analysis of all. The qualitative analyses showed that the schools that saw gains had four practices in common, and that these practices were poorly implemented or absent in those that did not improve. These were:

  • Leadership, shared responsibility, and professional collaboration
  • Intentional practices for improving instruction
  • Student-specific supports and instruction for all students
  • School climate and culture

Importantly, these practices were not specific interventions or programs. Schools employed many strategies to implement these practices in different ways depending on their context and needs. But several years of similar findings reinforced that implementing these four practices well was the key to successful school turnaround in Massachusetts.

Because of their central importance for successful turnaround, these four practices now thread through all of our agency’s supports for the state’s lowest performing schools. For example, when a school is designated as a Priority School, we commission independent researchers to conduct a site visit and develop a baseline assessment of the existence of the practices within the school. The results of that assessment inform the development of the initial three-year school turnaround plan, which describes specifically how the school will implement the four practices.

After implementation of the turnaround plan, the researchers return to the school annually to assess the extent to which the practices are becoming sustained and institutionalized. To address specific challenges a school might be experiencing with implementation, we provide targeted assistance, including professional development, funding (particularly SIG grants), consultation, access to turnaround partners, and other supports, all aligned to the turnaround practices. This maintains a high level of focus on these evidence-based practices.

The four turnaround practices touch upon the expertise of many offices within our agency, and as such, our Office of District Support must collaborate across the agency to bring to bear the best practices in each of those areas. The Center for Instructional Support, for instance, provides content specialists in English language arts and mathematics to support the second turnaround practice, intentional practices for improving instruction. Similarly, the Office of English Language Acquisition and Academic Achievement contributes specialists who enhance the implementation of the third practice, student-specific supports and instruction for all students.

As we look forward, we see opportunities to expand our within-agency collaboration to two new areas. First, we are working to more strongly connect our turnaround practices to our statewide strategy for improving curriculum and instruction, in particular by focusing on early grades literacy and middle grades mathematics.

Second, we have identified our monitoring of state and federal programs such as IDEA, Title I, and civil rights as a body of work that can become more aligned to the turnaround practices as well. Too often those forms of monitoring focus solely on compliance, and in Massachusetts, we currently use the same protocol for monitoring every LEA, irrespective of their outcomes for students. We plan to introduce a tiered monitoring system that will focus efforts on the lowest performing schools and districts, with an emphasis on both compliance and outcomes and with the turnaround practices as a unifying framework.

For Massachusetts, turning around our lowest performing schools has meant not just gathering evidence on effective practice, but embedding those practices throughout the agency’s work. Collecting data is a good start, but creating structures that consistently reinforce use of the findings throughout the agency’s work is what has really enabled our success.

Carrie Conaway is Chief Strategy and Research Officer, and Russell Johnston is Senior Associate Commissioner at the Massachusetts Department of Elementary and Secondary Education.

____________________________

The American Youth Policy Forum (AYPF), a nonprofit, nonpartisan professional development organization based in Washington, DC, provides learning opportunities for policy leaders, practitioners, and researchers working on youth and education issues at the national, state, and local levels. AYPF events and publications are made possible by contributions from philanthropic foundations. For a complete list, click here.