On October 29-30, 2018, the American Youth Policy Forum (AYPF) convened a discussion group Building a Research Office that Supports Continuous Improvement under ESSA. Attendees included state-level policymakers, researchers, and staff from a number of foundations. Three states, each at various levels of development, presented a dilemma they are currently facing in establishing or re-designing their research office, driven in part by the Every Student Succeeds Act (ESSA) that requires states to use research evidence in their policies and practices. This blog post presents a summary of the discussion held, shares challenges and suggestions for establishing and re-designing a research office, and highlights the progress some states have made.
State Education Agencies (SEAs) are at Different Stages in Developing Research Offices
The three SEAs that shared their stories were each at different stages in the development of their research office:
- One SEA that currently does not have a research office and has little financial or human resource support is seeking to implement evidence-based programs under ESSA with the limited capacity and resources they have.
- The second SEA is redesigning their research office to explicitly focus on research, evaluation, data analysis, and impact, and is seeking to build their agency’s capacity to leverage data and to establish partnerships with local research institutions.
- A third SEA has an established research office but is working on shifting from strict compliance and accountability to using data to support continuous improvement, and is seeking to incorporate the use of data and research across their agency to further bridge policy and data.
Although there was significant variation among the three SEAs that shared their stories, there were also many overlaps in experiences and challenges. The discussions following the state presentations stimulated collaborative peer-to-peer learning and enthusiastic feedback based on others’ trials and successes, providing valuable takeaways for all participants. Below are challenges and possible remedies discussed by participants in the meeting.
Challenges and Remedies
1. Confronting Limited Resources
SEAs are familiar with the challenge of limited resources whether it is with regard to funding, staff, or capacity. Research offices are often not seen as essential within SEAs and therefore are not prioritized. Even well-established SEAs have difficulties in attaining additional resources to expand their research efforts and better inform school improvement efforts.
Several solutions for SEAs struggling with limited resources involve allocating scarce resources more efficiently. While SEAs value randomized controlled trials (RCTs), which are the strongest research design available for studying impact but are also costly, state leaders also noted that there is value in studying implementation of programs and policies as that often provides more useful and usable information. Implementation analysis can go beyond whether the program worked or did not work and instead can provide answers related to the following: for whom it worked, why it did or did not work, and strategies for improving the program. As one participant commented, “You’re paying attention to implementation – you are modeling that. You’re saying it is not enough to choose an evidence-based approach, you have to pay attention to what’s happening on the ground and how the implementation is rolling out.” Cost effective analysis relies on asking the right types of questions. SEAs should be asking and answering questions such as “what problem are we trying to solve? What decisions are coming up where research might be helpful? What are we trying to learn this year? What is the best type of research design to use to answer these questions?”
For SEAs unable to dedicate staff to focus solely on research, forming a “coalition of the willing” is a realistic alternative. The “coalition of the willing” is a strategy for utilizing existing staff from various departments who are most interested in and able to contribute their capacity to research office efforts.
2. Ensuring Effective Research-Practice Partnerships
Another challenge highlighted by states was how best to enter into and work within a research-practice partnership (RPP). RPPs are long-term, mutually beneficial, and intentionally organized collaborations between researchers and practitioners that can assist SEAs in conducting rigorous research to inform their policies (additional information on RPPs can be found on our resource page).
Acquiring the resources for establishing an RPP is a challenge in itself, but this challenge is followed by additional obstacles such as setting up an RPP that effectively meets the needs of the SEA and informs programmatic work.
Participants emphasized the importance of effective communication strategies, observing that it is important to establish continuous communication between all involved parties – the research office, program offices, and external researchers. Laying out a timeline with checkpoints is a practical way to ensure effective communication throughout a research study. Checkpoints provide an opportunity for the research and program offices to see how the results are shaping up and to clarify data that may be misunderstood or overlooked by the researchers.
Hiring external and internal researchers who have experience working in an education context, are culturally competent, and are willing to work with a SEA’s research agenda are all equally important factors in producing meaningful research. It is also necessary for researchers to understand the implications of their research findings with regard to issues of equity, power, and privilege.
3. Strategically Structuring a Research Office
SEA research offices are often associated with evaluation, monitoring, and accountability, and therefore can be perceived as threatening by program offices, schools and districts, but this does not have to be the case. Research offices are uniquely positioned to have a much broader view and understanding of how agency work is interrelated and should leverage this position to promote collaboration across the agency and build relationships based on trust and support. One participant commented, “We work collaboratively to ensure that all teams and work are valued – we don’t want to devalue the reports, the numbers and percentages, because without that we can’t do our work. It’s a domino effect – you collect the data, report the data, you analyze, you research, and then you go and talk about policy. If one of those things isn’t there, then nothing works.”
Rephrasing “research agendas” as “learning agendas” shifts how program offices, districts, and schools perceive the research process. Rather than feeling the need to defend programs, regardless of whether or not the programs yield better student outcomes, program offices, districts, and schools should operate under the understanding that the research office is there to work with them, learn alongside them, and adjust the interventions as necessary to increase positive results.
Connecting program offices with one another and developing joint research agendas aligns work in a coherent manner. Brokering these inter-office relationships and potentially hiring someone specifically to act as a broker, maximizes research efficiency and can enhance focus on the bigger picture to more effectively inform policy.
Program offices need to have an understanding of how research findings can be used to inform their programmatic work. Implementing structured meetings and providing learning spaces to allow for involved parties to make sense of the work the research office is doing can lead to the effective application of evidence-based practices. Creating opportunities for this type of learning contributes to a strong research culture that can significantly improve agency work.
During the meeting, a representative from Massachusetts shared that twelve years ago, Massachusetts was the only state with someone working in a research capacity within their SEA. In 2018, this discussion group brought together over 30 participants and representation from ten SEAs working in research capacities. Especially heartening was one participant’s observation at the end of the meeting: “My takeaway is that our office needs to prioritize and think about the sequencing of what we’re going to do. I’m learning from other states that had a much more laser-like focus and were really thoughtful and strategic around things, so that’s what I’m thinking about for our team and what will lead our agenda for the agency with our limited resources.” The use of research evidence in policies and practices is becoming embedded in SEA work nationwide. SEAs are rising to the challenge of establishing and managing an effective research office, while research offices are using their position in their agency to advocate for better use of data and evidence. Beyond implementing new practices, research offices are changing how SEAs, districts, and schools value research, a cultural shift that we hope will have a significant impact on future student outcomes.