Overview
Participation in high-quality “Expanded Learning Opportunities (ELOs)” is associated with an increase in academic achievement, school attendance, time spent on homework, enjoyment and effort in school, and better student behavior (Anderson-Butcher, Newsome, & Ferrari, 2003). While having ELOs available is important for youth development, ensuring quality programming is essential to maximizing positive outcomes. This forum, the third in a series of forums sponsored by the W.T. Grant Foundation, focused on the Youth Program Quality Intervention (YPQI) system and the impact of the Youth Program Quality Assessment (YPQA) used to assess the quality of ELOs.
Charles Smith, Director of the David P. Weikart Center for Youth Program Quality,discussed the Center’s work in research, replication, and field services to improve youth experiences in ELOs. One of the main concerns of the Center is, “How can we use accountability metrics to measure adult-child interactions and how do we get the most of out them,” Smith explained. To address these concerns, Smith discussed how to hold youth programs and their staff accountable for their work. One assessment that Smith presented was the point-of-service (POS) construct using the YPQA. The construct presented four levels of a pyramid at the point of service. Programs must provide safe and supportive environments before meaningful interactions and youth engagement can occur.
Smith continued to explain the importance of assessing quality at the point at which services are delivered. Smith believes that it is important to assess quality at the point of service where program experiences are co-constructed by youth and adults. In addition, the point of service is embedded into management and the system accountability environment (SAE). The YPQI aims to build management competencies in order to develop an embedded adult learning system where site managers and site staff base their work off an assessment metric. Smith stated, “Quality is malleable, but is quality movable, and can you take it to scale?”
Lee Pearson, Research Services Manager at the David P. Weikart Center for Youth Program Quality, described the first two years of a three-year study focused on building Professional Learning Community (PLC) capacity and quality. The first year of the study was dedicated to the YPQA tool and the second year concentrated on the point-of-service. A random assignment of 100 afterschool programs to control and treatment groups across five out-of-school time (OST) networks in four states provided baseline and follow-up data through surveys, interviews, and external observations.
Pearson described two main research questions of the study:
-
Did the YPQI cause changes in management practices of site supervisors and direct staff?
-
Did the YPQI cause changes in the quality of instructional performances in afterschool programs?
Pearson explained that while the five OST networks presented a diverse range of characteristics, baseline YPQA scores were not significantly different between treatment and control groups, thus providing a good baseline for research. PLC-level behaviors were measured on self-reports of program observation, planning, and training. POS-level behavior outcomes were measured on YPQA scales of skill building, planning, choice, and reflection, among others. The results indicated that in year two, managers and direct staff at YPQI sites were significantly more likely than those in control sites to formally observe staff sessions with youth, to conduct program planning using assessment data, and to attend trainings focused on youth worker methods. Pearson added that offerings in treatment sites scored significantly higher than those in control sites on total YPQA scores, planning and choice subscales, and total practices index.
Lorraine Thoreson, Educational Consultant with the Michigan Department of Education, described efforts to take YPQI to scale using their Technical Assistance and Coaching Support System (TACSS) “to grow a culture of performance accountability and an infrastructure for continuous quality improvement and sustainable change.” TACSS is comprised of three main components: annual routine services, grantee intensive training and technical assistance planning, and individualized technical assistance and coaching.
Thoreson explained that they use TACSS to improve academic and developmental outcomes for children through Michigan’s 21st Century Learning Centers (21stCCLC) program. She stated, “We needed to grow a culture so that they felt it was important to be held accountable. They will do it so they can make themselves better.” Initially, the process was introduced on a small scale and voluntary basis, programs were not required to use the YPQA, but it was open to programs that chose to use the YPQA. “Once people thought that it was beneficial, they wanted to use it. Once they knew what higher quality looked like, they wanted to get there,” Thoreson asserted. The YPQA was later made a mandatory online self-assessment, but Thoreson pointed out that the YPQA was not required to receive funding. Thoreson added that it was important to separate the lines of communication regarding YPQA results and coaching interventions between funders and grantees. She believes that this has allowed workers to fully utilize the YPQA, thus alleviating concerns about how the results will affect program funding.
Annual routine services, such as the 21st CCLC kickoff event and sequence content training, build continuity according to Thoreson. Thoreson discussed how technical assistance (TA) is provided to program sites and discussed coaching and mentoring services. She explained that TA is a system support that involves administrators focusing on organizational issues using data to plan change. As a direct service, coaching/mentoring services provide individualized support focused on individual performance for sites and individual staff. Coaching/mentoring provides on-site, on-line and regional supports for grantees.
Ge Xiong, Vice President of Community Programs for Camp Fire USA, Minnesota Council, discussed how they support OST programs, such as clubs and camp programs, by using the YPQA. Through partnerships, clubs reach youth through schools, housing communities, libraries, and community centers. Using a research-based curriculum, staff work with youth to meet social, emotional, and academic needs. Xiong asserted that the YPQA is important to their work especially because of the high staff turnover, an inevitable consequence because they employ mostly college students. By using the YPQA, new and veteran staff can continue to provide quality service despite high turnover.
Xiong described the findings of the Minnesota Council quality assessment report of their programs. Findings showed that programs were providing safe and supportive environments while interaction scores were average and engagement scores were even lower. “At first, staff had trouble with [the YPQA] but now it is automatic,” stated Xiong. She continued to explain that the YPQA is used to reflect on programs and to examine critical needs. Xiong also emphasized the importance of hiring staff for specific duties in order to address indentified needs.
Conclusion
Mr. Smith concluded the forum by briefly summarizing improvement efforts on ELO quality. “Staff performance and quality can be changed. How do we align players so that we can focus on the point of service because that is the important contact point,” asserted Smith. He also stressed the importance of cultivating an environment of improvement. Smith stated, “We can empower individuals if we can get them to want to do better. We can create environments where there are people who are trying to keep getting better.” Smith added that many ELOs also have standards that do not address higher quality aspects such as interaction and engagement. Smith believes that with assessment tools such as the YPQA, standards can be evaluated for improvement.
Highlights from Question and Answer Period
A question was asked to describe, in further detail, how a culture of reflection and desire to improve is developed. Thoreson explained that it is important for administrators to be cheerleaders. In addition, Thoreson stated that a considerable amount of time was spent talking about quality and how to make improvements. Looking at rubrics using a common language facilitated discussion between participants. Smith added that it was important to instill intrinsic value by helping PLCs to understand how YPQA can benefit their work.
BIOGRAPHIES:
Lee Pearson is the Research Services Manager at the David P. Weikart Center for Youth Program Quality, where she leads data collection, analysis and reporting for a variety of research and evaluation projects, including the Youth Program Quality Intervention, a multi-state randomized field trial of setting quality funded by the W.T. Grant Foundation. Before joining the Weikart Center, she held research positions at the National Center for Education Statistics and Policy Studies Associates, Inc. (PSA) in Washington, DC. At Policy Studies Associates, her research focused on the role of out-of-school time programs in promoting positive social and educational outcomes for youth. Lee led the longitudinal evaluation of Citizen Schools in Boston, MA and contributed to several PSA evaluations of out-of-school time systems including: The After-School Corporation (TASC) in New York City, The Out-of-School Time Programs for Youth Initiative sponsored by the New York City Department of Youth and Community Development, and New Jersey After 3. Lee received her M.A. in sociology from The George Washington University.
Charles Smith is the Director of the David P. Weikart Center for Youth Program Quality, a joint venture of the Forum for Youth Investment and High/Scope Educational Research Foundation. Under his leadership, the Weikart Center is positioning point of service quality as a powerful public idea through a program of advanced research, consulting and adult learning technologies.
Dr. Smith’s current research interests include observational assessment methodology and modeling multi-level relationships between setting characteristics and individual outcomes. In recent years, he has led development and validation of the Youth Program Quality Assessment (Youth PQA) and directed leading-edge work in the areas of setting measurement and performance accountability policies for out-of-school time programs. Dr. Smith is currently the principal investigator for the Youth Program Quality Intervention (YPQI) study, a randomized trial designed to raise the quality of staff performance in out-of school time settings. Previously Dr. Smith worked as a research associate for the High/Scope Foundation. In this position, he coordinated a quasi-experimental longitudinal impact evaluation of the Michigan School Readiness Program as well as the Grantee Evaluation Support Project, an initiative designed to build capacity for data-driven continuous improvement in Michigan’s state funded preschool programs.
Dr. Smith received his degrees (BA, MA and Ph.D.) from Wayne State University in Detroit and was a co-founder of the Youth Urban Agenda Civic Literacy Project.
Lorraine Thoreson is an Education Consultant with the Michigan Department of Education working with the 21st Century Community Learning Centers. She has a Bachelors degree in Education from Wayne State University and a Masters in Counseling and Human Development from Michigan State University.
Ms. Thoreson serves on the executive board of the Michigan AfterSchool Association (MAA formerly MiSACA). She also serves as Secretary of the board of directors of the National AfterSchool Association. Lorraine is also a member of the steering committee for the Michigan After-school Partnership which is a Mott funded out-of-school time network aligning partners interested in out-of-school time programs.
Ms. Thoreson has worked in the field of out-of-school time for more than 20 years as a program director for school based programs in several different school districts in Michigan. Lorraine has presented professional development workshops for local, state and national conferences on a diverse array of topics relevant to out-of-school time programming. She has worked at the local, state and national level to promote the importance of out-of-school programs and Extended Learning Opportunities (ELO) for children and youth, including serving on the Council of Chief State School Officers ELO task force.
Ge Xiong, Vice President of Community Programs, has worked at Camp Fire USA Minnesota Council for five years. She graduated from Mount Holyoke College in Massachusetts with a major in Anthropology and Biology. Her passion to work with youth helped propel her to initiate her own youth program focused on Southeast Asian girls in high school. Ms. Xiong has also worked and volunteered with several non-profit educational agencies: TRIO, GEAR UP, and Admission Possible. Ms. Xiong also worked as an admission counselor at Concordia University in Saint Paul where she earned a Master’s in Education with a concentration in Youth Development. Ms. Xiong was recently named to Saint Paul’s Second Shift Commission.
- AYPF Publication: Learning Around the Clock!
- Fact Sheet – College and Career Ready Basics
Presenters
Lee Pearson
Research Services Manager
The Center for Youth Program Quality
7064 Eastern Avenue, NW
Washington, DC 20012
Dr. Charles D. Smith
Director
David P. Weikart
Center for Youth Program Quality
124 Pearl Street, Suite 601
Ypsilanti, MI 48197
734-961-6900 x 204
734-961-6904 fax
Lorraine Thoreson (MDE)
Michigan Department of Education
21st CCLC Consultant
P.O. Box 30008
Lansing, MI 48909
Ge Xiong
Vice President of Community Programs
Camp Fire USA Minnesota Council
2610 University Ave. W.
St. Paul, MN 55114
651.632.9196
