Using Assessment Tools to Evaluate Afterschool Programs: A Look at the Youth Program Quality Assessment (YPQA)

A Forum — November 2, 2007

More and more frequently, policymakers, funders, and youth program directors are looking for ways to ensure that the dollars and efforts they are allocating to programs are indeed having an impact on the youth they are serving. As a result, more youth programs, including afterschool programs, are turning to evaluation. 

As a response to this evaluation trend, multiple tools have already been created by organizations and other entities to assess afterschool program quality. These tools vary in how they are used and what they measure, due to the broad spectrum of youth program types that exist.  The Youth Program Quality Assessment (YPQA), which was developed by the High/Scope Educational Research Foundation, is one such tool that is becoming widely used by the afterschool field.  This tool allows program staff (or external specialists) to conduct evidence-based research on programs via observation and interview methodologies. Notes, observations, and interview data are used as evidence to score items, and these item scores are combined to create an overall program quality profile.

This AYPF forum provided attendees with a chance to learn more about why quality assessment is becoming so vital for youth-serving programs like afterschool programs. As an example of how the field is currently working to evaluate itself, forum presenters discussed the YPQA and its impact on focusing efforts to improve afterschool program quality.  Attendees heard from the director of New York State about that state’s efforts to implement evaluation in its afterschool programs, as well as from a program director from rural North Branch, Minnesota, who has been using the tool for the past two years. These speakers spoke both about the positive aspects and the challenges of assessing afterschool programs, as well as the policy implications of evaluation endeavors, at both the state and local levels.

Nicole Yohalem, Program Director, Forum for Youth Investment, opened the session by discussing the importance of focusing on program quality.  “Quality matters.  We have good reason to believe that high quality programs are more likely than low quality programs to have positive impacts on kids,” stated Yohalem.  She added that quality is indeed measurable, and that it is much more efficient to have programs measure their programmatic quality and leave the research community to link those indicators to youth outcomes.  Furthermore, Yohalem noted that quality is malleable, meaning that quality can be changed.  

The YPQA is one of several assessment tools that are highlighted in a guide created by the Forum for Youth Investment that compares youth program assessment tools.  According to Yohalem, the YPQA is a strong assessment from the scientific perspective because of strong validity and reliability coefficients.  The YPQA is becoming widely used in the afterschool field and represents a good example of the new class of assessment tools that produces useable data, provides continuity that is place-based, links accountability policy with workforce development policy, and offers more efficient and effective use of resources.  Specifically, Yohalem described these strengths of the YPQA:

• Observation items are specific and detailed, which, in turn, help staff learn specific skills that the YPQA is assessing.

• The YPQA builds a common language to talk about practice.

• It details how the program is working and develops accountability.

• It focuses on improving the program rather than focusing solely on youth outcomes.

Charles Smith, Director of the Youth Development Group at the High/Scope Educational Research Foundation, described the YPQA in further detail.  As an observational assessment, the YPQA observes and measures at the point of service.  Smith emphasized that measuring at the point of service allows a program to better hone in on the specific components to tweak in order to result in “greatest value added,” as well as to assess the dynamic between the adult and youth.  Another strength of the YPQA, according to Smith, is that it can be used in several different content areas because of the validity and reliability of each measure. 

The YPQA is a low-cost assessment tool that takes approximately one-to-two hours to complete, and training is available but not required to administer the tool.  Using the YPQA involves watching program activities and interactions and noting observations with regard to the items of the YPQA.  Once completed, observations are scored and measured on the given rubric.  Smith noted that too many programs focus mainly on building the program rather than also looking equally at the program offerings and the interactions that occur between adults and youth.  

Smith expressed that the purpose of using the YPQA should be for programs to increase youths’ access to the experiences that motivate them to attend and become engaged.  In addition, staff can learn from the metrics, improve performance, and develop more effective accountability standards.

Jeanne C. Leland, Director of the North Branch Area Public Schools, Minnesota, Community Education Department, described North Branch as a small rural community that has decreased in population since gasoline prices started to increase.  Leland stated that minimal local levy taxes for youth programs were used to leverage additional funds for programs through participant fees, and that for some families, access to the school age care program is difficult due to the costs. 

Leland spoke of the importance of evaluating youth programs, expressing that programs can always do more to enrich the lives of the youth who are being served. She also asserted her belief that when a program is high quality, improved youth outcomes indeed follow.  Leland described her use of the YPQA in assessing the quality of the North Branch school-age care afterschool programs, where the desire to improve upon the offerings for kids prompted the decision to look for an assessment tool that would help create the needed change.  She expressed that the YPQA provided a new way in which to observe afterschool programs.  After using the self-assessment, Leland was able to use the data for intentional quality improvements to engage youth at all ages.  Small steps were taken to empower staff to implement creative and purposeful changes.  Recognizing the need for educational support, monthly themed curricula were adapted to include math and more and better science-based activities.  Leland also discussed the positive impact of small group clubs and using older students as mentors to younger afterschool participants.  Despite initial support by staff for program changes, Leland noted that challenges indeed existed.  For example, staff and student habits were notably difficult to change, so additional staff meetings and funding for them had to be implemented in order to facilitate staff trainings on strategies to improve quality.

Judy Mills, Director of the New York State Advantage After School Program, discussed the multiple efforts that New York is undertaking to infuse quality assessment into their afterschool programs.  She noted three key roles for state policy:

• Set clear priorities about quality.

• Fund provider networks to support implementation of quality policy.

• Meet provider-level demand for more effective forms of accountability.

There are almost 300 Advantage After School Programs in the state, and 20 of them have recently implemented the YPQA.  In describing the positive aspects of YPQA’s implementation, Mills said that the tool greatly helped staff to reflect on thought processes.  She added that the addition of youth voice and engagement components at many of the programs have led to tremendous improvements in youth outcomes. In terms of challenges, Mills noted that because there are so many quality “trains” moving concurrently in the state, it has been difficult to coordinate their efforts in a systematic way and ensure their compatibility with one another.  Aligning federal assessment and reporting requirements would also help New York’s situation, she added.

In summarizing the session, Smith noted five important take-away points for policymakers and practitioners:

• The field is looking for more effective ways to spend resources that are already dedicated to accountability, evaluation, and professional development. Observational quality assessment metrics are increasingly the answer.

• Assessment tools allow us to measure the most important elements of programs, which raises the possibility of increased returns on existing investments.

• With the right kind of supports, this form of accountability can be an effective and sustainable part of programs that serve children and youth.

• Constituencies of state government are currently seeking out ways to be accountable in ways that demonstrably change the quality of services available to youth.

• State and local governments are already experimenting with new accountability policies based on the YPQA—and they are affordable.

Highlights from the Question and Answer Session

A question was asked regarding the relation of the YPQA to youth outcomes.  Smith expressed that no formal research studies have been conducted, but that relationships have been observed between the results of YPQA assessments and social-emotional outcomes. Specifically, Smith said that positive youth outcomes have been noted for programs in Grand Rapids, Michigan using the tool.

A question was asked regarding the subjectivity of the YPQA because the measures rely heavily on observation.  Smith stated that the YPQA has acceptable reliability and validity coefficients in addition to acceptable inter-rater reliability.      

Resources

PowerPoint Presentation, November 2, 2007

Yohalem, Nicole and Alicia Wilson-Ahlstrom with Sean Fisher and Marybeth Shinn (March 2007)  Measuring Youth Program Quality:  A Guide to Assessment Tools. The Forum for Youth Investment

Granger, R., Durlak, J. A., Yohalem, N., & Reisner, E. (April, 2007).
Improving after-school program quality. New York, N.Y.: William T. Grant Foundation.

Speaker Biographies

Jeanne C. Leland is the Director of North Branch Area Public Schools, Minnesota, Community Education Department. Providing a comprehensive offering of enrichment, recreation, and personal development classes during the day, evenings, and weekends for preschoolers, youth, adults, and senior citizens at the public schools supports the goal of tapping into the human resources of the community at the tax-supported school facilities.  Working with the citizen advisory council and staff from local and regional agencies ensures that collaboration is emphasized while addressing various community issues. Jeanne received her Master of Education degree from the University of St. Thomas in St. Paul, Minnesota, and her Bachelor of Arts degree in Elementary Education at the University of Wisconsin, River Falls. She and her husband have two children in college.

Judy Mills is the Director of the New York State Advantage After School Program and Project Director for a demonstration project involving the collocation of child welfare and substance abuse services. As the New York State lead for the High/Scope Youth Program Quality Intervention Project, Judy is overseeing the implementation of the Youth Program Quality Intervention in 20 Advantage After School Program sites in upstate New York. As a member of the New York State Afterschool Network, Judy is on the NYSAN Quality Assurance and Professional Development Committees. Judy joined the New York State Office of Children and Family Services in 1978 and was instrumental in the development of the state’s Homeless Housing and Assistance Program and the Runaway and Homeless Youth initiatives. Prior to this, she was part of a small group of community activists who created the Zacchaeus Community Kitchen and Hospitality House in Washington, D.C., now known as the Community for Creative Non-Violence. Raised in Arlington, Virginia, Judy attended American University and is a graduate of Millersville University and Sage College. She currently lives in Delmar, New York with her husband and son.

Charles Smith is director of the Youth Development Group at the High/Scope Educational Research Foundation, where he manages consulting and research related to High/Scope’s youth-level work.  Dr. Smith's current research interests include accountability systems, quality and effects of learning environments, and methodology. He is principal investigator for the Youth Program Quality Intervention Study, a three-year randomized trial focused on setting change in out-of-school time organizations. Dr. Smith presents, writes, and consults on alternatives to current accountability models. At Wayne State University in Detroit he co-founded the Youth Urban Agenda/Civic Literacy Project, a civic learning initiative for secondary and undergraduate students operating since 1993. 

Nicole Yohalem is Program Director at the Forum for Youth Investment, where she leads the Forum’s work on a range of issues including out-of-school time, youth engagement, and high school reform. Prior to joining the Forum’s staff, Nicole served as a youth development specialist at Michigan State University, where she developed, implemented, and evaluated community-based youth programs and provided training and technical assistance to programs statewide through the Cooperative Extension Service. From 1990 to 1995, Nicole worked in the adolescent division of the High/Scope Educational Research Foundation, where she directed the Foundation’s residential program for teens and developed training materials for youth programs. Over the past several years, Nicole has also served as a consultant to the World Bank on education reform in the Latin American and Caribbean region. She received her Master of Education degree from the Harvard Graduate School of Education.

American Youth Policy Forum   1836 Jefferson Place NW,  Washington DC 20036
Telephone 202-775-9731      |      Fax 202-775-9733      |     Email aypf@aypf.org