Overview
Afterschool/Out-of-School Time (OST) programs provide extended learning opportunities to help youth succeed academically and develop important social, personal, civic, and employability skills. Using a comprehensive holistic approach, OST programs offer a safe environment to supplement an academic curriculum while also addressing individual needs outside of school. Although the popularity of afterschool programming has increased at the federal, state, and local levels in recent years, and the number of existing programs has exploded, many afterschool programs struggle to maintain high quality standards in their practices due to capacity issues. Thus, improving and enhancing capacity should result in more high quality OST programs, with more qualified and knowledgeable staff, improved processes, sustainability, and dynamic curricula.
The April 24, 2008 event was the second in a three-part forum series, sponsored by the Charles Stewart Mott Foundation, examining capacity-building efforts in OST programs with a focus on New York City’s (NYC) 2005 OST initiative. Mayor Michael Bloomberg’s vision has drawn on resources in both the public and private sectors to extend high-quality services to all youth in NYC, with a priority on serving those in high-need neighborhoods. The City’s Department of Youth and Community Development (DYCD) has focused on building capacity using four organizing principles: Collaboration, Redefining Quality, Accountability, and Sustainability. This forum highlights NYC’s efforts to build program capacity, results from the second year of a longitudinal evaluation, and a ground-level perspective on how the OST initiative is helping improve local OST programs.
Forum Highlights
Jeanne Mullgrav, Commissioner, DYCD, asserted that OST represents the largest block of time youth spend outside of school, encompassing summer, school holidays, weekends, and evenings. Mullgrav explained the importance of redefining OST because of the opportunity for youth to develop social and academic skills. As the OST initiative began, Mayor Bloomberg asked the DYCD to help the city rethink and reshape existing afterschool programs. Mullgrav described existing youth-serving agencies as fragmented and commented that data was not being uniformly collected. Mullgrav pointed out another concern regarding quality. “How do we make sure that we have even quality across all programs so that a mother can expect, when she walks in, that there are service standards and a certain level of quality? How do we make sure that is replicated across the city?” asked Mullgrav.
With the help of the Wallace Foundation, investments were made in building the infrastructure of the new OST system. Mullgrav explained that the Wallace Foundation was interested in building a system that would serve as the glue that holds the system together. In order to target resources to reach isolated areas, five indicators were used to identify high-need zip codes: Percentage of youth population, youth poverty rate, English language learners, numbers of single parents, and numbers of disconnected youth. Currently, 63% of OST programs operate in priority zip codes.
Mullgrav continued to describe how the four organizing principles of the OST initiative have helped establish the current OST system. To use program dollars more efficiently, Mullgrav explained that by accessing public-use facilities, OST resources could be focused on human capital. Sixty-five percent of OST programs are located in public schools while others collaborate with the NYC Housing Authority, NYC Parks Facilities, homeless shelters, and Department of Juvenile Justice facilities. By collaborating with other public resources and operating more efficiently, programs are able to leverage public dollars with private dollars, explained Mullgrav.
To redefine quality, Mullgrav stated, “Research has shown that high quality OST programs offer a broad range of activities.” Although there has been pressure on programs to increase academic student performance, Mullgrav said research has shown that OST programs that offer social, personal, civic, and career development activities can have a positive effect on academic outcomes. Mullgrav also believes that regular attendance is a key factor in positive outcomes.
Accountability has been directly integrated with the OST Online data management system. OST Online is designed to track attendance, participation, and a range of individual participant characteristics such as food allergies, participation in activities, and who can pick up a participant from the program. Information from OST Online can be used by a range of users in a variety of ways. Mullgrav explained, “It is not enough just to collect the data. It is important to look at the challenges that our programs are having in meeting their goals.” Mullgrav continued to explain that evaluators go to sites and work with staff and participants to address areas of need. According to Mullgrav, this process builds capacity and sustainability in a consultation-oriented manner.
Elizabeth Reisner, Principal, Policy Studies Associates, Inc. (PSA), discussed the second year findings of the longitudinal evaluation of the OST initiative in NYC. Reisner pointed to three lessons learned from the second year evaluation:
- It is possible to launch, support, and grow a very large OST initiative that is not centrally focused on improving test scores.
- Even operating at scale, an OST initiative can adopt and implement improvement strategies that quickly demonstrate positive results.
- Rigorous evaluation can reveal program findings that can be fed back to programs through administrative tools.
Reisner continued to discuss key findings of their second year evaluation. When discussing program goals, OST program directors said their primary goals were to provide youth a safe environment and help them develop socially. In terms of staff quality, the evaluation found that directors had high levels of education and less difficulty hiring staff with diverse qualifications and experience partly because of positive word-of-mouth hiring referrals. DYCD also worked to align training and technical assistance with identified needs of the programs, which improved staff quality.
With respect to programs, PSA found that the main activities offered in elementary grades were homework help, in collaboration with school-day teachers, and subjects no longer focused on during the school day such as visual arts and crafts. Middle grades added a focus on team sports, while high school activities emphasized civic and social development. Reisner also noted that what youth liked best was the exposure to new experiences such as art activities, dance, field trips, and different physical activities. Self-reports by youth participants revealed a strong sense of belonging regarding their OST programs and a moderate level of engagement in prosocial behaviors.
The evaluation also found that programs achieving the highest attendance rates were those that:
- Employed college or high school students or school-day paraprofessionals or aides, thus achieving higher staff-to-youth ratios.
- Actively reached out to families via parent liaisons and other means.
- Delivered activities focused on academic enrichment and support.
Richard Berlin, Executive Director, Harlem RBI, spoke from a ground-level perspective and explained that Harlem RBI’s goals are holistic and comprehensive. Key outcomes are physical and mental health, high school graduation, college matriculation, and the development of work and life skills. This work all starts with basic physical and mental health. “We translate that into sexual health,” stated Berlin. Recognizing community concerns and issues, Berlin explained that teen pregnancy is a community issue that leads many youth to dropout out of school and earn low-paying wages. Using baseball and softball as a youth development tool, Harlem RBI strives to help youth graduate from high school, attend college, and become work-ready, while learning to be good teammates, friends, family, and active citizens. To achieve goals, Berlin discussed the importance of having high quality staff and trained professionals to meet kids where they are.
To accomplish this, Harlem RBI utilizes OST funds to support their Team Enrichment Program. Designed for youth ages 13-18, three age-specific components operate to achieve program goals. Berlin described the three different groups:
- Teambuilders (ages 13-14): Utilizes team-based experiential learning, providing opportunities for youth development in a safe, supportive and engaging environment.
- TeamWorks (ages 15-16): Develops job readiness skills, academic responsibilities, and leadership abilities such as planning, building consensus and public speaking.
- DreamWorks (ages 17-18): Focuses on developing college knowledge and engages parents in their children’s efforts to matriculate to college.
Berlin explained that the OST initiative has helped Harlem RBI to build organizational capacity. Not only does the OST initiative provide funding, additional components create a sustainable system. The OST Online management system helps program staff to track data while also creating a community of afterschool providers. In addition, Berlin believes that the OST initiative helps to add credibility and attracts private funding needed to successfully operate and manage programs.
Highlights from the Question and Answer Session
A question was asked why test scores were not used as a measure of success in the evaluation. Reisner responded by stating that academic improvement is not the major outcome desired, and that growth on indicators of positive youth development supersedes academic improvement among OST program goals. Reisner did mention that a quasi-experimental design is being constructed to look at academic progress in the third-year evaluation. Berlin reiterated that from a practical perspective, “This is really about youth development.” And that was a key promise DYCD make to the provider community in NYC.
Another question was asked regarding the focus on attendance as a major indicator of quality and other measures of program quality. Mullgrav commented that quality is assessed at different levels from staff credentials to what parents think about the program. In addition, Mullgrav believes that attendance is one of the many things that are looked at because if youth are not there, then they cannot be served. Christopher Caruso, DYCD Associate Commissioner for OST Programs for NYC, commented on the change in culture with regard to evaluation. Caruso stated that before the OST initiative, program reviews were punitive if quality standards were not met. OST contracts are now performance-based, so programs are still accountable for meeting enrollment and attendance targets. Through the new OST system, a contract manager goes to the site and is knowledgeable about high quality afterschool programming. Utilizing monitoring tools that look at quality, evaluators can serve in a coaching role to DYCD staff, who, in turn, work with site staff to make improvements, before punitive measures need to be taken.
Presenters
Jeanne B. Mullgrav was appointed by Mayor Michael R. Bloomberg to be Commissioner of the New York City Department of Youth and Community Development (DYCD) in 2002. As the City’s lead agency for administering youth and community programs, DYCD invests public funds in experienced community-based organizations that impact neighborhoods throughout the five boroughs. Prior to her appointment, Mullgrav served as Vice President for External Relations at The After-School Corporation (TASC), a not-for-profit organization dedicated to providing quality after-school programs. Previously, she was the Deputy Director of Intergovernmental Relations and held several senior management positions at Victim Services (now “Safe Horizon”), a major not-for-profit provider of services to crime victims. Mullgrav holds a B.A. from Swarthmore College and a J.D. from New York University School of Law.
Elizabeth Reisner is one of two founders and managers of Policy Studies Associates, Inc., a Washington-based firm that conducts research and evaluation in education and youth development. Ms. Reisner serves as principal investigator of evaluations of several large-scale after-school program initiatives, including the Out-of-School Time Programs for Youth Initiative and the Beacon Centers Middle School Initiative in New York City, New Jersey After 3, Citizen Schools, and Save the Children’s U.S. Programs. Ms. Reisner also oversees the evaluation of Partners in Excellence, a college training program for after-school staff, sponsored by the Center for After School Excellence, an affiliate of The After School Corporation. She serves as an evaluation advisor to the Edna McConnell Clark Foundation.
Richard Berlin, Executive Director at Harlem RBI since 1997, is also a founding member of Harlem RBI’s DREAM Charter School. Mr. Berlin began his connection with Harlem RBI as a volunteer baseball coach in 1994. Under his leadership, Harlem RBI has grown from a seasonal recreation program with one staff member to a thriving community based institution with a budget of $6 million. During his tenure, Harlem RBI has been recognized with numerous awards on a city, state and national level. Mr. Berlin holds a Bachelor’s Degree in Political Science from the University of Wisconsin at Madison and attended a Master’s Program in Political Theory at the London School of Economics and Political Science in London, England. Additionally, he has completed certificate programs at Columbia University’s Institute for Not-for-Profit Management in both Non-Profit Management and Leadership Development. Mr. Berlin has also been a Leadership Fellow at the Citizen’s Committee for the Children of New York City. He is an adjunct faculty member at The New School University’s Robert J. Milano Graduate School of Management and Policy, currently serves as the Co-Chair of the Council of Community and College Leaders at the Center for After School Excellence, and sits on the Program Council at The Partnership for After School Education and the National Advisory Board of the Johns Hopkins University’s Center for Summer Learning.
Resources
New York City’s Out-of-School Time Initiative Powerpoint»
Richard Berlin
Executive Director
Harlem RBI
333 East 100th Street
New York, NY 10029
212-722-1608 x. 226
Jeanne B. Mullgrav
Commissioner
NYC Department of Youth & Community Development
156 William Street
6th Floor
New York, NY 10038
212-442-6053
Elizabeth Reisner
Principal
Policy Studies Associates, Inc.
1718 Connecticut Ave., NW
Suite 400
Washington, DC 20009
202-939-9780 x. 323
