Outcomes for Children and Youth in the Out-of-School Time:
What the Evidence Says
A Forum — April 30, 2004
Interest in out-of-school time (OST) programs for youth has grown enormously over the last few years. These programs embrace a wide variety of goals, such as providing safe and caring environments critical to the healthy development of youth and promoting youth academic development through remediation, reinforcement, and application of academic skills. The concept of universal access to OST programming is gaining ground and federal, state, local, and private funders have devoted considerable resources to expanding these programs and ensuring their quality. The federal government allotted $1 billion to one particular approach to out-of-school-time programming, the 21st Century Community Learning Centers Program for FY 2004.
To ensure that positive outcomes for youth are achieved through this and other OST programs, policymakers, funders, and practitioners need access to reliable evidence about program effectiveness. At this American Youth Policy Forum, a panel of researchers discussed the available data on the effectiveness of various OST programs and the challenges associated with conducting research on OST programs. Representatives of several OST programs spoke about their programs, goals, strategies used to achieve goals, and problems encountered. Through their presentations and discussions with the audience, panelists identified problems faced by practitioners and researchers, and issued recommendations for policymakers, funders, and OST program leaders.
Part One: Research Perspectives
Nicole Yohalem, program director, Forum for Youth Investment, opened the forum with a description of the general goals of positive youth development. These goals span a range of domains including the physical, intellectual, psychological, emotional and social. For instance, Yohalem said, it is essential to their psychological and emotional development that youth learn coping skills and positive self-regard; in addition, they need to develop risk management and decision making skills and the capacity to navigate in a variety of cultural contexts. Yohalem reminded the audience that if we are to promote healthy development of youth, we must focus on the full range of outcomes; unfortunately, sometimes policymakers and others focus on the cognitive at the expense of other domains, and researchers and practitioners must be sure to emphasize that the cognitive domain is only one of many that are necessary for healthy development. Fortunately, OST programs can and do emphasize multiple domains; again, though, those involved with these programs have to be careful to resist the pressure that is sometimes exerted on them to focus solely on the cognitive domain.
If we consult public opinion, we see that people generally agree positive youth development spans a wide range of domains and includes acquisition of “soft” skills and attitudes, such as collaboration and tolerance, in addition to the traditional trio of reading, writing and arithmetic. People generally agree that schools should be teaching the range of 21st century skills; however, few believe that schools currently do a good job in this area. The public believes that other institutions and agencies should also play a role if the goals of youth development are to be achieved.
Out-of-school-time programs are a means to achieving these multiple goals for youth necessary to their healthy development. What, then, are the characteristics of highly effective out-of-school-time programs for youth? According to the National Research Council, developmentally positive settings promote youth’s physical and psychological safety, offer appropriate structure and supportive relationships, provide opportunities to develop a sense of belonging and build skills, expose students to positive social norms, and integrate family, school and community support systems. While it is clear that out-of-school-time programs with these characteristics can and do play an important role in positive youth development, researchers and practitioners need to gather more data about OST programs, in particular, who participates, the intensity, duration, and breadth of their participation, and the effectiveness of various types of programs.
Those who are interested in OST programs are wrestling with a difficult question: What is realistic to expect OST programs to achieve? Overall, the panel is cautiously optimistic about the outcomes that can be and are achieved. We must keep in mind, though, that it may not be appropriate to think that all programs can achieve every positive youth development outcome.
As we think about OST programs, we must also attend to the policy context within which these programs are situated. The standards and accountability movement has placed enormous pressures on OST programs; they are now expected to show their impact on academic outcomes. This is problematic for several reasons. First, academic development is only one of many essential to healthy growth and OST programs rightly focus on domains other than or in addition to the academic. Second, these programs are only one of many variables that affect youth development and youth spend a limited time in the programs. We must look at school, OST programs, family, and other community services working together to create desired outcomes. On a positive note, there seems to be a growing awareness amongst policymakers that these programs are only one of many influences on youth development.
Beth Miller, senior research advisor at the National Institute on Out-of-School Time (NIOST) and author of Critical Hours: Afterschool Programs and Educational Success, spoke about the links between OST and positive youth development. Her analysis of the research (presented in Critical Hours) found that “afterschool programs can increase engagement in learning, increase educational equity, and build skills for long-term success.” Afterschool programs are particularly good at strengthening student engagement in the learning process. Miller cautioned that we should not be so focused on producing immediate improvements in academic achievement that we lose sight of the more this more broadly important educational goal.
How do afterschool programs increase engagement in learning? They help build stronger relationships between youth and adults, and foster better work habits, increased feelings of personal efficacy, and higher educational aspirations. Research also shows that afterschool programs can increase educational equity by providing youth with experiences that their more affluent peers may get through other sources. OST programs also offer students opportunities for leadership and mentoring, and subject them to high expectations by staff who come to know them personally.
Afterschool programs help students acquire the “new basic skills” such as “literacy and numeracy, teamwork, problem-solving and analysis, communication, working with diverse people, and technology skills,” Miller said. Research shows that these programs foster “better teamwork skills, better conflict resolution skills, increased self-control and self-respect, greater skills in analysis and problem-solving, and greater skills in oral and written communication.”
Achievement of the positive results that some programs have reached, however, is not automatic. To produce such results, program leadership must have intentionality (they must know what they are trying to achieve and how to achieve it), high quality staff who have positive relationships with youth, challenging and interesting activities for youth, and a certain intensity of youth participation. People who are committed to and engaged with the program are essential to its success; however, if the appropriate infrastructure is not in place to support their work, they will burn out and the quality of the program will suffer. Programs must have appropriate evaluation procedures, adequate public engagement, sufficient financial support, staff development opportunities, and supportive community partnerships.
The full text of Critical Hours and an executive summary are available at the Nellie Mae Education Fund website: www.nmefdn.org.
Elizabeth Reisner, co-founder and manager of Policy Studies Associates (PSA), discussed the preliminary findings from PSA’s recent evaluation of two after-school programs, The After-School Corporation (TASC) in New York City and the Citizen Schools Program in Boston. The programs are similar to one another and typical of many afterschool programs: they are school-based but use external staff and support, serve more disadvantaged students than the district norm, and provide services to promote student growth in academic and non-academic areas; they are also under considerable pressure to demonstrate concrete academic outcomes while operating with limited financial resources.
According to Reisner, both TASC and Citizen Schools place high value on social outcomes but stakeholders increasingly emphasize academic outcomes due to high-stakes testing and budgetary constraints on after-school services. As a result, each program has had to make their case for external support based on student academic outcomes, while at the same time trying to educate their stakeholders about the other important outcomes that these programs produce for youth.
Preliminary findings from the evaluation of the TASC program show improvements in student school attendance rates relative to non-participants, with larger gains among frequent participants. There have also been gains in math achievement for participants, with larger gains among disadvantaged students. There were no overall gains in reading across all TASC programs although individual TASC programs did show improvements in reading for their participants.
Principals who were interviewed and surveyed said TASC programs brought parents into the schools and improved their schools’ overall effectiveness. They also said TASC exposed students to positive learning opportunities they would not otherwise have had (such as art and music), since these programs have been cut from many schools, due to the academic focus of recent school accountability measures. Students had positive things to say about their participation in the TASC programs, including that they provided a community characterized by trust, caring, and respect, exposed them to new learning opportunities, and helped them learn skills that made them better students.
Preliminary findings for the Citizen Schools’ evaluation also show promise. Eighth grade participants chose to apply to more challenging high schools compared to a similar group of non-participants; sixth-grade participants had more positive outcomes than a similar group of non-participants on measures such as school attendance, school suspension rates, English language arts (ELA) scores on the state test, ELA course grades and grade level promotion. In addition, students, staff, and volunteers reported that they were very satisfied with the program and that they had experienced good adult-student and student-student relationships. Students reported frequent engagement in oral presentations and writing, as well as numerous opportunities to become engaged in leadership, decision-making, and active learning.
Looking across the evaluations of these two out-of-school-time programs, it is possible to draw some general conclusions, according to Reisner. We know that these programs can “create the conditions for accelerated student academic growth and yield gains in key academic domains.” Stakeholders whose primary focus is on academic measures may find this result of particular interest, said Reisner. At the same time, though, they and others should keep in mind that student growth is visible in non-academic areas as well. This growth contributes to academic growth but “is valuable in its own right.”
Robert Granger, president of the William T. Grant Foundation, reviewed the results from four out-of-school-time program evaluations—21st Century Community Learning Centers, TASC, Extended-Service Schools Initiative, and the San Francisco Beacons Initiative—and a number of ongoing studies conducted by the Foundation’s grantees. He offered some cautions about what we definitely know from research. “Are after-school activities of various types related to youth development outcomes?” asked Granger. “Yes. We know that what young people do after school is related to how well youth do.”
At the same time, he warned that the relationship may not be one of cause and effect. “Do after-school programs affect important youth development outcomes? Sometimes,” he said. It appears that after-school programs can make a difference, but the average program probably does not, and it needs to be improved.
“Are there best program practices? Yes,” he assessed, “There is reasonable agreement among practitioners and researchers on the program practices that are related to youth outcomes; we do know what good programs look like and some of today’s practitioner panelists will describe such programs and their attributes.” On the other hand, we have limited knowledge about how to improve programs to bring about enhancements in student outcomes. He said, “We have no solid evidence and not much theory on how to advance the quality of programs. It is here that there is a need for better information.”
The lack of results-proven techniques for improving programs should not leave us paralyzed, though, he said. Given our consensus on the features of quality programs, we should test interventions to see if we can improve “everyday” programs. While doing so, we should be careful not to over claim that we are sure that after-school programs make a difference, and we should not expect that the programs can do it all.
Granger also shared some observations about youth participation rates. He said that while we cannot assume that high rates of participation alone will lead to positive outcomes, low participation rates in a program likely mean that the program is not making much of a difference in the lives of youth. He argued that we should try to promote high participation rates by improving program practices and putting programs where they are needed.
How can we improve programs? Granger suggested a number of areas for further study, including adjusting policy to reward program improvement, providing ongoing on-site coaching for staff, giving program staff help with curriculum development and alignment, and encouraging ongoing program, staff and participant evaluation. Granger closed by calling for a focus on staff. “When we want to improve a program, we should focus on changing what program staff do with the youth. Staff behavior is key to improving outcomes and we should keep this in mind at all times.”
Panelists responded to questions from the audience on a variety of topics. Issues related to the federal No Child Left Behind Act were of particular interest and concern to both panelists and the audience. Panelists agreed that No Child Left Behind is having a definite impact on out-of-school-time programs and that this impact must be taken into account by researchers. Reisner argued that there is a need for more research on OST programs. We need to know more about what is currently the case in OST programs. This will help us anticipate what might happen to these programs given the likely direction of policy and its influence on them, Reisner said. We need to design high quality experimental studies on a small scale to be able to do this. We also need research, argued Reisner, which is deliberately and carefully designed to show us whether it is OST programs that are responsible for producing changes in student outcomes.
Panelists stated that we should avoid an overly narrow understanding of evaluation. Evaluation should not focus solely on holding OST program staff accountable; rather, it should provide information that can be used to help staff figure out how to do what they do even better than they do it now. Some concerns about the implications of No Child Left Behind for OST programs were voiced. Granger argued that it may not be helpful to create policies that put in place strict certification requirements for hiring program staff. It is those who are passionate about their work with youth who often make the difference in outcomes produced and we do not want to block their entry into the field. Miller expressed the concern that the accountability movement might lead us to expect too much from OST programs.
Panelists also addressed how they would choose to invest additional resources in OST programs if these resources were to become available. Miller, Reisner, and Granger all said that investments in staff should be a priority. Reisner agreed that OST program staff need to feel that they are part of a profession and investment in profession-related activities would be very helpful. Miller said that investment in staff, hiring quality professionals and offering sufficient staff development would be an effective way to leverage improvements in OST programs.
Part Two: Practitioner Perspectives
In the second half of the forum, representatives from several out-of-school-time programs responded, providing information about the historical development of their programs, outcomes sought and strategies employed.
Earl Martin Phalen, chief executive officer, BELL (Building Educated Leaders for Life), spoke about his program, a community-based non-profit organization that operates OST programs for children living in low-income communities in the Boston, New York City, and Washington, D.C. areas. BELL targets children in grades K-6 from underperforming, high-poverty schools and offers two types of OST programs, one during the afterschool hours and the other during the summer. Each program is designed to increase children’s mastery of basic academic skills, raise their academic expectations and self-esteem, empower parents, and provide effective mentoring relationships between children and positive adult role models.
An evaluation of the summer program conducted in 2003 showed very positive results. Students who participated in the BELL program experienced significant gains of seven months in their reading and math skills as measured by results on the Stanford Diagnostic Tests, and parents and teachers noted improvement in scholars’ academic abilities, self-esteem and social skills. Parents were engaged with and satisfied with the program. An additional evaluation is planned for the 2004 and 2005 summer programs; this study will be the first ever use of an experimental design to evaluate a multi-site summer program.
Phalen identified several characteristics as essential to the success of the BELL program, including strong leadership, small teacher to student ratio, caring role models who believe in the capabilities of their students, rigorous curriculum, and strong partnerships with parents, schools, and the community
Phalen spoke about the impact of the current policy context on BELL and other OST programs. No Child Left Behind has had a “major, major impact on the program,” he said. With the funding associated with the provision of supplemental services to children in low-performing schools, BELL has been able to expand participation to provide services to 5,000 students. This has been a wonderful opportunity. On the other side, though, many of the for-profit groups recognized as approved OST supplemental service providers that are capturing significant funding do not have programs that serve youth well. Their programs tend to be too short and they do not produce long-term academic gains or allow youth to develop sustained relationships with positive adult role models.
Davon Russell, youth services director, Women’s Housing & Economic Development Corporation (WHEDCO), described WHEDCO, a community-based multi-service organization that provides year-round comprehensive multidisciplinary school-based programs for K-8 students at a magnet school in the New York City area. WHEDCO offers a summer day camp and an afterschool program during the school year. While the primary focus has been on the arts, WHEDCO believes there are important natural connections between the arts and academics. As children experience enjoyment in the arts programs when they become involved and engaged, they build confidence which carries over into and supports academic engagement and success.
Over time, the organization has moved to incorporate a more direct emphasis on academics, realizing that some of their students do not have access to resources needed to be successful in high school and beyond. WHEDCO has begun to hold a high school fair to help middle school students make informed decisions about which high schools to attend and provides assistance during the middle school years to prepare students to get into the high schools of their choice.
According to Russell, the important characteristics of WHEDCO programs include: a diverse staff with a teacher to student ratio of one to ten, strong parental involvement, a full-time coordinator whose sole responsibility is to manage the program, a safe environment, and positive attitudes among all staff involved with youth.
Adrian Haugabrook, executive director, Citizen Schools University (CSU) for Citizen Schools in Boston, spoke about the Citizen Schools approach that provides after school services within existing schools. Citizen Schools build on in-school learning to help students strengthen community connections and social networks, develop leadership skills and self-knowledge, gain access to technology, mentors, and valuable institutions and organizations, and develop writing, oral presentation and data analysis skills. Students participate in activities to build community within and outside the school and have opportunities to engage in structured explorations of novel topics and take part in special apprenticeship classes.
CSU has grown over time to include new sites in Boston and in other states. CSU continues to increase the number of schools in which they provide services. According to Haugabrook, there simply are not enough quality OST programs available to meet the current public demand. Haugabrook identified challenges facing Citizens Schools, including those shared by other OST programs, such as how to most effectively link after school activities with the curriculum of the school, and how to maintain excellence and develop multiple models of excellence as the program expands to new sites.
Jeff Wicklund, associate director, Youth+Entrepreneurship=Success (YES) program, spoke about his experiences with this innovative OST program. YES is an after school entrepreneurial education program of the Enterprise Center in Philadelphia that serves predominately African American high school students. The general mission of the Enterprise Center is to nurture entrepreneurs and support the growth of new companies. The Center was not founded to work with youth; however, when youth began reaching out to use the Center’s services, the Center recognized the importance of investing in its “future market” early and added a youth component.
The organization’s work with youth focuses on nurturing their entrepreneurial spirit and preparing them for success in the business world. There is an emphasis on helping to bridge the gap between high school and higher education; students are given support through test preparation, coaching and visits to college campuses. They are immersed in a professional and nurturing environment that allows them to explore entrepreneurship with an eye toward operating their own businesses in the future. The program also helps youth see the connection between the skills they have already learned in school and the skills needed in business. Coming to see this connection helps students master the academic curriculum because they are intrinsically motivated to learn; they are thinking about the future, how to achieve their goals, and what they need to learn if they are to achieve these goals.
Wicklund said that potential staff burnout is the most serious concern facing the program. Program leaders must juggle numerous roles to keep the program afloat and they are “spread pretty thin,” he said. Given additional resources, he would invest in additional staff, one of whom could provide technological support.
After the roundtable discussion, panelists responded to questions about their programs. Asked about challenges related to sustainability, Haugabrook and Wicklund said that funding issues are problematic. They must constantly engage in a “hustle” to acquire and sustain resources; they sometimes do not know how much money they will have from month to month or what will happen to their programs. Phalen said that No Child Left Behind has been a boon to them since it has meant an influx of additional funds. NCLB in general presents “an enormous business opportunity.” With a large number of low-performing schools, many students will have vouchers to spend: there is a market upon which providers can capitalize. The biggest challenge, he said, is that “many families do not know that they are eligible for services and what services are available to them. OST program providers need to reach out to them and let them know they are there.” Panelists generally agreed there is an enormous need for quality OST programs right now and this need will not evaporate any time soon. Wicklund said one challenge associated with funding is deciding which funders to pursue. “You can’t pursue everyone and their priorities,” he said. “You have to decide: this is what we do, this is what we do well, and then focus on doing it.”
Haugabrook said that expanding to serve new communities can be a serious challenge. Some modifications are necessary given differing community needs and assets but some elements must be non-negotiable if quality is to be retained, he said. Russell said that one challenge is to educate parents about what works for their children. Haugabrook agreed: Parents sometimes focus solely on academic outcomes and want immediate results, but this is not always what’s best for students. Phalen noted a similar problem with funders and said that this can be very frustrating. It is a barrier to serving youth and may unfortunately serve to push some nonprofit OST program providers out of business.