2022-23 and 2021-22 School Year Analysis
Katie James, Ph.D., Director of Research
Kelsey Rocha, Ph.D., Research and Data Consultant
Neha Pandit, Senior Manager of Research and Program Operations
Summary
This study examined the impact of PowerMyLearning’s Social Emotional Learning (SEL) program and Family Playlists on student math and SEL outcomes and teacher practice across the 2022-23 and 2021-22 school years. PowerMyLearning’s SEL program includes professional development and coaching focused on instructional practices that support SEL. Family Playlists, PowerMyLearning’s tech tool, extend children's learning beyond the classroom through fun, off-line, standards-aligned learning games. The combination of Family Playlists and the SEL program is designed to help students build foundational math skills while developing positive math mindsets, such as optimistic thinking, to set them up for success in future math learning.
Math Achievement Outcomes
There was a significant difference in the proportion of K-2 students who exceeded or met national grade-level math achievement benchmarks before and after Family Playlists and the SEL program (+28 percentage points (pp), p<.001).
There is also evidence that we are successfully reaching Black and Hispanic students. There was a significant change in the proportion of Black students who exceeded or met math benchmarks before and after Family Playlists and the SEL program (+31pp, p<.001). For Hispanic students there was also a significant difference in the proportion of students who exceeded or met math benchmarks before and after Family Playlists and the SEL program (+22pp, p=.01).
Social Emotional Learning Outcomes
PowerMyLearning’s Family Playlists and SEL program were related to significantly better SEL (+7pp, p =.02, ηp2 =.02) for kindergarten students who were assigned Family Playlists and whose teachers participated in the SEL program. When looking at K-4 students whose teachers participated in the SEL program, the SEL Program was associated with significantly better self-awareness (+6pp, p =.02, ηp2 =.02).
The SEL program had an even more pronounced impact on SEL for high needs students, including students with learning differences and Hispanic students. Specifically, K-4 students with special education status had significantly better overall SEL (+10pp, p<.01, ηp2 =.17) after the
program. Hispanic students had significantly better relationship skills (+12pp, p =.01, ηp2 =.13). Consistent with the overall student population, Black students had significantly better self-awareness (+6pp, p=.03, ηp2 =.02).
Teacher Practice Outcomes
Following the SEL program, teachers showed significant improvement on the instructional practices targeted by the program: inclusive community (+53pp, p<.001, ηp2=.66), culture of learning (+50pp, p<.002, ηp2=.65), and SEL-supporting classroom routines and procedures (+26pp, p=.002, ηp2=.50). For example, one 2nd grade classroom made major progress towards building an inclusive community that was noted by the coach as well as school administrators. At the end of the year, there was a clear sense of belonging among students in the classroom and the students and teacher consistently used a calm and respectful tone. Affirmations such as “thank you for being honest” and “don’t worry there are a million things to do” were commonly heard. The teacher consistently used a facilitative stance with only minimal, guided teacher talk.
To support a culture of learning, one teacher began implementing a “make things right” lesson to address student conflict and support self-reflection. Similarly, another teacher started using restorative circles to productively address community issues as a group.
As a result of the program, 100% of teachers implemented practices that strengthened their learning relationships with students and families. There was also a 23 percentage point increase in teachers’ connectedness to their students’ families and a 26 percentage point improvement in the effectiveness of teachers’ efforts to engage families in students’ learning during the program.
Research Context
This study examined the impact of our SEL program and Family Playlists on student math and SEL outcomes and teacher practice at an elementary school in New York during the 2022-23 and 2021-22 school years. This school has 99.3% minority enrollment, including 75% Black, 19% Hispanic, 3% Asian American Pacific Islander, and 1% White students, and 77% of students are eligible for free or reduced-price lunch.
PowerMyLearning has partnered with the elementary school since 2016. Strong school leadership shaped the focus of the partnership. As SEL became a key priority for the district, the school's leadership sought out our SEL program and Family Playlists to support teacher and student learning needs around SEL and set them up for success on this city and district priority.
During 2022-23 and 2021-22, all 16 K-5 teachers participated in the SEL program. During 2022-23, the seven K-2 teachers additionally assigned Family Playlists to their 176 students. During 2021-22 spring semester, the three Kindergarten teachers also assigned Family Playlists to their 38 students. Details of the study and our impact are provided in the research methods section.
PowerMyLearning SEL Program
The elements of PowerMyLearning’s SEL program, called “Nurture Student Growth Through Social Emotional Learning (SEL)” program are designed to build teacher capacity by leveraging best practices for teacher professional learning. Our educator workshops for teachers are always followed by small-group coaching to extend learning into daily practice.
Across the 2022-23 school year and 2021-22 spring semester, the partner school received the following:
- Four Educator Workshops designed to provide strategies for interpreting and addressing challenging behaviors in and out of the classroom, and for building an inclusive community of motivated learners. Workshop topics included: Trauma-Informed Practices that Benefit All Learners, Mindfulness and More for Kids and Adults, Build a Classroom Community Where All Student Feel They Belong, and Transform Discipline into a Learning Opportunity.
- Coaching to extend learning from the Educator Workshop into daily practice. Each teacher participated in four cycles of three coaching sessions (12 sessions total). Coaching sessions were 45 minutes each, resulting in nine hours of coaching per teacher.
- Four Family Workshops delivered in Spanish and English that were designed to provide strategies for families to support their children’s well-being, as well as their own.
- Family Engagement Action Planning designed to set goals and develop an implementation plan around engaging families as partners in their child’s learning. The Community Schools Director participated in a total of ten sessions of 1:1 consulting and three professional learning sessions with family-facing staff from other schools in the district.
Family Playlists
Family Playlists are hands-on, game-based math extension activities. Teachers schedule weekly Family Playlists assignments aligned with what they are teaching in the classroom. PowerMyLearning provides custom Family Playlists alignment to each district’s curriculum to support the assignment process. Family Playlists are sent to the mobile device or email of each student’s learning partner, a trusted adult in the child’s life, who completes the assignments with them. The assignments consist of two parts: a learning game and an “explain what you learned” activity. The learning game is a short off-line game that students complete with their learning partner using easy- to-find materials. After playing the learning game with the student, the learning partner records and uploads a video of the student explaining what they learned and provides feedback to the student’s teacher. The teacher can then use the student video and feedback data to tailor their instruction.
K-2 teachers assigned Family Playlists throughout the 2022-23 school year. Kindergarten teachers additionally assigned Family Playlists during spring semester of the 2021-22 school year. At the beginning of the 2022-23 school year, K-2 teachers and school leaders also attended Family Playlists launch sessions which are designed to provide training and resources to support effective implementation.
Research Methods
In this section, we will describe the study design, instruments, data analysis, and results. We will then discuss the implications and limitations of the results and next steps of this work.
Design
We used a pre-post study design to examine the impact of the SEL program and Family Playlists on student math achievement and SEL competencies as well as teacher practice.
Instruments
Math Assessment. Math achievement data was based on school-administered iReady math diagnostic assessments. IReady is an online, adaptive assessment. The iReady data used in the current study is overall grade-level placement. Overall grade-level placement uses national grade-level norms to determine the grade-level at which each student is currently performing.
SEL Assessment. Teachers completed the DESSA SEL assessment, which is regularly administered by the school district, for each student in their class. DESSA is a nationally standardized, norm referenced SEL assessment that is aligned to the CASEL framework. The DESSA includes 72 questions focused on eight SEL competencies: Goal-directed behavior, self- awareness, optimistic thinking, decision making, social awareness, relationship skills, self- management, and personal responsibility. Each subscale includes between seven and eleven questions. For each question, the teacher is asked to indicate on a five-point scale how often the student engaged in each behavior over the past four weeks.
Teacher Practice Observation Rubric. PowerMyLearning coaches conducted classroom observations to examine changes in teacher practice related to the learning environment practices targeted by the SEL program: inclusive community, culture of learning, and SEL supporting classroom routines and procedures. Observations were guided by a rubric that provided a five-point rating scale for each practice with definitions for each level and exemplars for each practice. Coaches also provided evidence from their observations to illustrate their scores.
Teacher Survey. At the beginning and end of the program, a survey was administered to all K-5 teachers, school leaders, and other staff. The goal of the survey was to understand teachers’ perspectives on the impact of the program, focusing on relevant instructional practices and teacher-student-family relationships. Survey data was collected from 15 teachers and staff members.
Data and Analysis
Math Data and Analysis. Math achievement data was collected from the school for K-2 students in Fall 2022 (pre) and Spring 2023 (post). Pre/post math data was available for 176 K-2 students, which included 147 Black students, 42 Hispanic students, and 25 students with learning differences.
We analyzed whether each student exceeded or met national grade-level math benchmarks at each timepoint, based on their iReady overall grade-level placement and actual grade-level, resulting in categorical data. We then conducted McNemar’s tests on the pre- and post-data for each assessment. We also calculated the percent of students who exceeded or met grade- level benchmarks at the beginning and end of the year.
SEL Data and Analysis. The DESSA produces eight scale scores, one for each SEL competency, and an SEL composite score. The composite score is the sum of the scale scores. SEL assessment data was collected from 21 teachers in grades K-4 in Fall 2021 (pre) and Fall 2022 (post). Pre/post data was available for 244 K-4 students, which included 38 Kindergarten students, 48 students with special education status, 209 Black students, and 47 Hispanic students.
We conducted repeated measures ANOVAs on the pre/post data for each SEL competency and the SEL composite score.
We also conducted the above math and SEL analyses for several subgroups, including students with special education status, Black students, and Hispanic students to ensure that our program had equitable outcomes across racial and ethnic groups and for students with learning differences.
Teacher Practice Data and Analysis. PowerMyLearning coaches conducted classroom observations for each of the 16 K-5 teachers at the beginning and end of the school year. These observations produced a score out of 5 for each targeted instructional practice at each time point as well as qualitative descriptions of the observed practices.
Repeated measures ANOVAs were conducted on the pre/post data for each practice. Themes and examples were generated from the qualitative data to illustrate key changes.
Results
Math Results. The findings indicate a statistically significant difference in the proportion of students who exceeded or met grade-level math achievement benchmarks before and after Family Playlists and the SEL program (+28pp, p<.001). After the program, 32% of students exceeded or met iReady math national grade- level norms as compared to 4% of students before.
There is evidence that we are successfully reaching the highest needs students, including Black and Hispanic students. For Black students, the change in the proportion of students who exceeded or met district benchmarks before and after Family Playlists and the SEL program was significant (+31pp, p<.001). After the program, 35% of Black students exceeded or met math achievement benchmarks versus 4% before. For Hispanic students there was a significant difference in the proportion of students who exceeded or met math achievement benchmarks before and after Family Playlists and the SEL program (+22pp, p=.01). After the program, 29% of Hispanic students exceeded or met as compared to 7% before. For students with learning differences, 17% met exceeded or met math benchmarks after the program as compared to 4% before. This 13 percentage point improvement was not statistically significant (p=.38)
SEL Results. The results indicate that PowerMyLearning’s Family Playlists and SEL program were related to significantly better overall SEL for Kindergarten students (+7pp, p=.02, ηp2 =.02). When looking at specific SEL skills, these students had significantly better self-awareness (+28pp, p<.001, ηp2 =.30), goal- directed behavior (+19pp, p<.01, ηp2 =.19), optimistic thinking (+16pp, p=.01, ηp2 =.17), and relationship skills (+15pp, p=.02, ηp2 =.14). The SEL program alone was associated with significantly better self-awareness among K-4 students (+6pp, p=.02, ηp2 =.02).
The SEL program had a more pronounced impact on students with learning differences. Indeed, students with special education status had significantly better overall SEL (+10pp, p<.01, ηp2 =.17) after the program. In terms of specific SEL skills, they had significantly better self-awareness (+26pp, p<.01, ηp2 =.17), personal responsibility (+22pp, p<.01, ηp2 =.15), optimistic thinking (+20pp, p<.01, ηp2 =.14), self- management (+18pp, p=.03, ηp2 =.10), social awareness (+16pp, p=.01, ηp2 =.12), and relationship skills (+15pp, p=.03, ηp2 =.09).
As well, Hispanic students had significantly better relationship skills after the program (+12pp, p=.01, ηp2 =.13). Black students had significantly better self-awareness after the program (+6pp, p=.03, ηp2 =.02).
Teacher Practice Results. Following the SEL program, teachers showed significant improvement on the three targeted instructional practices: inclusive community (+53pp, p<.001, ηp2=.66), culture of learning (+50pp, p<.002, ηp2=.65), and SEL-supporting classroom routines and procedures (+26pp, p=.002, ηp2=.50).
For example, one 2nd grade classroom made major progress towards building an inclusive community that was noted by the coach as well as school administrators. At the end of the year, there was a clear sense of belonging among students in the classroom and the students and teacher consistently used a calm and respectful tone. Affirmations such as “thank you for being honest” and “don’t worry there are a million things to do” were commonly heard. The teacher consistently used a facilitative stance with only minimal, guided teacher talk.
To support a culture of learning, one teacher began implementing a “make things right” lesson to address student conflict and support self-reflection. Similarly, another teacher started using restorative circles to productively address community issues as a group.
As a result of the program, 100% of teachers implemented practices that strengthened their learning relationships with students and families. There was a 23 percentage point increase in teachers’ connectedness to their students’ families. After the partnership, 80% felt very or extremely connected to their students’ families as compared to 57% before. There was also a 26 percentage point improvement in the effectiveness of teachers’ efforts to engage families in students’ learning. At the end of the program, 83% rated their efforts as very or extremely effective versus 57% at the beginning.
Discussion
PowerMyLearning is committed to building evidence around the impact of our work. This study found statistically significant effects on math achievement, multiple aspects of SEL, and teacher practice related to PowerMyLearning’s Social Emotional Learning (SEL) program and Family Playlists. The SEL findings suggest that the combination of Family Playlists with the SEL program provides an even stronger impact on student SEL than the SEL program alone. One limitation is that these results are correlational rather than causal. Indeed, these findings reflect the impact of all learning experiences during the 2021-2022 and 2022-23 school years and cannot be directly attributed to our intervention. Beginning in Fall 2023, a randomized control trial is planned to examine the impact of PowerMyLearning’s SEL program and Family Playlists, which will enable us to more precisely determine the extent to which our programs are having causal effects. We will also analyze SEL outcomes for the 2022-23 school year, when data becomes available, to corroborate these findings.
Acknowledgments
This study was generously funded by the Robin Hood Foundation.
