2021-2022 Analysis by Katie James, Ph.D., Director of Research, and Neha Pandit, M.A., Senior Manager of Program Research and Operations
This study examined the impact of PowerMyLearning’s education technology innovation, Family Playlists, on the Social Emotional Learning (SEL) outcomes of Kindergarten students. Family Playlists are designed to activate the Triangle of Learning – a unique approach that we have been pursuing and studying over the last few decades. The playlists are fun, curriculum-aligned extension activities delivered to families each week via text message in over 100 languages. At the end of each playlist, the child explains an academic concept to a family partner who submits a video of this activity along with additional feedback to the teacher. This step draws on the “Learning by Teaching” research (e.g., Fiorella & Mayer, 2014) that has demonstrated improved student mastery and recall. This step also provides teachers with actionable feedback about student understanding and student and family lived experiences.
PowerMyLearning is founded on the belief that the Triangle approach─where teachers and families work together as allies to support student learning─should result in stronger student outcomes. Prior to this study, we had evidence that Family Playlists had a statistically significant impact on middle school students’ SEL outcomes related to student agency (Martin, 2021). This study is our first with Kindergarten students and indicates that students who received Family Playlists had significantly better SEL in terms of self-efficacy and growth mindset compared to the control group.
This study included two stages: a short-cycle RCT and a delayed Family Playlists implementation with both the control and implementation groups. Findings from the short-cycle RCT indicate that students in the implementation condition had significantly better self-efficacy (+13%; p=.04) and growth mindset (+11%; p=.07) compared to the control group. Findings from the pre/post analysis of the delayed implementation stage indicate that students’ SEL skills were significantly better for both the control group and the implementation group.
This study examined the impact of PowerMyLearning’s Family Playlists on Kindergarten teachers, students, and families at one partner school in San Jose, California during the 2021-2022 school year. With respect to demographics, 67% of students at the school identify as Hispanic, 29% identify as AAPI, and 78% qualify for free or reduced-price lunch. The school had 67 Kindergarten students and four Kindergarten teachers during the 2021-2022 school year. This study examined student-level data for all 67 students collected from surveys completed by the Kindergarten teachers. Details of the study and our impact are provided in the research methods section of this report.
Family Playlists are weekly game-based, collaborative extension activities developed with K-2 students in mind. Family Playlists are assigned by the teacher, in alignment with their instructional scope and sequence, and completed by the student with a family partner at home. Each playlist culminates with the child explaining an academic concept to the family partner, drawing on the research on “Learning by Teaching”, to deepen student learning and foster SEL competencies. When students teach the content of a lesson, they develop a deeper and longer-lasting understanding of the material than students who do not teach it (Fiorella & Mayer, 2014). At the end of each Family Playlist, families submit a video of their child explaining the concept they learned and are given the opportunity to submit feedback to teachers via a short, integrated form. The student videos and family feedback provide teachers with insights about student understanding of concepts learned in the classroom and the students’ and families’ lived experiences, cultural frames of reference, and funds of knowledge (González et al., 2006). Teachers can then use these insights to inform their teaching practices at the whole class and individual student level.
In this section, we will describe the study design, SEL assessment, data analysis, and results. We will then discuss the implications of the results and next steps of this work.
Study Design. This study included two stages: a short-cycle RCT and a subsequent delayed Family Playlists implementation. Each stage was three weeks long. During the short-cycle RCT, PowerMyLearning researchers randomly assigned the 67 Kindergarten students to one of two conditions: implementation (N=35) and control (N=32). Students in the implementation condition received three weeks of Family Playlists and students in the control condition received business-as-usual instruction (no Family Playlists). During the subsequent delayed implementation, students in both conditions received three weeks of Family Playlists. Thus, the implementation condition received a total of six weeks of Family Playlists (via the short-cycle RCT and subsequent delayed implementation) and the control condition received a total of three weeks of Family Playlists (via the subsequent delayed implementation).
SEL Assessment. PowerMyLearning constructed a survey to measure five areas of SEL (Self-Management, Self-Efficacy, Growth Mindset, Social Awareness, and Student Engagement). The survey was adapted from the Panorama Teacher Perceptions of Students’ SEL guide. A question around growth mindset was adapted from the Panorama SEL User guide as there was no teacher-rated survey item that captured growth mindset. The survey included one question per construct.
The SEL survey was completed by Kindergarten teachers before the short-cycle RCT (time 1), after the short cycle RCT (time 2), and after the delayed implementation (time 3). Each teacher completed the SEL survey for each student in their classroom. Responses to each survey question were scored on a 5-point scale.
Data Analysis. To analyze the data, we conducted a mixed ANOVA with a Bonferroni correction for each construct. The mixed ANOVAs included the between-subjects factor of condition (implementation v. control) and within-subjects factor of time (time 1, time 2, time 3). We also conducted simple effects tests to examine hypothesized effects both between- and within-conditions. Thus, this analysis enabled us to examine differences between conditions at each time point and analyze change over time within each condition.
Results. Results were considered statistically significant at p<.10. For brevity, we will only report the results of the simple effects tests. The first set of findings relate to differences between conditions that were observed during the short cycle RCT. The results indicate that, after three weeks of Family Playlists (time 2), students in the implementation group had significantly better self-efficacy (p=.04) and growth mindset (p=.07) compared to the control group. At time 2, there were also positive but non-significant trends between conditions in self-management (p=.13), engagement (p=.21), and social awareness (p=.22). There were no differences between conditions at time 3. At time 1, students in the implementation group had significantly better self-efficacy (p=.01), growth mindset (p=.01), engagement (p=.04), self-management (p=.09) than the control group. There was also a non-significant trend for social awareness at time 1 when comparing the implementation group to the control group (p=.11).
The next set of results pertain to pre-/post-effects within each condition. We will first report findings related to the impact of three weeks of Family Playlists (via the delayed implementation) on students in the control condition (time 3 v. time 2). The findings indicate that, after three weeks of Family Playlists (time 3), students in the control condition had significantly better self-efficacy (p=.004), growth mindset (p=.01), self-management (p=.001), engagement (p=.001), and social awareness (p=.005) than before the implementation (time 2).
In the next section, we will focus on results related to the impact of six weeks (via the short cycle RCT and delayed implementation) of Family Playlists on students in the implementation condition (time 3 v. time 1). The findings indicate that, after six weeks of Family Playlists (time 3), students had significantly better self-efficacy (p=.01) and growth mindset (p=.09). There was also a non-significant trend for self-management (p=.15). There was no effect for engagement and social awareness.
Discussion. This study provides evidence of the impact of PowerMyLearning’s Family Playlists on multiple aspects of SEL. The findings indicate that there were significant differences between conditions at both time 1 and time 2. Thus, before receiving any Family Playlists, students in the implementation condition had better SEL than students in the control condition. Importantly, students were randomly assigned to condition and teachers were blind to condition at the time the surveys were completed. Thus, the difference between conditions at time 1 is almost certainly due to chance. We used an omnibus analysis (mixed ANOVA), which included the data from all three timepoints in a single analysis, and this ANOVA showed statistically significant differences between conditions at time 2. The findings that students in both conditions had significant pre-/post- SEL improvement after three- and six-weeks of Family Playlists provides additional evidence that Family Playlists likely contributed to the observed differences. The next step of this work is to conduct additional randomized control trials in different contexts and with additional participants to corroborate the results of this study.
This study was generously funded by the Leon Lowenstein Foundation.