A recent study found Family Playlists had a statistically significant impact on students’ New York State math test scores, equivalent to four months of additional learning.
How effective are Family Playlists?
The study found a statistically significant effect on 2018 seventh grade standardized math test scores for students who were assigned Family Playlists as compared to seventh grade students in the prior year who had the same teachers but were not assigned Family Playlists (p<0.001). The effect sizei (Cohen’s d) was .33—which is well beyond the federal What Works Clearinghouse criteria of .25 for a “substantively important” effect—and equates to four months of additional learning.ii.jpg?width=602&height=303&name=Evidence%20of%20FP%20Math%20Achievement%20Progress%20Dial%20(1).jpg)
Three important notes: (1) Family Playlists were very early in their development during this study (the 2017-2018 school year was the first full school year of implementation), and a large number of improvements have been rolled out since then; (2) These results demonstrate a big return for a relatively light touch intervention (teachers are expected to assign a Family Playlist at the end of each unit–approximately once every two weeks–and each Family Playlist takes about 40 minutes to complete); and (3) Family Playlists are “additive” because they are a homework intervention that leverages out-of-school time and can therefore be easily layered on top of other academic interventions without a degradation in impact.iii
Who was included in the study?
The treatment group included 132 students from two district-run schools in the Bronx who were in the seventh grade in 2017-18 and whose teachers began assigning Family Playlists for the first time that school year. The comparison group included 150 seventh-grade students who had the same teachers the prior year (2016-17) and were not assigned Family Playlists. This research methodology was developed with input from the Robin Hood Foundation, an early funder of Family Playlists. The student demographics across both schools in the 2017-18 school year were: 66% Hispanic; 30% African American, 7% English Language Learners, 93% student in poverty, and 24% students with disabilities.
What data were used?
The study used student-level New York State standardized math proficiency ratings for the treatment group (2017 and 2018) and for the comparison group (2016 and 2017). Math proficiency ratings are numeric scores between 1.00 and 4.50 that reflect performance on the statewide math exam and can be compared year over year to show growth in proficiency in grade-level standards. We obtained data for all students for whom we had signed consent forms, per Institutional Review Board guidelines.
What kinds of analyses were done for the study?
This study performed an Intent-to-Treat (ITT) analysis, which is considered a more conservative estimate of impact, and included all seventh-grade students who were assigned Family Playlists, regardless of their level of participation. This study used an analysis of covariance (ANCOVA), with group (treatment vs. comparison group) as the independent variable, post-test score as the dependent variable, and pre-test score as the covariate.
What is the underlying evidence base for Family Playlists?
Family Playlists replicate the most successful elements of the “Teachers Involve Parents in Schoolwork” (TIPS) program from Johns Hopkins University, which has shown more positive and frequent interactions between students and their families and better achievement and emotional outcomes over the control condition.iv Family Playlists also leverage the research on the protégé effect (or learning-by-teaching), which has shown that students who teach others have stronger comprehension and recall of the material than students who do not teach others.v Of the learning strategies described in the book “Learning as a Generative Activity: Eight Learning Strategies that Promote Understanding,” learning-by-teaching had one of the largest effect sizes.vi
What is PowerMyLearning’s Strategic Evidence Plan?
PowerMyLearning’s Strategic Evidence Plan (SEP) is a five-year plan to guide our activities around continuous improvement and ongoing evidence building. In Year 1 (2018-19), our primary focus was on understanding the factors influencing teacher adoption of Family Playlists, with the goal of helping us ensure maximum uptake in as cost-effective a manner as possible. As part of this work, we conducted an A/B randomized control trial to test the impact of personalized “nudges” for teachers, and found promising results that are informing development going forward. In Year 2 of the SEP (2019-20), our primary focus will be optimizing the triangle learning relationships between students, teachers, and families by providing additional supports to help teachers implement Family Playlists at the highest levels of fidelity. We will also conduct research with a professor who has expertise on the protégé effect in order to learn how this learning strategy is impacting student achievement. Years 3 through 5 will include additional work around adoption, optimizing the triangle of learning relationships, and optimizing students’ academic and social-emotional learning outcomes.
References
i Cohen’s d is an effect size used to indicate the difference between the mean values of two sets of data— one treatment and one control—and is measured in units of standard deviation.
ii To convert effect size to months of learning, we relied on a toolkit from the Education Endowment Fund (EEF): https://educationendowmentfoundation.org.uk/public/files/Toolkit/Toolkit_Manual_2018.pdf.
iii In “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring” [http://www-inst.eecs.berkeley.edu/~cs301/fa14/resources/Bloom_The2SigmaProblem.pdf], Benjamin Bloom wrote about how districts should look for a set of interventions that together can achieve a large impact on student achievement. He described homework interventions as additive to classroom instruction interventions because when these two interventions are implemented concurrently, they have no overlap and add up as expected (e.g., “Homework Intervention” (0.8 SD) + “Classroom Instruction Intervention” (0.7 SD) = 1.5 SD).
iv Voorhis, Frances L. Van. “Adding Families to the Homework Equation: A Longitudinal Study of Mathematics Achievement.” Education and Urban Society, vol. 43, no. 3, 2010, pp. 313–338., doi:10.1177/0013124510380236. https://journals.sagepub.com/doi/abs/10.1177/0013124510380236.
v Learning as a Generative Activity: Eight Learning Strategies that Promote Understanding.” https://www.amazon.com/Learning-Generative-Activity-Strategies-Understanding/dp/1107069912.
vi Fiorella, Logan, and Richard E. Mayer. “The Relative Benefits of Learning by Teaching and Teaching Expectancy.” Contemporary Educational Psychology, vol. 38, no. 4, 2013, pp. 281–288., doi:10.1016/j.cedpsych.2013.06.001.
