A Randomized Control Trial (RCT)

The Effect of UMIGO Transmedia Products on First and
Second Grade Students’ Math Ability:
A Randomized Control Trial (RCT)
Michael Cohen Group, LLC 375 W Broadway Suite 502, New York, NY 10012
The
Project
The goal of the UMIGO Project is to develop educational media products that help young children achieve basic
competencies in mathematics. In 2011, the Nation’s Report Card (NAEP), a national assessment of educational
progress, revealed that 53% of American fourth graders failed to attain proficient mathematics standards. This early
numeracy failure is related to other long-term academic outcomes. The UMIGO Partnership recognizes the importance
of providing all children with a solid foundation in mathematics, especially children in low-income households. The
UMIGO project aims to create a media-based supplemental mathematics curriculum for young children in first and
second grade. Initial components of the UMIGO property developed at the time of the study* included music videos,
songs, web games, and iOS games targeting children’s ability in a number of areas of math.
Study Objective
The purpose of this study was to evaluate the effects of the UMIGO transmedia products on first- and second-grade
students’ math ability. Products from four math areas were chosen for inclusion in this study: early arithmetic,
inequalities (greater than/less than/equal relations), capacity, and height & length.
Overall Findings
•
Transmedia properties can be an effective means of educational intervention.
•
Children randomly assigned to interact with the UMIGO property showed significant gains in their understanding of
capacity and inequalities (ps< .05), compared to children assigned to interact with a comparable commercially available
property (control condition). These effects did not differ by grade, gender, or geographic location.
•
Children who interacted with the UMIGO property did not show significantly greater gains compared to children who
interacted with the control property in their understanding of early arithmetic and height & length (ps > .05).
Pre-test and Post-Test Assessment
Measure Scores: Over All Math Areas
Chart 1 shows the scores (averaged across all
four math areas) for children in the UMIGO and
control groups.
Assessment Measure Gains:
From Pre-Test to Post-Test by Area of Math
Chart 2 shows the improvement from pre- to post-test on
each of the domains, for the UMIGO and control groups.
Users of UMIGO materials showed significantly greater
improvement than controls in inequalities and capacity.
*The UMIGO properties included in this study were created during the first phase of the grant, and are preliminary to the UMIGO Appisodes currently in
development.
Research was funded by the U.S Department of Education as part of the Ready To Learn program award number U295A100026. The contents of this
report were developed under a cooperative agreement from the U.S. Department of Education. However, these contents do not necessarily represent the
policy of the Department of Education, and you should not assume endorsement by the Federal Government.
Methodology
Participants: Students were recruited from five Title One schools in Bridgeport, CT; Memphis, TN; and SedroWooley, WA. (In Title One Schools, a minimum of 40% of students are from low-income families.) A total of 514
first- and second-grade students from 31 classes participated in the study.
Intervention
Students were randomly assigned to use either the UMIGO transmedia suite (experimental condition), or a similar,
commercially available transmedia suite (control condition). In addition to usual classroom instruction, students
interacted with the media products, unmediated by instruction, for 19 days, 25 minutes/day (a total of 8 hours).
Experimental Condition
Control Condition
Students used the UMIGO transmedia suite of
products which included touch screen apps,
computer games, music videos, and paperpencil/board games.
Students used a commercially available
transmedia suite of products which included touch
screen apps, an online world with web based
games, and paper-pencil activities.
Assessment
Prior to exposure to the UMIGO or control materials, students’ math knowledge and ability was assessed to provide
a baseline. Following the intervention, students were re-assessed using the same instrument to test for changes in
performance.
A custom-designed assessment instrument was developed. The assessment included questions from the following
scales: The Assessment for the California Mathematics Standards Grade 1; California Standards Tests Grade 2
Mathematics; and the Georgia Criterion-Referenced Competency Tests (CRCT) 1st and 2nd Grade Math Tests,
supplemented by custom designed measures.
Results
Significant gains from pre- to post-test were seen for both experimental and control groups in all curriculum
areas. Of particular interest, children who used UMIGO materials showed significantly greater learning gains in
inequalities (d = .24) and in capacity (d = .48) than children in the control. These gains in learning are uniquely
attributable to use of the UMIGO materials.
Student Outcome Performance
on Pre-Test and Post-Test by Area of Math
Chart 3: Standardized T-Scores for outcomes (capacity; early arithmetic; height &
length; and inequalities).
Effect Sizes for Student Gains
of UMIGO Condition Relative
to the Control Condition
Chart 4: Effect sizes (Cohen’s d) for the
difference in post-test scores between
UMIGO and Controls. Capacity scores had
a medium but significant effect size (d=.48)
and inequality scores had a small effect size
(d=.24). Effect sizes of height & length and
early arithmetic did not reach significance.
As indicated(*), UMIGO users had significantly higher gains than controls on capacity and inequalities measures (ps < .05).

Download Report