Learnings from NEEV Live Class Program
Summary:
NEEV Live Class Program is an online, at-home Math remediation program. One-hour lessons are conducted three times a week by Avanti teachers through online live classes. These classes have been rolled out in 80 schools across 8 districts of Haryana for 5788 ninth-grade students. In grade 9th, around 4500, i.e. 77% students have attended the program at least once, and 1736, i.e. 30% of students attend more than 2 hours of live classes every week.
The program has shown different levels of impact for students of different academic levels. Thus, to study the impact of the program on student learning, we have classified students academically at three levels- L1 (at grade level), L2 (moderately remedial) and L3 (deeply remedial), based on their performance in our baseline assessments. When taught in heterogeneous batches composed of students of all aforementioned academic levels, the program is most beneficial for deeply remedial students. These students show an improvement in scores ranging from 10 to 24 percentage points from baseline to endline. The results of moderately remedial students and grade level students are mixed, mostly due to the low level of content being taught. To maximize the impact of the program for all levels of students, we have redesigned the program so that the teaching level and pace is more customized to their academic levels.
Fig 1: Glimpse of NEEV Live Class program
Importance of Mathematics in Life, Education and Career
Mathematics serves as the foundational base of human thought and provides an effective way of building mental discipline, mental rigor and logical reasoning. It also plays an instrumental role in understanding other fields of study like Science, Economics, Music, etc. It is hence no surprise that multiple research studies have proven that mastery of basic mathematical concepts are strong predictors of later achievements in life[1], intelligence scores, academic motivation and duration of education[2]. In India, mastery of 8th to 10th grade level Mathematics concepts is also crucial for students to clear both basic and advanced level entrance and competitive exams for entrance into top colleges and government jobs.
NEEV Live Class program
So what does it take to ensure maximum students who graduate grade 10th do so at grade level in Mathematics? The following points answer the question for us:
Mathematics is a learning progression of concepts and skills that build upon one another such that fundamental concepts of lower grades gradually build and expand in more complex concepts taught in middle and high school
73% of Grade 9 students are at basic and beginner level at Mathematics and lack mastery over the aforementioned fundamental concepts.[3]
Taking into account both these points, a remediation-focussed mathematics program has the maximum potential of benefiting student education as well as longer term life outcomes. Over the past 4 years of Sankalp project, Avanti has collaborated with the Government of Haryana to find a scalable solution to remediate middle-school students in Haryana Government Schools.
This report focuses specifically on NEEV Live Class program. Out of all the interventions piloted till now, NEEV Live Classes is of special interest for the following reasons:
It is a high-intensity at-home online intervention (3 hours a week). This makes it novel
It is highly scalable operationally
It is extremely cost-effective, given that there is no need for the students to stay back in any physical location
Being complementary to school teaching, it seeks to reduce the workload of school teachers rather than add to it.
NEEV Program Description:
NEEV curriculum, developed by Avanti’s academic team is a curriculum which maps every concept of grades 9th and 10th linearly to foundational concepts of lower grades. These foundational concepts need to be mastered by students in order to attain grade-level mastery. The objective of this curriculum, thus, is to bridge the gap in students’ foundational learning and bring them to grade level.
This curriculum is taught in 80 Haryana government schools through NEEV Live Class Program. This math remediation program is an annual program consisting of 75 classes conducted across the academic year. It covers 125 concepts from grades 3 to 8. Each student receives 3 one hour-long classes per week conducted by an Avanti teacher.
Fig 2: In-School student orientation
This is primarily a direct-to-student program where students are on boarded through physical school visits conducted by Avanti Program Associates and parent orientation through telecalling done by Avanti’s counselors, while the school teacher plays an initial supportive role in validating the program and setting up a Whatsapp group for Live Class communications. Telecalling has played a major role in the program with regular counseling calls being done for students who miss classes.
Fig 3: Monthly school visits with student celebration and attendance followups
Research Infrastructure:
NEEV Live classes are conducted in 80 schools spread over 8 districts of Haryana. It is part of an RCT study being conducted with JPAL and USAID. In addition, Avanti’s internal Research and MEL team conducts quarterly internal assessments to monitor program health. We also conduct more detailed research in a set of 8 Sandbox schools in order to improve program adoption and efficacy.
Program Adoption:
The current work has seen the following adoption:
Fig 4: NEEV Live Class Adoption.
*Regular students are students with >=50% attendance. Previous assessment results have proven that this is the minimum attendance required for students to benefit from NEEV live classes meaningfully
77% of school enrolled students attend a NEEV Live Class at least once
34% of the enrolled population participate every week in the program (Active Users)
30% of the enrolled population consume >2 hours of live classes every week (Regular Users)
These engagement rates are significantly higher than typical regular usage rates for such high-intensity digital interventions (typically sub 5%).
Program Impact- Main Learning:
Our main learning from the last 1.5 years of NEEL Live Class research is that online teaching is most beneficial to students when customized to their academic levels. Without taking into account the needs of the students in terms of content difficulty and teaching pace, the benefit to students is insignificant or negative.
We now recognise our students are at three academic levels, as described in the table below:
Fig 5: Students categorised as per their academic levels and learning requirements
Assessment Strategy:
A baseline and endline is conducted before and after each unit for a cluster of schools using cluster sampling methodology. Both baseline and endline assessments have 20 multiple choice questions (MCQs), with one mark assigned for each correct answer and zero marks for incorrect and unanswered questions. The questions were appropriately mapped to the remedial content taught in the NEEV live classes. All internal assessments are conducted offline within the school premises under the invigilation of one or more Avanti team members, and wherever possible, school staff.
Assessment Results:
NEEV Live Class program has been piloted and scaled up in Haryana government schools since October 2021. In this duration, a total of 4 major internal assessments have been conducted in selected treatment schools. In all assessments, it has been observed that only students with attendance of more than or equal to 50% classes benefit meaningfully from the program. Hence, only the results of these students, hitherto known as Regular students, is considered to measure impact of the program on student learning outcomes.
Fig 6: Learning gains in multiple rounds of internal baseline and endline assessments. Students benefited the most when teaching was leveled at their specific academic level (in this case level 3 students)
This table highlights the difference between baseline and endline scores of three academic levels of students as a result of NEEV Live Class program and NEEV Workbook Program. The first three columns are of grade 9th students’ performance data from 80 RCT treatment schools where RCT is going on, while the last column shows the result of NEEV Live Class program in grade 10th of experimental Sandbox schools. The latter function outside of the RCT study.
In RCT treatment schools teaching content and pace was determined by the pre-set academic calendar of the program, while in experimental Sandbox schools, as a result of ongoing research, we were able to adjust teaching content and pace as per their academic levels.
Our results show that students need to be taught at their academic level to gain from the live class program. This is proven by:
Last year content was being taught at L2-3 level, and hence both L2 and 3 students gained from the program.
This year in NEEV LC, our content focused on L3 level, and hence, there was no measurable benefit for L2 students
L1 students do not benefit from remediation program since they are already at grade level
When teaching is customized to students’ level, as in Sandbox schools, all levels of students benefit from the program
Emerging Plan for NEEV 2.0:
Recognising that students need to be taught at their academic level, next year we are dividing batches so that students of different academic levels can be taught separately:
In April, students of the lowest tercile, L3, will be taught the basics of grade 3-5 content in a foundation program
In May, students of L2 and L3 levels will be combined to teach remediation content from grades 6 to 9
At the same time, students of L1 levels will be taught in separate batches, remediation and at grade content from grades 8-10th.
Fig 7: Structure of NEEV 2.0, allowing for more homogenous batches with targeted teaching levels
References:
MSDF Common Assessments 2021-22 Endline Report - Avanti Fellows, February 2022
School readiness and later achievement, Duncan, Dowsett et al., Developmental Psychology, 43(6), 1428–1446, 2007
Enduring Links From Childhood Mathematics and Reading Achievement to Adult Socioeconomic Status, Association for Psychological Science, Volume 24, Issue 7
