NEEV Hybrid Program: Making A Case For Bichronous Online Learning

Why at-home digital learning?

It takes a village to raise a child and definitely more than the traditional school system to prepare them for today’s competitive career landscape. Particularly in the current times where effective teaching hours in schools are much below that mandated by experts[1], COVID lockdowns have had a substantial detrimental effect on students’ academic levels[2] and competition for quality colleges and careers is increasing[3].

One of the best bets to address this gap in effective learning time for a typical school going Indian child to actually achieve successful life outcomes is at-home digital learning. In a quest to add more meaningful learning hours to students’ academic life, Avanti has tried synchronous digital learning solutions with proven success in the form of live classes. Today, more than a thousand students attend Avanti live classes every day making it one of India's largest live class programs in this specific demographic. 

Fig 1: A glimpse of NEEV Live Class Program

From Synchronous to Asynchronous Learning:

In Haryana, live classes are conducted as part of the NEEV Live Class program where hour-long live classes are conducted three times a week for math remediation of middle school students. These live classes focus on 125 foundational Mathematics learning outcomes for grade 9th and 10th government school students. Roughly 30% students attend these classes meaningfully, i.e. attend more than 2 hours of live classes every week. This is much higher than similar high-touch models across the education sector, where the typical attendance is around 5%. Students who attend these classes regularly typically improve their performance by 10 to 24 percentage points from baseline to endline.

Given the positive results of our synchronous digital at home learning intervention, our next step was to find a lighter-touch, more scalable version of the remediation program. To do this, we decided to substitute some synchronous components of the NEEV program with asynchronous components like our Quiz product or Plios.

Fig 2: A Glimpse of Avanti’s Quiz Product

Who are the primary users for asynchronous learning products?

Is asynchronous learning suitable for all learning types? Our previous projects with asynchronous learning told us no. Asynchronous learning is a low touch model with minimal human intervention. It also is most likely to be adopted by a user set showcasing demonstrated intent of using solving the problem the product seeks to resolve.  Which of our students have demonstrated intent towards self-directed learning with minimal human touch? 

Based on our experience of working with them during live classes, the answer was our academically proficient students. Based on our selection, this would be the top tercile of students as per their performance in our baseline assessment.
What is the persona of these students? What are these students like?

Fig 3: Characteristics of Academically Proficient Students

Typically academically proficient students are:

1. Self-motivated to learn: They typically seek academic resources outside of typical classroom learning. As per our internal survey, a number of them use YouTube videos to study.

2. Need autonomy in how they learn: Compared to other student types, academically proficient students are most likely to be selective about their learning experience, choosing their own mode and facilitator of learning.

3. Curious and/or Competitive: Academically proficient students are mostly both curious and competitive and seek to know where they stand academically. They also are more likely to demonstrate meta-cognition skills and be more interested in tracking their learning journey. This is a good opportunity to build leaderboards and user profiles.

4. Need Quick Gratification: Most of these students are impatient and need quick replies to questions, doubts, etc.  

Student-Centered Product Design:

Given that academically proficient students seek to design their own learning experiences, if we wanted adoption of our product, we needed to co-design it with them. Thus started a series of conversations with 13 students of GSSS Chhapra and GSSS Balsamand co-design the hybrid NEEV Program.

The 1st Question: Which is the most suitable mode of asynchronous learning?

Given a choice of assessments, videos, and plios, the students selected plios to be the most suitable form of asynchronous learning. Plios are interactive videos where a video is overlaid with questions at regular intervals. As a user watches a video, questions pop up that they can answer (or skip) and continue watching the video. 

The students’ choice of plio also corroborated with Avanti’s earlier learnings where we found student engagement on plios to be five times higher than regular YouTube videos.

The 2nd Question: Which plio type would students prefer?

Till date Avanti has experimented with two types of plios. The first type are concept plios which are regular animated or non-animated videos with little to no human presence. The second type are exam plios where, similar to live classes, a teacher is featured in the video and uses a whiteboard to teach the concept.

Based on conversations with students, as well as previous qualitative and quantitative data, students prefer videos with human presence in them.

The plios designed for the hybrid program are hence similar to teachers teaching a live class, with pop up questions substituting the real time Q&A session. Here’s a sample of a hybrid program plio. Only one feature was missing- Doubt Resolution.

The 3rd Question: What should be the program structure? 

The super enthusiastic students of GSSS Chhapra and GSSS Balsamand, after multiple rounds of trials and feedback cycles, helped us arrive at the following structure:

Fig 4: Weekly structure of the synchronous NEEV program vs. the hybrid NEEV program with asynchronous and synchronous components

After a round of open-ended interviews with students, the structure that emerged was substituting live classes teaching new concepts with plios, followed by a live class for doubt resolution. 

Through previous pilots we know only around 14% students of this profile ask doubts organically. Hence, to make doubt classes more effective, plio results are shared with teachers beforehand to inform them of which topics were potential doubt areas. In addition, an assessment on both concepts is also conducted weekly and acts as a formative assessment for teachers to plan doubt classes. 

The final structure for the program was concept teaching via plios on alternate days in the first half of the week. This would be followed by a weekly assessment and a doubt class to resolve doubts and low-scoring concepts covered in the plios and assessments. All throughout, the center for communication for all learning sessions and followups were whatsapp groups. 

Fig 5: An example of social influence- Students sharing plio results on whatsapp communities motivates other students to attempt plios too

A Higher Adoption Rate

Since it was being co-designed with students, the NEEV hybrid intervention was piloted in three stages. Overall, we saw a healthy adoption of more than 50% regular users in each stage of the pilot. For reference, this percentage of regularity is 10 percentage points higher for when these same students attend live classes. This is because of the flexibility asynchronous learning provides in terms of time and consumption.

Pilot 1: July 2022: The MVP was codesigned and piloted in 2 schools with 16 grade 10 students over a duration of 2 weeks. 69% students were weekly active users (WAU) who attended at least 1 learning session per week, 62% students were regular students who attended >50% learning sessions per week.

Pilot 2: August-September 2022: The second pilot was conducted in 3 schools with 32 schools over a duration of 4 weeks. 78% students were weekly active users (WAU) who attended at least 1 learning session per week, 78% students were regular students who attended >50% learning sessions per week.

Pilot 3: November - December 2022: The program was piloted with 124 grade 10 students over a duration of 5 weeks. 100 students were onboarded to the program, i.e. they attended at least one learning session. 80 students were active weekly and 65 students attended more than two learning sessions per week.  In the table below, the attendance of the same set of students in live classes conducted at the end of the year is shown for comparison of adoption in a 100% synchronous program.

Fig 6: Adoption of Hybrid NEEV Program vs. NEEV Live Class Program

Out of the three components of the hybrid program, plios have the highest usage, with the ratio of average users per component being 35:26:29 for plio, assessment and doubt class, respectively. The average duration of our plios is 20 minutes, out of which the average captured watchtime in 72% or 14.5 minutes.

Fig 7: Comparative adoption and feedback on individual components of hybrid program

Impact on Learning:

In the 2nd pilot, the 25 students (78%) who engaged with the program regularly showed an increase of 28pp on score from baseline to endline.

To test the impact on learning on a larger cohort, we conducted a second baseline assessment during the third pilot. The baseline was conducted prior to beginning the intervention and scores were compared with the corresponding endline assessment. The positive impact of the program is clearly visible as the number percentage point increase from baseline to endline increases with level of student engagement with the program. 

Passive students who did not engage with any component of the hybrid program showed an increase of 15 percentage points, while irregular students who engaged with less than 50% of learning sessions per week showed an increase of 19.5 percentage points. Students who engaged the most with more than 50% of learning sessions per week showed an increase of 25.15 percentage points.

Fig 8: Impact on learning increases with meaningful engagement with the Hybrid Program

Fig 9: Increase in pp change from baseline to endline assessment with increasing engagement

Other Observations:

• Online communities keep students engaged: Whatsapp groups served as online communities where students shared their results, asked doubts and communicated with teachers and each other. Online research also shows that when synchronous communication features are integrated with asynchronous features, the online course is more engaging, increasing learning outcomes, positive attitudes, and retention.[4]

• Online communities boost adoption: As students of this academic category are eager and competitive, the sharing of scores by one batch member triggers the rest of them to also complete their learning session and share scores

• Results are teaching style-agnostic: Adoption and results of the intervention were agnostic to the teacher type. This pilot was conducted with three teachers, each very different from the other in terms of teaching style and level of interaction with students. By design, even the teacher featured in the plio and the one taking the doubt class of the batch was different.

• Low Offline Field Engagement: Unlike NEEV live class program, which required monthly school visits to engage with students and boost adoption, school visits did not play a major role in this program. The team visited schools once in two months, only for the purpose of conducting assessments.

Way Forward: Next, we plan to scale the intervention to 15 schools, around 400 grade 7th students in a year-long intervention and check for impact on long-term learning. The model will also be tested in Avanti’s Test Prep program for grades 11th and 12th.  

References:

1.  Involvement of Teachers in Non-teaching Activities and its Effect on Education, NIEPA, Ministry of Human Resource Development

2.  Impact of COVID-19 Pandemic on Lives of Children, Volume 1, Issue-1, Children First, DCPCR

3.  Is The Pressure Of Competitive Exams Ruining An Entire Generation?, Atrijo Banerjee, Jun 28, 2021, Youth Ki Awaaz

4. Bichronous Online Learning: Blending Asynchronous and Synchronous Online Learning, Martin et al, Educause Review

Learnings from NEEV Live Class Program

Summary:

NEEV Live Class Program is an online, at-home Math remediation program. One-hour lessons are conducted three times a week by Avanti teachers through online live classes. These classes have been rolled out in 80 schools across 8 districts of Haryana for 5788 ninth-grade students. In grade 9th, around 4500, i.e. 77% students have attended the program at least once, and 1736, i.e. 30% of students attend more than 2 hours of live classes every week. 

The program has shown different levels of impact for students of different academic levels. Thus, to study the impact of the program on student learning, we have classified students academically at three levels- L1 (at grade level), L2 (moderately remedial) and L3 (deeply remedial), based on their performance in our baseline assessments. When taught in heterogeneous batches composed of students of all aforementioned academic levels, the program is most beneficial for deeply remedial students. These students show an improvement in scores ranging from 10 to 24 percentage points from baseline to endline. The results of moderately remedial students and grade level students are mixed, mostly due to the low level of content being taught. To maximize the impact of the program for all levels of students, we have redesigned the program so that the teaching level and pace is more customized to their academic levels. 

Fig 1: Glimpse of NEEV Live Class program

 Importance of Mathematics in Life, Education and Career  

Mathematics serves as the foundational base of human thought and provides an effective way of building mental discipline, mental rigor and logical reasoning. It also plays an instrumental role in understanding other fields of study like Science, Economics, Music, etc. It is hence no surprise that multiple research studies have proven that mastery of basic mathematical concepts are strong predictors of later achievements in life[1], intelligence scores, academic motivation and duration of education[2]. In India, mastery of 8th to 10th grade level Mathematics concepts is also crucial for students to clear both basic and advanced level entrance and competitive exams for entrance into top colleges and government jobs. 

NEEV Live Class program

So what does it take to ensure maximum students who graduate grade 10th do so at grade level in Mathematics? The following points answer the question for us:

1. Mathematics is a learning progression of concepts and skills that build upon one another such that fundamental concepts of lower grades gradually build and expand in more complex concepts taught in middle and high school

2. 73% of Grade 9 students are at basic and beginner level at Mathematics and lack mastery over the aforementioned fundamental concepts.[3] 

Taking into account both these points, a remediation-focussed mathematics program has the maximum potential of benefiting student education as well as longer term life outcomes. Over the past 4 years of Sankalp project, Avanti has collaborated with the Government of Haryana to find a scalable solution to remediate middle-school students in Haryana Government Schools. 

This report focuses specifically on NEEV Live Class program. Out of all the interventions piloted till now, NEEV Live Classes is of special interest for the following reasons:

• It is a high-intensity at-home online intervention (3 hours a week). This makes it novel

• It is highly scalable operationally

• It is extremely cost-effective, given that there is no need for the students to stay back in any physical location

• Being complementary to school teaching, it seeks to reduce the workload of school teachers rather than add to it.

NEEV Program Description: 

NEEV curriculum, developed by Avanti’s academic team is a curriculum which maps every concept of grades 9th and 10th linearly to foundational concepts of lower grades. These foundational concepts need to be mastered by students in order to attain grade-level mastery. The objective of this curriculum, thus, is to bridge the gap in students’ foundational learning and bring them to grade level.

This curriculum is taught in 80 Haryana government schools through NEEV Live Class Program. This math remediation program is an annual program consisting of 75 classes conducted across the academic year. It covers 125 concepts from grades 3 to 8. Each student receives 3 one hour-long classes per week conducted by an Avanti teacher.

Fig 2: In-School student orientation

This is primarily a direct-to-student program where students are on boarded through physical school visits conducted by Avanti Program Associates and parent orientation through telecalling done by Avanti’s counselors, while the school teacher plays an initial supportive role in validating the program and setting up a Whatsapp group for Live Class communications. Telecalling has played a major role in the program with regular counseling calls being done for students who miss classes. 

Fig 3: Monthly school visits with student celebration and attendance followups

Research Infrastructure:

NEEV Live classes are conducted in 80 schools spread over 8 districts of Haryana. It is part of an RCT study being conducted with JPAL and USAID. In addition, Avanti’s internal Research and MEL team conducts quarterly internal assessments to monitor program health. We also conduct more detailed research in a set of 8 Sandbox schools in order to improve program adoption and efficacy.  

Program Adoption:

The current work has seen the following adoption:

Fig 4: NEEV Live Class Adoption.

*Regular students are students with >=50% attendance. Previous assessment results have proven that this is the minimum attendance required for students to benefit from NEEV live classes meaningfully

• 77% of school enrolled students attend a NEEV Live Class at least once 

• 34% of the enrolled population participate every week in the program (Active Users)

• 30% of the enrolled population consume >2 hours of live classes every week (Regular Users)

• These engagement rates are significantly higher than typical regular usage rates for such high-intensity digital interventions (typically sub 5%).

Program Impact- Main Learning:

Our main learning from the last 1.5 years of NEEL Live Class research is that online teaching is most beneficial to students when customized to their academic levels. Without taking into account the needs of the students in terms of content difficulty and teaching pace, the benefit to students is insignificant or negative.

We now recognise our students are at three academic levels, as described in the table below:

Fig 5: Students categorised as per their academic levels and learning requirements

Assessment Strategy:

A baseline and endline is conducted before and after each unit for a cluster of schools using cluster sampling methodology. Both baseline and endline assessments have 20 multiple choice questions (MCQs), with one mark assigned for each correct answer and zero marks for incorrect and unanswered questions. The questions were appropriately mapped to the remedial content taught in the NEEV live classes. All internal assessments are conducted offline within the school premises under the invigilation of one or more Avanti team members, and wherever possible, school staff.

Assessment Results:

NEEV Live Class program has been piloted and scaled up in Haryana government schools since October 2021. In this duration, a total of 4 major internal assessments have been conducted in selected treatment schools. In all assessments, it has been observed that only students with attendance of more than or equal to 50% classes benefit meaningfully from the program. Hence, only the results of these students, hitherto known as Regular students, is considered to measure impact of the program on student learning outcomes.

Fig 6: Learning gains in multiple rounds of internal baseline and endline assessments. Students benefited the most when teaching was leveled at their specific academic level (in this case level 3 students)

This table highlights the difference between baseline and endline scores of three academic levels of students as a result of NEEV Live Class program and NEEV Workbook Program. The first three columns are of grade 9th students’ performance data from 80 RCT treatment schools where RCT is going on, while the  last column shows the result of NEEV Live Class program in grade 10th of experimental Sandbox schools. The latter function outside of the RCT study.

In RCT treatment schools teaching content and pace was determined by the pre-set academic calendar of the program, while in experimental Sandbox schools, as a result of ongoing research, we were able to adjust teaching content and pace as per their academic levels. 

Our results show that students need to be taught at their academic level to gain from the live class program. This is proven by:

• Last year content was being taught at L2-3 level, and hence both L2 and 3 students gained from the program.

• This year in NEEV LC, our content focused on L3 level, and hence, there was no measurable benefit for L2 students

• L1 students do not benefit from remediation program since they are already at grade level

• When teaching is customized to students’ level, as in Sandbox schools, all levels of students benefit from the program 

Emerging Plan for NEEV 2.0:

Recognising that students need to be taught at their academic level, next year we are dividing batches so that students of different academic levels can be taught separately:

• In April, students of the lowest tercile, L3, will be taught the basics of grade 3-5 content in a foundation program

• In May, students of L2 and L3 levels will be combined to teach remediation content from grades 6 to 9

• At the same time, students of L1 levels will be taught in separate batches, remediation and at grade content from grades 8-10th.

Fig 7: Structure of NEEV 2.0, allowing for more homogenous batches with targeted teaching levels

References:

1. MSDF Common Assessments 2021-22 Endline Report - Avanti Fellows, February 2022

2. School readiness and later achievement, Duncan, Dowsett et al., Developmental Psychology, 43(6), 1428–1446, 2007

3. Enduring Links From Childhood Mathematics and Reading Achievement to Adult Socioeconomic Status, Association for Psychological Science, Volume 24, Issue 7