Context
When our latest OnboardAI cohort wrapped up, we wanted to move beyond a single completion number and understand what actually happened inside the cohort. The questions we set out to answer:
- Where were our highest drop-offs, and why?
- Who completed successfully, who didn't, and why?
- What was the reported perceived improvement and satisfaction?
- Where could we predict engagement?
Data sources
The analysis combined three sources:
- Baseline forms capturing learner profiles and starting confidence.
- Endline forms capturing perceived improvement and satisfaction.
- MentorAI conversation data from the agent that helps students complete tasks on the platform.
The dashboard
To answer these questions in one place, I built a cohort analysis dashboard with several views.
Where learners dropped off
A task-by-task completion view that shows exactly where in the journey learners stopped.
Engagement
Cohort engagement measured through time spent, so we could see where effort was concentrated.
Individual learner drill-down
A learner-level view with an LLM-generated summary of each task. It also shows how each learner worked with MentorAI: messages sent, average message length, and time spent per task.
Top performers vs. learners who dropped off
A side-by-side comparison of the profiles of top performers and learners who were left behind, looking at factors such as college attended, first-generation learner status, comfort with English, and the devices learners had access to.
HEART scores
Scores from Google's HEART framework (Happiness, Engagement, Adoption, Retention, Task success), shown at both cohort and individual level.
Perceived improvement
A baseline-to-endline comparison showing change in self-reported confidence and command over skills.
What we found
Drop-off clustered at three points
The highest drop-off was between Task 8 and Task 9, where 3 learners left. The earliest drop-off came between Task 2 and Task 3 (2 learners), and a mid-journey dip came between Task 5 and Task 6. Learners were leaving at the start, the middle, and the end of the course, so no single fix would cover all three.
English comfort separated top performers from the rest
On average, top performers reported higher comfort in English than the bottom performers who did not complete. Since tasks and MentorAI conversations were in English, language was a likely barrier.
First-generation learners were over-represented among non-completers
Half of the bottom performers were first-generation learners, a much higher share than among the top performers. That points to a gap in support rather than ability.
A platform failure ended one learner's attempt
One learner dropped off because MentorAI was rate-limited when they started a session. They couldn't complete the task, lost interest, and did not return. That drop-off had nothing to do with the learner.
What I'd change next
These are the changes I recommended for the next cohort, matched to what the data showed.
Fix the drop-off points
- Check-ins where learners falter. Reach out proactively to learners who stall around Tasks 2 to 3, 5 to 6, and 8 to 9, the points where drop-off clustered.
- A buddy system. Pair learners so each has someone to compare notes with and keep them accountable.
- Gamification. Add a leaderboard to encourage participation, along with progress bars so learners can see how far they've come.
Make the course accessible in English and beyond
- Local-language support. Let learners read task explanations in their own language.
- Simplified language with translate-on-demand. Simplify task wording and explanations for learners with lower English proficiency, with an option to translate whenever they want.
- A separate track. Consider placing lower-proficiency learners in their own cohort with stronger task-explanation support.
Design support for first-generation learners
- Design a separate intervention for first-generation learners, since they were so over-represented among non-completers.
Make the platform reliable and more engaging
- Handle MentorAI rate limits gracefully. A learner should never be stuck mid-task because of a usage cap.
- Go beyond chat. Add more interactive ways to complete tasks than conversation alone.