[Your Name]
← Back to case studies
[MONTH YYYY] IN PROGRESS

Ontum Content Studio

Product requirements doc — AI-assisted content generation for a multi-product edtech org

PRD 0→1 product AI / LLM evals cross-functional

Context. Ontum runs marketing content across five channels — four product lines (Kosh, Kul, Kavach, OnboardAI) plus a general EdTech channel — written by different people at different times, with no shared record of what had already been said. Drafts risked repeating an old angle, contradicting an earlier claim, or drifting from the underlying product docs depending on who wrote them and how recently they'd reviewed the source material.

My role. [Describe your role here — e.g. scoped the problem, wrote this PRD, working with the team through implementation]. Below is the PRD as I wrote and iterated it — lightly cleaned up for this write-up, structure and content otherwise unchanged.

Status: v1 drafted, Phase 1 currently being implemented with the team.

1. Overview

Ontum Studio is an internal tool that generates on-brand marketing drafts for each of Ontum Education's product lines (Kosh, Kul, Kavach, OnboardAI) plus a cross-product "General EdTech" channel — grounded in real product documents, aware of what's already been posted, and refined through a feedback loop before anything is treated as final.

2. Problem statement

Content for Ontum's five channels is currently created without a shared memory of what's already been said — so there's no built-in way to know whether a new draft repeats a past angle, contradicts an earlier claim, or leaves an obvious follow-up story untold. Voice and grounding-in-fact also vary by whoever's writing and how recently they've reviewed the underlying product notes.

3. Goals

4. Non-goals (v1)

5. Users

Internal tool, used by multiple people across Ontum — not a solo tool — spanning roughly marketing, program management, and the founding team. That's a real requirement, not a footnote: Phase 1 needs genuine Supabase Auth with RLS scoped per user (or role, if roles end up mattering), not the permissive single-user policy originally assumed — see the updated note in §7.

6. Feature requirements

6.1 Product folders

Five folders — Kosh, Kul, Kavach, OnboardAI, General EdTech — each with its own document library, tone setting, and post history. Switching folders scopes everything to that product.

6.2 Document library (per product)

6.3 Generation

6.4 Feedback & iteration

6.5 Finalize (memory commit)

6.6 Coverage ledger

6.7 Retrieval (phase 2)

7. Non-functional requirements

8. Success criteria

Being finalized with the team; the current working list:

9. Architecture reference

Full technical design (Supabase schema, Edge Function logic, retrieval layer) lives in a companion technical spec — this PRD intentionally doesn't duplicate it.

10. Risks & open questions

11. Phasing

12. Open design question raised by Phase 3

Auto-posting and performance feedback both imply the tool holds live credentials to real social accounts and can take real public actions — a different risk profile than the read/write-your-own-database tool Phase 1 is. Worth treating Phase 3 as its own smaller PRD/spec pass when the time comes, particularly around: what "human confirmation" actually requires (a click, or a review window), and whether performance feedback should ever silently change future drafts' tone/angle without a human noticing the shift.

13. Evals: validating generation quality

The first piece of Phase 1 built out past the PRD stage was an eval system for judging generated content quality, starting with OnboardAI and Kul social content — since "grounded, on-brand, non-fabricated" (§6.3) is a claim that needs a way to check it, not just assert it.

13.1 Rubric first

Defined what "good" means for social content across six judgment dimensions, each scored 1–5 with anchors at 1, 3, and 5:

Alongside this, a separate set of deterministic pass/fail checks was defined: character limits, required elements (CTA, hashtags), banned terms, and platform-format fit. These gate shipping regardless of judgment scores.

13.2 LLM-as-judge prompt

Turned the rubric into a system prompt that scores any given post against the six dimensions and returns structured JSON — a 1–5 score plus a one-line reason for each dimension. This is the core scoring engine the rest of the system calls.

13.3 Runnable eval script

run_eval.py wires the deterministic checks and the LLM judge together into one pipeline:

13.4 Test fixtures from real data

This is where most of the actual work went, in two rounds.

Good examples. Took 7 real, already-published posts (Kul and OnboardAI, across LinkedIn and the Wix blog) and split each into a sources component (the real facts — numbers, dates, partner names — grounding the post) and a draft component (the actual published text). judgeFocus for these is set to "no scripted failure — calibration check": if the judge scores these low, the judge itself is miscalibrated, not the content.

Bad examples. Built 3 synthetic drafts on top of the same real sources, each engineered with a specific, named failure:

judgeFocus for these spells out exactly what should be caught (e.g. "claims X, unsupported by the source above"), so these test judge sensitivity rather than content quality.

13.5 Format iteration

The fixture schema was reshaped twice as the design was refined, ending on a single flat gradedExamples array — label: "good"/"bad", one draft string, sources, and judgeFocus — replacing an earlier nested sources/priorDrafts structure. Simpler to loop over in the runner.

Net result

A working judge, a script to run it, and 10 labeled examples (7 good, 3 bad) to sanity-check that the judge rewards what real good posts do and penalizes what bad ones do — before trusting it on live, unlabeled generations.

Appendix

Where this stands. Phase 1 implementation is underway; the first piece built past the PRD was the eval system described in §13 — a rubric, an LLM-as-judge scorer, and a labeled set of good/bad test fixtures for OnboardAI and Kul content, so generation quality can be checked rather than assumed. [Add further updates here as Phase 1 progresses — what else got built vs. this draft, any decisions that changed once engineering started, what you'd revise about the PRD itself in hindsight.]