Every use case so far has AI playing the counterpart so students can practice. This chapter inverts the role: AI conducts the assessment. It is the newest pattern in this playbook, the one your assessment committee will ask about, and the one with the best-documented deployment to learn from. It also exists because of a crisis worth naming plainly.
Why oral exams are back
Large language models broke take-home assessment. Students submit polished essays, problem sets, and code they cannot explain under two minutes of follow-up questioning. Instructors know it; surveys of faculty describe overwhelmed departments with no consensus on what an AI-proof assessment looks like.
The countermeasure emerging across universities is the oldest assessment there is. Cornell professors now attach Socratic oral defenses to written problem sets — TAs grade the 20-minute defense, not the submission. Penn’s teaching center describes “a massive shift toward in-person assessments” and runs faculty workshops on oral examination. The logic is airtight: a viva requires real-time reasoning in your own voice, and cannot be outsourced to a model.
The problem is the same one from Chapter 1: oral exams never scaled. Twenty minutes per student per assessment, examiner variance between the 9 a.m. and 4 p.m. sessions, scheduling across a 200-person lecture. Which is why the most instructive experiment of the past year is a professor who decided to fight fire with fire.
The NYU Stern deployment
Panos Ipeirotis, at NYU’s Stern School of Business, built an AI examiner for the final exam of an AI product-management course, and published the entire system, including its failures (the blog write-up grew into arXiv:2603.18221, with all prompts as appendices). The shape of it:
The exam. Two parts, about 25 minutes total, taken from home at a time the student chose. Part one: the examiner probes the student’s own capstone project — decisions, trade-offs, “why did you choose that model?” Part two: the examiner picks a case discussed in class and pushes the student to reason through it live. Personal defense plus transfer: a clean template for almost any discipline.
The architecture. Not one agent but a workflow of specialists: an authentication agent that verifies student identity and refuses to proceed without it, then a project-discussion agent with the student’s project details injected as per-student variables, then a case-discussion agent running structured questioning. Decomposition kept each phase focused — the same sub-agent logic that any complex conversation design converges on.
Personalization as integrity. Because each exam is generated from the student’s own submission plus a rubric, no two exams are alike and there is nothing to leak. The full exam structure was shared with students in advance (Chapter 4’s practice-is-learning property, deployed for real). Preparing for the exam and understanding your own project are the same activity.
The grading. Transcripts went to a council of three model families — Claude, Gemini, and GPT — which scored independently, then read each other’s evaluations and revised, with one model chairing the synthesis. Inter-rater reliability landed at Krippendorff’s α = 0.86, above conventional thresholds for human rater agreement. The professor reviewed outputs and retained final grading authority.
The cost. 36 examinations, $15 total, $0.42 per student, versus hundreds of dollars of TA time for the human-run equivalent. At that price, the strategic move is not replacing the final: it is attaching a ten-minute comprehension check to every assignment, making “explain your submission” a routine part of submitting.
What broke, and the lesson it teaches
The published failure modes matter more than the successes, because they generalize:
- The agent stacked questions. Despite explicit, repeated prompt prohibitions, the examiner would bundle two or three questions into one turn, overwhelming students. Prompting did not fix it.
- It could not randomize. Asked to pick cases at random, the agent gravitated to the same few. Randomness needs to come from code, not from a model’s intentions.
- The cloned voice backfired. The examiner spoke with a voice cloned from a professor, intended as familiar; students experienced it as aggressive. Persona warmth is a design variable you must test with real students, not assume.
The recurring lesson, in the authors’ own words: behavioral constraints on LLMs must be enforced through architecture, not prompting alone. One question per turn is a turn-taking rule the system enforces; case selection is a random draw the code performs; identity is a gate, not a request. Chapter 19 turns this into an operations checklist.
The student experience, honestly
Students largely endorsed the format: 70% agreed it tested genuine understanding. They also found it harder: 83% reported more stress than a written exam — and 83% had never taken any oral examination. Those numbers are a design brief, not a verdict. The stress is partly unfamiliarity, and unfamiliarity is fixable with the tools this playbook already gave you: publish the structure, provide the same examiner as an unlimited-use practice scenario for weeks beforehand, and grade the first one formatively. An oral exam that students have privately rehearsed ten times is a different psychological event from an ambush. Chapter 16 treats the equity and accommodation questions this raises.
There is also a role-play framing worth making explicit to students: the examiner is a persona, the exam is a scenario, and the skill being assessed — explaining your reasoning under questioning — is the same one the rest of this playbook builds. Programs that run interview drills (Chapter 5) will find their students walk into AI vivas already fluent in the format.
On Tough Tongue: the examiner pattern is a scenario with the student’s submission attached as context, a rubric attached for scoring, resistance configured to probe two layers past every answer, and session recordings retained for review and appeals. The same scenario published in practice mode doubles as the revision tool, which is the cheapest stress mitigation available.
Exercise
Draft the rubric for a 15-minute viva on the last major assignment you graded. Four criteria, each with a one-line description of what a top answer demonstrates live (not what the written submission contained): e.g., can explain why alternatives were rejected; can extend the work to a variant the assignment didn’t cover; can identify their own weakest assumption; can answer a factual probe about their own text. Then test it the honest way: pick the submission you most suspect was AI-written, and ask whether your four criteria would catch it in fifteen minutes of conversation. If not, sharpen them. This rubric is the seed for Chapter 19’s dry run.