Chapter 9 described the anatomy of an AI oral exam and the pedagogy behind it. This chapter is the operations manual: the checklist you need before your first live viva, the policies that belong in your academic-integrity statement, and the engineering lesson that keeps emerging — the one where the AI does exactly what you prompted and still does the wrong thing.
The operations checklist
Run through this before the first scored exam. Every item below failed at least once in a published deployment.
Identity and access
- Identity verification. At minimum, the student identifies themselves by ID number before the session begins, and the system checks it against the roster. The NYU deployment used a dedicated authentication agent as its first step. The more productized version integrates with SSO (university single sign-on), which avoids the shared-account problem. Until SSO is wired, a unique session link per student plus a verbal ID check is the floor.
- Scheduling window. Give a window (e.g., “complete your viva between Monday 8 a.m. and Friday 10 p.m.”), not a fixed slot. The flexibility is a feature for students, but the deadline is necessary: open-ended windows leak information as early completers debrief.
- One attempt for summative. Practice is unlimited. The graded attempt is one shot, explicitly signaled: the student enters knowing this is the assessment, and the system marks it as such.
Per-student context injection
The exam should know what the student submitted. The project-discussion agent at NYU received the student’s project details as dynamic variables — name, project title, decisions made, tools chosen — so the examiner asked informed questions, not generic ones.
The next step is retrieval over the student’s submitted report, slides, or code, so the examiner can quote and probe precisely (“on slide 7 you wrote X — walk me through why”). This eliminates the “tell me about your project” warm-up and starts the exam at the level of specificity that separates understanding from recitation.
Exam structure and phase control
Decompose the exam into phases (authentication, project discussion, case discussion, close), each handled by a separate sub-agent or a structured workflow stage. The decomposition is not about sophistication; it is about preventing phase contamination: the authentication agent should not start asking academic questions, and the case-discussion agent should not drift back to the project.
Each phase should have explicit transition conditions (authentication confirmed → proceed; project discussion complete → move to case) and a time budget. Without time budgets, an examiner will spend twenty minutes on one question; with them, it covers the rubric.
The architecture-over-prompting rule
The NYU deployment’s most generalizable failure: despite explicit, repeated prompt instructions, the agent stacked questions — bundling two or three into one turn and overwhelming students. Prompting said “one question at a time.” The model ignored it.
The lesson applies to every behavioral constraint in an AI exam:
| Constraint | Prompting alone | Architecture |
|---|---|---|
| One question per turn | Agent sometimes ignores | System enforces turn structure; new question only after response received |
| Random case selection | Agent gravitates to favorites | Code performs the random draw and injects the case |
| Time limit per phase | Agent loses track | Timer fires a transition signal |
| Stay in scope | Agent wanders | Each phase has a separate agent with a scoped prompt |
| Don’t reveal answers | Agent hints under pressure | Answers are not in the agent’s context |
The rule of thumb: if the constraint matters enough to put in the rubric, it matters enough to put in the code. Prompts express preferences; architecture enforces requirements.
Voice and persona
The NYU team cloned a professor’s voice for the examiner, intending familiarity. Students found it aggressive. The lesson: test the persona voice with five representative students before the graded run. Warmth is a variable, not a given, and it is specific to the voice model, the pacing, and the phrasing — not to the prompt’s instruction to “be warm.”
A neutral, professional, non-cloned voice is the safe default. If you clone, clone someone students do not have a pre-existing authority relationship with.
Academic integrity policy language
Your syllabus needs a paragraph covering AI-conducted exams specifically. It should address:
- The exam is recorded and transcribed; the recording is an education record.
- The student may not use AI tools, notes, or other assistance during the graded session (or: may use specific permitted resources — be explicit).
- The transcript may be reviewed by the instructor and authorized graders.
- The grading process (automated draft, human review, council pattern if used) is described.
- The appeal path: a student who disputes a score can request a human review of the transcript; the recording is available for both parties. This is the fairest appeal process in educational assessment, because the evidence is complete and unambiguous.
What to do when the AI misbehaves
It will. Mid-exam. The agent stacks questions, fixates on one topic, gives away an answer, or produces a non-sequitur. Your protocol:
- The student reports the incident via a flagging mechanism (a button, an email address, a form) that timestamps the report.
- The session is reviewed by the instructor against the transcript and recording.
- If the misbehavior materially affected the student’s performance, the student is offered a retake with a different case seed, or the affected section is regraded by human review alone.
- The incident is logged and the scenario is adjusted before the next cohort.
Do not promise perfection. Promise a fair response to imperfection.
The dry-run checklist
Before the first live graded viva, run these five steps:
- Self-test. The instructor takes the exam themselves, twice: once well, once poorly (Chapter 17’s self-test). Confirm it scores correctly.
- Volunteer beta. Five volunteer students take the exam ungraded and give feedback on clarity, difficulty, pacing, and persona tone.
- Rubric calibration. Grade the five volunteer transcripts by hand, compare to automated scores, and adjust criteria where they diverge.
- Infrastructure test. Confirm identity verification, scheduling, and recording work with the actual student accounts.
- Policy review. The syllabus paragraph exists, the academic-integrity committee has seen the format, and the appeal path is documented.
On Tough Tongue: exam scenarios use the same scenario builder with session recordings stored for review. Per-student context injection uses dynamic variables or attached documents, and the instructor dashboard shows completed sessions with transcripts and scores. Flagged sessions route to the instructor for human review.
Exercise
Draft the dry-run checklist for your first AI-conducted viva with five volunteer students. For each of the five steps above, write the specific action, the person responsible, and the date it happens relative to the exam. Then draft the two-paragraph academic-integrity policy addition for your syllabus, covering recording, permitted resources, grading process, and appeals. These two documents, once written, are the operational pre-requisites for Chapter 20’s pilot.