Can a conversation reveal more about someone’s capability than a polished written answer?

I came across a fascinating new paper through Dr Philippa Hardman 's excellent newsletter on how AI is quietly reinventing assessment.

Researchers at NYU used AI voice agents to conduct oral examinations with students. The AI questioned them about their work, probed their reasoning and asked follow-up questions. The resulting conversations were then assessed against a rubric by several different language models.

The study is small and the approach still needs proper validation against expert human judgement. But its central premise is important: when AI can produce a polished essay or report, the finished artefact tells us less about what someone genuinely understands. A live, adaptive conversation makes it much harder to conceal the quality of the thinking behind it.

I was particularly interested because this is very close to the approach I have been developing with Leadership Skill Builder.

Rather than asking people to describe their leadership capability in a questionnaire, it places them in a realistic situation and challenges them to explain what they would do and why. Their responses are evaluated across five areas of leadership capability, with feedback grounded in what they actually said.

The intention is not simply to generate a score. It is to turn assessment into development: identify a skill worth strengthening, provide focused learning and create another opportunity to practise it.

I would not describe one research paper as a validation of the whole approach. But it is encouraging to see serious academic work exploring the same underlying idea: that adaptive conversation can provide richer evidence of capability than static assessment.

The beta version of Leadership Skill Builder is free to try.

Be warned: the assessor will challenge your answers. It will not give you an easy ride.