Using AI to generate follow-up questions in an oral assessment

In a previous post, I discussed the need to consider the logistical and pedagogical implications of running an oral assessment as an extra assurance that the learner’s work is authentic and representative of their understanding of the content (Sotiriadou, 2019). In the post, I outlined several logistical considerations, including where and when to run this specific type of assessment, the need to consider the weighting of the assessment to ensure it is taken seriously, the need to teach the students how to do well in the assessment and not assume that they will naturally be able to perform, as well as the need to train the tutors or those running the assessments to create equity in the types of questions and provocations asked of the students.

The University of Sydney’s wonderful resource on using AI potentially helps address this last consideration. It outlines this innovative approach: ‘During the assessment: Use AI-driven (+/- examiner co-piloted) platforms to perform the oral test, such as to generate follow-up questions to the student.’

This potentially eliminates the lack of equity in having varying levels of skill amongst tutors asking questions to different students. However, I would want to see this run in real life to see how effective it is, and whether the questions would actually improve the reliability of the difficulty level of questions being asked.

Another innovative example of theirs is to have the oral test observed by the examiner but performed between the student and a simulated client (custom AI) that has been trained to function in a specific way relevant to the learning outcomes being assessed. Again, the testing out of this would be useful in determining its validity.

What is relevant here though is the need to know what you are looking for to know if it is beneficial. Like all AI output, in order to know if it is of value it pays to be highly knowledgeable in the area explored. If we are going to be asking our students to be critical of the outputs of gen-AI, we too as educators must possess this skill. We must be able to determine if the types of questions that gen-AI is generating in an oral test are relevant and equitable and be able to step in and adapt if not. To do so, we have to practise this skill. We have to align the questions with the assessment outcomes and understand the gradation necessary to differentiate between responses. We have to run mocks of the test and in real time analyse and critically reflect on the AI output. Only then will we be assured that it is an efficient and fair thing to do.

I believe it is going to become increasingly relevant to run an oral assessment as a way to certify learning. Knowing what constitutes good questioning is a skill that needs learning and developing. Only then will gen-AI augment the process.   

References

Sotiriadou, P., Logan, D., Daly, A. and Guest, R. (2019). The role of authentic assessment to preserve academic integrity and promote skill development and employability. Studies in Higher Education, 45(11), pp.1–17. doi:https://doi.org/10.1080/03075079.2019.1582015 .

‌

I’m Paul Moss. I’m a learning designer at the University of Adelaide. Follow me on Twitter @edmerger