Tag Archives: examinations

Replacing human readers in examinations

We have completed phase 1 of investigating the possibility of replacing human readers in examinations. This involved analysing past exam papers for suitability with text-to-speech applications (i.e. TextHelp).

  • Some subjects are unproblematic for TextHelp.These subjects are Anthrolopogy, Applied Social Studies, Business & Law, Education, English, Philosophy, Politics and Sociology. A couple have minor issues as a result of formatting (e.g tables reading down instead of across).
  • Some are completely unsuitable (e.g. scientific formulae, computer-language syntax, language not English). These subjects are Nua Ghaeilge, Chemistry, German, Electronic Engineering, French, Computer Science, Mathematics, Experimental Physics, Maths Physics and Spanish.

I suggest that with both of these sets of subjects, no further investigation or testing is required. Several subjects show potential and need to be further examined: History, Media Studies, Ancient Classics, Geography, Biology, Finance, Economics and Music. These subjects are ambiguous for several reasons;

  • Some comprises of unproblematic papers and problematic papers (e.g. Music – text-based papers OK, papers with musical notation unsuitable).
  • Some have a mix of text with significant use of mathematical or other symbols within questions (e.g. Economics, Biology).
  • Some have heavy use of Latin words both within questions and as part of other parts of questions (Ancient Classics).

Our next move is to work out what risks are associated with using technology to read the papers? We will need to consult both the departments and student-users on this. For example, if a candidate uses TextHelp to read a question with, say, the symbol for infinity, the software won’t read that symbol correctly. We need to establish whether this is a serious issue for the candidate on two fronts:

1. Exam integrity: Could a misunderstanding occur which would result in a candidate making a avoidable error in the exam. How would it be different if a (non-specialist) human reader was present? (Academic Dept/Exams Office)

2. Effect of SLD/disability. Can we reasonably say that a candidate with an SLD would be aware that he/she would need to pay special attention to items such as these, but yet benefit from the technology with the remaining text in the paper which is unproblematic? Could awareness of the limitations of the software be part of the training they receive from us? Are there issues around exam stress/anxiety that need to be considered? (Yes, I’d imagine)

Interesting times!  As we have a site licence for TextHelp, we need to go with that (and PDFAloud as it helpfully pops itself into Acrobat) until we are able to consider others software combinations.