Replacing human readers in examinations

We have completed phase 1 of investigating the possibility of replacing human readers in examinations. This involved analysing past exam papers for suitability with text-to-speech applications (i.e. TextHelp).

  • Some subjects are unproblematic for TextHelp.These subjects are Anthrolopogy, Applied Social Studies, Business & Law, Education, English, Philosophy, Politics and Sociology. A couple have minor issues as a result of formatting (e.g tables reading down instead of across).
  • Some are completely unsuitable (e.g. scientific formulae, computer-language syntax, language not English). These subjects are Nua Ghaeilge, Chemistry, German, Electronic Engineering, French, Computer Science, Mathematics, Experimental Physics, Maths Physics and Spanish.

I suggest that with both of these sets of subjects, no further investigation or testing is required. Several subjects show potential and need to be further examined: History, Media Studies, Ancient Classics, Geography, Biology, Finance, Economics and Music. These subjects are ambiguous for several reasons;

  • Some comprises of unproblematic papers and problematic papers (e.g. Music – text-based papers OK, papers with musical notation unsuitable).
  • Some have a mix of text with significant use of mathematical or other symbols within questions (e.g. Economics, Biology).
  • Some have heavy use of Latin words both within questions and as part of other parts of questions (Ancient Classics).

Our next move is to work out what risks are associated with using technology to read the papers? We will need to consult both the departments and student-users on this. For example, if a candidate uses TextHelp to read a question with, say, the symbol for infinity, the software won’t read that symbol correctly. We need to establish whether this is a serious issue for the candidate on two fronts:

1. Exam integrity: Could a misunderstanding occur which would result in a candidate making a avoidable error in the exam. How would it be different if a (non-specialist) human reader was present? (Academic Dept/Exams Office)

2. Effect of SLD/disability. Can we reasonably say that a candidate with an SLD would be aware that he/she would need to pay special attention to items such as these, but yet benefit from the technology with the remaining text in the paper which is unproblematic? Could awareness of the limitations of the software be part of the training they receive from us? Are there issues around exam stress/anxiety that need to be considered? (Yes, I’d imagine)

Interesting times!  As we have a site licence for TextHelp, we need to go with that (and PDFAloud as it helpfully pops itself into Acrobat) until we are able to consider others software combinations.

Advertisements

One response to “Replacing human readers in examinations

  1. I will say that this is one area where WYNN has big advantages, reading diagrams, and mathematical and scientific language. Still it is never perfect. I read one book a few years ago on Second Language acquisition. It used the terms “L1” and “L2” consistently, and nothing I could do could make it anything but, “one pound” and “two pounds.”

    I think the biggest thing here is choice, I’d almost ALWAYS prefer the computer. The human reader has always robbed me of the privacy I need to engage my thinking process. But that’s me. I’ll make those adjustments and look out for the issues you talk about. Of course it helps that I have a dozen years of experience using computer readers, and have built up my skills.

    As you say, interesting times. It is just hard to successfully mix the range of human learning needs with mass evaluation, and with “single software” paradigms.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s