Australian test finds robot essay assessors on par with human teachers
Add teaching to the 'well-educated middle-class jobs threatened by automation' list
Software has emerged as the equal of humans when it comes to marking essays in an Australian study.
The test of test-marking software was conducted by the Australian Curriculum, Assessment and Reporting Authority, which administers standardised tests called the National Assessment Program – Literacy and Numeracy (NAPLAN).
NAPLAN tests include an essay-writing component, and in the 2012 edition of the tests some students were asked to type those essays into computers.
ACARA believes those essays are suited to automated marking because they have “criteria that target lexical properties of essays (sentence structure, paragraphing, punctuation and spelling), criteria that target semantic properties of essays (audience and ideas) and criteria that explicitly target successful interplay of lexical and semantic features of writing (test structures and cohesion).”
Automated essay-marking software, ACARA's research paper (PDF) on the subject says, is proven to and designed for assessing those qualities, so the Authority decided to give four vendors a run and compare their assessments to those of human essay markers.
To help train their systems the four vendors were provided with 1,014 essays plus scores provided by human markers. The vendors were then handed 339 essays to mark with their automated systems.
There's a heap of methodological explanation in the research paper, but the TL;DR conclusion is that “The investigation showed that at both the rubric criteria and total score levels, the four marking systems provided satisfactory (equivalent or better) results relative to human marking.”
The result is significant in Australia, because ACARA is also planning to administer NAPLAN online, instead of with printed tests. It's probably significant globally, because teachers can often pick up extra cash marking exams.
Teachers also spend lots of time reading and marking essays. If software can take over some of that work, perhaps it might free teachers for other activities. Less pleasant scenarios? Try students using essay-reading-as-a-service to fine-tune their output without having to do much thinking. Also teachers missing out on casual marking gigs, and perhaps a reason for downsizing as another middle-class job requiring a high level of education is threatened by automation. ®