Criterion Online Writing Evaluation Service
Educational Testing Service launched the Criterion Online Writing Evaluation Service around 2003, a web-based tool that used natural language processing and machine learning to score student essays automatically, providing formative feedback intended to supplement instructor assessment.
Background
By the early 2000s, researchers had been working on automated essay scoring for several decades. The earliest systems, going back to Project Essay Grade in the 1960s, had relied mostly on surface features: word count, sentence length, vocabulary range. They could approximate a holistic score, but they did not look closely at what a student had actually written.
ETS, the Educational Testing Service, had been developing a more sophisticated approach through the 1990s. Their e-rater engine used natural language processing, the field of getting computers to analyse written and spoken language, to go further than surface counting. It looked at syntactic structure, discourse organisation, and lexical features to produce a score. By the late 1990s e-rater was being used alongside human raters in high-stakes testing, but that work was happening inside ETS’s research and testing operations. Classroom teachers did not have access to anything like it.
What teachers had instead was time pressure. A writing teacher working with several classes might receive hundreds of drafts across a week. Giving each one a useful written response, and returning it quickly enough to help students revise, was genuinely hard. Formative feedback, the kind meant to help a student improve a piece of writing before it is graded, was the thing most likely to get squeezed.
What happened
In 2003, ETS made the e-rater technology available to schools and universities through a commercial web service called the Criterion Online Writing Evaluation Service. Students submitted essays through a browser, and the system returned both a score and diagnostic feedback on specific writing features: grammar, usage, mechanics, style, and organisation. The idea was not to replace a teacher’s judgement, but to give students a way to get feedback on drafts at any hour, as many times as they chose to revise.
The underlying e-rater engine, described in ETS research published in 2004, worked by extracting features from a submitted essay and comparing them against models built from large sets of human-scored writing. Version 2.0 of e-rater incorporated a range of linguistic features and used a linear model to combine them into a score. The 2004 research report details the system’s agreement with human raters across several essay types, though exact agreement figures varied by prompt and population.
The CALICO Journal review from 2016 describes Criterion as it operated in practice: a tool aimed at second-language writers as well as native English speakers, positioned for use in composition courses and language learning programmes. The web-based delivery meant institutions could offer it without installing software locally, which was still an unusual convenience at the time. Students could submit a draft, read the automated feedback, revise, and submit again, all within a single session.
Why it mattered
Criterion represented one of the first large-scale commercial deployments of automated essay scoring in an educational context, moving NLP-based writing assessment from research settings into routine classroom use. Its underlying e-rater engine, developed at ETS, contributed to the evidence base for automated scoring validity and influenced subsequent work on machine-scored writing assessment at scale. The service also raised persistent questions about the pedagogical effects of teaching to an automated scorer, prompting a research literature that continues to inform the field.
Organisations
Sources
- A Review of Criterion: An Online Essay Evaluation System.CALICO Journal.Secondary
- Automated Essay Scoring with e-rater V.2.0.ETS Research Report Series.Primary source
Cite this page
AI Achievements. (2003). Criterion Online Writing Evaluation Service. Retrieved 2026-08-22, from https://achievements.ai/milestone/criterion-online-essay-evaluation
@misc{achievements_criterion_online_essay_evaluation,
title = {Criterion Online Writing Evaluation Service},
author = {{AI Achievements}},
year = {2003},
url = {https://achievements.ai/milestone/criterion-online-essay-evaluation}
}