ACT Learning and Professional Services

As a machine learning model, CRASE is a type of AI model, but it differs greatly from other popular models in that it is specifically trained based on ACT’s writing rubric, utilizing tens of thousands of real ACT writing tests to complete the specialized task of rating essays.

One way to evaluate the accuracy of CRASE is to send a sample of essays through the engine and determine the percentage of essays that received the same score from both CRASE and the human scorer. This percentage is called the exact agreement rate. ACT requires that the exact agreement rate between any two raters (human or computer) be at least 60%, a common threshold for interrater reliability in the scoring of standardized performance assessments.

For the CRASE models created for the ACT writing test, all human-CRASE exact agreement rates exceeded 68%, surpassing ACT’s requirement.

In a separate study, CRASE researchers compiled the human-human and human-CRASE exact agreement rates for 173 essay scoring models produced for various customers between 2016 and 2019. We compared the human-human and the human-CRASE exact agreement rates. A significant majority of the human-CRASE exact agreement rates were higher than their human-human counterparts.

See the CRASE5 for ACT Writing Technical for additional reading on CRASE’s performance.

Some news articles have highlighted the potential for bias to exist in automated scoring engines, expressing concern that such bias may inadvertently advantage or disadvantage an examinee because of their gender or ethnicity.

The CRASE research team takes these concerns seriously, and our responses with respect to the CRASE models developed for the ACT writing test include the following:

• We reviewed examinee demographic information throughout engine training to ensure that selected subgroups were fairly represented in the model-building process.
• We computed various agreement metrics for selected subgroups to confirm that scoring accuracy thresholds were met for all studied subgroups. 
• We used a new evaluation technique called differential feature functioning to determine whether the essay features used in scoring were behaving consistently across studied subgroups.
• We continue to research new methods for identifying potential subgroup differences in automated scoring models.

All analysis to date has shown that any observed differences between subgroups are minimal and should not directly advantage or disadvantage the diverse population of examinees taking the ACT writing test.

CRASE can automatically identify certain types of unusual responses, such as essays that are blank, essays written in a language other than English, and essays containing certain types of “non-attempts.”

CRASE has been used by ACT to score the ACT writing test in growing capacity since late 2022. For examinees, the best approach is to write with a focus on producing a compelling argumentative essay; the criteria established in the writing rubric reflect concepts necessary to do so. Students should concentrate on producing their best writing.