Examining Differences in Examinee Performance in Paper and Pencil and Computerized Testing - Gautam Puhan, Keith Boughton, & Sooyeon Kim - The Journal of Technology, Learning, and Assessment, Volume 6, Number 3, November 2007

Abstract: The study evaluated the comparability of two versions of a certification test: a paperand- pencil test (PPT) and computer-based test (CBT). An effect size measure known as Cohen’s d and differential item functioning (DIF) analyses were used as measures of comparability at the test and item levels, respectively. Results indicated that the effect sizes were small (d < 0.20) and not statistically significant (p > 0.05), suggesting no substantial difference between the two test versions. Moreover, DIF analysis revealed that reading and mathematics items were comparable for both versions. However, three writing items were flagged for DIF. Substantive reviews failed to identify format differences that could explain the performance differences, so the causes of DIF could not be identified.

Full Text: PDF

Automated Essay Scoring Versus Human Scoring: A Comparative Study -  Jinhao Wang & Michelle Stallone Brown - The Journal of Technology, Learning, and Assessment, Volume 6, Number 2, October 2007

Abstract: The current research was conducted to investigate the validity of automated essay scoring (AES) by comparing group mean scores assigned by an AES tool, IntelliMetric™, and by human raters. Data collection included administering the Texas version of the WritePlacer Plus test and obtaining scores assigned by IntelliMetric™ and by human raters. The research sample of 107 participants was drawn from a Hispanic serving institution in South Texas. A One-Way Repeated-Measures ANOVA was conducted to examine the difference between the AES mean score and human raters’ mean score. Results of the test indicated that the mean score assigned by the AES tool IntelliMetric™ was significantly higher than the faculty human raters’ mean score on WriterPlacer Plus test. This finding did not corroborate previous studies that reported non-significant mean score differences between AES and human scoring.

Full Text: PDF