Author Information
Makiko Kato, Tohoku University, JapanAbstract
Although many rubrics have been developed to assess English summary writing, most have focused on intermediate to advanced learners. Recognizing that learners with different proficiency levels produce summaries with diverse characteristics, particularly when working with different source texts, this study examines the validity of a four-category analytic rubric designed for learners across a wide range of proficiency levels. Using multivariate generalizability theory, the study analyzed sources of score variance, rating reliability and stability, relationships among rubric categories, and the effects of the number of raters on score reliability. The data consisted of two English summaries written by 70 Japanese university students based on two different source texts. Six experienced English teachers served as raters and were divided into two groups, each evaluating summaries produced by 35 students using the same analytic rubric. The results showed that the contributions of score variance differed across rubric categories and rater groups. In particular, the Paraphrasing category exhibited relatively large variance attributable to examinees, indicating that it consistently reflected learners’ summarization ability. In contrast, other categories were more influenced by examinee-by-task interaction and residual variance, suggesting greater sensitivity to evaluation conditions. Rating reliability also varied by rater group and generally improved as the number of raters increased, while reliability decreased substantially when only one rater was used. Overall, the findings indicate that the analytic rubric is effective for capturing learner differences in specific categories, while highlighting the importance of rater composition and assessment conditions in its application.
Comments
Powered by WP LinkPress