Author Information
Makiko Kato, Tohoku University, JapanAbstract
With the introduction of summary writing in the EIKEN Test, the importance of teaching the task has increased. However, integrated writing tasks are more complex to assess than independent writing such as essays, placing a greater burden on teachers. In actual educational settings, teachers have limited time to participate in rater training due to extensive non-instructional duties. As a result, teachers with varying levels of experience evaluate student writing independently. Ideally, analytic scoring rubrics should be employed precisely because summary writing is a complex skill, as they enable providing feedback to students. A valid rubric should also minimize severity differences among raters. This study is part of validation research for a four-category analytic rubric developed to capture diverse characteristics observed in English summaries produced by learners at various proficiency levels. English summaries written by 70 Japanese university students were evaluated by six raters and analyzed using the Many-Facet Rasch Model (MFRM) to examine: facet effects and fit indices; rater severity; independence of rating categories; difficulty of summary tasks based on different source texts; interaction effects; and measurement precision of writer ability estimates. Overall, the results demonstrated that this rubric validly measures summarization ability even within the constraints of a nested rater design. While differences in rater severity and task difficulty exist, rater training and adjusted scores are essential for ensuring fair assessment. Although statistically significant difficulty differences were found between the two tasks, these differences were practically small and correctable through MFRM, suggesting both tasks can be considered comparable
Comments
Powered by WP LinkPress