On improving Kaiwa (会話) assessment: Incorporating JF Standard descriptors and MFRM

Abstract

Despite the available rubrics, assessing speaking objectively has been a debatable issue to language assessment experts mostly due to the dependence on the raters’ authority. Scoring speaking performance often results in unfairness since subjectivity may come into play. Kaiwa (Speaking) is one of the four competencies examined in Japanese language assessment. On one hand, objective and accurate speaking assessment is badly needed. On the other hand, raters tend to overrate or underrate at times. Using Many-Facets Rasch Measurement (MFRM) and JF Standard descriptors, this study aimed to evaluate the Japanese speaking (Kaiwa) assessment. To this end, a cohort of 75 freshmen, consisting of 28 males (37%) and 47 females (63%), were assessed on the five-rubric scale (comprehension, vocabulary, structure, fluency, and pronunciation). These students’ age ranged from 18 to 20 years of age and their Japanese proficiency level was equal to N5. Two raters were involved in the assessment. The result revealed that: (1) 29 biases were found in rater-student interaction and rater-component interaction; (2) different patterns of rating behaviour were discovered. Rater 1 was more lenient than rater 2 but rater 2 was more consistent; (3) pronunciation and fluency are components that contributed the most to bias while structure was the most objective component being scored. For examiners, this result implies that scoring moderation should be held before grading students. For policymakers, the implication of the study suggests that modifications in the assessment rubric and statistical control be made so that fairer ratings could be achieved.