Analysis of Evaluator Reliability for the Raters' Calibration Training

Kim,  jooah; Shin,  Yooseok; Seo,  Jeong Taeg

J Korean Dent Assoc > Volume 58(5); 2020 > Article

The Journal of The Korean Dental Association 2020;58(5):284-291.
Published online May 31, 2020.

채점자 조정(calibration) 교육 제안을 위한 평가자 신뢰도 분석

김주아¹, 신유석², 서정택³

^¹연세대학교 치과대학 치의학교육학교실
^²연세대학교 치과대학 보존과학교실, 구강과학연구소
^³연세대학교 치과대학 치의학교육학교실, 구강생물학교실

Analysis of Evaluator Reliability for the Raters' Calibration Training

jooah Kim¹, Yooseok Shin², Jeong Taeg Seo³

¹Department of Dental Education Yonsei University College of Dentistry
²Department of Conservative Dentistry and Oral Science Research Center Yonsei University College of Dentistry
³Department of Dental Education and Department of Oral Biology Yonsei University College of Dentistry

Abstract

This study analyzed the change in the rater reliability based on the student's practice evaluation process conducted at Yonsei University College of Dentistry. Through this, we suggest the significance of the rater calibration training in the student's practical evaluation of dental college. Nine professors from the department of Conservative Dentistry, Yonsei University College of Dentistry, analyzed the results of class II restoration cases twice in 2017 and once in 2018. Intra Class Correlation (ICC) which is a statistic used to determine the consistency of raters with three or more scores, was also calculated. ICC values increased as raters participated in rater calibration meetings and grading experiences. This shows that the rater reliability is related to the grading experience and feedback from calibration meeting. Based on the results of previous studies that grading experiences and rater calibration training can cause a meaningful change in rater behavior, we propose to conduct rater calibration training to ensure the evaluator reliability.

Key Words: reliability;rater calibration;intra-class correlation;