Assessment Issues in an On-Line Environment - Addressing Rater Reliability Issues
Abstract: Assessment in a large scale on-line environment has potential reliability issues when two or more independent assessors are scoring assignments Determining the consistency of the scoring across the assignment can be challenging Ensuring that students are not subject to the luck of the draw for the assessor requires preparation of the assessors before the scoring begins The assessment process is subject to error from an assessor who is not prepared or who incorrectly interprets scoring guidelines Interpreting the performance of students with confidence and reliability among assessors is possible when all assessors are prepared for scoring in a consistent manner This discussion will focus on the process of training teaching adjuncts in a large scale on-line graduate program to assess the work in electronic portfolios The e-portfolios are presented as a comprehensive assessment of the work of graduate student throughout the program of study