One of the persistent challenges in teaching online is designing assessments that are credible, reliable, and resistant to academic dishonesty. In smaller courses, this can be done by avoiding exams and turning to papers, blogs, creative writing, and other kinds of activities that can be gathered into a final portfolio. As much as I would love to assess students using something other than quizzes and exams in Online Rome, it just isn't feasible given the large enrollments that we are required to maintain. We do make a point of making written work some part of the grade, in the form of essays and short answer exam questions. But, ultimately, about 50% of the final grade is determined by the students' performance on quizzes and exams.
I've been waiting for a reliable and affordable online proctoring service to come on the market. With today's announcement that Instructure (Canvas) has partnered with Verificient, an online proctoring service, it looks like we are making some progress on this front. This was an obvious next move for Instructure, now that their LMS is being used as a platform for online course delivery of all sorts, including MOOCs. The absence of a proctoring service was a serious gap that, it seems, is now being filled. We'll see how effective it is--I never underestimate the creativity of students and their ability to outsmart any monitoring system. But, at least, it is a good start. As online courses become more common, we need a way to protect the integrity of the grade. The system doesn't need to be perfect. After all, academic dishonesty happens all the time in face to face classes. But instructors need to feel reasonably confident that the grades they are awarding were, in fact, earned. This is especially true as universities move towards a policy of not distinguishing between online and classroom-based courses on transcripts.
The quizzes, which appear at the end of each module, are not proctored. We tried to disincentivize dishonesty by writing questions that are application of facts rather than regurgitation of facts (and therefore difficult to Google or find in a textbook). The quizzes are timed and each question has three variations, so it is unlikely that students working together will have very many questions in common. The quizzes are also not worth all that much of the final grade. Ultimately, even if a student cheats on 1-2 questions/quiz, it is unlikely to make any difference in the final course grade.
Still, I would have preferred for the quizzes to be mastery quizzes rather than graded quizzes. When we tried that approach in the Fall 2014 version of the course, however, students didn't study for them and, ultimately, were not learning the material as well as they needed to in order to perform well on midterm exams. By switching to graded quizzes, we were able to get the students to take them seriously. The performance data looked almost identical to the data produced by my classroom-based students, which suggests to me that there was not much cheating going on. Likewise, we saw significant improvements in the exam scores--another indication that the students were studying for the quizzes. Finally, there were no obvious cases where a student had high quiz scores and low exam scores--not a sure sign of academic honesty, but a decent indication that any cheating was small-scale.
Online Rome also had three midterm exams that were administered on campus, in a proctored environment. We included these, in part, as a way to ensure the course's credibility. I suspect that, at some point, these midterms can be eliminated. I suspect that the quiz data will demonstrate that it is a reliable indicator of a student's mastery of the content; and that the instructor can design other, more engaging activities that require students to make connections across the course. A key skill in the study of Ancient Rome is the ability to see patterns and connections across time. Studying for midterm exams is one way that students begin to see broader patterns--but there are many other ways that this could happen.
For now--and, I suspect, the next several years, midterm exams will be a part of the course. For all sorts of reasons, it would be easier to administer these exams via Canvas. This spring, we required that all students--even non-UT students--take the exam on campus or in an accredited testing center. We had to resort to this after several problems with academic dishonesty in the fall semester, when we did allow distance students to take the exams online (there was an oral component to the exam, which was largely just a pain for everyone and did not work especially well). Since the majority of the students who currently take Online Rome are Austin-based, it has been fairly easy to administer the exams on campus. As we make a real effort to expand the enrollment to non-Austin based students, however, it would be great to have an effective proctoring service built into Canvas.
No comments:
Post a Comment