Formative Assessment and Mastery Learning in Online Rome

I have no formal training in the discipline of Instructional Design and, when I started the design process for Online Rome, my university did not have any instructional designers who had worked in the online medium.  I felt a bit at sea but, intuitively and because I've taught for close to 20 years, I had some ideas about what would work for my content.  One thing I did at the start was take nothing as a given.  One hears various "rules" like "never make a video/podcast longer than 3-5 minutes."  In the end, I found it much more useful to talk to others who had taught either blended or online courses in the humanities; and to go with my gut instincts and then adjust based on student behaviors.

I approached the design process very thoughtfully and with a clear idea of what I wanted the course to accomplish.  I knew that much of it was going to be constructed to train students to be self-regulated learners.  I knew that mastery learning and frequent formative assessment would be critical.  I knew that a key component was frequent and timely feedback.  But I had only a vague idea of how to translate all of these concepts and practices into the course.  It was difficult to find clear guidance, mostly because I didn't know who to ask or where to go looking (if only I had started talking shop with Laura Gibbs a few years ago!)   Likewise, I did not have charts like the one above sitting in front of me.  It turned out that it didn't matter.  In fact, my final course design follows Bloom's Mastery Learning Process (MLP) step by step.

Notably, the first version did not--but, based on student feedback and my own (and the instructor's) sense of how the students were working through the course, we made a number of adjustments to the design.  The end result is a design that perfectly models Bloom's MLP.  I think there's a useful lesson here: even though faculty often don't know the jargon of instructional design (three years ago, I was a total ignoramus when it came to terms like summative vs formative assessment), thoughtful instructors do understand the concepts and frequently apply them in their teaching.  We can pick up the lingo pretty easily with some guidance.

I will dedicate another post to the roles that frequent and timely feedback; and self-regulated learning played in the course design.  In this post, I want to focus on the the process of mastery learning, and how I used it to structure the course modules for Online Rome.

Each module is sub-divided into several parts, usually 5-6.  For each sub-module there are typically some textbook readings (generally quite short).  As well, inside the module, students are asked to read and answer questions about primary texts or objects.  We included all the readings in a .pdf file on the landing page of each module, so that students could more easily access and review them.

Example of Readings in Sub-Modules

Each sub-module is structured as a mix of questions of various kinds (multiple choice, matching, ranking, short answer) and bits of content "delivery" in the form of links to outside video clips, short podcasts, or blocks of text.  There are usually 15-20 questions per sub-module.  The sub-module begins with a short introduction to orient students, make connections, and highlight critical issues. 

Introduction of sub-module on Tiberius Gracchus

This constant re-orientation is time-consuming but is an important structural principle.  I've found that, especially in online courses, orientation (and the feeling of security that orientation creates for the student) is an essential piece of the puzzle.  These frequent re-orientations also provide the designer with the opportunity to re-emphasize important parts from previous sub-modules; and lets us foreshadow things to come.  So, in the introduction picture above, we not only introduced the student to Tiberius Gracchus but also to his younger brother Gaius.  That way, when the students come to Gaius Gracchus in a later module, they have already been primed to connect him to his brother Tiberius and to think about how their political careers differed.

We end each sub-module with a short conclusion that summarizes the key points of the sub-module and also points forward to the next sub-module.  One reason for including the discussion of what's to come: realizing that students don't necessarily sit down and work through an entire module in one pass.  We don't know exactly how they work through the modules--this is information I'd love to have--but it's clear that they tend to do 1-2 sub-modules at a time.  So the other purpose of this constant re-orientation is to help them pick up where they left off.  If they read a conclusion to a sub-module, they have some sense of what is coming; and then, when they start the next sub-module, and read its introduction, they are re-oriented in the material.

Typical sub-module

The image above illustrates the organization of a typical sub-module: we ask the students to read a short selection from the biographer Plutarch and then answer a question about the reading.  The next question also refers them first to a short reading and then asks a question.  These questions are followed by a text block, where we give them more detailed information about Numantia.

We also included podcasts on a range of topic, in this case the topic is the complex Gracchan Land Reforms.  More questions follow the podcast, asking students to apply knowledge they have acquired from readings in the sub-module, podcasts, or text blocks.  We often asked multiple questions about important bits of content, with these questions distributed throughout the module.  An intial question might only ask the student to recall a fact; subsequent questions would require analysis and application of those facts.   As well, towards the end of the modules, we would ask questions that required students to make connections between the bits of content in the sub-module (or even, with content that they had learned in earlier parts of the course).  Repetition with variation was a key principle of design in Online Rome. 

As they worked through the sub-module, students received instant feedback from Canvas.  They were told whether they got the right answer but, whether their answer was right or wrong, all students received more information about the question in a feedback box.  To give one example:  A multiple choice questions asks, "According to the Lex Licinia, how much public land could a Roman legally cultivate?"  There are five possible answer choices.  The feedback box that pops up after the student submits their answer choice elaborates and reinforces the content that the question is testing: "The Lex Licinia allowed all Roman citizens to cultivate up to 500 iugera (about 300 acres) of public land (ager publicus). This land was owned by the Roman state but could be used free of charge by citizens to grow crops."

In building the course, we used the feedback box strategically, to clarify common misunderstandings, to repeat content that was being tested in the question, and to elaborate on the content.  The feedback box was another place where we "delivered" content.  We regularly asked questions later in the sub-module from content that was given in the feedback boxes, in part to remind students to read the boxes even if they got the question right.

To receive full credit for their module work, students had to earn a 90% on the automatically graded questions.  They could re-do the submodules as many times as they wanted to, until they were satisfied with their grade.  The entire goal was mastery.  This was explained to students in the orientation module at the start of the class and repeated several times during the first few weeks.  Each module ended with a 15 question graded, MC quiz.  A 90% or higher on the graded quiz earned full credit; 70-89% earned half credit.  Although the quiz was graded, it was still very low-stakes, ultimately.  Students had the opportunity to work through a practice quiz before taking the graded quiz.  The practice quizzes usually had 30-50 questions and were useful for assisting students in identifying gaps in their mastery.  Several of the questions on the graded quizzes were directly taken from the modules or practice quiz, or were variations that tested the same content.  Everything about the module work emphasized mastery of content over grades; and formal assessment was used primarily to motivate students to review and consolidate the content from the module before heading into the more high-stakes midterm exams.

We did include summative assessments, in part to ensure that students were doing the work themselves and learning the content.  If a student was working through the course as it was designed, however, the "high-stakes" midterms would not be exceptionally difficult.  They would have had plenty of practice with the content and with answering questions about the content.  The other type of summative assessment we included was a short (500-750 word) essay at the end of each module.  Each student was required to write five essays over the course of the semester.  The essay prompt required students to apply and analyze.  But again, the essay prompts were simply extensions of the short answer questions that the student should have answered in the module work.  The essay work also helped to prepare the students for the short-answer section of the midterm exams.

Because orientation and security are key factors for successful student learning in an online class, we found that this emphasis on mastery learning worked extremely well.  Our students in the first two iterations of Online Rome demonstrated that they were willing to work hard when expectations and pathways were clearly articulated and rational.  They appreciated the progression of activities, the frequent practice and feedback, that supported their mastery of the course content.  Interestingly, one of the favorite activities was the essays at the end of the modules--a huge surprise since, in my experience, students don't love to write.  But several students from the Spring 2015 iteration mentioned that they liked the essays.  I suspect this is because, having mastered the facts and having done some basic analysis, they enjoyed the opportunity to do more in-depth analysis of key concepts/events in the course.

By building the course around the concept of mastery learning, using frequent formative assessments, and providing instantaneous feedback, we were able to create a comfortable learning environment for students.  As well, by including the graded quiz at the end of each module, we incentivized students to consolidate their learning module by module instead of midterm by midterm--and this made a substantial difference in the exam grades of the "middle of the pack" students who tend to perform poorly when they cram for exams.  Although the graded quizzes were not worth that much of the final course grade, students took them seriously--the distribution of grades mirrored pretty closely the distributions I see in my f2f classes.  In its final version, Online Rome looks very much like its blended counterpart, mutatis mutandis.