Formative Assessment and Mastery Learning in Online Rome

I have no formal training in the discipline of Instructional Design and, when I started the design process for Online Rome, my university did not have any instructional designers who had worked in the online medium.  I felt a bit at sea but, intuitively and because I've taught for close to 20 years, I had some ideas about what would work for my content.  One thing I did at the start was take nothing as a given.  One hears various "rules" like "never make a video/podcast longer than 3-5 minutes."  In the end, I found it much more useful to talk to others who had taught either blended or online courses in the humanities; and to go with my gut instincts and then adjust based on student behaviors.

I approached the design process very thoughtfully and with a clear idea of what I wanted the course to accomplish.  I knew that much of it was going to be constructed to train students to be self-regulated learners.  I knew that mastery learning and frequent formative assessment would be critical.  I knew that a key component was frequent and timely feedback.  But I had only a vague idea of how to translate all of these concepts and practices into the course.  It was difficult to find clear guidance, mostly because I didn't know who to ask or where to go looking (if only I had started talking shop with Laura Gibbs a few years ago!)   Likewise, I did not have charts like the one above sitting in front of me.  It turned out that it didn't matter.  In fact, my final course design follows Bloom's Mastery Learning Process (MLP) step by step.

Notably, the first version did not--but, based on student feedback and my own (and the instructor's) sense of how the students were working through the course, we made a number of adjustments to the design.  The end result is a design that perfectly models Bloom's MLP.  I think there's a useful lesson here: even though faculty often don't know the jargon of instructional design (three years ago, I was a total ignoramus when it came to terms like summative vs formative assessment), thoughtful instructors do understand the concepts and frequently apply them in their teaching.  We can pick up the lingo pretty easily with some guidance.

I will dedicate another post to the roles that frequent and timely feedback; and self-regulated learning played in the course design.  In this post, I want to focus on the the process of mastery learning, and how I used it to structure the course modules for Online Rome.

Each module is sub-divided into several parts, usually 5-6.  For each sub-module there are typically some textbook readings (generally quite short).  As well, inside the module, students are asked to read and answer questions about primary texts or objects.  We included all the readings in a .pdf file on the landing page of each module, so that students could more easily access and review them.

Example of Readings in Sub-Modules

Each sub-module is structured as a mix of questions of various kinds (multiple choice, matching, ranking, short answer) and bits of content "delivery" in the form of links to outside video clips, short podcasts, or blocks of text.  There are usually 15-20 questions per sub-module.  The sub-module begins with a short introduction to orient students, make connections, and highlight critical issues. 

Introduction of sub-module on Tiberius Gracchus

This constant re-orientation is time-consuming but is an important structural principle.  I've found that, especially in online courses, orientation (and the feeling of security that orientation creates for the student) is an essential piece of the puzzle.  These frequent re-orientations also provide the designer with the opportunity to re-emphasize important parts from previous sub-modules; and lets us foreshadow things to come.  So, in the introduction picture above, we not only introduced the student to Tiberius Gracchus but also to his younger brother Gaius.  That way, when the students come to Gaius Gracchus in a later module, they have already been primed to connect him to his brother Tiberius and to think about how their political careers differed.

We end each sub-module with a short conclusion that summarizes the key points of the sub-module and also points forward to the next sub-module.  One reason for including the discussion of what's to come: realizing that students don't necessarily sit down and work through an entire module in one pass.  We don't know exactly how they work through the modules--this is information I'd love to have--but it's clear that they tend to do 1-2 sub-modules at a time.  So the other purpose of this constant re-orientation is to help them pick up where they left off.  If they read a conclusion to a sub-module, they have some sense of what is coming; and then, when they start the next sub-module, and read its introduction, they are re-oriented in the material.

Typical sub-module

The image above illustrates the organization of a typical sub-module: we ask the students to read a short selection from the biographer Plutarch and then answer a question about the reading.  The next question also refers them first to a short reading and then asks a question.  These questions are followed by a text block, where we give them more detailed information about Numantia.

We also included podcasts on a range of topic, in this case the topic is the complex Gracchan Land Reforms.  More questions follow the podcast, asking students to apply knowledge they have acquired from readings in the sub-module, podcasts, or text blocks.  We often asked multiple questions about important bits of content, with these questions distributed throughout the module.  An intial question might only ask the student to recall a fact; subsequent questions would require analysis and application of those facts.   As well, towards the end of the modules, we would ask questions that required students to make connections between the bits of content in the sub-module (or even, with content that they had learned in earlier parts of the course).  Repetition with variation was a key principle of design in Online Rome. 

As they worked through the sub-module, students received instant feedback from Canvas.  They were told whether they got the right answer but, whether their answer was right or wrong, all students received more information about the question in a feedback box.  To give one example:  A multiple choice questions asks, "According to the Lex Licinia, how much public land could a Roman legally cultivate?"  There are five possible answer choices.  The feedback box that pops up after the student submits their answer choice elaborates and reinforces the content that the question is testing: "The Lex Licinia allowed all Roman citizens to cultivate up to 500 iugera (about 300 acres) of public land (ager publicus). This land was owned by the Roman state but could be used free of charge by citizens to grow crops."

In building the course, we used the feedback box strategically, to clarify common misunderstandings, to repeat content that was being tested in the question, and to elaborate on the content.  The feedback box was another place where we "delivered" content.  We regularly asked questions later in the sub-module from content that was given in the feedback boxes, in part to remind students to read the boxes even if they got the question right.

To receive full credit for their module work, students had to earn a 90% on the automatically graded questions.  They could re-do the submodules as many times as they wanted to, until they were satisfied with their grade.  The entire goal was mastery.  This was explained to students in the orientation module at the start of the class and repeated several times during the first few weeks.  Each module ended with a 15 question graded, MC quiz.  A 90% or higher on the graded quiz earned full credit; 70-89% earned half credit.  Although the quiz was graded, it was still very low-stakes, ultimately.  Students had the opportunity to work through a practice quiz before taking the graded quiz.  The practice quizzes usually had 30-50 questions and were useful for assisting students in identifying gaps in their mastery.  Several of the questions on the graded quizzes were directly taken from the modules or practice quiz, or were variations that tested the same content.  Everything about the module work emphasized mastery of content over grades; and formal assessment was used primarily to motivate students to review and consolidate the content from the module before heading into the more high-stakes midterm exams.

We did include summative assessments, in part to ensure that students were doing the work themselves and learning the content.  If a student was working through the course as it was designed, however, the "high-stakes" midterms would not be exceptionally difficult.  They would have had plenty of practice with the content and with answering questions about the content.  The other type of summative assessment we included was a short (500-750 word) essay at the end of each module.  Each student was required to write five essays over the course of the semester.  The essay prompt required students to apply and analyze.  But again, the essay prompts were simply extensions of the short answer questions that the student should have answered in the module work.  The essay work also helped to prepare the students for the short-answer section of the midterm exams.

Because orientation and security are key factors for successful student learning in an online class, we found that this emphasis on mastery learning worked extremely well.  Our students in the first two iterations of Online Rome demonstrated that they were willing to work hard when expectations and pathways were clearly articulated and rational.  They appreciated the progression of activities, the frequent practice and feedback, that supported their mastery of the course content.  Interestingly, one of the favorite activities was the essays at the end of the modules--a huge surprise since, in my experience, students don't love to write.  But several students from the Spring 2015 iteration mentioned that they liked the essays.  I suspect this is because, having mastered the facts and having done some basic analysis, they enjoyed the opportunity to do more in-depth analysis of key concepts/events in the course.

By building the course around the concept of mastery learning, using frequent formative assessments, and providing instantaneous feedback, we were able to create a comfortable learning environment for students.  As well, by including the graded quiz at the end of each module, we incentivized students to consolidate their learning module by module instead of midterm by midterm--and this made a substantial difference in the exam grades of the "middle of the pack" students who tend to perform poorly when they cram for exams.  Although the graded quizzes were not worth that much of the final course grade, students took them seriously--the distribution of grades mirrored pretty closely the distributions I see in my f2f classes.  In its final version, Online Rome looks very much like its blended counterpart, mutatis mutandis.

More Thoughts on the Small Pilot vs Immediate Scale-Up Dilemma

I love social media, especially Twitter and blogging, for many reasons but especially for the way that it facilitates the creation of communities dislocated in space.  When I first started to experiment with the flipped class model three years ago, I really had no idea what I was doing.  I didn't even know that this thing that I was wanting to do actually had a name and that there were communities of teachers out there who knew a lot about how it worked (mostly in K-12 and in math and sciences, but still).  I don't remember exactly why I got on Twitter, but I immediately discovered all sorts of interesting and helpful conversations that I could insert myself into (I also learned, rather quickly, to not be a shy lurker).  Since that time, I have benefited enormously from the connections and conversations I've made via Twitter.  I can honestly say that I learn something new every single day thanks to the plethora of smart people, through their comments or through their curation of interesting essays, articles, and blog posts.  This has been an especially critical outlet for me because my own university is still very much in the preliminary stages of thinking about the kinds of questions that interest me.  And, since I'm not an administrator, I'm generally not part of the conversation even at my own university.  Twitter is where I've found people who inspire me, engage with me, and help me think through the nuances of different important issues.

Over the weekend I wrote a post about my own experience of "jumping into the deep end" with course development projects.  In both cases, I had planned to do a pilot of the new course and then, over time, scale up the enrollments.  In both cases, for different reasons, this isn't how things played out. The first iteration of each project ended up being taught at scale (400 students for the flipped class; about 330 for the online class).  I've become convinced that one of the reasons our development process was successful--despite the enormous stress caused by the immediate scale up--is that we were able to troubleshoot very effectively.  By the second iteration, we had a class that ran smoothly and produced high levels of learning.  Thus, when I watched the e-Literate TV case study of ASU and saw that they were embracing a "jump into the deep end" approach, it resonated with me.  Likewise, when I read Phil Hill's comments on the "to pilot or not to pilot" question, I was struck by his observation that many pilots on academic campuses end up in what he calls "pilot purgatory" (a fabulous turn of phrase!).

I was really pleased to see that Matt Reed (aka @deandad) had taken up the issue in his regular column for Inside Higher Education.  He made a number of really important points about this important question.  Perhaps most usefully, he reminded readers that context matters.  When talking about a single course that is largely self-contained and not especially resource-intensive, the immediate scale-up makes more sense (in my case, the additional resources required were negligible); but if the project is costly and depends on the collaboration of multiple units, taking it a bit slower often makes sense.  Likewise, if total failure is a real possibility and if people can be irreversibly harmed by that failure, then common sense and basic morality dictates a slower development process.  The main dilemma is distinguishing between projects that would benefit from a faster scale-up; and projects that, for various reasons, need to be developed more slowly and under less risky and potentially chaotic circumstances.

It is great to see a nice conversation developing from e-Literate's ASU case study.  The question of how to develop projects, and especially how to do first iterations, is a major one on campuses these days.  It seems to me that this conversation has gone a long way towards helping experimenters make informed choices. 
One thing that came up in a Twitter exchange between me and Phil Hill: industry is much more willing to take risks in developing new projects.  Academia tends to be extremely conservative and, in general, fails to understand the cycle of iteration.  Academics (and administrators) see any kind of failure as a bug instead of a feature.  Failure is how we learn. 

Yet it is difficult for professors to put something out there that is not, in their mind, perfect.  It is even more difficult for them to deal with the inevitable setbacks and obstacles that arise during the first iteration of a new course design (including basic student resistance to change).  Thus, a lot of money is spent to develop projects and get them to the testing phase.  Yet very few ever reach the "developed" stage.  It is challenging for faculty to change their mindset, to understand that everything we do--including our teaching--is a work in progress and should be constantly evolving.  One of the great gifts of my experience with these course development process is an increased tolerance for error; and decreased perfectionism.  It can be incredibly difficult to put something out that you know isn't as good as it could be--but to accept that the fastest way to improve it is to just put it out there and let people go at it.  But this gets easier, as one understands that it is just part of the process.

Matt is absolutely correct about the risks of running a project at scale that requires lots of collaboration across different parts of the institution.  With my flipped class, everything was very self-contained.  I could handle problems myself.  With the online class, I was suddenly trying to collaborate with a number of units across campus to keep things running: the registrar, the deans in my College, IT/Canvas support, my department, our Extended Campus school; our Liberal Arts Development Studio staff, and my instructor.  It was nothing short of a circus, especially because the first iteration of the online class happened to coincide with a big increase in the number of faculty using Canvas as their LMS.  Canvas crashed a few times during the semester and had a number of odd but critical glitches. 

It was not easy to try to keep everything running and ensure that everyone was doing what they needed to be doing.  Balls definitely got dropped.  It didn't help that I ended up having unplanned surgery in early November and was out of commission for about three weeks.  One thing I noticed, however, was that it was easier to get these different units to stop dragging their feet and take action precisely because we were live with over 300 students.  They knew that if they didn't do their job, a large number of students were going to be adversely affected.  So, while it can be a massive challenge to get all these different partners pulling their weight, it was actually a bit easier to pressure them when so many students were involved.  It certainly put an enormous amount of pressure on me, as project leader, to ensure that everything got done and no major balls got dropped; and I spent a lot of time nagging people (and, I'm sure, annoyed many administrators!).  Still, in the end, I was able to get things done--largely, I think, because so much was at stake.

There are absolutely times when it makes sense to hold back and develop ideas more incrementally.  At the same time, it seems that too often, the default is to start with the small pilot with the plan of figuring out how to scale up someday.  I suspect that more of these pilots would take hold if a more aggressive experimental process was followed; and if faculty were more willing to embrace a tolerance for failure (or, rather, see failure as an essential part of the process).

Scaling Up

Teaching a large enrollment class (250+ students), whether in a classroom or online, is not unlike herding cats.  Actually, based on my considerable personal experience in both activities, I'd choose herding cats over the challenges of managing the logistics of a large enrollment class any day.  Shake a bag of treats and the cats come running.  Large classes, well, they take quite a bit more effort to prevent a decline into total chaos. 

Earlier this week, I watched e-Literate TV's latest installment on personalized learning at Arizona State University with quite a lot of interest. While I recognize the deeply concerning labor issues that underpin the ASU model, and hope that over time a more just staffing model is implemented, especially with regard to non-tenure track faculty; I also find Michael Crow's efforts to create the New American (Public) University compelling (see this condensed interview in the New York Times about his plans and motivations).  There's no question that the public university, even flagships like my own, are undergoing slow but fairly extreme transformations.  I like a lot of what Crow is doing, not least of which is emphasizing broad access without losing sight of the need to provide intensive support for many of these students who would not otherwise have had access to a four-year university.  I admire the clarity of his approach and the creative solutions he has found to create new revenue streams in the face of ever-shrinking state appropriations.

He is operating under extremely difficult circumstances, but takes that as a challenge.  He has a "take no prisoners" mentality, and he has the decided advantage of being at least five years further along in the process of evolution than just about any other institution.  He was already taking the first steps to re-imagine and re-invent the American (Public) University while most public universities (and their leadership) were unaware of how rapidly the landscape of higher education was shifting--and would continue to shift--even as the economy has begun to recover.  Many academics are critical of his vision.  For his part, Crow seems oblivious to his critics and powers ahead with his plans.  It seems that, about once a month, a new partnership or initiative is announced by ASU.

Crow has hired some very smart and experienced people to help him realize his transformative vision for ASU (and the people of Arizona).  In watching Phil Hill interview Adrien Sannier and others to capture ASU's approach to personalized learning, I was deeply impressed by several things.  But what stood out the most was the rejection of the traditional "small pilot" model of course development.  Phil Hill nicely captures Timothy Harfield's rejection of the standard pilot model--and his own observations about the general failure of this model to lead to sustaining innovations.  Typically, experimental course designs are piloted to small numbers of students, with the aim of identifying problems before "scaling up."  At ASU, when it came to implementing their remedial math course (and, I gather, other similar courses), they skipped the piloting stage and jumped straight to scale.  This is exactly the way to do it.  [But see Matt Reed's thoughtful response in Inside Higher Education, including his important point that--in some circumstances--it does make sense to start small, especially when taking on complicated projects that involve collaboration across several different parts of the institution.]

By accident, my own course development projects have both followed this same pattern.  The first time, when I flipped my Introduction to Ancient Rome course, I had planned to teach it to 200 students; but ended up with an enrollment of 400 students when my university needed more seats in introductory level courses.  I had no idea what I was getting into, and the extra 200 students added about 5 times as much work and complexity to the logistics of the course.  While it was, without question, a very challenging semester, the advantages of going straight to scale were quickly apparent: every problem became evident very quickly; we were pushed to maximize efficiency in every aspect of the course; we received a tremendous amount of feedback on every part of the course.  I spent the winter break retooling the course based on the feedback from students and teaching assistants; and my own experience of teaching the course.  I taught the revised version in the spring semester.  It went off without a hitch.  [as an aside: one other important factor in course development projects is teaching the course at least 2-3 semester in a row, in order to make the adjustments while everything is still fresh in the memory.]  I can honestly say that the first iteration of the flipped version was one of the most difficult teaching experiences of my life; but, by skipping the pilot phase, I think we were able to create a very strong course very quickly.  It was a painful first iteration, but there was a big payoff in subsequent iterations.

When we went live with the asynchronous online version of Introduction to Ancient Rome in Fall 2014, we assumed that it would be a small pilot.  We had not advertised the course at all.  It wasn't even listed and open for registration until four days before classes started.  We guessed we might get 10-15 random students register for it.  In fact, we ended up with over 300 students.  Once again, I found myself in a challenging situation.  Not only were we still doing significant development work on the modules; but I was also training an instructor who was new to the world of online education (and who had never taught a class over the size of about 30 students).  Once again, though, we found that the large size of the class immediately highlighted the design flaws and encouraged us to figure out how to maximize efficiency in everything.  It was a stressful semester for me as well as the course instructor, not least because we were rushing to finish modules and release them to students in the course of the semester.  However, we learned everything we needed to know about the course's design flaws.

I spent this spring semester revising some parts of the overall course design (e.g. adding more structure, including a graded quiz at the end of each module); and then making revisions to the individual modules.  As with the flipped version of the course, I was able to avoid a drawn-out process of identifying and eliminating the design flaws by jumping straight to teaching it at scale.  In one semester, we figured out what parts of the course either didn't work at all or needed to be revised; and also identified the features that were crucial to the course's success (e.g. clear expectations; an extended orientation to the course; strong online presence of the instructor; integration of weekly f2f review sessions on campus).   In addition, because I'd had the experience of making this scary leap before, it was a lot easier to manage the chaos and remind myself that nobody was going to die.

In theory, pilots make a lot of sense.  It seems like a way to contain the inevitable problems of a new course. In reality, however, it doesn't actually make much sense to pilot courses that need to be scaled to small groups of students.  A small class and a large enrollment class are entirely different in their character and their challenges.  Many of the challenges of large enrollment courses are logistical and are a direct result of scale.  In order to identify and remediate them, the "pilot" of the course has to be run at scale.  It is a bit scary to take this approach, to be sure; but one quickly learns that students are resilient and reasonably forgiving of our learning curve.  Most of the problems are not devastating to student learning; they just get in the way.  When I taught the flipped class for the first time, the course evaluations were mixed--about half the students loved the class and half hated it.  But their hate was a result of the flipped model, not problems in course design.  Interestingly, the student evaluations on the first iteration of the online course were much more positive.  While they identified parts of the design that needed more attention, the responses were in general very positive.  Behind the scenes, it often felt like total chaos poised on the edge of disaster.  But we were pretty good at keeping all of that behind the scenes (including our own high stress levels).  As a result, the students had an overall good learning experience while also helping us figure out how to make the course better.

One of the most significant challenges for public universities in the coming decade will be figuring out how to support learning at scale.  In particular, we have to figure out how to design and implement online courses that can be taught at scale.  This is not to say that there is not an important place for smaller, seminar-style online classes--those are also going to be an essential part of any future curriculum at a public institution.  But at least some courses will need to be large-enrollment, probably asynchronous (though could be synchronous) in order to meet growing demand, especially as institutions increase the size of their student population.  So how do we teach at scale while supporting high levels of student learning?  F2f, large enrollment lecture classes are not the answer, as universities are slowly coming to realize. The next several years will probably see a shift from faculty delivering content in the classroom to faculty building online courses that have the capacity to personalize learning (if adequately and appropriately staffed).

ASU has a good model in place for teaching at scale--a model in which the roles of the instructor and TAs in particular are re-imagined.  One of the many positive things about teaching at scale is that, with a well-designed (and tested) course, it is much easier to identify the students who need our attention.  The majority of the students are actually very capable of working on their own (often with a group of peers).  They benefit from feedback on their work but don't require frequent interventions to keep them on schedule or help them understand content.  With an adequately staffed course and skilled teaching team (and, ideally, an easy to use dashboard), members of the teaching team can focus on those students who most need intensive mentoring to advance successfully through a course.

The keys to success in teaching at scale?  A good course and, most importantly, a skilled teaching team with clearly defined roles.  ASU has figured out that the success of ALL students and not just the top 30% requires a significant structure of support, from an instructor to graduate assistants to undergraduate mentors.  Too many people, including those in administrative positions, only pay attention to the first part of this equation--the quality of the course design--and don't realize the importance of the teaching team to student success.

Liberal Arts of the Future: A Thought Experiment

Source: Dr. Roopsi Risam

"Colleges of arts and sciences are going to have to evolve a bit."

--Dr. Larry Singell, Executive Dean of the College of Arts and Sciences," Indiana University, Bloomington

As this recent Inside Higher Education article highlights, colleges of arts and sciences/liberal arts at public institutions are in a state of crisis around the country. This so-called "Crisis of the Humanities" has been developing for many years, slowly but surely. Faculty and administrators in these colleges have been remarkably slow to recognize the seriousness of the crisis and its multiple causes; and to develop coherent and effective strategies for addressing its causes. Instead, in the face of reduced instructional budgets and cuts in soft money (the kind of money that is used to fund graduate students as well as lecturers and adjuncts. It is also used to fund faculty/grad travel to conferences, research assistantships, etc.), the prevailing strategy has been to institute cuts in hiring of all sorts (tenure track faculty but also lecturers and adjuncts) and to reduce the size of graduate programs. In addition, faculty and staff have received scant raises in close to six years (with the exception of promoted faculty, who received outsized raises in the past two years, effectively placing their salary among the lower-paid full-professors and completely leap-frogging over the senior associate professors).


If these budget shortfalls were a short-term problem, this approach might make sense. It would cause some temporary difficulties for departments but those problems would be reversible once the funds were restored. In the past, this was how it worked: in lean years, hiring freezes were instituted, no merit raises were given and everyone waited for better times. Inevitably, those better times came. This was the pattern of my first 6 or so years at my university. Then, when the economy collapsed and legislatures aggressively cut allocations to public universities (in part, at least, using the economic collapse as an excuse for these cuts), everything changed.


Now, some 7-8 years later, we have to face the fact that things aren't going to return to normal. In addition, faculty in the humanities/arts and sciences need to come to terms with the fact that part of what has happened is a deliberate reallocation of institutional resources, with liberal arts on the losing end. The result: colleges of arts and sciences that are starving for resources while facing ever-growing declines in enrollment. Some blame the widespread implementation of the Responsibility Centered Management (RCM) budgeting model, in which each unit has to pay for itself. Cross-subsidies and/or distributions of supplementary funds from the provost's office are a thing of the past. In a lot of ways, RCM is a good model. It encourages units to be fiscally responsible, strategic, and entrepreneurial. The downside is that, in effect, it pits colleges within the university against each other in a competition for students. This competition for students is far less of a problem at schools whose admissions offices make an effort to ensure a good distribution of admits for all colleges. If that doesn't happen, however, inequities ensue.


In the past, however, colleges of arts and sciences could rely on the fact that, even if more students majored in engineering or computer science or business, they still had to take several courses in the arts and sciences to fulfill graduation requirements. Even if public colleges of arts and sciences have little to no control over admissions and so tend, they could count on the fact that they could keep their enrollments up by offering a plethora of service courses. Thus, at most large, public institutions, every department offers several introductory level, large enrollment courses for non-majors. These courses meet some general education requirement and the students are predominantly non-majors (in my courses, approximately 35% are liberal arts students of some ilk).


So now we get to the real problem for colleges of arts and sciences these days: enrollments in our courses by non-liberal arts students (and even non-major liberal arts students) have dropped precipitously in recent years, especially the past 3-4 years. This drop isn't minor, and it is increasing at a nearly exponential rate with each passing year. What is happening? In basic terms, two things: first, non-liberal arts students are finding other sources of courses to fulfill their graduation requirements (AP exams, credit by exam, dual enrollment, online, community college, even other 4 year university courses). With the dramatic increase in the availability of introductory online courses in arts and sciences, campuses have taken a major enrollment hit.


It is critical to understand what motivates students to seek out these alternatives to our campus courses. It is often said that they are just looking for an easier class and, certainly, that is sometimes the case. More often, though, it is for other reasons. First, flexibility. Especially STEM students with lab courses have tight schedules and appreciate the flexibility of an online course, especially asynchronous online courses that nevertheless have *some* structure to help them stay on schedule. Secondly, students are taking their required courses during the summer. In the past, they would take these during summer sessions on campus. Now, faced with rising costs and a desire to avoid taking on more debt than is absolutely necessary, many students are working in the summers and cannot take courses that are scheduled during the day. Many of them return home to live rent-free with their families. These students are also taking courses that are offered online or at community colleges at a time that fits their schedule (and are less expensive than courses at the university).


My own university--and department--has seen a precipitous dropoff in summer enrollments, yet has made few if any changes to the curriculum. We do not offer tuition discounts; we continue to offer most courses during the workday instead of in the evenings; we have few online courses on offer. Similarly, we've made few changes to our regular semester curriculum. We are beginning to offer online courses, though most are synchronous and require students to log-on for a quiz each class period at a certain time. The basic strategy to prop up enrollment tends to be to create obstacles for students who are bringing in outside credits rather than to create positive incentives for them to take our courses.


The other cause of the enrollment hit in colleges of arts and sciences is more obvious: fewer students are choosing to major in our degree programs. Some of this may have to do with the students who are admitted. I don't know statistics for my own university, but a common complaint is that admissions offices admit more students who want to major in professional degrees like business; or STEM fields rather than arts and sciences students. This means that, from the start, colleges of arts and sciences are at a disadvantage. In addition, it is becoming more common for colleges and schools within universities to aggressively recruit majors to their degree programs in order to increase their enrollments (and, thus, their share of the pot of institutional funding).

Source: Dr. Roopsi Risam

It is common for colleges of arts and sciences to claim "critical thinking" as its value proposition. This is not incorrect. Our courses and degree programs do, in fact, emphasize training and practice in critical thinking. Employers are clear that this is an important job skill. Yet, for some reason, we liberal arts folks have not been very good at moving beyond the "critical thinking" claim to more fully rationalize why it makes sense for an undergraduate student to major in English or History or Classics when entry-level jobs are hard to come by even for biology and chemistry majors.


In general, we do a terrible job of articulating distinction between short-term, jobs based skills and first jobs; and longer term, "soft" skills (like written communication, problem solving. creativity) that, over time, result in equally lucrative careers for liberal arts majors. We need to do a better job at tracking our graduates over time and tracking salaries. We need to be able to argue our case from data rather than anecdote. At the same time, we do need to pay more attention to equipping liberal arts majors for an increasingly competitive job market. It's all well and good to talk about long-term benefits, but that means little when you can't get a first job, can't pay the rent or buy food despite having a college degree.


If Larry Singell is correct that colleges of arts and sciences must evolve, as I think he certainly is, then what does/should that evolution look like? One approach that has already been taken is to add new degree programs that are more pragmatic and likely to appeal to job-focused students (and their parents), things like neuroscience, health sciences, data analytics.  We also need to offer a wider range of courses, from first year seminars to upper division electives, that might appeal to majors in other disciplines.  To give just one example, a friend of mine who teaches in an English Department is offering a Race and Cyberspace seminar for first year students at her university.  The students are all computer science majors. There are all sorts of ways that we can broaden our offerings, whether through collaborative teaching or simply by spending some time to design a new course that is likely to appeal to students outside of our department or college.  [And, ideally, any course development would be adequately supported by the university....]

Digital Humanities clearly has some role to play, and Institutes or Centers for Digital Humanities (or, alternatively, Digital Studies) are already thriving on many campuses. Digital Humanities won't save the humanities (or arts and sciences more broadly), but it is certainly one way to make the work we do in liberal arts more appealing and accessible to our students. It is also a way to equip our undergraduate and graduate students with key digital skills that are transferable to any number of careers. I don't want to over-emphasize the value of undergraduates (and graduate students) attaining some training in digital research tools as well as, for graduate students, digital teaching and learning--but these are both areas where we could do a lot to both better prepare our students for the realities of the jobs markets; and also open their eyes to new ways of thinking about familiar research questions and developing new research questions that emerge from these new ways of "seeing" data.


We also need to pay more attention to integrating such skills as project management, teamwork, and oral/written communication into all of our courses. We do pretty well, most of us, with oral/written communication in smaller classes, less well in large classes. We need to devise creative ways to get our undergraduate students involved in our research. This is a major challenge in liberal arts, and it has something to do with the kinds of research we do, the kinds of questions we ask. I'm not suggesting that we entirely re-formulate our research to make it possible to integrate undergraduates into a "research team;" rather, I'm suggesting that we get creative. For example, I've found that one of the best ways to have undergraduates doing significant research work in my field is to have them work with me on course development.


Finally, in the spirit of thinking of degrees more along the lines of a project, we ought to make better use of e-portfolios. When a student graduates with their BA, they should have a portfolio of work that documents their skills and highlights their best work. This work should not just be research papers--after all, it is increasingly true that most of our students will not continue their studies in graduate school. We are not simply preparing students to do research at the graduate level; we are preparing the vast majority of them to enter the workforce in a range of different jobs. It isn't fair to our students to pretend that this is not the case. E-portfolios might also encourage departments to do more to rationalize their curriculum, to connect their different courses and make an effort to have students learn and practice different kinds of skills in a range of major courses. It should be that, at graduation, a student can articulate in clear terms exactly what skills they learned during their college career.


At public institutions, colleges of arts and sciences are in crisis. This crisis is, to be sure, in part the result of deliberate decisions made by legislatures, regents, and even university administrators. But it is also in part the result of a failure to evolve and address pressing issues like declining enrollments. Thus far, the primary response has been to institute cuts--to the number of tenure track lines, lecturers and adjuncts; and to cut the size of graduate programs. As a result, some departments are now unable to staff enough courses to meet the existing student demand--which further drives students to seek alternative sources for these courses. Departments have to choose between staffing lower division courses that tend to be large enrollment; or upper division courses that serve their majors and keep their major going strong. It's a kind of Sophie's choice.


At this point, most colleges of arts and sciences are as lean and efficient as they can get. They cannot withstand further cuts as a response to additional budget cuts. It's already a nearly impossible to keep our department and programs running. There has to be some other, positive and proactive approach to the problem of decline enrollments. It's a tough time to be a faculty member in liberal arts, but it's also--potentially--an exciting time. We have the opportunity to re-imagine and modernize the liberal arts, not just because it will help our graduates find gainful employment but because it will improve our courses and degree programs. Many faculty will resist any kind of change, but we have reached the point that we either change or wither into irrelevance. We need to rethink such basic things as how we offer courses, especially required courses. But we also need to think about how to teach skills in addition to critical thinking.  I'll conclude with a chart that illustrates why we need to teach our students, undergraduates and graduates, to operate competently in technology-rich environments.  This is a challenge for most of us since it's a set of skills many of us don't have and didn't need.  But we need to figure out how to instill them in our students if we hope to attract undergraduates to the liberal arts.

Update 6/12/2015: An interesting essay by Steve Mintz, Director of the UT System's Institute for Transformational Learning, on the need for a more integrative approach to undergraduate education.  From his perspective, the humanities disciplines need to think about how to build bridges to STEM disciplines; and our curricula need to include humanities courses, but in an integrated and rational way (e.g. medical ethics, narrative medicine).  It's still not clear exactly what this would look like, but I do think this is one step that liberal arts colleges need to make if they are going to thrive.

Source: OECD Education

Why Universities aren't Dead....and Won't Be

 

Libraries, or at least the open stacks libraries of my youth and middle age, may well disappear over the next decade.  The space of the library will be reconfigured for different activities, even if these activities are ultimately serving the same purposes as the libraries of my undergraduate and graduate years.  I'm very interested to see what Jim O'Donnell, a well-known classicist and transformative thinker, does as the new head of libraries at Arizona State University.  I was lucky enough to write my dissertation with Jim and to have him as a mentor for all these years.  He has some fascinating, truly innovative ideas.  I'm excited to see what sticks.  What Jim and other university librarians have concluded--from traffic and usage patterns over the past several years--is that the open stacks library on the university campus has become an inefficient use of space when space is at a premium.  Many faculty, even in the humanities, rarely browse stacks anymore, and most of our students venture into them even less often and only under compulsion.  In some cases, faculty barely set foot in the library--most of what we need for our teaching and research is available online or, at least in my case, so essential that I purchase my own copy of the book.  The same has become increasingly true for our students.  Something essential is lost when stacks are moved into storage, to be sure, but it seems to be a growing inevitability as libraries re-think their function in the university.

Libraries as repositories of books may well go the way of the typewriter over the next decades, as their space is reconfigured to support faculty-student interaction and collaboration.  The space itself will continue to nurture curiosity and support learning, but will do so in a way that reflects the 21st century realities of college/university usage patterns and learning practices.  For instance, it's not difficult to imagine a transition to storing most books off campus (or otherwise out of sight) and allowing users to request them electronically with a 24 hour turnaround.  In place of bookshelves, we will find things like 3-D printers, Digital Studies Institutes, and expansive spaces for collaborative work between students and between faculty and students.

In the same way that the campus library is undergoing transformation, so is the university itself.  It is increasingly apparent that we are living in a period of expansive change to higher education, some of it necessitated by state defunding of public education; and some of it the result of deliberate choices by high-level administrators to shrink the size of the tenure-track faculty,  The focus the shifts to small seminars for lower-division students; upper division courses for majors; and graduate education.  It is more and more clear that public universities, even flagships, can no longer afford to educate students for a full four years.  The first two years of college--"grades" 13-14--are being grouped with the last two years of high school (grades 11-12) to form a unit that, currently, falls into a gap.

The private sector, in particular, is racing to fill this gap as students respond by cobbling together credits from AP exams, credit by exam, online courses from a wide range of sources (including 2 and 4 year schools), and f2f classes from community colleges as well as 4 year universities and colleges.  When I was a college student in the 90s, we had AP exams and, rarely, we might transfer in credit from community college; but I took more than 128 credit hours during my 3.5 years (plus summers) in college.  When I first began to teach at UT Austin, the general pattern of my own college experience continued to hold.  Students might take some required courses over the summer at a school near their home, but many also remained on campus to take classes during summer sessions.

I have not seen enough internal data to know exactly when things changed; but the change became very noticeable about 4 years ago, not long after the Great Crash of 2007.  Summer enrollments on campus plummeted.  Now we struggle to get even 15 students for classical civilization courses that used to enroll 50+.  Our first year Latin classes have not run for at least three years.  It's not that students aren't taking courses during the summer--it's that they are doing it while living at home and working.  They need flexibility.  Courses scheduled in the middle of the day; or are scheduled to take up most of the day (with morning and afternoon sessions) aren't going to cut it.  Many current college students can't afford to remain in Austin, so they take courses at schools near their homes or online.  Increasingly, more of them are doing this during the semester as well, so that they are taking their major courses on campus but anything that they can take elsewhere, they are (either because we can't offer enough sections of courses to meet demand or because they need the flexibility of an evening or online course).

Many universities have been very slow to recognize and respond to this troubling trend.  As a result, we continue to, in effect, outsource the instruction of many of our lower-division courses.  This has a wide range of consequences for students as well as for the university.  For students, it means that they are not necessarily being prepared for subsequent courses in that field (though this is often a non-issue since many courses are "one-offs" and are taken to fulfill some core curriculum requirement).  For the university, it means that we are losing a significant number of credit hours.  This is especially troublesome for colleges of arts and sciences/liberal arts.  These colleges have long depended on the high enrollments of their "service" courses to subsidize the smaller courses for majors. As our semester credit hours shrink, we lose funding for new tenure-track positions as well as lecturers.  This then means that we cannot offer enough sections of courses to meet student demand, thus shrinking credit hours and driving students to other "providers."  It is a vicious circle.

The death of the (public) university has nevertheless been greatly exaggerated.  It's not dying, even if it sometimes feels that way.  But it *is* transforming.  This transformation was enabled by the massive disinvestment of states over the past 7-8 years.

As this map illustrates, the disinvestment was almost universal and on a huge scale.  Even as students have taken on higher debt loads because of rising tuition, it has been impossible for universities to make up the losses in state funding.  There is general agreement that things are never going to return to "normal," and that the days of viewing an education as a public good are in the past.  As a result, it is imperative for universities to find new ways of fulfilling their missions.  This presents a significant and, at times, nearly overwhelming challenge. 

One result of state disinvestment, I suspect, is that we will see public universities increasingly directing resources away from the traditional large lecture courses that tend to dominate the first two years of college for most students.  For one thing, the large lecture class is not a very effective way for students to learn.  For another, we are learning how to deliver those classes more efficiently but also in ways that produce better learning outcomes.  If ASU has gone to one extreme with the creation of the Global Freshman Academy, in which students take MOOCs from EdX for credit, most public universities will likely resort to some variation of this.  Some freshmen may do an entire year from home, via online courses and MOOCs (though one hopes that this spurs improvements to the design and student support in MOOCs).  This isn't a disaster--in many cases, a well-designed and well taught online class can easily outpace what we can do in a campus classroom (this is certainly the case with my Online Rome class).

Most residential freshmen, however, will likely be taking a mix of hybrid courses (which have a significant online component but also involve f2f interactions with the teaching team on campus); and small seminars that emphasize experiential learning.  Online courses are not going to kill the university, as many before me have observed.  Rather, they will highlight the value proposition of the residential college experience.  Specifically, they will require faculty to develop learning experiences that can ONLY be done in a f2f classroom, that leverage the f2f.  Experiential learning will play a big role in the university of the future.  This article on the future of college makes a similar point.

Liberal arts won't be eliminated (though we may need to get a lot better at being able to talk about the value of our courses beyond "teaching critical thinking.")  Regional public universities may move more towards emphasizing pre-professional subjects and "trades" instead of liberal arts, to be sure; but liberal arts should continue to have some place in this 21st century university.  Still, it will require a lot of hard work to reconceptualize what that place looks like and how it works with other disciplines. We will need to be creative and collaborative.  We will need to accept that we have to teach in ways that we ourselves were never taught.  We will have to work on shifting from a model of content delivery to a serious emphasis on mentoring students on how to learn.  Our students live in a world in which content is available in seconds, at all times.  It is no longer enough to fill their heads with content and send them on their way.  We have to do the hard work of teaching them how to work with that content, how to develop a set of learning behaviors that are transferable to everything they do.

Robots are great at making widgets.  Students are not widgets, and learning is not linear.  By necessity, all learning is to some degree personalized and adaptive.   Sophisticated student learning requires the support of skilled experts aka professors.  It also requires well-designed courses and, ideally, substantial interaction with a teaching team.  The kinds of courses, like large lectures, that do not support this kind of learning will go away--not least because they can be done at least as effectively online (and, I suspect, with time and the insights of the learning sciences, they can be done far better than what is currently on offer).  But the kinds of courses that privilege face to face interaction, like hybrid courses or courses built around experiential learning, will thrive on our campuses.

I imagine that this transition and transformation will be rocky for everyone.  At the same time, it's difficult to see any other future (generally speaking) for public institutions.  Most important is that, as institutions, we are able to articulate our value proposition, articulate to students what it is that they will get from the time they spend on campus--besides parties and football.

For more on the current enrollment declines and budget deficits in colleges of liberal arts/arts and sciences, see IHE's "Arts and Sciences Deficit"  (h/t Phil Hill)

The Challenges of Ongoing Course Development

Reconstruction of the Arch of Titus in the Circus Maximus.  Source:  http://www.telegraph.co.uk/news/worldnews/europe/italy/11638975/Massive-triumphal-marble-arch-built-by-Romans-to-honour-Emperor-Titus-discovered.html

As is my early morning ritual, I was browsing my Twitter feed while trying to wake up the other day.  I paused on an interesting post: the remains of a second Arch of Titus had been excavated in the eastern end of the Circus Maximus.  This is pretty big news.  We knew this arch existed and even had a copy of its dedicatory inscription, but to actually find large pieces of it is kind of exciting.  In my field of classics, we don't often find the remains of famous monuments.  Intrigued, I read the news articles and, over the next days, several blog posts about the find and its significance.

Remains of the Arch of Titus in the Circus Maximus.  Source: http://www.telegraph.co.uk/news/worldnews/europe/italy/11638975/Massive-triumphal-marble-arch-built-by-Romans-to-honour-Emperor-Titus-discovered.html

Remains of the Circus Maximus

Finding these blocks of marble doesn't actually change anything for ancient historians.  The arch's existence was documented in medieval sources, which also preserved its dedication to Titus.  We know that it remained standing until the 12th century, when the Circus Maximus was converted to agricultural use and inhabited.  The arch was used to provide support for an aqueduct that provided water to the fields in the Circus Maximus.  It was assumed that, around this time, the arch was plundered for building materials.  Once the arch is reconstructed, we will have a better sense of its fate.  We will be able to see how much of it remained intact until it was buried; and, possibly, the archaeological context will provide additional information.

One of the great things about an online class is the ease with which they can be updated and modified.  If I had produced a printed textbook, there would be nothing to do except keep a list of updates for a second edition.  With the Online Rome class, I can make revisions in minutes.  The discovery of this second Arch of Titus is exactly the sort of thing that I will add to my version of the course.  Confession: although I am a very good classicist and ancient historian, I don't know everything.  I didn't actually know that Titus had two arches constructed in his honor by his brother and successor, Domitian.  More to the point, the current version of Online Rome perpetuates the traditional "textbook" narrative: the Arch of Titus at the start of the Roman Forum, near the Flavian Amphitheater, is a triumphal arch that commemorates Titus's victory over the Jews and Judea.

In reading the articles and blog posts about the newly discovered Arch of Titus in the Circus Maximus, I realized that this traditional narrative is wrong in some important ways.  The best source for understanding how the two Arches of Titus related is the detailed discussion on The History Blog.  A couple of important details: first, it is this Arch of Titus at the entrance of the ancient Circus Maximus (an arena where chariot races were held) that was, in fact, the triumphal arch for Titus's victory over the Jews.  It was dedicated to Titus by the Roman Senate soon after Titus's untimely death.  The dedication makes clear that the arch was erected in honor of Titus's military victory over the Jews and his sack of Jerusalem.  This is the arch's dedicatory inscription:

Senatus populusque Romanus imp(eratori) Tito Ceasari divi Vespasiani f(ilio) Vespasiani Augusto pontif(ici) max(imo), trib(unicia) pot(estate) x, imp(eratori) XVII, [c]o(n)s(uli) VIII, p(atri) p(atriae), principi suo, quod praeceptis patri(is) consiliisq(ue) et auspiciis gentem Iudaeorum domuit et urbem Hierusolymam, omnibus ante se ducibus regibus gentibus aut frustra petitam aut omnino intem(p)tatam, delevit.

The Senate and People of Rome [dedicate this arch] to the Emperor Titus [snip many titles], because by his father’s counsel and good auspices, he conquered the people of Judaea and destroyed the city of Jerusalem, which all of the generals, kings, and peoples before him had either failed to do or even to attempt.

 The Arch of Titus at the start of the Via Appia in the Roman Forum, by contrast, is not a standard triumphal arch despite the allusions to Titus's victory over the Jews in some of the relief panels.  As The History Blog post observes, "The central panel in the single arch’s soffit relief depicts Titus being carried to the heavens by an eagle. The inscription also emphasizes the recently deceased emperor’s divinity: “SENATUS/ POPULUSQUE ROMANUS/ DIVO TITO DIVI VESPASIANI F(ilio)/ VESPASIANO AUGUSTO” (The Senate and People of Rome [dedicate this arch] to the divine Titus Vespasian Augustus, son of the divine Vespasian)." 

Source: http://en.wikipedia.org/wiki/Triumphal_arch

This second Arch of Titus was erected in 82 by Domitian and, like the first arch, was dedicated to the memory of the deified Titus by the Roman Senate.  Note the much less elaborate inscription in the second arch, the omission of Titus's many titles and offices.  This second arch is clearly not a standard victory arch but rather, a kind of celebratory arch that, as much as anything, celebrates the divinity of Vespasian and Titus; and helps to legitimate Domitian's status as their successor.  But how, exactly, are we to understand the meaning and occasion of this second, surviving Arch of Titus?  How would a Roman living during Domitian's rule have understood these two arches?  How would their relationship have been conceptualized?

It is interesting to observe that Titus's official triumphal arch was erected at the entrance of the Circus Maximus, a public space for entertainment.  This choice seems especially meaningful when we remember that it was the Flavians who constructed the massive Flavian Amphitheater (Colosseum), not too distant from the Circus Maximus.  In fact, Domitian would make significant renovations to the massive Amphitheater and held elaborate games to re-inaugurate it.  So why erect a second arch to his deceased brother on the edge of the Roman Forum?  Suddenly, the Arch of Titus in the Roman Forum, a monument that seemed to have a straightforward story to tell, becomes complicated.  Suddenly, the traditional treatment of the extant Arch of Titus as a victory arch is not so easy to defend. 

In this instance, a historian more versed than I am about Flavian archaeology might have discussed the two arches in the first place.  Nobody knows everything, however, and the brilliance of the digital medium is the ease of revision.  It forgives ignorance and allows easy correction.  At the same time, important questions are raised by an online course that never come up when an instructor is teaching from a printed textbook: who is responsible for keeping an online course updated, especially if the original designer is no longer involved in the project?  Is it the responsibility of the course coordinator?  Should someone be brought in to provide updates and revisions to the content every year or two?

In my field of Classics, the number of necessary updates is likely to be fairly small compared to something like Contemporary US Government.  Still, the original course cannot be left unrevised for years.  There are new discoveries as well as new theories of old evidence every year.  One of the reasons it is so important to have a content expert continuously involved with the running of the course is precisely that someone needs to stay aware of these new findings in the field (and be able to understand the significance of the new findings). 

People frequently compare online courses to textbooks.  In some ways, they do function like enhanced textbooks.  They nevertheless differ from textbooks in some important ways, most especially in the fact that they are organic and can be modified.  In addition, at least in my Online Rome course, there is no "textbook author" voice that is distinct from the voice of the instructor.  From the perspective of the student, there is only the voice of the instructor; and the instructor can modify the modules as they wish, to reflect their own expertise and interests (within reason).  An online course is a kind of living creature.  It is never "finished"; it can be and should be constantly updated, added to, subtracted from.

I don't know if this will happen, but it is my hope that, even after the course has been handed off to the Classics Department to run, that the course coordinator as well as the instructors will continue to work on the content, adding something here and taking away something there.  Perhaps they might have a new idea for a game.  Although the course I handed over is complete, polished, and ready to go, I hope that future instructors will recognize that one of the benefits of the digital medium is the ability to make modifications and corrections. 

I imagine coming back to the class in a decade to find that Online Rome has continued to evolve over that time to adapt to the needs of students; incorporate new content (new discoveries or theories of existing evidence but also, e.g., videos, animations, virtual reality experiences, and games); and incorporate new learning tools as they become available.  It is important that everyone involved with the course recognize the opportunity they have to put their own mark on the course; but also, the responsibility to ensure that it is regularly and systematically updated.  With every passing month, it seems, there is more and more content available, some of it very good.

For instance, I just noticed that the MAV, a museum which creates virtual reality experiences from archaeological remains, has just posted a reconstruction of the baths at Herculaneum.  The MAV page on Facebook is a great resource for these virtual reconstructions (they also have reconstructed the Villa of the Papyri), especially because it is nearly impossible for students to imagine what ancient bath complexes looked like from the scant remains that we can show them.  These reconstructions allow them to imagine the scale and magnificence of the original public buildings.  Keeping track of new digital content and deciding what to integrate into the course will be an essential role of the course coordinator (or someone).  Without revisions, additions, and modifications, Online Rome will indeed function more like a textbook--a stable object that slowly becomes outdated in a range of ways.