Exam Wrappers

One of the most significant additions to my Spring 2013 flipped class was exam wrappers.  These were relatively short surveys, posted on Survey Monkey, with questions designed to induce the students to review their exams; reflect on their preparations and study strategies; and think about how they might address any deficiencies in their performance.  They were also intended to encourage students to recognize the role that some of the graded and ungraded formative assessments played in preparing them for the exam.

We returned the graded exams one week after they were taken and posted the exam wrapper.  Students had a week to do the exam wrapper and received 2 extra credit points on the subsequent exam.  As well, if they did ALL of the surveys during the semester (there were a lot since we were trying to learn as much as possible about their learning), they received two points added to their final grade.

The exam wrapper for Midterm #2 can be found here (just enter random numbers for the ID info and click next to see the questions; here are some other examples on the Carnegie Mellon Eberly Center site).  These were created by an assessment specialist at UT Austin's Center for Teaching and Learning, Dr. Stephanie Corliss.  I reviewed them and made small suggestions.  The point of the exercise was a. to encourage students to  pick up their exam and look at it; b. to have to spend even five minutes thinking about how they studied and, hopefully, recognize for themselves what strategies worked and didn't work; and c. to remind them of all the different learning tools available to them to prepare for the exam (practice quiz questions, recorded lectures, etc.)

What most surprised me about the exam wrappers was the extent to which they did my job for me, but far more effectively.  Normally, I would offer a short review of the class performance on the exam and then talk about common problems.  As well, with each exam, I would point out to them ahead of time ways that the content might post particular challenges or require particular approaches to mastery.  Between the weekly quizzes and the exam wrappers, however, I found that the students were largely able to self-regulate and self-correct.  Semester after semester, I have told classes that cramming for my exams is not a particularly effective strategy, particularly for exams 2 and 3.  This went in one ear and out the other.  But when these students saw these difficulties crop up in small doses on their weekly quizzes, they adjusted on their own.  Likewise, when I read through their comments on the exam wrappers, it was absolutely clear that they had identified the sources of trouble for themselves.  Sometimes it was too little preparation; more often, though, they had important insights into how their preparation strategies were wrongheaded; and a clear and effective plan for adjusting them.

I have also used exam wrappers in graduate seminars to extremely good effect, particularly in a class that focuses on improving students' abilities to read large amounts of Latin quickly.  They instill in students a habit of self-reflection and self-regulation.  They encourage students to see midterm exams (or quizzes) as formative assessments, as offering feedback on their learning strategies.  Most importantly, they remind students of the resources available to them to learn the course material; and emphasize that there is still time to make adjustments and improve their performance.  I expected my graduate students to become more reflective learners.  I was, quite frankly, very surprised to see how effective the exam wrappers--together with the weekly quizzes--were at helping my undergraduates to reflect on their learning strategies and make appropriate adjustments.  These adjustments were clear in their grades, as they improved even as the difficulty level of the course increased.

A Response to Anant Agarwal's Call for Teachers to Join the Revolution

This past weekend Anant Agarwal, the president of edX, published an op-ed in The Observer titled "Online Universities: it's time for teachers to join the revolution."   There is much in his rather self-serving op-ed to criticize (and parody, as Jonathan Rhees has done so brilliantly), but perhaps the most important thing to point out is the false premise of his title.  Current teachers are cast as Luddite resisters who have dug in their heels and are opposing something that is improving education.  This is a complete misrepresentation of practicing educators at all levels, many of whom regularly use a range of education technologies as well as tried and true pedagogical methods to produce learning in their students.  In fact, education technology has been a presence in K-12 as well as post-secondary classrooms for at least a decade, and has expanded tremendously, especially on college campuses, thanks to the availability of fast broadband connections.

To suggest that the pre-MOOC university professor was teaching with chalk and yellowed notes is an absurd caricature whose sole purpose is to position MOOCs as the shiny new thing that will revolutionize education.  Agarwal declares that the "days of the old ways of teaching are numbered".  While there are certainly examples of "the old way of teaching" on every college campus, I suspect that they are vastly outnumbered by the many innovative, technologically savvy, and dedicated faculty who are working every day to improve student learning.

MOOC supporters repeatedly point out that the platforms and price point (free) improves accessibility.  I will leave it to others to explicate why this isn't quite true (and here).  Certainly, by putting courses on an open access platform, their is the potential to improve access.  But it is far from clear that MOOCs really are doing much beyond making lifelong learning more convenient; and targeting gifted learners in third world countries.

The most significant source of my irritation with the self-promoting MOOC rhetoric is centered on the claims that the mode improves learning--and, specifically, that it can support better learning than can a campus-based, large enrollment course.  In basic terms, there's no evidence to support this claim, at least not yet.  Furthermore, given the demographics of current MOOC users, it seems to me unlikely that even extensive study of  data from individual courses or users is going to be generalizable beyond a particular course or a small subset of users.   This is a big claim to make, and it requires clear data, not anecdote and declarations.

Working teachers need to know how particular students at particular universities in particular courses learn.  Much more useful would be funding projects to study currently enrolled students, especially a public universities.  As well, we need to know how best to help our beginning students learn HOW to learn, especially how to learn in a post-secondary setting where instructors aren't teaching to tests.  Since the majority of  current MOOCs users are degree holders and experienced learners--that is, they are people who already know how to navigate a course, study, be self-motivated, etc--the data being collected is unlikely to yield much of value.  This will change as more of the students are actual college students with all the complexities that college students present, when the data sets can be narrowed down to "current UT students taking X course" but that is years in the future.  As Rob Reich explains, "If MOOCs promise to enhance student learning, they must show that they deliver at least as powerful outcomes as traditional lecture classes in universities and community colleges.  If not, their virtue is their democratizing potential; but they will only be better than nothing."

Agarwal writes, "MOOCs are also improving the quality of education.  Online learning promotes active learning, where the learning watches videos and engages in interactive exercises."   I agree that MOOCs are improving the quality of education--but they aren't doing it alone and they aren't doing anything revolutionary.  They are simply hacking the flipped class model--and replacing the in class part of the formula with a (generally chaotic and ineffective) discussion board.  Furthermore, the notion that interspersing a few questions into a video creates an active learner is absurd.  It's better than simply lecturing at students; but it takes a lot more than a few questions in a video to create active learners.  Likewise, it takes a lot more than a few i>clicker questions in a lecture to truly engage students.  I've observed many a lecture class where students are shopping for clothes but still able to stop and click in on questions.

Indeed, much of what Agarwal has to say about how MOOCs facilitate learning imagines an ideal learner who uses the course precisely as it was designed to be used.  Yet teachers  know that this isn't how it works in real life.  It's absolutely true that videos allow for self-paced learning, but that doesn't mean students will use it that way.  In fact, in my own class, I found out that very few of them did.  Sure, in an ideal world, they would watch the videos carefully, engage with the embedded questions, and review confusing points.  The problem is, students rarely behave like ideal learners and so do not get the benefits that they could from the technology. We educators may understand that a key to learning is practice with retrieval; but students don't know that and will often find ways to avoid doing it unless their grade depends on it.  But again, MOOCs aren't the first to recognize the value of retrieval.  Faculty have been using LMSs like Blackboard to do practice quizzes and the like long before the advent of MOOCs.  As Glance et al. note in "The Pedagogical Foundations of Massive Open Online Courses," "MOOCs are in essence a restatement of online learning environments that have been in use for some time." 

I am similarly puzzled by the claim "students have always been critical of large lecture halls where they are talked at and declining lecture attendance is the result."  This is not at all my experience.  Quite the opposite.  Students enjoy a good lecture, one where they can scribble down the notes on the PPT, memorize them, and do well on exams.  It is unclear to me what the benchmark for declining student attendance is--this observation is meaningless without more context and data.  Student attendance has little to do with the quality of a course, in my experience, and much more to do with time of day, time of semester, whether they are commuting, etc.  In truth, one of the major challenges in moving from a lecture model to a model that emphasizes active learning is having to train students to be engaged learners.  Still, many complain that they would rather just be lectured to.  Yes, the ideal student would recognize the value of being an engaged learner; but most of us are not teaching these ideal students.  In fact, most of us have to spend a considerable amount of time and energy teaching our students who to be students, teaching them (and then repeatedly reinforcing) good learning strategies.

After declaring that the revolution has arrived, Agarwal turns to the flipped class model.  Indeed, it's become increasingly clear that the aim of the MOOCs vis-a-vis brick and mortar campuses, ultimately, is to provide content for flipped classes.  I've written elsewhere about why this is misguided and misleading.  In a nutshell, because the difficulty of a flipped class isn't moving content outside of class, it's doing everything inside of class.  Yet, by fetishizing content delivery and the professors who do it, the MOOC Incs have encouraged the belief that universities can save money by hiring non t-t faculty to run these flipped classes.  This is a recipe for disaster and demonstrates how little the MOOC Incs understand (or care about) the pedagogy of the  flipped class.

The op-ed concludes with the declaration that "it's time for teachers to rethink learning methods."  It is unfortunate that Agarwal doesn't realize (or won't admit)  that this was happening long before MOOCs came on the scene.  MOOCs are doing nothing new, apart from experimenting with large scale delivery.  Certainly, they have encouraged an engaged discussion about technology and pedagogy; but thus far, they are using methods that have long been used on brick and mortar campuses--indeed, if anything, MOOC pedagogy feels a bit retro to most of us (breaking up lecture with questions; discussion board; quizzes).  I can imagine a time when MOOCs might advance our understanding of student learning and make real contributions to pedagogical theory, but they aren't yet there.  The MOOC Incs would do well to recognize this point and refrain from perpetuating ridiculous and unfair caricatures of teachers. 

Graphic: A Closer Look at MOOCs

This morning, the Los Angeles Times published the first part of what they are labeling a Roundtable Discussion about MOOCs. Of the participants, two are faculty who have or are preparing to teach a MOOC on Coursera (Al Filreis at Penn and Cathy Davidson at Duke); one is a professor who teaches online courses at University of Illinois, Springfield (Ray Schroeder). The fourth is a scholar and cultural critic who has regularly worked to draw attention to the labor issues that are raised by the push to accredit MOOCs (Ian Bogost at Georgia Tech). In this first installment, the four faculty members issued position papers. There was essentially no engagement by the participants with the particular points raised by the other papers (outreach and increased accessibility were the themes of the first three; economic/labor implications was the theme of the fourth). Hopefully, the follow-up tomorrow will include some efforts by the three supporters of various forms of online learning to engage with the real concerns raised by Bogost. And, in turn, one hopes that there might be a way to talk about ways that the outreach and accessibility issues--both real benefits of the MOOC model--can be addressed without contributing to the ongoing efforts to outsource teaching at public colleges and universities.

MOOCs are multifaceted, with clear benefits. They are also a kind of Pandora's Box that, rightly, many are afraid to open. We seem to have a good sense of what the pros and cons are, as well as the politics around each position (including the refusal to acknowledge that the decision to teach on a MOOC platform is, in a way, already taking a position in the debate). Now we need to figure out how to move the conversation beyond this outlining of issues, to find some strategies for managing the inevitable disruption that large-scale, platform-based teaching is going to cause at once state-supported institutions.

Thank you to Allison Morris for providing me with this wonderful graphic that outlines the pros and cons of MOOCs so beautifully (graphic produced by OnlineCollegeCourses.com).

My Summer MOOC: Archaeology's Dirty Little Secrets

I was delighted to learn that Dr. Sue Alcock would be offering a short MOOC this summer on Coursera's platform.  Dr. Alcock, a renowned classical archaeologist and gifted teacher, directs the Joukowsky Institute for Archaeology and the Ancient World at Brown.  The course, "Archaeology's Dirty Little Secrets" is well-suited to the MOOC mode.  Archaeologists have long been experts at public outreach, in part because their work is expensive to fund and very frequently depends on private donations.  As well, because they work on material culture, on stuff, their work is of far more general interest than, say, analyses of Vergilian metrics.  It is also a good course to run in the summer, when the teaching team is at work on their sites scattered around the world.  Besides introducing takers to some of the basic concepts of archaeological study, it is sure to inspire wanderlust!

The course is well-designed, not least because it doesn't even try to be a college-level course.  It does very well what it is trying to do: provide outreach to the general community; inform the public about some of the major issues at play in cultural preservation; raise awareness about (and, perhaps, funds for) Pompeii--an amazing site that the Italian government is letting fall into ruin; and give users a sense of what academic archaeology is all about and perhaps encourage them to take a real class at some point.  This course reminds me very much of the free lecture series that my department does from time to time for (mostly) retired adults.  It is meant to generate interest and provide an intellectually satisfying experience.  It is not meant to be a credit-bearing course even if it is full of interesting content.

As an aside, I wonder if part of the frustration and confusion about outreach efforts like "Archaeology's Dirty Little Secrets" is the fact that they are presented like actual, credit-bearing courses: there is a syllabus; assignments are due on particular days; there are quizzes; there is a professor lecturing on content.  Users are called students.  Certainly, some MOOCs really are designed to replace a credit-bearing course, particularly in math and CS.  In the humanities, however, they are generally not intended to do so.  Dr. Alcock's course certainly isn't--it's a shell of a course she taught for credit on the Brown campus and lacks much of the serious content and assessment that her for-credit course surely had.  That's not to say that the MOOC version is bad--in fact, it's a fantastic introduction to archaeology and the ethics of cultural preservation; and a great way to raise public awareness about the need to preserve such sites at a time when so many are being destroyed in North Africa and the Middle East.

So, some great things about this MOOC: first, it is not especially lecture based.  Each unit does have about 30 minutes of lecture from Dr. Alcock, but it is not the centerpiece of the unit.  The unit also involves demonstrations from Dr. Alcock, informal in tone and held in her very messy office.  There are conversations with various young faculty/postdocs about their own projects; and, finally, short slideshows about selected topics done by the teaching team.  This last feature, "Peoples, Places, Things" is my favorite and is something I am going to steal and integrate into my own hybrid (and, eventually, online) class.  The integration of different voices, different faces, is a big plus for the course (and, interestingly, is a tactic used by one of the few women chosen to use the Coursera platform).  I also liked that there are options for each assignment, ranging from writing an essay to interviewing friends and family to a more hands-on exercise.  Many people won't take the time to do these assignments, but they are well-conceived and a fantastic way to get users to engage (and another thing I am going to use to inspire my ongoing development of such activities in my own class).

There is amazing content in the course and it models some effective ways to teach material culture to undergraduates.  It lays out the major questions that any Intro to Archaeology course ought to include.  At the same time, the mode--a MOOC--means that it is a struggle to support the kind of engaged student to student interaction as well as instructor to student interaction that really pushes individual students to engage with the big questions on their own terms, with push back from content experts.  There is a discussion board and it has some interesting threads on it.  Overall, though, the board is used primarily for procedural questions.  It's a challenge to get students to discuss in a face to face classroom.  That challenge is magnified many times over on a discussion board--especially a discussion board for thousands of students from a wide range of backgrounds.  Dr. Alcock opts for an informal tone that works for her.  She comes off as smart, goofy, and accessible.  Yet, ultimately, she's not accessible.  I can't knock on her office door and have a conversations with her because she's in Petra, working on her research--as she informed me in a recent email.

All to say, this course is a wonderfully conceived way of using the Coursera platform to accomplish a range of important and useful objectives--none of which were to offer a course that replaced a credit-bearing course.  The motivated student will have a great time learning about archaeology and engaging in the various activities conceived by Dr. Alcock and her teaching team.  They might even have some productive discussions with others in the discussion forum.  At the same time, it is important to reiterate the point that this is not a college-level course on ancient archaeology.  Rather, it's a kind of teaser that, ideally, will encourage students to take such a course at campuses around them.  It might also encourage the general public to understand why archaeology matters and be more inclined to support the funding of preservation and research projects.

When a Butteryfly in Cambridge Flaps its Wings

A good friend and former grad school colleague of mine is an Associate Professor of Classics at a small liberal arts college in the Pacific Northwest.  Her college is tuition-dependent, has a very small but spectacular classics department, and depends on faculty to teach not only a 3-3 load but also to take on several additional directed readings courses as well as thesis students.  In addition, she is very involved in the college community through committee work as well as supervising several campus student groups.  During the academic year, she is incredibly, crazily busy.  My job at an R1, in a department with a strong graduate program, takes a very different form from my friend's job, most obviously a lighter teaching load (2-2).  Still, I would assert that I am just as busy as she is, albeit with different kinds of tasks.  Over the years, our phone calls during the semester always seem to revert to a kind of friendly competition over whose job is more exhausting.  At some point, I realized it was impossible to determine because our jobs are simply too different to compare--despite the fact that we are both Associate Professors of Classics.  To really understand one another's position, we would need to trade jobs for a year (at least).

In conversations with this friend and others in my field who teach at a range of different colleges and universities, I repeatedly observe just how different our jobs are--apples, oranges, peaches, and bananas.  This is an important point at this particular historical moment, as we all try to make sense of the MOOC phenomenon and do so largely in terms of our own limited departmental and institutional experience.  The tendency to assume that all jobs are basically the same, apart from the "research" vs "liberal arts" distinction, clouds the issue in some significant ways; and, I think, it's part of what is preventing a more constructive conversation about MOOCs among the academics most invested in the roles they now and will eventually play in higher education.

The conversation around MOOCs has become noticeably polarized in recent months, fueled by academics (including me) who are skeptical of the mode and concerned that it is being viewed as a magic bullet by state legislatures and regents in particular.  Faculty who are providing their courses on the Coursera or EdX platform have been criticized for their complicity in what seems to be a persistent move towards the outsourcing of teaching at public institutions.  They understandably feel defensive and unfairly criticized. The situation was not helped when the Chronicle published an article suggesting that MOOC faculty as a group had given little thought to the potential impact of a more widespread distribution of their courses to other institutions, where they would be offered for credit.  Many faculty, who decided to offer their course as a MOOC--for very little by way of financial compensation and motivated primarily by a desire to increase access to their course--are justifiably feeling victimized by the increasingly hostile, anti-MOOC blowback.

Cathy Davidson recently posted "Clearing up Some Myths about MOOCs", in which she reviews a number of criticisms of MOOCs and offers her response ("so here's an account of what I've learned about the MOOC myths from what I've done and learned first hand.  I don't know what of these things is true beyond my experience but the point of first-hand experience is to counter myths with at least some substance and reality, however limited").  She has several great points to make and many of these points could be expanded on by those who have been students of MOOCs or who have other kinds of experiences with larger scale online instruction.  Yet, at the end, the post only invites those with first hand MOOC teaching experience to contribute their views. Still, it is a far more reasonable and thoughtful engagement with serious criticisms of the MOOC model than the dismissive views of Northwestern Journalism professor Justin Martin.

It is worth pointing out that several of the particular anxieties discussed by Davidson (e.g. concern that MOOC faculty are going to get super rich) are not the source of most of the anti-MOOC anxiety/rage/well-founded concern.  Some of the myths are related to the major criticisms of skeptics--high dropout rate, unproven pedagogy that seems to rely largely on lecture in the majority of MOOCs--but they only skirt around the elephant in the room.  What is that elephant?  Accreditation. If MOOCs had remained in the realm of outreach and non-credit continuing education; if they had simply offered access to skills-based courses to those who would otherwise have no access, most (all?) with an investment in the future of higher education would be supporters.  The problem is, they are positioning themselves as "disruptive" to the traditional system of accreditation (and Congress is listening).

The problem is, the MOOC Incs need to return a profit to their investors, and they are not going to do this without a revenue stream.  That revenue stream requires accreditation that leads to some end (e.g. certificate, degree).  Whether it is by some sort of licensing agreement with less prestigious universities (San Jose St. and California more generally is ground zero for this model); by creating very inexpensive degree programs (e.g. the MS in CS at Georgia Tech, with corporate sponsorship from AT&T); or by forming partnerships with selected public institutions, the apparent intention of the MOOC Incs is to function as a middleman between the producer and consumer of course content. I agree with the defenders that the current MOOCs--many of them only 6-7 weeks long--are unlikely to be packaged and marketed to less prestigious institutions (though some may be).  Far more likely to happen: Coursera, Udacity, and EdX will trade on the hype generated by their elite partners in order to persuade less elite partners to select a small number of "superstar" faculty to use their platform to develop courses in house and deliver them to thousands of students at a time, across entire state systems.

Typically, faculty don't need to worry about what is happening at other institutions around the country or world.  What happens at Cal State Northridge has little to do with what is happening at Harvard or Oxford. Yet, in the specific instance of MOOCs, institutions (and faculty) do not operate in isolation but rather, as part of a larger and even global ecosystem. Suddenly, what happens at Harvard may well have quite a lot to do with what happens at Cal St. Northridge. It is not enough to assume that academic standards will dictate administrative decisions; and that face to face instruction will always remain the most common form of instruction just because it produces the best learning outcomes. Still, the extent to which individual faculty--much less institutions--are responsible to the community at large will be debated.  But it should be debated rather than sidestepped.

In the face of severe budget shortfalls and an apparent unwillingness of states to invest in education, public institutions are desperately searching for ways to cut costs.  The MOOC model (not necessarily the existing MOOCs themselves, but the model of large scale courses) is, at least at first glance, very appealing: expertly delivered content that only requires a teaching assistant to walk the students through the details of the course.  While this may strike faculty at elite, private institutions as an extremely unlikely outcome, I suspect it will begin to happen in various forms over the next several years at public institutions like my own.  Exactly what role faculty at Harvard and Yale and Penn and Duke are going to play in all of this is unclear (and, most likely, up to them).  More clear is the fact that public institutions are determined to cut instructional costs.  They have to continue to teach large numbers of students with fewer faculty than ever.  Expertise is expensive and public institutions have long led the way in scaling up class sizes as a way of cutting instructional costs.  It is inevitable that large-scale, platform-supported courses will become one way of addressing this problem, at least to start.  The effect this will have on important factors like student learning, retention rates, and  time to degree are entirely unclear at this point. 

I agree with Cathy Davidson, Al Filreis, Noel Jackson, and others that support cautious experimentation with MOOCs and other kinds of larger scale online instructional modes.  I appreciate the potential benefits of MOOCs in terms of increasing accessibility, but my immediate concern is more personal: the role that MOOCs will play in the curricula of public institutions like my own.  We need to know better how these work (and don't work); and we need to ground the discussion in evidence rather than speculation (one hopes that the MOOC Research Hub, funded by the Gates Foundation, will be useful in this regard).  We need to be developing a good sense of best practices for different disciplines and types of content.  We need to understand the differences between learners who come to a MOOC with a college degree and well-practiced learning behaviors versus those without any of these skills.  We need to better understand what we gain and lose when we shift to a large-scale mode of content delivery/instruction with little opportunity for student engagement (because, if we could afford the instructional support that is required for meaningful student engagement, we wouldn't be resorting to this format in the first place).  For all of this to happen, we need to be having a thoughtful, engaged, sometimes combative conversation that involves everyone who has a stake in the outcome.

******************
6/14/2013: Roundtable discussion of MOOCs in LA Review of Books.  Perfectly illustrates this phenomenon of supporters and critics speaking at cross-purposes.  Everything each person says is valid and true, but they do not advance the discussion of how to deal with the MOOC incursion into public higher ed.

Not to pick on CD, but this tweet captures my point brilliantly: there's no debate at all (and it's not live, it's in the very dead form of print).  These are four "position papers" in which each party carves out their space in the debate.  There's no real debate at all.  Now, hopefully, that debate will come in the follow-up tomorrow and in the form of comments from readers.  But thus far, because there's so little attempt by anyone to engage with the arguments of others, it seems more likely that people will endorse positions they already hold rather than truly collaborate on a set of approaches, best practices, etc.  But perhaps I am being pessimistic...

6/13/2013: Colleen Flaherty, "It's My Business" ("MOOCs – or massive open online classes championed by many as a way to democratize education and lower costs, and criticized by others for what they see as the courses’ potential to diminish educational quality and supplant the role of the professor – and other online education initiatives have forced an alliance between professors across disciplines, Nelson said, in their growing concerns about intellectual property and academic freedom.")

Peter Schmidt, AAUP Sees MOOCs as Spawning New Threats to Professor's Intellectual Property

The Myth of the Super Professor

Yesterday I wrote about the myth of the bad professor, a myth that has been a central feature of the narrative of a crisis in higher education.   The myth has been a driving force behind the daily calls for the reform of higher education--preferably by Silicon Valley VCs who are entirely outside of the system (and who, on the whole, seem to understand few of the complexities of higher education in the US).  The topic evidently hit a nerve with many because, in about 24 hours, the post had almost 500 hits.  Among other things, I think that points to the fact that so many of us are fed up with the constant attacks on our abilities and efforts.  It is enough of a challenge to inspire often undermotivated students to do the hard work of learning; the last thing we need is to defend ourselves from the politically motivated attacks of those who have never walked in our shoes and who have no idea what is actually happening on the ground.

Yet, to an extent, we have left ourselves vulnerable to these attacks because, in all the years that college teaching has been professionalized, we've never really developed a coherent and consistent system for measuring student learning in our classes.  In many respects, we ourselves are guilty of perpetuating the myth of the bad professor because we have not adequately challenged the myth of the great professor.  If the bad professor is phoning it in, the great professor is the charismatic sage whose students hang on to his (it's usually a he) every word.  When faculty are evaluated for annual raises, tenure, and promotion, student evaluations are consulted; teaching awards are often a result of student nominations and student recommendations.  At no point in this process is any significant attention paid to student learning.  The best professor is the one whose students love him and want to impress him (and, to be fair, are hopefully inspired to work hard in the course because of this love and desire to impress).  But shouldn't the best professor be the one who produces the highest learning outcomes in a cohort of students?  Shouldn't we be evaluating the quality of instruction not on the performance but on the results of the performance?

Most faculty will agree with the research that repeatedly demonstrates that teaching evaluations are not a very effective measure of instructor quality or student learning.  There are a range of age and gender biases at play.  One recent study found, unsurprisingly, a strong correlation between the attractiveness of the professor, the grade the students thought they were getting, and the positivity of evaluation of the professor and course.  We all know these things are true, yet we continue to play along.  One of my favorite stories about gaming course surveys goes as follows: during the semester, slip into the class discussion commentary on each of the survey questions, telling the students exactly how you are doing whatever the question asks (e.g. emphasizing repeatedly your accessibility).  Do this over and over.  By the end of the semester, the students will have internalized the narrative and will rate you and your course highly.  In my own case, I know that if I want to have outstanding course evaluations, I just have to let everyone think they are getting an A up until the time they submit the evaluation--and then use a high stakes final exam to sort them out.  I've never actually done this, but know plenty of people who do.

So what do we need to do to shift the focus from the personality of the professor--and the cult of personality more generally--to evidence-based arguments about instructional quality and student learning?  First, we need to assess our courses and students much more deliberately.  We need to be able to demonstrate in some terms the value added by our courses.  Yes, I know this plays into the rhetoric of the "outcomes" crowd.  It will inevitably undervalue all sorts of difficult to evaluate skills like critical thinking; and it assumes that a course's value is immediately known at the end of the semester (though this second issue could be addressed with follow-up surveys).  Many of us feel more comfortable with the current system, even while acknowledging its imperfections--including me.  At the same time, by essentially allowing ourselves to be evaluated based largely on features of our personality, we are laying the foundation for our demise (because there's always someone else who is even more accessible, entertaining, etc.).  MOOCs perpetuate the notion that good teaching is equivalent to skilled public performance rather than demonstrable learning outcomes.  An important first step in responding to the pedagogical claims of MOOCs is to insist that courses and instructors demonstrate student learning outcomes.

As part of the redesign of my Intro to Rome class this past year, I instituted a thorough system of assessments of the course.  Students had multiple opportunities to comment on all parts of the course.  This was interesting and instructive for a range of reasons, not least of which was because we also had data that allowed for a comparison between their self-reports and direct assessment of various behaviors.  It quickly became clear that there was a significant gap.  One place where this gap was enlightening to all of us was in the assessment of the implementation of an "ethics flag."  Three large enrollment classes implemented the flag in Fall 2012.  One class was lecture-based, 220 students.  The second was a mix of lecture and discussion sections, with 300 students.  Mine was flipped, with discussion as primary feature, with 400 students.  When the students were assessed, they reported the most satisfaction and learning in the lecture-based class; then the mixed class; with my class in last place.  When the flag implementation committee did a direct assessment of their work, the results were exactly the reverse.  They may have "liked" my class the least, but they learned the material much better--not surprising since I was requiring them to engage with it actively rather than passively.  This spring, the discussion section course and my course were offered again and the results were the same, but with a slightly smaller gap and enough student comments to be pretty sure that perceived workload was inversely correlated with student satisfaction. Alas.

When we saw these results, they made sense--but were also an important reminder that student self-reports often reflect their sense of comfort.  Many of them are still far more comfortable learning passively via lecture than in more active forms.  Indeed, as my campus has worked to "blend" a number of gateway courses through the Course Transformation Program, a consistent feature of the blended courses has been lower student evaluations.  There is a clear and indisputable inverse correlation between level of student engagement required and student satisfaction with the course.  What we are learning is that the techniques that produce the greatest learning gains and best prepare students to progress through degree programs and graduate on time aren't necessarily the courses (and instructors) that they love the most.  Often, this is because such courses require significant and consistent engagement.  At the same time, these instructors are clearly doing their job of producing significant learning gains in their students.

The time has come to abandon the myth of the great professor--the lecturer who keeps the audience rapt in their seats, scribbling down his every word, chuckling at his witty jokes, in awe of his brilliance.  Certainly, many professors are skilled performers and many of the students in their classes learn well. But there are plenty of boring, bad performers who are very good at designing courses and whose students learn at very high levels--even if those students didn't necessarily love the professor.   Teaching is ultimately about student learning.  If we continue to insist on defining our best professors as those who are the most gifted performers, but without any way to quantify what this means, we make it difficult to defend ourselves from the accusations of bad teaching.  If we shift to defining good teaching through demonstrable student learning, we make it very difficult for these outside (and sometimes inside) attacks to carry much weight.  Of course, this shift has to start at the highest levels of the university, by creating a system that evaluates and rewards student learning more than instructor likeability.  Likeability matters.  It matters that students enjoy a class.  But, at the end of the day, what matters most is that they learned the course content.

[disclaimer: my students generally like me and give me high course evaluations.  I'm not writing this because of sour grapes.  Rather, after a decade+ of teaching, I've seen repeatedly that the system is set up to incentivize a kind of teaching that does not always serve the students' best interests.  And, at the moment, this system has left faculty very vulnerable to charges that they aren't "good", with no clear way to rebut such charges.]

The Myth of the Bad Professor

Back in my youth (not that long ago...really!), teaching was generally viewed as a respectable profession, filled with dedicated and patient professionals who regularly went above and beyond the call of duty to look after the welfare of their students.  Sure, there was the pot smoking 9th grade American history teacher who had us spend more time doing guided meditations (nap time!) than learning about the Civil War.  But what I remember were the teachers who assigned several essays/week (the grading!) and taught me how to write before I went to college; the teachers who coached us in after school activities and served as advisers to clubs; the teachers whose doors were always open and who offered great advice for college applications. I grew up in California, at a time when California invested in public education at all levels.  For the most part, it worked pretty well and I arrived at college very prepared to succeed.

These days, though, barely a day goes by without some new indictment of the teaching profession.  Teachers are incompetent, unmotivated, phoning it in, living off the fat of hard-working Americans thanks to the power of labor unions.  At least for awhile, it was the schools who were underfunded and failing; but now the blame has been shifted to the teachers.  It isn't state legislatures who are failing students (by slashing education budgets); it is badly performing, unaccountable teachers who are failing.  If students are being left behind, we are told, it's not because of larger socioeconomic factors or underfunded schools; it's because teachers aren't doing their job.

If K-12 teachers are bad, college professors are even worse.  They only care about research.  They devote as little time as possible to undergraduate education.  Their "grading standards" are irrational. They only care about money--and especially about making more money than their colleagues.  They use grad students as pawns to carry out their personal, interdepartmental vendettas.  Managing faculty is like herding cats (ok, this one might actually be true).  These caricatures have given rise to a large set of personalizable "professor" memes on http://www.quickmeme.com.  It's not funny to praise a professor for doing a great job of inspiring and facilitating learning in trying times.  No, it's far more funny to seize on outdated caricatures of professors assigning their own textbooks to make a profit off of their students; of professors assigning too much work (or, alternatively, constantly canceling class).  I'm sure that every university still has a few professors who live up to these caricatures.  What is missed in all of this, however, is the extent to which classroom teaching, even in large lecture classes, has improved substantially on campuses across the country in the past decade.

The myth of the aged, rumpled (usually male) professor reading his yellowed, dog-eared notes is just that....a myth.  Sure, this character used to be common--I can think of a number of such characters who taught my undergraduate classes (including one professor who still used purpled exams produced on a mimeograph...in the early 90s).  But I suspect that, even on R1 campuses that put a bounty on research productivity, most instructors are, in fact, very good teachers.  Many are outstanding.  It is certainly true that many of us would benefit from additional professional development, in part because quality teaching can benefit from expertise with educational technology tools well beyond an LMS.  Many of us would benefit from a system that placed more value on high-quality undergraduate (and graduate) teaching.  Still, given the current state of affairs, the remarkable thing isn't how many bad professors there are out there, but how many good ones.

As Melonie Fullick so aptly articulates, "What I find deeply uncharitable (and inaccurate) is to generalize this experience even to the majority of university teaching faculty. I’ve given academic lectures in various classes, and I can assure you there’s no reason to assume the speaker is simply “transmitting” information. One of the main underlying issues here is the assumption of passivity in the students, and of a transmission model of communication that has long been critiqued by communication theorists. Another is the generalization about faculty approach, as if those doing the speaking aren’t working to make their presentation engaging and responsive—and as if there’s nothing but lecturing going on in a course."

This is an important point, especially when it comes to large enrollment courses.  The bad lecturer is the straw man invoked by the MOOC Incs and others who want to "scale up" classrooms in the thousands.  The argument goes roughly as follows: "If most classroom-based, large enrollment classes are terrible--because, according to our caricature of the bad professor, they must be--then we can at least select out the "best" lecturers (whatever that means), put them on video, and stream them over the internet."  So we take something that we have deemed bad, claim that it is bad because of the professor's incompetence rather than the context (one person trying to teach hundreds of students), and then purport to solve the problem by finding competent professors.  This logic completely elides the fact that, as research has demonstrated, students don't learn more or better from good lecturers (in fact, they sometimes learn less well because they have a false sense of confidence/mastery).  It ignores the fact that teachers don't create learning, we facilitate it.  The failings of the lecture class are, for the most part, a consequence of the large numbers.  Increasing those numbers exponentially is very unlikely to fix the problems and far more likely to exacerbate them.

Perhaps even more worse, though, is the damage that this "myth of the bad professor" does to faculty morale.  It encourages the general public, including students, to view professors as disinvested  narcissists who are a drain on taxpayer money.  Far from being a noble vocation, teaching at a public university leaves one open to the lampoons of politicians eager to justify their disinvestment in higher education.  In reality, professors (as well as K-12 teachers) are doing more with less, year after year.  They are working harder than ever, largely because they genuinely care about their students and believe that education is a social good.  In the meantime, legislators and even institutional administrators hang on to the myth of the bad teacher in order to justify a shift to very large-scale, lecture-based education using the so-called best professors.

If we can leave behind this myth of the bad teacher; if we can recognize and acknowledge all of the creative, engaged teaching that is happening at campuses large and small around the country, we might be able to begin the challenging work of brainstorming different solutions to the "large enrollment, lecture-based courses don't work very well to create learning" problem.  Specifically, in large enrollment classes, we might be able to experiment with ways to exploit ed tech to minimize the structural problems of the environment; through careful assessment and redesign, we might be able to find alternative models that work very well to teach larger numbers of students effectively.  To do this, though, will require the direction of money and technical support to campus-based teaching.  It will require recognizing that the teaching faculty are a university's greatest asset.  Furthermore, they are an asset that deserves to be respected and supported, not universally caricatured as incompetent, disinterested in undergraduate teaching, and easily replaced with videotaped (or live-streamed) lectures of select Super Professors from around the country (and world).

*************
If this post resonated with you, please also read Melonie Fullick's excellent post on the same topic (written back in October 2012).  In her wise words: " Do we even have a way of “measuring” student learning to show what works? The context of all this is not a neutral one, it’s not just “let’s improve learning by finding out what will help the most”. The loss of resources including government funding has created serious material pressures. The urgency of the rhetoric about “change” should also remind us that there will be winners and losers in the education game, and futurologists have stakes in predicting something that could become a self-fulfilling prophecy—because prediction inspires present action."  Exactly.  And, as Fullick points out, much of this is driven by the basic assumption that on the ground university teaching is of poor quality, but with no data to support such a claim.
**************
More on the connection between the myth of the bad professor and the replacement of professors by large-scale modes of course delivery in this post by The Homeless Adjunct.