I was chatting the other day with a colleague who works in our College of Undergraduate Studies. We got on the topic of MOOCs and the idea that they are claiming to provide free education to the world. As those trained in Latin are wont to do, we discussed the etymology of the word education: educere, to lead out or raise up. The Romans used the verb to talk about raising their children. When we educate our students, we truly are acting in loco parentis. But how do we effectively nurture tens of thousands of children at once? Can this job be reduced to a series of algorithms that require little human interaction? Can we rely on the older kids to step in and raise their younger siblings because we parents are too busy making sure there's food on the table and the laundry is done?
The phrase "education at scale" has been attached to MOOCs--initially by Coursera, Inc. co-founders Daphne Koller and Andrew Ng in their talk "The Online Revolution: Education at Scale"--with seemingly little thought about what it actually implies. In fact, the phrase itself highlights a paradox at the root of Coursera, a paradox that Koller has alluded to when she rehearses the impetus for founding Coursera. On the one hand, Koller was interested in improving pedagogy in her own classes by using online tools like pre-recorded, interactive videos to deliver content outside of class, thereby freeing up class time for higher quality interactions with her students. In other words, she was interested in blending (or perhaps flipping) her campus-based classes to improve the learning experience of her students. On the other hand, Ng was interested in globalizing his classroom and (massively increasing the instructor:student ratio). While Koller was driven by improving student education, Ng seems to have been driven by a desire to scale-up his audience. In their home discipline of computer science, these two impulses are somewhat less contradictory than they are in, say, English or History of Classics.
I don't mean to say that it is not possible to teach more students with better learning outcomes by using technology more effectively--it obviously is and it is what many K-12 and university instructors are trying to figure out how to do. But it is important to recognize that the phrase "education at scale" is not what most MOOCs currently do. Rather, MOOCs deliver content at scale. It is commonplace to point out that printed books have been performing this same role for centuries; but it is worth observing that MOOCs can go to places where books in large numbers still cannot (particularly third world countries). As well, for a generation raised on images rather than the printed word, I can imagine that it is easier for them to learn from the embodied rather than written "textbook" (even if it is true for some/many (as Mary Beard argues) that content is easier to digest in written form).
Putting content online--even if it has embedded quizzes and assignments--is not the same as educating. I can understand how two extremely intelligent and creative Computer Science researchers might not recognize this distinction, however. In a problem-based field where instruction is highly structured and can easily be packaged in terms of mastery of a series of concepts of increasing complexity; and where assignments can be machine-graded, the claim that a MOOC student who completes all the work and uses the discussion forums as well as peer study groups is receiving something approximating the classroom education of a paying Stanford student is more believeable. I can't imagine that all MOOC students come away with a strong grasp of the conceptual underpinnings of the course content, but obviously this is a small cost by comparison to the benefits for students who otherwise would have no access to a particular course. I suspect that statistics or a differential equations course work in fairly similar ways. I also imagine that, as we move further down the spectrum, towards disciplines that are less problem-based, that don't have a single correct answer to a question, we move further away from anything vaguely resembling education. Taking a MOOCified version of my Intro to Rome class won't hurt anyone; but I could not say with a straight face that they were getting 90% of the learning experience that my UT students were getting. If they were, then my UT students would be right to think I was an overpaid dinosaur.
This would all be low-stakes, disciplinary quibbling if it weren't for the fact that legislators and even university administrators salivate at the phrase "education at scale." They imagine that they will be able to teach tens of thousands of students with a small stable of "the best professors" (whatever that means). Heck, maybe they will just license courses from other institutions and hire low-paid adjuncts to run the course on campus and call it education. This is the point where a public-lecture friendly platitude can actually become incredibly dangerous to the mission of higher education, particularly at public universities and in disciplines that do not fit the current MOOC platforms all that well (i.e. liberal arts).
States and the federal government have cut appropriations for education year after year, for decades. We are at something of a crisis point, to be sure, but it is a crisis that is man-made and is not a crisis in higher education per se. It is a crisis created by government's refusal to fund education properly; administrators' inclination to use adjuncts and lecturers instead of tenure-track faculty (thus giving themselves more control over annual budgets since they have effectively lowered the fixed costs of paying tenure-track faculty salaries). Fewer instructors means that fewer courses are offered and fewer seats are available, particularly in high-demand and lower division courses. This problem could be solved in a number of different ways: states could appropriate reasonable money for their public institutions; philanthropists, especially those wealthy Silicon Valley folks who are now so interested in higher education, could endow professorships across disciplines. Tuition could be increased, effectively acknowledging the fact that states are no longer sponsoring so-called state universities.
Instead, as many others have noted, we have venture capitalists and others with a horse in the race declaring that there is a crisis in higher education; and that only technology can solve the problem. Oh, and to adopt that technology will cost a massive investment of $$ by institutions who are too impoverished to hire tenure-track faculty. The regents of my own institution recently created an Institute for Transformational Learning and gave its director somewhere around $50 million to allocate. In the meantime, my department is not permitted to replace retiring and departing colleagues, to the point that most of our core courses will be taught by grad students, adjuncts, and lecturers in the coming year. State lawmakers, regents, and even administrators want to believe in the myth of education at scale because it seems to provide a solution to their budgetary woes that does not require a long-term investment in faculty.
Until MOOCs make serious advances in their pedagogy, however, it is false advertising to say that the vast majority of them are providing education at scale. They provide many other benefits, including allowing for much higher quality student-instructor interaction by shifting content delivery to outside of class; opening the gates of the university to anyone who is interested (and perhaps persuading more Americans that professors are not all pipe-smoking, latte drinking, leftist slackers); and encouraging Americans to be more intellectually engaged in general. These are all good things, but they are not interchangeable with a quality university classroom experience--not yet and perhaps not ever.
But let's not throw the baby out with the bathwater. It *is* possible to teach at scales larger than what we have currently been doing, and to do it more effectively than ever before, by making better use of education technology. I have done this in my own large lecture class. Enrollment has doubled from 200 to 400 students; but those 400 students are learning more and better. I know this because we are doing careful studies of their learning and comparing it to previous cohorts. To accomplish these improved learning outcomes at a larger scale has required a large and sustained investment of time and energy from me and a tremendous amount of classroom support (a team of 4 ace teaching assistants and 2 undergraduate graders). I have worked with an excellent team of learning and assessment specialists from around campus and have consulted with several other faculty who are doing similar things with their courses. I have spent a year developing content for the course and will continue to do so over the summer. Effective teaching requires enormous energy and engagement--and this doesn't get easier or cheaper over time (at least not until a computer can be programmed to respond with a certain kind of intervention to a certain kind of comment on a discussion board, for example). There *are* advantages to teaching 400 students instead of 200, but they are related to issues like time to degree and better use of campus resources.
When we talk about education at scale, we can't assume that this is what a MOOC does, at least not without evidence of learning outcomes for MOOC students. In the meantime, though, we need to be experimenting, figuring out what the limits of scale are for our classes and our disciplines. What is the largest number that we can teach and continue to see improved learning outcomes? At what point does that curve turn downward? The limits of scale will surely vary quite a lot between disciplines and individual courses. The more theoretical and "subjective" (in a good way) the content, the more difficult it will be to increase size without serious sacrifices to learning outcomes. Education at scale does not simply happen by making course content available and interactive (though I am willing to concede that it can sometimes happen in a computer science or calculus course with a highly motivated and self-sufficient student). Moreover, in most disciplines, education at a scale of tens of thousands, much less hundreds of thousands, of students will never happen.
Addendum 3/28/13: A critique of the problem of scale even in a math course on the MOOC platform.
Addendum 4/1/2013: Steve Krause on Duke's English Comp Course and the problems of teaching writing at scale (scroll down the page for comments)
Friday, March 15, 2013
Wednesday, March 13, 2013
Another Reason I Love Lecture Capture
Among the many challenges of teaching my Intro to Rome class is the fact that, well, I have to be the one to teach it. On the one hand, because I have recordings of all of the content archived, this isn't really a problem. If I am sick, I can post a recording and know that the students have at least been able to listen to me lecture on the content. In some real way, though, this misses the point of the class, which is to provide the students the opportunity to practice and apply the content. Delivery of content is a fairly small part of what I do in any class session. My PPTs are a mix of i>clicker, peer discussion, group discussion, and content slides. I ask the students to consider questions from previous lectures as well as from the content I have just delivered. I review main points of their Piazza discussions. It is not really possible to ask a colleague who has never taught this way to step in and take over for me.
Unfortunately, thanks to a dental emergency just before spring break, I had to do just that. Fortunately, I had already created the PPT presentation. I had a recording of me lecturing on the content slides. If possible, I wanted class to meet as usual; so I asked one of my teaching assistants to step in for me. This had several advantages, not least of which was the fact that he was used to the large audience and the method of teaching. He could take my PPT, listen to my lecture and take notes, and then try to approximate that in class in my absence.
Typically, graduate teaching assistants struggle with the lecture format. It's not an intuitive way to teach and requires very good time management skills as well as an ability to be clear, focused, but also entertaining. Too often, grad students spend too much time on early material and never get to the final third of their planned lecture. In this case, my TA delivered the lecture perfectly. He got through all of the material; he was lively but got all the key content across to the students; and he injected his own observations from time to time. It struck me that this mode of teacher training is far more effective than our more usual "here's a topic; create a lecture and deliver it" mode. In this instance, the TA could see the things that I thought were important to emphasize but he could also add his own twist. It was ultimately his lecture, not mine; yet it covered all the necessary content and kept the class on track.
Many professors I know are taking MOOCs for much the same reason: we want to see how someone else teaches the same material we teach. This is going to be a valuable contribution of the MOOC. Too often, once we get jobs, we get little useful feedback on our teaching and rarely have conversations with others who teach our same courses (typically because we are the only person at our institution to teach that course). The internet has changed this to an extent, in that we can now look at syllabuses from colleagues' courses; but being able to audit them adds an entirely new dimension. In the same way that (I think) my TA was able to learn from "watching" me and then doing it himself, I have learned a lot from watching my colleagues at other institutions teach courses on topics like Greek mythology.
Unfortunately, thanks to a dental emergency just before spring break, I had to do just that. Fortunately, I had already created the PPT presentation. I had a recording of me lecturing on the content slides. If possible, I wanted class to meet as usual; so I asked one of my teaching assistants to step in for me. This had several advantages, not least of which was the fact that he was used to the large audience and the method of teaching. He could take my PPT, listen to my lecture and take notes, and then try to approximate that in class in my absence.
Typically, graduate teaching assistants struggle with the lecture format. It's not an intuitive way to teach and requires very good time management skills as well as an ability to be clear, focused, but also entertaining. Too often, grad students spend too much time on early material and never get to the final third of their planned lecture. In this case, my TA delivered the lecture perfectly. He got through all of the material; he was lively but got all the key content across to the students; and he injected his own observations from time to time. It struck me that this mode of teacher training is far more effective than our more usual "here's a topic; create a lecture and deliver it" mode. In this instance, the TA could see the things that I thought were important to emphasize but he could also add his own twist. It was ultimately his lecture, not mine; yet it covered all the necessary content and kept the class on track.
Many professors I know are taking MOOCs for much the same reason: we want to see how someone else teaches the same material we teach. This is going to be a valuable contribution of the MOOC. Too often, once we get jobs, we get little useful feedback on our teaching and rarely have conversations with others who teach our same courses (typically because we are the only person at our institution to teach that course). The internet has changed this to an extent, in that we can now look at syllabuses from colleagues' courses; but being able to audit them adds an entirely new dimension. In the same way that (I think) my TA was able to learn from "watching" me and then doing it himself, I have learned a lot from watching my colleagues at other institutions teach courses on topics like Greek mythology.
Quizzes as a Tool for Calibrating Student Expectations
With every passing semester, I believe ever more firmly that one of the fundamental keys to a successful class (like any good relationship) is the clear communication of expectations. More recently, as I move to a much more student-centered classroom and model of instruction, I have come to see that, for me to communicate my expectations clearly, I need to have a good understanding of what my students' baseline expectations are. This is particularly true when it comes to grades; and particularly true in a class with 75% underclassmen in the state of Texas. These students gained admission because they were in the top 10% (or 8% or 9%) of their graduating high school class. This probably means that they have rarely if ever received a grade lower than an A since middle school--if then.
This spring, working together with a colleague in UT's Center for Teaching and Learning, I administered a backgrounds, goals, and expectations survey to the students during the first week of the semester. Nearly the entire class completed the survey (there was a big incentive attached). I recently received an overview of the results. In many respects, I was surprised: fewer of them work than I expected (only 25%); more of them were in the class because they were interested in the topic than I would have guessed (only about 25/335 registered for purely pragmatic reasons, e.g., "it fit my schedule"). The biggest shock was the grade that they expected to get in the course: 96% expected an A of some kind. Only .6% expected a grade lower than a B+. Historically, somewhere between 35-40% earn some kind of A in the course. Another 30% earn some kind of B and about 20% earn a C. I fail very few students because they drop the course, even after it is over; or withdraw. This particular cohort seems to be very good (I wrote about their performance on the first midterm recently). They are performing at a high level on quizzes and midterms. This means that more like 50% are in the A range, perhaps 55%. I am fairly sure, though, that 96% of the class will not be earning some kind of A.
I'd be curious to know how this same cohort would answer that question now, at the midway point of the semester. They have now sat for five quizzes and a midterm exam and have completed half of their graded discussion posts. With the quizzes, they get weekly feedback on their performance and have the opportunity to calibrate their effort but also their expectations. Despite the hassles of administering scantron quizzes (to limit cheating) to a class of nearly 400 students, I am a completely sold on their many benefits--and these benefit extend well beyond motivating regular and consistent study. I would imagine that, with each quiz, the student can see how they are doing. I post the questions with the correct answers and also review questions that were missed by more than 25% of the class (and frequently include those questions on future quizzes). I also post a summary of the class performance, so that students have some sense of where they stand relative to the rest of the class.
When I first started teaching a decade ago, I was reluctant to ever post data that showed student performance relative to their classmates. I wanted students to focus on themselves and not compare themselves or be hyper competitive. Since I don't grade on a curve, it doesn't matter how someone else did. Over the years, though, I've realized that this comparative data has an important role to play in helping my students calibrate their expectations, face reality, and grasp that they are no longer in high school. They may well be a small fish in a very big pond full of much larger fish. It cuts back on complaints when they get a glimpse of just how many big fish are swimming around with them.
The other advantage of frequent graded assessments: they force students to confront reality. When I used a 3 midterm system for my lecture class, I regularly had students operating in extreme denial. Even at the end of the semester, they believed that they were going to get a much better grade than they actually received (I once calculated this from the course evaluations). They persuade themselves that they will do better on the next exam, regardless of how they have done on previous exams. I am stunned at the degree of denial that I see on a regular basis. My sense is that these weekly quizzes are going a long way towards dispelling that denial and forcing students to fish or cut bait. They are also giving them very specific information about their learning behaviors and reminding them that, if they don't keep up, it will be very bad for their grade.
We will do an end of semester survey that asks them to reflect on their performance in the course. I am now very curious to read those surveys, and especially to see whether, over the course of the semester, the frequent graded feedback actually does help them to adjust their expectations but also realize that, if they want an A, they are going to have to work very hard for it. I would not be surprised if this cohort ends up earning significantly higher grades than I usually give--but it will be because they worked hard for them. They knew how hard they would need to work because, each week, they got feedback on the success of their learning strategies.
This spring, working together with a colleague in UT's Center for Teaching and Learning, I administered a backgrounds, goals, and expectations survey to the students during the first week of the semester. Nearly the entire class completed the survey (there was a big incentive attached). I recently received an overview of the results. In many respects, I was surprised: fewer of them work than I expected (only 25%); more of them were in the class because they were interested in the topic than I would have guessed (only about 25/335 registered for purely pragmatic reasons, e.g., "it fit my schedule"). The biggest shock was the grade that they expected to get in the course: 96% expected an A of some kind. Only .6% expected a grade lower than a B+. Historically, somewhere between 35-40% earn some kind of A in the course. Another 30% earn some kind of B and about 20% earn a C. I fail very few students because they drop the course, even after it is over; or withdraw. This particular cohort seems to be very good (I wrote about their performance on the first midterm recently). They are performing at a high level on quizzes and midterms. This means that more like 50% are in the A range, perhaps 55%. I am fairly sure, though, that 96% of the class will not be earning some kind of A.
I'd be curious to know how this same cohort would answer that question now, at the midway point of the semester. They have now sat for five quizzes and a midterm exam and have completed half of their graded discussion posts. With the quizzes, they get weekly feedback on their performance and have the opportunity to calibrate their effort but also their expectations. Despite the hassles of administering scantron quizzes (to limit cheating) to a class of nearly 400 students, I am a completely sold on their many benefits--and these benefit extend well beyond motivating regular and consistent study. I would imagine that, with each quiz, the student can see how they are doing. I post the questions with the correct answers and also review questions that were missed by more than 25% of the class (and frequently include those questions on future quizzes). I also post a summary of the class performance, so that students have some sense of where they stand relative to the rest of the class.
When I first started teaching a decade ago, I was reluctant to ever post data that showed student performance relative to their classmates. I wanted students to focus on themselves and not compare themselves or be hyper competitive. Since I don't grade on a curve, it doesn't matter how someone else did. Over the years, though, I've realized that this comparative data has an important role to play in helping my students calibrate their expectations, face reality, and grasp that they are no longer in high school. They may well be a small fish in a very big pond full of much larger fish. It cuts back on complaints when they get a glimpse of just how many big fish are swimming around with them.
The other advantage of frequent graded assessments: they force students to confront reality. When I used a 3 midterm system for my lecture class, I regularly had students operating in extreme denial. Even at the end of the semester, they believed that they were going to get a much better grade than they actually received (I once calculated this from the course evaluations). They persuade themselves that they will do better on the next exam, regardless of how they have done on previous exams. I am stunned at the degree of denial that I see on a regular basis. My sense is that these weekly quizzes are going a long way towards dispelling that denial and forcing students to fish or cut bait. They are also giving them very specific information about their learning behaviors and reminding them that, if they don't keep up, it will be very bad for their grade.
We will do an end of semester survey that asks them to reflect on their performance in the course. I am now very curious to read those surveys, and especially to see whether, over the course of the semester, the frequent graded feedback actually does help them to adjust their expectations but also realize that, if they want an A, they are going to have to work very hard for it. I would not be surprised if this cohort ends up earning significantly higher grades than I usually give--but it will be because they worked hard for them. They knew how hard they would need to work because, each week, they got feedback on the success of their learning strategies.
Tuesday, March 12, 2013
Andrew Ng Comes to Austin: What MOOCs Can and Can't Do
As part of his global lecture tour touting the miracle of MOOCs, and Coursera in particular, Dr. Andrew Ng made a stop at UT Austin. The presentation itself did not offer anything new to anyone who has been following the MOOC conversation over the past year. It rehearsed the birth of Coursera at Stanford and offered a show and tell of some of the platform's functionality. It was also packed with vague platitudes about learning, education, and pedagogy. Dr. Ng is clearly an intelligent guy; yet, if this had been a presentation for one of my classes, it would have earned a solid C with exhortations to be specific and argue from evidence. I was especially frustrated with the absence of a clear "so what". That is, what, exactly, is the rationale for Coursera? What does it do and how does it do it? I've taken a few Coursera courses and, for the most part, enjoyed them (while recognizing the extreme limitations of the mode of instruction). What about student motivation (and the high dropout rate)?
On the one hand, Dr. Ng tells us, education is a human right; and his goal is to educate everyone, regardless of social origins or financial status. Everyone, he thinks, is entitled to the kind of first rate education that his institution, Stanford, has to offer (I'll leave my thoughts on that particular issue for another post). Never mind that "an education" at Stanford is far more than the 4 years of courses taken by its students. Let's assume for now that, in fact, the "best professors" do in fact teach at the most prestigious (i.e. highly ranked) universities (a remarkably bad assumption). I am sympathetic to the goal of making knowledge open access, particularly in places where students can't afford textbooks and don't have easy access to libraries. I can also see how, for certain disciplines like computer science, a MOOC might work pretty well. Dr. Ng repeatedly stated that a MOOC could certify someone's learning and allow them to get a better job. Again, this makes sense to me in a field like computer science--a field which relies a lot on certifications of various kinds. A smart kid in India can take a MOOC, learn the content, get a certificate from Stanford, and then use that to get a better job. This strikes me as an enormous social good.
The problem is, it only makes sense for highly technical fields which rely on certifications. That is, for fields which might easily be taught in technical schools rather than at universities. It makes no sense whatsoever for most disciplines currently taught at universities. Sure, disciplines like political science or classics can package a course as a MOOC--but it is basically just a group of lectures with multiple choice quizzes. Now, to be fair, this is often the same product that is being delivered on campus to smaller audiences. Yet, because the learning outcomes aren't easily quantifiable and measurable using machines, it is impossible to test deep learning or certify much of anything. Certainly, getting a certificate in Greek Mythology is not going to position anyone for a better job.
Dr. Ng and Coursera are very proud of their apparent solution to teaching course content that requires assessments that can't be machine graded (e.g. analytical essays): peer grading. Suddenly, the talk is all about grading rather than learning. Whereas the discussion of hard science/math disciplined focused on certifying skills (aka learning), now it's about grades. One study has shown that peers grade roughly the same as professors. All well and good, if the point of assigning an essay was to get a grade. It is at this point that Dr. Ng's (and Coursera's) lack of familiarity with and understanding of humanities disciplines becomes a serious impediment. They seem totally unaware that written work is generally assigned as a way to assess student progress and help them improve. 300 word essays are not going to do any of that; as well, despite lauding peer grading, the fact is that it has yet to be a real success or do much of anything apart from assigning an arbitrary letter grade to a piece of writing. Serious humanities scholars ought to be outraged and scared by this aspect of Coursera's self-presentation. It is very clear that the platform cannot support the delivery of a serious humanities course, that involves real and demonstrable learning. In fact, Coursera seems to not take humanities courses very seriously. They want to have some of them on their menu, of course; but, as is often the case in bricks and mortar universities, these courses are treated as second-class citizens, where it is grades rather than learning that matter. I don't mean to single out Coursera for this attitude; unfortunately, all other MOOC platforms share this approach and nothing is going to change unless humanities scholars get serious about figuring out how to deliver our content in a pedagogically sound way to large audiences.
So, to sum up: certain classes have the capacity to perform a real social good. These are classes where the content is highly structured and relatively easy to transfer and assess if student motivation is high. Dr. Ng's class is a perfect example of one such class. Coursera's platform is not well-designed for a serious humanities class. If the aim is to broadcast a series of lectures and check retention with short quizzes, fine. But if the aim is to do serious instruction, well, Coursera isn't there yet nor does it seem to really care. In fact, it is convincing itself that peer grading is the ideal solution.
The real value of Coursera isn't really for the supposed "target audience" but instead, for the paying customers at universities. In fact, when one registers for a Coursera course, one is volunteering to be a research subject in a great experiment on learning. As Dr. Ng admits, they are amassing an incredible amount of high specific data that is giving important insights into student learning--and, because of the scale, is making visible patterns that would otherwise be invisible to instructors. As someone who has experienced this phenomenon on a smaller scale, I know how much my teaching has changed thanks to data about student learning behavior.
But there's an important point to be made here: the real benefit of these MOOCs isn't for the 40,000 students taking them--even if a few students will, in fact, benefit. It is for the 40 students who are taking that course on campus. Thanks to the data gathered from the MOOC audience, an instructor can improve content delivery; identify and address trouble spots; and make better use of in class time by having students watch content outside of class. I am quite certain that, in this regard, MOOCs will pay off great dividends for paying customers in the form of much better classes and more meaningful engagement with the instructor. But it is important not to confuse this benefit with the claim for a larger, more global benefit of making a Stanford (or Princeton or Harvard) education available to the masses. MOOCs are not doing that. They are simply giving the masses the opportunity to sit at the feet of a living textbook for a few weeks. That is not teaching and is not a model that is particularly supportive of learning. Indeed, Dr. Ng tacitly concedes this point when he points out the benefits for the paying students, that is, the students who will actually benefit from evidence-based pedagogy that supports learning.
I am a big fan of universities doing a better job of sharing their resources with the outside world; and I do think MOOCs have an important role to play in education, especially continuing education. The data about learning behavior that they are collecting is going to be an incredible resource as we instructors continue to learn how to better teach our students. But I also feel strongly that we all need to acknowledge what MOOCs can't do (deliver courses where learning can't be machine-graded); and companies like Coursera need to be more honest about what they actually are doing: providing the resources to improve dramatically the quality of on campus education for paying customers; offering universities the chance to advertise their wares (and professors the opportunity to sell vast numbers of their own books); and perhaps appealing even more directly to their alumni (just as alumni cruises and lecture series do). These are worthy goals; but they should not be confused with delivering high quality, online instruction. Even more, they shouldn't distract university administrators from investing in platforms and the development of courses which can, in fact, deliver the serious online learning that MOOCs promise but fail to deliver.
On the one hand, Dr. Ng tells us, education is a human right; and his goal is to educate everyone, regardless of social origins or financial status. Everyone, he thinks, is entitled to the kind of first rate education that his institution, Stanford, has to offer (I'll leave my thoughts on that particular issue for another post). Never mind that "an education" at Stanford is far more than the 4 years of courses taken by its students. Let's assume for now that, in fact, the "best professors" do in fact teach at the most prestigious (i.e. highly ranked) universities (a remarkably bad assumption). I am sympathetic to the goal of making knowledge open access, particularly in places where students can't afford textbooks and don't have easy access to libraries. I can also see how, for certain disciplines like computer science, a MOOC might work pretty well. Dr. Ng repeatedly stated that a MOOC could certify someone's learning and allow them to get a better job. Again, this makes sense to me in a field like computer science--a field which relies a lot on certifications of various kinds. A smart kid in India can take a MOOC, learn the content, get a certificate from Stanford, and then use that to get a better job. This strikes me as an enormous social good.
The problem is, it only makes sense for highly technical fields which rely on certifications. That is, for fields which might easily be taught in technical schools rather than at universities. It makes no sense whatsoever for most disciplines currently taught at universities. Sure, disciplines like political science or classics can package a course as a MOOC--but it is basically just a group of lectures with multiple choice quizzes. Now, to be fair, this is often the same product that is being delivered on campus to smaller audiences. Yet, because the learning outcomes aren't easily quantifiable and measurable using machines, it is impossible to test deep learning or certify much of anything. Certainly, getting a certificate in Greek Mythology is not going to position anyone for a better job.
Dr. Ng and Coursera are very proud of their apparent solution to teaching course content that requires assessments that can't be machine graded (e.g. analytical essays): peer grading. Suddenly, the talk is all about grading rather than learning. Whereas the discussion of hard science/math disciplined focused on certifying skills (aka learning), now it's about grades. One study has shown that peers grade roughly the same as professors. All well and good, if the point of assigning an essay was to get a grade. It is at this point that Dr. Ng's (and Coursera's) lack of familiarity with and understanding of humanities disciplines becomes a serious impediment. They seem totally unaware that written work is generally assigned as a way to assess student progress and help them improve. 300 word essays are not going to do any of that; as well, despite lauding peer grading, the fact is that it has yet to be a real success or do much of anything apart from assigning an arbitrary letter grade to a piece of writing. Serious humanities scholars ought to be outraged and scared by this aspect of Coursera's self-presentation. It is very clear that the platform cannot support the delivery of a serious humanities course, that involves real and demonstrable learning. In fact, Coursera seems to not take humanities courses very seriously. They want to have some of them on their menu, of course; but, as is often the case in bricks and mortar universities, these courses are treated as second-class citizens, where it is grades rather than learning that matter. I don't mean to single out Coursera for this attitude; unfortunately, all other MOOC platforms share this approach and nothing is going to change unless humanities scholars get serious about figuring out how to deliver our content in a pedagogically sound way to large audiences.
So, to sum up: certain classes have the capacity to perform a real social good. These are classes where the content is highly structured and relatively easy to transfer and assess if student motivation is high. Dr. Ng's class is a perfect example of one such class. Coursera's platform is not well-designed for a serious humanities class. If the aim is to broadcast a series of lectures and check retention with short quizzes, fine. But if the aim is to do serious instruction, well, Coursera isn't there yet nor does it seem to really care. In fact, it is convincing itself that peer grading is the ideal solution.
The real value of Coursera isn't really for the supposed "target audience" but instead, for the paying customers at universities. In fact, when one registers for a Coursera course, one is volunteering to be a research subject in a great experiment on learning. As Dr. Ng admits, they are amassing an incredible amount of high specific data that is giving important insights into student learning--and, because of the scale, is making visible patterns that would otherwise be invisible to instructors. As someone who has experienced this phenomenon on a smaller scale, I know how much my teaching has changed thanks to data about student learning behavior.
But there's an important point to be made here: the real benefit of these MOOCs isn't for the 40,000 students taking them--even if a few students will, in fact, benefit. It is for the 40 students who are taking that course on campus. Thanks to the data gathered from the MOOC audience, an instructor can improve content delivery; identify and address trouble spots; and make better use of in class time by having students watch content outside of class. I am quite certain that, in this regard, MOOCs will pay off great dividends for paying customers in the form of much better classes and more meaningful engagement with the instructor. But it is important not to confuse this benefit with the claim for a larger, more global benefit of making a Stanford (or Princeton or Harvard) education available to the masses. MOOCs are not doing that. They are simply giving the masses the opportunity to sit at the feet of a living textbook for a few weeks. That is not teaching and is not a model that is particularly supportive of learning. Indeed, Dr. Ng tacitly concedes this point when he points out the benefits for the paying students, that is, the students who will actually benefit from evidence-based pedagogy that supports learning.
I am a big fan of universities doing a better job of sharing their resources with the outside world; and I do think MOOCs have an important role to play in education, especially continuing education. The data about learning behavior that they are collecting is going to be an incredible resource as we instructors continue to learn how to better teach our students. But I also feel strongly that we all need to acknowledge what MOOCs can't do (deliver courses where learning can't be machine-graded); and companies like Coursera need to be more honest about what they actually are doing: providing the resources to improve dramatically the quality of on campus education for paying customers; offering universities the chance to advertise their wares (and professors the opportunity to sell vast numbers of their own books); and perhaps appealing even more directly to their alumni (just as alumni cruises and lecture series do). These are worthy goals; but they should not be confused with delivering high quality, online instruction. Even more, they shouldn't distract university administrators from investing in platforms and the development of courses which can, in fact, deliver the serious online learning that MOOCs promise but fail to deliver.
Midterm #1: The Results
My "stealth-flipped" Intro to Ancient Rome class sat for their first midterm a few weeks ago. Honestly, I was dreading this exam and the aftermath. In the fall, everything was going extremely well until after the first midterm. After the midterm, students suddenly slacked off, started to complain about having to attend class, and ranted on Facebook about the flipped class model. In a matter of week, it went from a class that I looked forward to teaching to one that I dreaded and resented. I spent several months talking to learning specialists, reading evaluations, and trying to figure out what went wrong and, more importantly, how to prevent the same thing from happening again. One clear answer (besides weekly quizzes and less heavily weighted midterms): a much more challenging first midterm. To this end, I added a fair chunk of course content (on archaeology) and expanded the chronological range of the exam. I made sure that the questions were challenging. My aim was to have the best students score in the mid-low 90s, with the rest of the class scattered in the Bs and Cs. If a student studied, they wouldn't fail the exam; but I expected a pretty good cluster of Bs and Cs. I wanted the exam to be a wake-up call of sorts, a reminder that they had to stay engaged and working hard if they hoped to earn a high grade.
I was stunned when the results came back. Despite my most concerted efforts--and a substantially more difficult first exam, this spring class outscored their two previous cohorts by a wide margin. Some rough data for comparison:
Fall 2011 (c. 220 students, traditional lecture): Avg: 80; Median 85
Fall 2012 (400 students, traditional flipped): Avg. 81.87, Median 87
Spring 2012 (400 students, stealth flipped): Avg. 86.92, Median 92
In Fall 2012, A (159); B (109); C (56); D (24); F (33) [numbers approximate]
In Spring 2013 A (215); B (93); C (34; D (17); F (22) [numbers approximate]
I am a tough grader and when I want to write a tough exam, it's a tough exam. In the comparison of Fall 2011 and Fall 2012, the flipped class scored only a few points higher than the regular lecture course--but the exam in Fall 2012 was substantially more challenging. The grades can't really be compared because, thanks to the flipped method, I was able to write a more difficult exam that tested deeper learning. I upped the ante once again in the spring; yet, somehow, this spring cohort aced the exam and significantly outpaced the fall cohort. In part, this may reflect the fact that it is the second semester for freshmen and they are better able to manage the challenges of college--but freshmen are only about 25-30% of my students. So why such a dramatic difference--and off the charts high grades on a very difficult exam?
First of all, weekly quizzes. The spring students had taken three weekly quizzes leading up to the midterm, covering all but the final week of course material. In addition, they had access to practice quizzes for all the content that would be covered on the exam. On Blackboard, we can see who tries the quizzes (I set it so that we don't see how they perform). Over 90% of the class tried all available practice quizzes. That is, they had a lot of practice answering multiple choice and "mark all correct" questions.
In addition to multiple choice and mark all correct questions, the midterm had 8 short answer questions. In the fall, these short answer questions tripped up a number of students, primarily because they did not answer them completely, directly, or with enough detail. We received a number of complaints from them and my "grade czar" had to deal with many dissatisfied students who did not understand why they had lost partial credit for vague and incomplete answers. This time, I made a short Echo video reviewing my expectations for the short answer questions and giving them sample questions and answers, with explanations. My head TA also did a lot of work with practicing the short answer questions during his weekly Supplementary Instruction sessions. Finally, we made available to the students a very long list of potential short answer questions. It was nothing that wasn't also available last semester--but this time, I said explicitly that all questions would be drawn from the (exhaustive) list, albeit perhaps in recombined form. By telling them that work on this document would be rewarded, they focused their attentions on answering the long list of questions rather than speculation. In Fall 2012, I told them that the questions would be drawn from in class questions and review questions embedded in the lectures; but, for some reason, this was never enough to motivate them to assemble a list and learn the material. By assembling the list, distributing it to the class, and encouraging them to create a Google doc and pool knowledge, we were able to get them to focus their energy on study rather than fretting.
In the days leading up to the exam, I also observed some notable changes in learning behavior. First, the class discussion board--Piazza--was very quiet. In the past, this was a place where students would pose questions whose answers could easily be found with a bit of effort. This had stopped (perhaps in part because I had warned them against doing so in the syllabus). From what I gather, Facebook was also fairly quiet. Instead of the usual pre-midterm anxiety venting that was a hallmark of the fall class, this group seemed to be studying. Their anxiety levels were noticeably lower (the teaching team was not inundated with emails) and they seemed to have a clear grasp of what to expect. The efforts I made to convey expectations as transparently as possible, as well as the practice they had with the quizzes, seemed to pay off.
Typically, in the 48 hours before an exam, there is a big spike in Echo views of recorded lectures. We waited and waited for the spike to come, but it never did. In addition, what we did see was the students going back into the recorded class sessions to check details or refresh their memories. I was delighted by this behavior--it was exactly what I wanted. In the fall, I pleaded with students to go back to the recorded class sessions, to recognize that exam questions were coming from class and prepare by reviewing the recordings. Few of them ever did this. Without a word from me, this spring cohort is using the recorded class sessions exactly as they were intended (and making it worth the expense and effort to record class meetings).
I am learning a lot from my current cohort of students. Two things stand out: the value of frequent, low-stakes assessments for supporting learning (and, more generally, a positive experience) in a large enrollment class; the importance of lecture--not so much as a means of knowledge transfer as much as a tool for orienting students, letting them get to know me, feel a connection to me, and experience my enthusiasm for ancient Rome.
We are now heading swiftly towards the second midterm. This is a tough exam--it's the part of the class that led me to flip the class in the first place; and inspired me to toughen up the first section of the course. Thus far, quiz grades have not dropped off at all and the students themselves have observed that the material is getting more difficult and requiring more work. In the past, I tried to warn classes about the increased difficulty level to no avail--midterm grades always dropped by 10 points. I have a feeling that this cohort will finally break that pattern, and I will not have said a word to them about the increased difficulty. They have figured it out themselves and adjusted their effort accordingly.
I was stunned when the results came back. Despite my most concerted efforts--and a substantially more difficult first exam, this spring class outscored their two previous cohorts by a wide margin. Some rough data for comparison:
Fall 2011 (c. 220 students, traditional lecture): Avg: 80; Median 85
Fall 2012 (400 students, traditional flipped): Avg. 81.87, Median 87
Spring 2012 (400 students, stealth flipped): Avg. 86.92, Median 92
In Fall 2012, A (159); B (109); C (56); D (24); F (33) [numbers approximate]
In Spring 2013 A (215); B (93); C (34; D (17); F (22) [numbers approximate]
I am a tough grader and when I want to write a tough exam, it's a tough exam. In the comparison of Fall 2011 and Fall 2012, the flipped class scored only a few points higher than the regular lecture course--but the exam in Fall 2012 was substantially more challenging. The grades can't really be compared because, thanks to the flipped method, I was able to write a more difficult exam that tested deeper learning. I upped the ante once again in the spring; yet, somehow, this spring cohort aced the exam and significantly outpaced the fall cohort. In part, this may reflect the fact that it is the second semester for freshmen and they are better able to manage the challenges of college--but freshmen are only about 25-30% of my students. So why such a dramatic difference--and off the charts high grades on a very difficult exam?
First of all, weekly quizzes. The spring students had taken three weekly quizzes leading up to the midterm, covering all but the final week of course material. In addition, they had access to practice quizzes for all the content that would be covered on the exam. On Blackboard, we can see who tries the quizzes (I set it so that we don't see how they perform). Over 90% of the class tried all available practice quizzes. That is, they had a lot of practice answering multiple choice and "mark all correct" questions.
In addition to multiple choice and mark all correct questions, the midterm had 8 short answer questions. In the fall, these short answer questions tripped up a number of students, primarily because they did not answer them completely, directly, or with enough detail. We received a number of complaints from them and my "grade czar" had to deal with many dissatisfied students who did not understand why they had lost partial credit for vague and incomplete answers. This time, I made a short Echo video reviewing my expectations for the short answer questions and giving them sample questions and answers, with explanations. My head TA also did a lot of work with practicing the short answer questions during his weekly Supplementary Instruction sessions. Finally, we made available to the students a very long list of potential short answer questions. It was nothing that wasn't also available last semester--but this time, I said explicitly that all questions would be drawn from the (exhaustive) list, albeit perhaps in recombined form. By telling them that work on this document would be rewarded, they focused their attentions on answering the long list of questions rather than speculation. In Fall 2012, I told them that the questions would be drawn from in class questions and review questions embedded in the lectures; but, for some reason, this was never enough to motivate them to assemble a list and learn the material. By assembling the list, distributing it to the class, and encouraging them to create a Google doc and pool knowledge, we were able to get them to focus their energy on study rather than fretting.
In the days leading up to the exam, I also observed some notable changes in learning behavior. First, the class discussion board--Piazza--was very quiet. In the past, this was a place where students would pose questions whose answers could easily be found with a bit of effort. This had stopped (perhaps in part because I had warned them against doing so in the syllabus). From what I gather, Facebook was also fairly quiet. Instead of the usual pre-midterm anxiety venting that was a hallmark of the fall class, this group seemed to be studying. Their anxiety levels were noticeably lower (the teaching team was not inundated with emails) and they seemed to have a clear grasp of what to expect. The efforts I made to convey expectations as transparently as possible, as well as the practice they had with the quizzes, seemed to pay off.
Typically, in the 48 hours before an exam, there is a big spike in Echo views of recorded lectures. We waited and waited for the spike to come, but it never did. In addition, what we did see was the students going back into the recorded class sessions to check details or refresh their memories. I was delighted by this behavior--it was exactly what I wanted. In the fall, I pleaded with students to go back to the recorded class sessions, to recognize that exam questions were coming from class and prepare by reviewing the recordings. Few of them ever did this. Without a word from me, this spring cohort is using the recorded class sessions exactly as they were intended (and making it worth the expense and effort to record class meetings).
I am learning a lot from my current cohort of students. Two things stand out: the value of frequent, low-stakes assessments for supporting learning (and, more generally, a positive experience) in a large enrollment class; the importance of lecture--not so much as a means of knowledge transfer as much as a tool for orienting students, letting them get to know me, feel a connection to me, and experience my enthusiasm for ancient Rome.
We are now heading swiftly towards the second midterm. This is a tough exam--it's the part of the class that led me to flip the class in the first place; and inspired me to toughen up the first section of the course. Thus far, quiz grades have not dropped off at all and the students themselves have observed that the material is getting more difficult and requiring more work. In the past, I tried to warn classes about the increased difficulty level to no avail--midterm grades always dropped by 10 points. I have a feeling that this cohort will finally break that pattern, and I will not have said a word to them about the increased difficulty. They have figured it out themselves and adjusted their effort accordingly.
Monday, February 25, 2013
MOOCs and the Liberal Arts
In his recent Salon article, Andrew Leonard put in words something that has been nagging at me for months, namely, the fact that humanities disciplines are badly served by the higher ed MOOC hype. It has been common to analogize the rise of the MOOC (and, more generally, the push to deliver more online courses as a way of lowering costs) to seemingly similar revolutions in the music and news industries. As Leonard perceptively observes, however:
"it’s become clear to me that there is a crucial difference in how the Internet’s remaking of higher education is qualitatively different than what we’ve seen with recorded music and newspapers. There’s a political context to the transformation. Higher education is in crisis because costs are rising at the same time that public funding support is falling. That decline in public support is no accident. Conservatives don’t like big government and they don’t like taxes, and increasingly, they don’t even like the entire way that the humanities are taught in the United States."
I want to emphasize Leonard's key insight: the transformation in higher education that is being touted has a decidedly political context that has been missing from transformations of other industries. Whereas changes in the music and newspaper business models were driven by economics, those in education are largely being driven by politics (for a similar argument on the economics that are driving the higher education revolution, see Predatory Privatization: Exploiting Economic "Woes" to Transform Higher Education). As well, the education of a populace has long been considered a social good in a way that the supply and medium of music or news delivery is not. Thus, when The New York Times declares 2012 "The Year of the MOOC" and proclaims the rapid proliferation of these "courses"--and, more to the point, the emergence of businesses like Coursera and EdX--a game-changer, we need to think long and hard about what this means (see also Audrey Watters' excellent overview). This is not a value-neutral declaration; and the consequences for the landscape for higher education are potentially vast.
A lot has been written, and more is published each day, about the shortcomings of MOOCs to match the quality of a class delivered in a "bricks and mortar" classroom. The dropout rates are massive (approaching 90% for most courses) and there's not a lot of quality control. Some MOOCs are high quality while others seem to function more like teasers to attract applicants to the university or college that produced the course; to sell copies of the instructor's book(s); and to attract donations from alumni. A great deal of a course's success depends on the instructor's ability to adapt the content and delivery of content for an enormous and international audience. It also depends on the platform's suitability to the course's content. The platforms were developed for math, computer science, and "hard" science courses and are reasonably good at supporting that content. As well, the MOOC environment is particularly well-suited to the sort of course that has clearly correct and incorrect answers; and where much of the learning can (but probably should not) be reduced to memorizing and applying formulas.
A reasonably intelligent and self-motivated student can buy the textbook, follow the recorded explanations of concepts, and then apply those concepts to homework assignments. Unclear concepts can be reasonably well-explained by peers, in part because it is reasonably clear when someone knows what s/he is talking about. A peer might not walk a student through a difficult problem as smoothly or straightforwardly as an instructor, but I suspect that this process happens with success on a regular basis. Reflecting on my own experiences with math, physics, and chemistry in college, I can also imagine that a peer sometimes has an easier time of identifying and clarifying misunderstandings because they, too, just worked through the exact same set of steps. In part, though, the success of this process depends on the fact that it is also obvious when a peer is NOT able to be helpful (because they fail to arrive at the correct answer). It also depends on the fact that solutions follow a predictable set of steps.
The process of learning in an introductory programming or math course is entirely different from that in an introductory writing or even psychology course. It's not that math/science courses have objective answers while the social sciences and humanities are subjective. It's that there are sometimes more than one correct answer; and the process of arriving at correct answers can vary from student to student. In addition, students often struggle to distinguish between good and poor advice from their peers in humanities courses. They are more prone to persuasive rhetoric because they are less familiar with the facts of an issue--indeed, if they already knew the facts, they wouldn't need to be taking the course. The learning process differs in significant ways, yet the MOOC platforms and modes of delivery don't account for this at all. If Blackboard tries to do too much, to be everything for every course, the Coursera and EdX platforms don't do enough (yet) to support the delivery of a humanities MOOC that is more than a kind of living textbook.
The most significant challenge for humanities instructors is the role of written "free response" to the learning process, whether in the form of single words; short answers; or longer essays. Those of us who use peer review in our courses can only stand agape when the Coursera founders naively suggest it as the solution to grading written work (Andrew Ng discussed this during a lecture at UT Austin in March 2013, c. min. 16). Certainly, peer review can work very well, but it requires an immense amount of structure and oversight in relatively homogenous student groups. In much larger groups, it's simply not a good approximation of instructor feedback (Audrey Watters on the problems with the Coursera peer review model). The point of peer review isn't about grades--preliminary studies indicate that peers do a reasonable job on this front. It's about providing sustained and thoughtful critiques that encourage deeper thinker and learning on the part of the writer. Sure, some learning happens in the process of writing; but most of it happens in the process of reading and responding to insightful comments. Peer review can be part of this process; but in a decade of using it as part of writing assignments at all levels, I've never seen it come close to replacing the comments of the instructor (or even teaching assistant) in an undergraduate course--and even in a graduate seminar, it is a nice supplement to rather than replacement of the specialist instructor's comments.
The implications of the lack of fit between current MOOC platforms and humanities content are significant. If MOOCs are here to stay, as seems to be the assumption, then humanities faculty need to take this problem seriously. They need to recognize the political dimensions to the rise of the MOOC; and recognize the potential threat these pose to the survival of the humanities at so-called state-funded institutions. They need to take seriously the challenge to develop functionalty that allows for the delivery of a serious and pedagogically sound humanities course to a large audience. This does not necessarily require the use of the existing platforms. Indeed, humanities faculty might well opt to develop their courses on a different platform, for instance, Instructure's Canvas LMS. We can hope that an enterprising, humanities-focused company might come onto the scene with a platform that better serves our needs.
If we do not take this problem seriously, however, we run the risk of sending the message to the public at large that humanities topics are merely fun and light (the college student's "blow off course"); and don't involve complexity of thought, analysis, and real learning. At the moment, MOOCs on humanities topics tends towards the sort of class that appeals to lifelong learners and those who want to do something fun but intellectual in their spare time. This is not because these instructors are lightweights or because their "brick and mortar" classes aren't serious and demanding. It's because the current functionality of the Coursera and EdX platforms is not sufficient for the kinds of things that support real learning in a humanities course. The answer, I think, is not to lament this failing but rather, to work actively with Ed Tech experts to figure out how we might deliver a reasonable approximation of what we do in the classroom to a larger audience.
***********
6/8/2013: Cara Reichard, MOOCs Face Challenges in Teaching Humanities ("Richard Saller, dean of the School of Humanities and Sciences, suggested that there are certain qualities of the humanities that are better suited to an intimate classroom setting than to a massive online format. “The humanities have to deal with ambiguity [and] with multiple answers,” Saller said. “The humanities, I think, benefit hugely from the exchange of different points of view [and] different arguments.”)
************
6/12/2013: Posthegemony revisits this topic of MOOCs and the Humanities, way that they are not a good fit.
"it’s become clear to me that there is a crucial difference in how the Internet’s remaking of higher education is qualitatively different than what we’ve seen with recorded music and newspapers. There’s a political context to the transformation. Higher education is in crisis because costs are rising at the same time that public funding support is falling. That decline in public support is no accident. Conservatives don’t like big government and they don’t like taxes, and increasingly, they don’t even like the entire way that the humanities are taught in the United States."
I want to emphasize Leonard's key insight: the transformation in higher education that is being touted has a decidedly political context that has been missing from transformations of other industries. Whereas changes in the music and newspaper business models were driven by economics, those in education are largely being driven by politics (for a similar argument on the economics that are driving the higher education revolution, see Predatory Privatization: Exploiting Economic "Woes" to Transform Higher Education). As well, the education of a populace has long been considered a social good in a way that the supply and medium of music or news delivery is not. Thus, when The New York Times declares 2012 "The Year of the MOOC" and proclaims the rapid proliferation of these "courses"--and, more to the point, the emergence of businesses like Coursera and EdX--a game-changer, we need to think long and hard about what this means (see also Audrey Watters' excellent overview). This is not a value-neutral declaration; and the consequences for the landscape for higher education are potentially vast.
A lot has been written, and more is published each day, about the shortcomings of MOOCs to match the quality of a class delivered in a "bricks and mortar" classroom. The dropout rates are massive (approaching 90% for most courses) and there's not a lot of quality control. Some MOOCs are high quality while others seem to function more like teasers to attract applicants to the university or college that produced the course; to sell copies of the instructor's book(s); and to attract donations from alumni. A great deal of a course's success depends on the instructor's ability to adapt the content and delivery of content for an enormous and international audience. It also depends on the platform's suitability to the course's content. The platforms were developed for math, computer science, and "hard" science courses and are reasonably good at supporting that content. As well, the MOOC environment is particularly well-suited to the sort of course that has clearly correct and incorrect answers; and where much of the learning can (but probably should not) be reduced to memorizing and applying formulas.
A reasonably intelligent and self-motivated student can buy the textbook, follow the recorded explanations of concepts, and then apply those concepts to homework assignments. Unclear concepts can be reasonably well-explained by peers, in part because it is reasonably clear when someone knows what s/he is talking about. A peer might not walk a student through a difficult problem as smoothly or straightforwardly as an instructor, but I suspect that this process happens with success on a regular basis. Reflecting on my own experiences with math, physics, and chemistry in college, I can also imagine that a peer sometimes has an easier time of identifying and clarifying misunderstandings because they, too, just worked through the exact same set of steps. In part, though, the success of this process depends on the fact that it is also obvious when a peer is NOT able to be helpful (because they fail to arrive at the correct answer). It also depends on the fact that solutions follow a predictable set of steps.
The process of learning in an introductory programming or math course is entirely different from that in an introductory writing or even psychology course. It's not that math/science courses have objective answers while the social sciences and humanities are subjective. It's that there are sometimes more than one correct answer; and the process of arriving at correct answers can vary from student to student. In addition, students often struggle to distinguish between good and poor advice from their peers in humanities courses. They are more prone to persuasive rhetoric because they are less familiar with the facts of an issue--indeed, if they already knew the facts, they wouldn't need to be taking the course. The learning process differs in significant ways, yet the MOOC platforms and modes of delivery don't account for this at all. If Blackboard tries to do too much, to be everything for every course, the Coursera and EdX platforms don't do enough (yet) to support the delivery of a humanities MOOC that is more than a kind of living textbook.
The most significant challenge for humanities instructors is the role of written "free response" to the learning process, whether in the form of single words; short answers; or longer essays. Those of us who use peer review in our courses can only stand agape when the Coursera founders naively suggest it as the solution to grading written work (Andrew Ng discussed this during a lecture at UT Austin in March 2013, c. min. 16). Certainly, peer review can work very well, but it requires an immense amount of structure and oversight in relatively homogenous student groups. In much larger groups, it's simply not a good approximation of instructor feedback (Audrey Watters on the problems with the Coursera peer review model). The point of peer review isn't about grades--preliminary studies indicate that peers do a reasonable job on this front. It's about providing sustained and thoughtful critiques that encourage deeper thinker and learning on the part of the writer. Sure, some learning happens in the process of writing; but most of it happens in the process of reading and responding to insightful comments. Peer review can be part of this process; but in a decade of using it as part of writing assignments at all levels, I've never seen it come close to replacing the comments of the instructor (or even teaching assistant) in an undergraduate course--and even in a graduate seminar, it is a nice supplement to rather than replacement of the specialist instructor's comments.
The implications of the lack of fit between current MOOC platforms and humanities content are significant. If MOOCs are here to stay, as seems to be the assumption, then humanities faculty need to take this problem seriously. They need to recognize the political dimensions to the rise of the MOOC; and recognize the potential threat these pose to the survival of the humanities at so-called state-funded institutions. They need to take seriously the challenge to develop functionalty that allows for the delivery of a serious and pedagogically sound humanities course to a large audience. This does not necessarily require the use of the existing platforms. Indeed, humanities faculty might well opt to develop their courses on a different platform, for instance, Instructure's Canvas LMS. We can hope that an enterprising, humanities-focused company might come onto the scene with a platform that better serves our needs.
If we do not take this problem seriously, however, we run the risk of sending the message to the public at large that humanities topics are merely fun and light (the college student's "blow off course"); and don't involve complexity of thought, analysis, and real learning. At the moment, MOOCs on humanities topics tends towards the sort of class that appeals to lifelong learners and those who want to do something fun but intellectual in their spare time. This is not because these instructors are lightweights or because their "brick and mortar" classes aren't serious and demanding. It's because the current functionality of the Coursera and EdX platforms is not sufficient for the kinds of things that support real learning in a humanities course. The answer, I think, is not to lament this failing but rather, to work actively with Ed Tech experts to figure out how we might deliver a reasonable approximation of what we do in the classroom to a larger audience.
***********
6/8/2013: Cara Reichard, MOOCs Face Challenges in Teaching Humanities ("Richard Saller, dean of the School of Humanities and Sciences, suggested that there are certain qualities of the humanities that are better suited to an intimate classroom setting than to a massive online format. “The humanities have to deal with ambiguity [and] with multiple answers,” Saller said. “The humanities, I think, benefit hugely from the exchange of different points of view [and] different arguments.”)
************
6/12/2013: Posthegemony revisits this topic of MOOCs and the Humanities, way that they are not a good fit.
Wednesday, February 20, 2013
Midterm #1: The No-Cram Exam?
The first midterm for my "stealth-flipped" Intro to Rome class took place yesterday. I was dreading this first exam, not only because I needed to make sure that the exam itself was more challenging than than one I gave in the fall; but because the 3-4 days before the exam were going to tell me how well my stealth flip was working to encourage a "learn as you go" approach. It would also tell me a lot about how well I'd done to manage the students' expectations for the exam. I focused a lot of energy on defusing potential anxieties: I created a short video about the short answer section of the exam (something that they had not done before in the class); the TA who is running Supplementary Instruction sections focused the week's meetings on preparation for the short answer questions; and I gave the students a list of all potential short answer questions and encouraged them to work collaboratively on it as a Google Doc. I reminded them that the exam questions might be worded differently, but that the questions would test the same content. I wanted them to focus their energy on study rather than fretting, speculating, and anxiety-posting on Facebook.
I suspect I was more nervous for the exam than some of my students. In the fall, I could accurately predict class performance on an exam from the lecture viewing data. The steeper the spike in the 48 hours before the exam, the lower the scores would be. I felt like I was watching a car crash unfold in a dream, completely helpless to change the course of events. (I wrote about the most spectacular of the two crashes here). As it turned out, the current class behaved nothing like the fall class. The weekend passed quietly with few questions posted on Piazza--and all of them legitimate and reasonable questions rather than the "I am too lazy to look this fact up" sort. Nobody on the teaching team, including me, was inundated with emails. When I received data about their viewing of pre-recorded and in class lectures, there were no notable spikes. Throughout the day on Monday, I was waiting for a spike that never came.
Instead, by all appearances, the students were refreshing knowledge that they already knew pretty well; and working through the study tools I provided (practice multiple choice questions for the content not tested by a quiz + potential short answer questions). Starting on Saturday, they were hard at work on the Google Doc, refining and adding details (and sometimes acting like trolls, but that's a topic for a separate post). Their behavior suggested that they knew what to expect on the exam and that they were focusing their energy on study rather than anxiety-driven, unproductive speculation. From what I heard, there wasn't much chatter on FB and Piazza remained almost completely silent throughout the lead-up to the exam yesterday afternoon.
I don't yet know how the students performed on the exam, but I am optimistic. It was a difficult exam--intentionally so, because I don't want them to get overly confident and then perform poorly on the next exam, which covers much more complex content. To be honest, I am a bit in shock that--at least so far as the data suggests--the class as a whole was not cramming. I'm sure some students did cram, but the vast majority clearly did not. Hopefully, they realized that the weekly quizzes kept them on top of their learning for the class and that preparation for the exam wasn't nearly so stressful because of this regular preparation along the way. Certainly, their behaviors before and during the exam suggest that, on the whole, they knew what to expect and what to do to prepare. They clearly felt oriented to the course expectations and understood that it was on them to put in the work to learn.
We are planning to do an exam wrapper after we return the exams next week, primarily to encourage the students to actively reflect on their study and learning process (but also to collect some preliminary feedback about their experience in the course thus far). I am eager to see what they say, to see if their own sense of their preparation process matches up with my observations of their behavior. I can say with a lot of confidence that this group approached the midterm in a completely different fashion from the group in the fall. I won't declare the stealth flip a success until we are deeper into the semester (and until their time management skills are really being challenged). Nonetheless, it does seem that the combination of weekly quizzes and providing a very long list of potential short answer questions has had a remarkable effect on changing student learning behaviors--and anxiety levels. Most of all, it lets students who have little experience with learning in the humanities know what to expect and how to study.
I can tell them that it is not possible for 90% of students to do well in my course with a "cram for the exam" approach until I am blue in the face, but everyone wants to believe they are in the select 10% who *can* learn this way. This false belief has been reinforced repeatedly during their secondary and even post-secondary education. As well, many (most?) of them lack the time management skills to discipline themselves to learn regularly even if they know better unless a grade is not involved. It is absolutely fascinating to me to see that, by creating an assessment structure that rewards a "learn as you go" approach; making a very big effort to orient the students in the class; and training them in the methods of learning in a humanities/history course, I am seeing significant and positive changes in their behaviors and (so it seems) attitude. Knock on wood.
I suspect I was more nervous for the exam than some of my students. In the fall, I could accurately predict class performance on an exam from the lecture viewing data. The steeper the spike in the 48 hours before the exam, the lower the scores would be. I felt like I was watching a car crash unfold in a dream, completely helpless to change the course of events. (I wrote about the most spectacular of the two crashes here). As it turned out, the current class behaved nothing like the fall class. The weekend passed quietly with few questions posted on Piazza--and all of them legitimate and reasonable questions rather than the "I am too lazy to look this fact up" sort. Nobody on the teaching team, including me, was inundated with emails. When I received data about their viewing of pre-recorded and in class lectures, there were no notable spikes. Throughout the day on Monday, I was waiting for a spike that never came.
Instead, by all appearances, the students were refreshing knowledge that they already knew pretty well; and working through the study tools I provided (practice multiple choice questions for the content not tested by a quiz + potential short answer questions). Starting on Saturday, they were hard at work on the Google Doc, refining and adding details (and sometimes acting like trolls, but that's a topic for a separate post). Their behavior suggested that they knew what to expect on the exam and that they were focusing their energy on study rather than anxiety-driven, unproductive speculation. From what I heard, there wasn't much chatter on FB and Piazza remained almost completely silent throughout the lead-up to the exam yesterday afternoon.
I don't yet know how the students performed on the exam, but I am optimistic. It was a difficult exam--intentionally so, because I don't want them to get overly confident and then perform poorly on the next exam, which covers much more complex content. To be honest, I am a bit in shock that--at least so far as the data suggests--the class as a whole was not cramming. I'm sure some students did cram, but the vast majority clearly did not. Hopefully, they realized that the weekly quizzes kept them on top of their learning for the class and that preparation for the exam wasn't nearly so stressful because of this regular preparation along the way. Certainly, their behaviors before and during the exam suggest that, on the whole, they knew what to expect and what to do to prepare. They clearly felt oriented to the course expectations and understood that it was on them to put in the work to learn.
We are planning to do an exam wrapper after we return the exams next week, primarily to encourage the students to actively reflect on their study and learning process (but also to collect some preliminary feedback about their experience in the course thus far). I am eager to see what they say, to see if their own sense of their preparation process matches up with my observations of their behavior. I can say with a lot of confidence that this group approached the midterm in a completely different fashion from the group in the fall. I won't declare the stealth flip a success until we are deeper into the semester (and until their time management skills are really being challenged). Nonetheless, it does seem that the combination of weekly quizzes and providing a very long list of potential short answer questions has had a remarkable effect on changing student learning behaviors--and anxiety levels. Most of all, it lets students who have little experience with learning in the humanities know what to expect and how to study.
I can tell them that it is not possible for 90% of students to do well in my course with a "cram for the exam" approach until I am blue in the face, but everyone wants to believe they are in the select 10% who *can* learn this way. This false belief has been reinforced repeatedly during their secondary and even post-secondary education. As well, many (most?) of them lack the time management skills to discipline themselves to learn regularly even if they know better unless a grade is not involved. It is absolutely fascinating to me to see that, by creating an assessment structure that rewards a "learn as you go" approach; making a very big effort to orient the students in the class; and training them in the methods of learning in a humanities/history course, I am seeing significant and positive changes in their behaviors and (so it seems) attitude. Knock on wood.
Subscribe to:
Posts (Atom)