Friday, March 14, 2014

Five, Four, Three... We're Rolling!

After a summer off-camera, I stepped back into the film studio this morning to film a 90 second "sizzler" video for my Introduction to Ancient Rome course.  These are the education equivalent of the nmovie trailer, but we are advertising our class (and ourselves).  It also shows the extent to which the line between education and entertainment is blurring.  Once the sizzler is ready, it will be posted on YouTube.  It's not intended to advertise the campus-based, blended version of the course--that one fills up very quickly every semester and has a long waiting list.  Rather, this is to advertise an online version of the course, which I'll be developing on the Canvas platform for launch in Summer 2014.  The current plan is to open it to UT System students, but perhaps also more widely to auditors.

There's a lot of behind-the-scenes work that goes into creating one of these sizzlers.

Tuesday, September 10, 2013

Festina Lente: Re-imagining UT Austin


Over the next semester, I am going to be writing opinion columns about the landscape of higher education in Texas and the consequences for the UT Austin campus.  This is the first, introductory column, where I describe the project (and reference the Roman emperor Augustus!)  I'll be visiting classes around campus, talking to instructors, and getting feedback from students in an effort to try to figure out what works and what doesn't, in both online and onsite classrooms.  I'm very excited about this opportunity, not least because it will give me an excuse to learn a lot more about what is happening in campuses around the UT Austin campus.

Saturday, August 31, 2013

The Real Magic Formula: Formative Assessment


Earlier this week, Dr. James Pennebaker and Dr. Sam Gosling, two UT Austin professors in the Department of Psychology, launched a live-streaming online course that they are calling a SMOC (Synchronous Massive Online Course).  The nomenclature is a clear riff on the MOOC (Massive Open Online Course); but the SMOC differs in a couple of ways.  First, it is not open.  In fact, it costs $550 (still, less expensive than a classroom-based course at UT Austin).  It is massive by most standards, in that it enrolls about 1000 students (they are hoping to increase that number to 10,000 eventually)--but not the multiple tens or even hundreds of thousands who register for MOOCs.  Finally, as they highlight, it is live rather than pre-recorded.  Students are required to sign in at the start of class to take short "benchmark" quizzes.  Since there is no participation grade, it does seem that students can opt to sign out once they have finished the quiz and watch the archived recording of the class at their leisure to prepare for the next benchmark quiz.  The course does include moments for live chats and polls, in an effort to engage students during the live broadcast.  Still, as someone who teaches a similar audience, I wonder how many will, in fact, engage regularly during the live broadcasts, especially as the semester goes on.

The start of the course has been accompanied by quite a lot of publicity: an op-ed in the Houston Chronicle; an article in Inside Higher Ed; several press announcements from UT Austin; a television appearance on Good Day Austin; The Daily Texan (and here); and, now, articles in the Wall Street Journal and the New York Times.  There is also a Twitter feed for the class: @PsyHorns.  The press coverage emphasizes the innovation of the model, in particular, the claim that the live streaming broadcast as well as the delivery platform (an in-house product developed by UT Austin and aptly named Tower) facilitate connectivity and active engagement in a way that the MOOCs and other large enrollment online classes have not.  This is certainly a noble aspiration, but a lot depends on the students.  This is especially true since no part of the final grade is connected to engagement in the live broadcast.  I do hope that, at the end of the semester, Pennebaker and Gosling will release at least preliminary data about student engagement: how many logged off immediately after the quiz?  How many engaged in one poll or pod discussion?  all of them?  Are there connections between final grades and participation in the live broadcast versus reviewing the archived broadcast?  These are important questions for faculty who are designing their own version of an online course.

I am grateful that I've been given the chance to audit this course.  Mainly, I am interested in experiencing Tower from the student perspective, as I work through the design of Rome Online.  I am especially interested in the question of connectivity--how to facilitate connections to me but especially between students in a virtual classroom that may well contain hundreds of students.  I am looking forward to seeing how Pennebaker and Gosling make use of the Tower platform; and how they manage the strengths and weaknesses of a synchronous broadcast to a large audience (though, at least in the current iteration, not any larger than they've taught in previous years in various configurations: c. 1000 students).

One tool I am going to be watching closely is the benchmark quiz.  The Tower platform has a very nice quiz tool that I am hoping to use in my own online course.  I am eager to experience it from the student point of view.  As I discovered in my own rather large (but only 400 students) course last spring, a key element for improving student learning is structure--and especially, frequent quizzes.  This point is made clearly in the WSJ article: "Recently, they moved one class of students online and gave them multiple-choice tests that delivered instant feedback. That group performed better than their offline class—and the online students' grades even improved in their other classes. The professors hypothesize that it is because the regular quizzes helped the students acquire better study habits."

There's an important point here: improved student performance in a course (whether the Intro to Psychology or any other large enrollment course) is not a consequence of any technological bells and whistles but rather, an instructional design decision--to overhaul the class assessment and put much more emphasis (and weight) on quizzes rather than large stakes midterms.  This was the case for Intro to Psych; it was also true in my Intro to Ancient Rome class.  In large classes, regardless of mode of instruction, students require a lot of structure.  In my Rome class, I continued to give midterms, so that I had ways to compare the different cohorts.  I suspect that, if midterms were still offered in the Intro to Psych class, they'd find that the benchmark quiz cohort performed better than the midterms only cohort.  There's a pretty basic reason for this improved performance: quizzes require students to study as they go rather than try to cram a month's worth of learning into a few days.

One thing that would be interesting to know is whether, in a "benchmark quiz" cohort, there's any correlation between grades on quizzes and engagement during the live broadcast.  If there is, that's a good argument for the high cost of producing a live streaming broadcast.  If there's not, that tells us that the value is in the formative assessment rather than the mode of delivery.  It's these sorts of questions that are important to test as we move forward with offering online versions of our campus-based courses.

Friday, August 30, 2013

Rome Online Course: The Challenge of Connecting Students in an Online Environment

The building that houses my campus office also includes several classrooms.  Students like to congregate in the halls, waiting for the previous class to finish.  My first office was directly across from a classroom.  It was nearly impossible to get any real work done at work between 9 am-3 pm, between the chattering students and the sounds from the classroom.  At one point, our office manager put signs up, to limited effect, to remind students not to talk.

These days, it's pretty quiet in the halls, especially in the early days of the new semester.  Students are staring at the phones, not making conversation with those around them.  Two friends might be chatting, but otherwise, silence and intense focus on their smart phone.  This scene repeated itself as I arrived at the auditorium where I teach my Rome class.  As the students made their way into the room and found seats, I was heartened to hear them talking to one another--not everyone, but enough that it was pretty noisy.  There's something deeply interesting about the fact that, these days, we aren't worried about students talking to one another too much but, rather, too little.

In large enrollment courses, students will have a much easier time engaging in the course--and even showing up--if they feel a connection to their classmates.  Back in the olden days (i.e. when I was in college in the 90s), going to class was a social experience.  It was where we connected with friends, made plans for after class, and shared experiences that we then rehashed, parsed, and joked about--and continue to, even all these years later.  We walked around campus in packs, frequently stopping to chat with friends we encountered along the way.  Only the most anti-social of us walked around with a walkman and ear phones.  But times and social mores change.  Now, as instructors, we often have to actively work to encourage our students to see the classroom as a place for experiencing and benefiting from the social aspects of learning.  This is especially true on a very large campus like UT Austin, and in the very large courses on these campuses.  The default mode for students, unless they already know some of their classmates, is to isolate, be silent and passive.

By incorporating a student response system (i>clickers), peer instruction, class discussion and a discussion board; as well as encouraging group work on a wide range of other activities, I feel like I have done about as well as I can to push students to get to know their classmates, to take advantage of the power of distributed learning.  Some choose not to to engage, and I let them be.  But I make sure that they are deliberately choosing to opt out of an established norm rather than following the norm.  Certainly, it's possible to take my class without ever engaging much with classmates--or even showing up to class for more than the weekly quizzes.  But this means sitting along, watching videotapes of lectures, not sharing in the communal laughter.  Most students quickly realize that it's more fun to be there.

I am coming to see that a big--if not the biggest--challenge in developing an online version of my Rome class is finding a way for the students to feel connected to one another.  Sure, it helps a lot if they feel connected to the course instructor; but the real focus of the course design needs to be on finding ways to encourage the to connect to one another, and to facilitate that connection at the very start of the course.  Watching the "Doc on the Laptop" alone is isolating.  For anyone whose normative educational experience involves sitting in the same room as the instructor and their classmates, it's a constant reminder of distance, inaccessibility.  There is some thought that synchronous live streaming lectures help to lessen this perception of distance.  I'm not sure.  For many students, it might well heighten it.

The central challenge is overcoming the entropy of the massive (whether 400 or 10, 000 or 40,000).  In a large course, I think, it is realizing that the primal connection is that which exists between peers rather than student to instructor. Or, to put it a slightly different way, it's recognizing that the instructor is just another person.  It seems that, at least early on, a key element of any course platform has to be a well-conceived chat function, that can group students but also let each student know with whom they've been chatting and have a way for them to Direct Message one another.  In a synchronous class, perhaps it makes sense to group students before the class starts and ask them to introduce themselves to one another.  In an asychronous class, it probably makes sense to group students into discussion groups and have them respond to prompts, with an instructor or TA helping to guide the discussion.

I suspect it might also be easier for students to take advantage of the synchronous elements of a class if, first, they have done some form of asynchronous conversation with their peers.  That is to say, given the expense of producing a synchronous course, it probably makes sense to use that element sparingly and only when it is going to produce a lot of "bang for the buck."  From a pedagogical perspective, it's likely to be a lot more effective once the class is underway, once students have already had a chance to engage with one another (and the instructor) in various asynchronous formats.  For instance, I can imagine having steady groups for each unit of the Rome course.  And at the end of each unit, doing a synchronous broadcast that asked students to engage in polls, chat with one another, etc.  And then shuffling the groups for the next unit, with some people staying together.

The key, though, is a very sophisticated chat tool.  Discussion boards are useful, but as the size increases, the boards are populated by a lot of individuals posting their individual thoughts without much attention to engaging with the thoughts of their classmates.  This seems inevitable, and the only real solution is to break large classes into smaller groups.  This is also true for chat groups during a synchronous broadcast--any more than 4 or 5 people and it is a bunch of people typing, saying the same thing, not interacting with one another in any real way.  The time lag only exacerbates the problem.

Thursday, August 29, 2013

It's Magic!

Sebastian Thrun, Founder of Udacity: "The thing I'm insanely proud of right now is I think we've found the magic formula."

Despite nearly twenty years of teaching experience, first as a grad student and then a professor, I still get the jitters before the first class meeting.  This is especially true when I teach my ginormous Introduction to Ancient Rome class: 400 students and a range of previous experiences with learning history and reasons for taking the course.  From demographic surveys, we know that most of them are there to fulfill a core curriculum requirement; but, for whatever individual reason, they've chosen to take this class instead of some other class that fulfills that requirement.  I love walking in on the first day of class and seeing that so many students are going to be learning about ancient Roman history and culture.  They are going to be learning about the evidence historians use and how we evaluate it.  They are also going to be learning that you can't blindly trust everything you read or see.  You have to think about the larger historical and cultural context that produced that text or monument. 

The first few weeks are all about orienting the students to the many moving pieces of the course.  Most especially, they are about setting expectations for the level of active engagement with the content that I am going to expect from them inside and outside of class.  This requires a tremendous amount of thought, organization and hard work from me and my teaching team (four graduate student TAs).  Going into this fall semester, I feel pretty optimistic that my structure for the class as well as the "protocol" in place will, on the whole, work to motivate students to make good choices (e.g. stay on top of the course material; attend class meetings).  If they make good choices, chances are in their favor that they will learn more and better; and earn a higher grade than they would have if they were in my traditional lecture version of the course.

Still, at the end of the day, everything depends on the students.  I can set up a course that motivates and rewards good learning behaviors; but I can't make them learn.  All to say, there is no "magic formula" in higher education.  Or, perhaps, better said: the real magic formula is investing a whole lot more money in public education, so that we are not teaching 400 students at a time.  The "magic formula" is a strong and positive relationship between instructor and student.  When we are talking about teaching hundreds or thousands of students at a time, however, we can do our best to set them up for success; but learning is ultimately their responsibility.

As well, in actual college classrooms, we don't have the luxury of improving our learning outcomes by changing our student demographic.  There's nothing novel or magical here.  Highly selective colleges and universities learned this trick a long time ago: if you want to boast about the success of your graduates, only admit the very best and most prepared students.  It's hard work--and risky--to admit students who are less well-prepared, who need a lot of guidance from experts.  As university professors, we work with what we have.  I am lucky, in that the make-up of my students doesn't change a whole lot from year to year.  Still, one of the challenges of the large class is finding ways to identify and provide help for the students who are struggling.  It's also knowing when and how to intervene. 

I wish there was a magic formula for this, but there's not, at least not yet.  Maybe someday a magic algorithm will be able to predict early in the semester that a student is going to struggle.  In the meantime, my TAs and I have to use the many combined years of experience we have to work with the range of students that we have.  We do a pretty good job, but it's hands on, a lot of time and effort from us as well as the students.  Then again, I've not yet tried to wave my magic wand and cast a spell...

Monday, August 26, 2013

Rome Online Course*: Re-imagining my Presence (and Planning for my Absence)

As I begin to think seriously about the design for the online version of my Intro to Ancient Rome class, which we are planning to launch on Canvas in Summer 2014, I find myself thinking a lot about my own role.  The MOOC mania gave rise to what Jonathan Rhees has aptly termed a class of Super Professors--professors who were suddenly performing in front of audiences in the tens of thousands instead of the usual few hundred.  Oftentimes, these mostly (though not always) very senior faculty were well-known for their academic achievements and, because of the nature of the MOOC Incs, taught at very prestigious institutions.  They were front and center of their courses, as much because the main pedagogical format was content deliver via pre-recorded lecture.  That is to say, a student wasn't just learning about the ethics of justice; he was learning Michael Sandel's version of that content.  In courses like coding, calculus, or even physics, the outsized presence of the instructor was lessened by the nature of the content; but especially in liberal arts topics, the course could be as much about the instructor as the content.  Of course, this is also true for traditional lecture courses taught in a campus classroom.

As a sometime student of MOOCs and other types of online courses, it seems to me that this model, this emphasis on the instructor, gets it exactly backwards.  While it makes some kind of sense for an instructor to rely on the personal charisma generated by their presence in a face-to-face class or even a blended class, it makes little sense to do it in a purely online class.  The absent-presence of the instructor speaking at the student through a computer is a constant reminder of the distance that separates instructor from student.  It is a constant reminder that, however accessible the professor seems, s/he is ultimately inaccessible.  People will argue that this is no different than the experience of sitting in a 500-student classroom, but I'm not so sure.  The more I teach a very large (400 student) class, the more convinced I am that a good design for an online course will move away from being an imperfect substitute for an interactive lecture course; and move towards something entirely different.

My own sense is that this different thing needs to have a place for the instructor to construct and practice his/her authority; but that, in some real sense, it's important to concede the point that no amount of fancy computer mediation, not even live streaming, can replicate the experience of being in a classroom with us.  I;m not sure I'd have believed this until I taught a pure flipped class last fall.  The students watched all the lectures outside of class; and class time was devoted to practicing the content.  About 30% thought this was great.  Those are the ones who would be a natural audience for an online class, I think.  But the other 70% were somewhere between hated it and "meh".  Those are the students who need us to take seriously the differences of online vs f2f course delivery.  There is already a rich body of research on this topic, and those of us who are or are planning to teach online courses need to know it and take account of it in our course design.

The other important thing that our course design needs to take account of is the possibility that we might not be the instructor.  In my own case, for instance, I plan to teach a few iterations of the course but then, most likely, will hand it off to others to teach most of the time.  Planning for this is extremely important and, again, suggests that less reliance on pre-recorded lectures from a single instructor is the way to go.  As I begin to make concrete plans, I am thinking hard about a. how to deliver content in ways other than lecture; and b. how to involve a number of voices in the content delivery, so that it doesn't seem to students like I am somehow "MIA" if I am not the instructor.  Having students construct and master content through modules and reserving lecture only for very difficult or amusing topics is one clear answer. I am also thinking about how to build into the design places for other instructors to incorporate their own material.  I am thinking about how to balance synchronous and asychronous elements of the course.

Part of the sustainability of my course depends on design decisions made at the start; and it depends on the recognition that student learning nearly always requires some sort of relationship between instructor and student--a relationship that is impeded if someone else is always appearing on the screen as the sage on the stage.  For this same reason, I am extremely skeptical that licensed MOOCs will be all that effective in the long run for subjects like mine.  Students don't put nearly as much weight on an instructor's professorial rank and status as most faculty and administrators do.  Most of them don't know the difference between a lecturer and a full professor, nor do they care.  They care that their course is taught in a way that facilitates their learning and, hopefully, provides an enjoyable experience.  Part of this experience is the connection they forge with the course instructor, whether during office hours on campus or via email and discussion boards and Google Hangouts in an online course.  It is essential than any online course design provide space for a new instructor to put their own imprimatur on a course, establish their authority, without the constant distraction of an absent, inaccessible sage on the stage.

*ROC (Rome Online Course).  I am planning to write regularly about my plans for developing my Intro to Rome class for an online audience, including the particular challenges of planning from the start for multi-modal delivery; and for handing the instruction over to others on a regular basis.

Sunday, August 25, 2013

MOOCs, SMOCs, Super Professors and Online Course Design

This weekend, two of my colleagues at the University of Texas, Austin wrote about their Introduction to Psychology SMOC (Synchronous Massive Online Course) in the Houston Chronicle.  For $550, anyone, anywhere can register for their course and earn credit from UT Austin.  The selling point of the SMOC over the MOOC is personalization, both via an "unparalleled teaching platform" designed in-house (known as Tower); and through peer discussion led by a peer mentor (a student who has already taken the course): "While the thousands of students watch our live lectures, they will each be in small "pods" with other students to discuss the class as it unfolds. Each pod will have a mentor who took the class last year."  As in most MOOCs, the instructors will lecture, but live rather than pre-recorded.  Unlike most MOOCs (though more are experimenting on this front), the class session will involve discussion amongst the "pods" of students and, presumably, some sort of computer-mediated "conversation" with the instructors.  What this all looks like in practice, and how well the technology holds up at scale, remains to be seen.  This much is clear: the course design of this SMOC aims to be more interactive than at least the stereotypical MOOC; but it retains essentially the same role for the instructors--and, more importantly, keeps them fairly removed from the individual students.  The may be "world famous instructors" but nearly every student in the course will experience them in the same way they experience a TV performer--at a substantial distance.

In reality, there's not much difference between this SMOC and a MOOC like Al Filreis's ModPo from Coursera, which films live discussions, takes questions, etc. Indeed, the main difference is that this SMOC is not "open".  Access is limited to those who pay the $550 registration fee.  The SMOC offers credit based on student performance on "benchmark quizzes" before each class whereas the MOOC offers non-credit "certificates of achievement" based on student performance on quizzes and peer-graded assignments.  A point that the instructors emphasize in their advertising for the course is the absence of large stakes assessments and the associated stress.  [Sidenote: I am a big fan of low-stakes assessments, but I'd like to see the midterms retained in the initial stages of developing this delivery model, as much so that it is clear to possible critics of the method that the students are learning as much and as well as students in other versions of the course.  This seems especially important at this point since, despite best efforts, it's still very difficult to control for cheating on these quizzes.  Students tend to be very good at finding ways around our best obstacles.]