Monday, May 25, 2015

Some Challenges in Implementing Technology in Teaching


I love Twitter for many reasons, but most of all because, in my own experience and use of it, it has been an excellent way to connect with people and talk through ideas.  I learn something new every single day.  I tend to avoid fights, though I do frequently express my views on controversial subjects.  For me, Twitter has been a space for networking with a range of people who share my interests in education technology, digital pedagogy, and higher education policy.  It's also been a great space for thinking through and adding nuance to my views through dialogue with smart interlocuters.

Yesterday I wrote a blog post about the need for faculty to get up to date on education technology.  Implicit in my post was the view that classroom technologies like student response systems make it much easier to implement best practices in our teaching.  It's not that I want faculty using technology for the sake of technology; but rather, that education technology is a crucial tool for updating one's pedagogy to integrate current best practices.  The growing array of ed tech tools--which seems to expand by the day--makes it easier for us to do our job well, but it requires faculty to be aware of what is out there and figure out how to use it.

To give just a few examples: student response systems are a fantastic way to check comprehension and evaluate what topics need clarification, what topics are already well-understood in large enrollment classes (and even in small classes).  I use i>Clicker questions to tell me what I need to spend time on in class.  Instead of just running through my prepared lecture, I figure out what students understood from their pre-class work, and what they didn't.  I then spend the majority of my time on the parts they didn't understand.  It requires a lot of improvisation, and has completely changed the way I teach large classes.
 i>Clickers are especially helpful because they are anonymous and you get a response from all students, not just the few who raise their hands. 

A second example: we know that practice and rapid feedback are crucial for learning.  This can be entirely automated by creating a database of practice questions that include feedback for the students.  These practice questions let students practice answering questions; and they let them know quickly where they need to do more review.  In addition, well-written questions can push the students to make connections across the course.  In all of my classes now, I have a database of automatically graded questions (MC, matching, ordering, etc.) for the students to practice on and build their mastery of the course content.  This small change, which required a substantial initial investment of time, has done wonders for student learning.  Most obviously, it has meant that students can know do much more higher-order thinking in class and in other activities.

The incredible array of ed tech tools  is all great if you are fortunate enough to teach at a university where a. your classrooms are "smart," i.e. equipped with the technologies you want to use, like lecture capture; and even a good projector or Smart Board.  As well, if you want students working online during class, the room needs to be able to support those wireless connections.  In large enrollment courses, this can be a huge challenge.  Even at UT Austin, we only have a few classrooms that can reliably support more than 100 or so wireless connections at one time.  This should change when Google Fiber comes to campus; but UT is, in this respect, a very advanced campus.  More typically, austerity has meant significant cuts to IT budgets, and this has had serious consequences for classrooms.  At many non-elite, public universities, classrooms are seriously out of date.

And then there's the question of accessibility for students.  As Megan Kelly gently reminded me this morning on Twitter, i>Clickers are an additional expense for students, one that many don't want to take on.  In addition, especially in more rural areas, students don't necessarily have reliable access to a broadband connection.  Even if our campus libraries and such provide that access, it may not be feasible for a commuter student who is working 30-40 hours/week to remain on campus to do their homework and studying.  Even at UT Austin, this issue occasionally crops up.  The first semester that I taught my Introduction to Ancient Rome as a blended class, I had a lot of students who lived in the campus dorms.  They paid for broadband based on their usage at that time and many were upset to be using their broadband bytes for school instead of gaming or Netflix.  Again, no matter how eager an instructor might be to integrate digital assets into their course, it may not be feasible if the students can't easily access the online content.

As Megan noted, in the classroom there are some ways around these limitations, using analog tools that let us implement the same pedagogies.  I will say, when I teach on campus, I always opt for analog over digital--partly because we are still at a stage where things go wrong with the digital.  So, for example, I give low stakes, weekly quizzes.  It would be much, much easier to do these online but I can't assume that all students have a device to get online; nor can I assume that the wireless will support 200+ students at one time, especially on a timed quiz.  Rather than deal with endless tech issues, I use scantrons.  It's more work but it saves the hassle of constantly troubleshooting tech issues.

I am also a huge fan of student response systems in class.  But there are a lot of ways to do this, including making laminated cards of different colors and having students hold these up.  I ended up not using them, but I have a box of laminated cards, four colors held together my a ring.  I was planning to hand these out to students at the start of the semester and then collect them at the end.  These analog methods are not quite as easy as using an i>Clicker but it accomplishes the same goal.  I also do a lot of "talk to your neighbor" with complex questions and then have the groups click in with their response.  This could also be done with laminated cards.

One of the things that my conversation with Megan clarified for me was that, in pushing my faculty colleagues to get up to date on education technology, what I was really advocating was that they get up to date on the latest pedagogy, as they are informed by the learning sciences.  It's always about the pedagogy, not the technology.  Ed tech tools can make it a lot easier to integrate good pedagogical techniques into our classes--most especially in the area of providing immediate feedback and frequent practice.  But ed tech does not replace pedagogy; and much of what ed tech does can be accomplished with analog tools.

At the same time, it is crucial that universities invest in keeping their IT and classroom technology up to date.  This also seems like an area where foundations ought to be looking to provide financial assistance.  In addition, libraries should be places where students can go to do any work that requires a quick and reliable broadband connection.  This doesn't entirely solve the problem for our students, many of who are commuting to campus and working long hours off campus; but it at least provides the option.

Foundations like the Gates Foundation, I have an idea for you: provide stipends for students who don't have access to broadband at home.  Make it possible for them to sign up for broadband so long as they are registered students.  Similarly, it must be possible for companies like i>Clicker to provide under-resourced public universities, especially in more rural areas, with used i>Clickers (e.g. first generation i>Clickers).  These first generation clickers are perfectly functional for what most of us need and it would be a great service to students and instructors.  Just a thought...

Sunday, May 24, 2015

Ignorance is Not an Excuse

I've been reflecting a lot on the issue of professors and education technology of late.  In particular, I've been thinking about the exact issue raised by the tweet pasted above: we have reached a point where it is no longer acceptable to be unable to use classroom technology and the extensive suite of learning tools available to support student learning.  We are horrified if a professor shows up to class unprepared; yet regularly turn a blind eye to colleagues who refuse to invest the time and energy necessary to master the latest ed tech tools and research; and integrate them into their classrooms.  Let me be clear: I'm not suggesting that all faculty need to learn how to code.  Rather, I'm suggesting--to give one obvious example--that it is malpractice to continue to teach by lecture and high stakes midterms when we now know that there are many other, more effective ways to teach large classes; and we now have the technological capacity to implement at least some of those tools (e.g. online practice quizzes; pre-recorded content lectures; student response systems like i>clicker).

Likewise, if one is in any kind of leadership role, with decision-making authority, it is simply inexcusable to be unaware of the vigorous conversation about the role of education technology in higher education.  In particular, it is essential for our college/university leaders to grasp what is at stake when a decision is made about, for instance, outsourcing online course development to the private sector.  Higher education, especially public education, has become deeply politicized.  Debates about the role of education technology are central to the politics of higher education, in part because there is a lot of money at stake.  As campus IT evolves into a complex (and expensive) part of the campus infrastructure, more and more resources are directed away from instructional budgets and towards IT departments (in various configurations).  This includes the creation of entirely new units in such areas as Learning Sciences and Data Analytics.  This isn't a bad thing; it has the potential to be quite a good thing.  But it is a big change.

This reallocation of resources is  largely invisible to most teaching faculty and even department chairs; yet an awareness of it is essential.  For one thing, it might finally demonstrate to departments that our universities are in the midst of significantly redefining the way they accomplish their mission.  Rank and order faculty remain ignorant of this important conversation at their own risk.  Departments with graduate programs have an ethical obligation to ensure that their graduate students are trained in the latest best practices, including a strong knowledge of online course design and implementation; and a basic knowledge of important ed tech tools.  The graduate students of today will be expected to have this skill set when they apply for jobs, and many of them will be asked to teach both in classrooms and online.  It is inexcusable to not prepare those who pursue academic jobs for this reality.

For many decades, it was possible for faculty, and especially for our department leadership, to be disconnected from the larger, national discussions about higher education.  It is surprising to me how few faculty have any real sense of the vigorous debates currently happening, especially around the issue of the private sector elbowing its way into both K-12 and higher education.  It is simply impossible for departments (and Colleges within universities) to make wise decisions about where to direct resources if you don't understand this encroachment by the private sector--and what factors are fueling it.  Most faculty have little sense of where their students are taking classes.  They don't realize that, over the past three years, the number of students taking courses outside of the university--either online, at community colleges, or even at other 4 year institutions--has increased exponentially, to the point that a college like mine (Liberal Arts) has already outsourced an unacceptable amount of our General Education/Core Curriculum courses.  We should be having vigorous conversations about how to reverse this trend, including how to integrate our own, high-quality online courses into our curriculum.  It is frustrating to me to see that, if these conversations are happening, they are happening at very high levels and largely exclude faculty.  It is equally frustrating, though, to see my faculty peers express so little interest in these important conversations.

I appreciate that faculty are busier than ever these days.  It would be helpful if institutions did more to ensure that faculty are educated about policy conversations but also, that faculty are held accountable for using best practices in their classrooms.  In any other profession, an individual who refused to remain up to date on basic technological developments would no longer be able to work in that field.  This has become a real issue in medicine, as more and more sophisticated diagnostic tools come out (e.g the Da Vinci robot for abdominal surgeries).  Physicians in procedure-intensive specialties regularly have to learn how to use new devices--otherwise they would lose patients and business.  Likewise, it would be considered malpractice for a physician to be unaware of the latest research and medications for various conditions.  Imagine a rheumatologist who ignored all the new studies about the dangers of long-term use of prednisone and continued to prescribe it at high doses to his patients!

Out of date or sub-par teaching won't kill anyone, to be sure. Eventually, though, students and their parents will become more savvy and less willing to tolerate poor learning experiences.  The result won't be that tenured faculty are fired for poor teaching, however.  Instead, the persistence of faculty resistance to ed tech and best practices will be used in higher education debates to persuade the leaders of our institutions that the teaching of, especially, first and second year courses can't be left to the luddite faculty who refuse to update their practices.  This will (and, in some places, already has) lay the foundations for outsourcing these courses to private-sector companies; and to the continued redirection of resources away from departmental instructional budgets.  If departments want to hire more positions, the best thing they can do is figure out how to reclaim the credit hours that we have lost--and will continue to lose in escalating numbers.  We are approaching a tipping point.  Soon it will be too late, if it isn't already too late.

One of the best historians of education technology is Audrey Watters.  She regularly posts her public presentations to her site hackededucation.com (see The Golden Lasso of Education Technology for a recent example).  I am also a huge fan of this collection of her essays, The Monsters of Education Technology (a bargain at $4.99 on Kindle; 9:99 for pb).  Another fantastic way to get caught up on the conversation about higher education and the politics of education technology?  Twitter.  Really.

5/26: See also The Ed Tech Curmudgeon: "If the educators that care about students don't find a way to respond, the future of education will belong to those who stand to profit economically or politically or both. Who will build the universities and colleges of the future - those who understand the history and share the values of the educational enterprise, or those who simply want to make a buck?"

Tuesday, May 19, 2015

Another Funny Thing Happened....

With thanks to Laura Gibbs for sending me this perfect cartoon!
You know something is up when you wake up at 7:30 am to an overflowing inbox and a large number of notifications on Twitter.  All my early bird friends let me know that Steve Kolowich at the Chronicle of Higher Education had picked up some bits of the Online Rome staffing saga.  The article has the somewhat odd title of "When your Online Course is Put up for Adoption."  It opens as follows: "Jennifer V. Ebbeler always knew that somebody else would end up teaching her online Roman-history course. But that didn’t make giving it up any easier."  Let me be clear, in case it hasn't been utterly clear to everyone who reads this blog: I have never taught the online course--I only designed and built it.  I never intended to teach it except randomly and in the distant future.  I had no problem giving it up; in fact, I couldn't wait to unload it. Indeed, I learned of the department's bizarre decision to hire a non-Romanist as course coordinator from a Facebook post in which I celebrated handing it off.  There was no regret, no second thoughts.  I WAS DONE. I bought myself a pair of expensive earrings to celebrate. 

It had been a long and difficult two years, made much more challenging by severe health issues over this past year.  Despite those issues, I felt like I owed it to my team to continue the project (thus letting them remain employed).  Especially this spring, it was incredibly difficult to find the energy to do the work on the course.  I motivated myself by thinking about the big picture: we needed the course to be finished this spring so that funding and staffing could be in place for the Fall 2015 semester, when the course would be in the Classics Department's possession.  I had also worked hard with my PM and the Asst Dean who oversees our development studio (LAITS) to get the funding in place.  This required a great deal of work and time and politics, especially on the part of Joe TenBarge, the Asst Dean.  The department was totally uninvolved in getting the funding lines in place.  But, when I finished the last of the development, it seemed like everything was finally coming together.  It felt like it was worth all the horrible days of working on it this spring.

Because I had designed the course for others to run and teach, I also put in place a transition plan.  It's not a typical "sage on the stage" MOOC-model course.  There are no video lectures; I appear nowhere in the course.  It focuses on active learning and, especially, the use of primary source materials.  Students work through modules and create their own narrative through a kind of Socratic method of course design.  We draw on documentaries, primary sources, and other things for content.  But thinking and questioning are key features of the course.  One of the reasons students like it, and why it produces high levels of student learning, is precisely because of this unusual model (unusual not because of the focus on active learning but because we do this at a scale of 1 instructor/100 students).

Let me be clear that my issue has nothing to do with wanting control or with wanting "my chosen successor" appointed.  In fact, what I suggested in a document dated 9 March is that the department continue the current instructor's appointment.  As I said, he couldn't succeed me in a job I never did--it's always been his job.  But, as the course transitioned to the department, I was no longer able to appoint him.  That became the responsibility of the department chair.  The main motivation for my suggestion was an interest in ensuring a smooth transition.  The plan I suggested would have had the current instructor be "course coordinator."  One thing that is not clear to most people: this is one course, but with multiple sections, taught to somewhere between 300-500 students (depending on how many sections are run; the College is requiring at least three/semester to keep the funding for the position).

So, in truth, it's about running a small business, working with at least some grad students who aren't content experts and have no experience in online teaching but who will be "instructing" sections of 100 students.  It's a model that can work, I think, but it needs for certain key components to be in place.  One of those is a course coordinator/mentor/supervisor who knows the course inside and out and knows how to teach online.  Of course, over the year, I expected that others would learn how to run the course through the experience of training under an experienced course coordinator.  My goal, always, was sustainability.  That requires many people with the ability to teach the course at any given time.

I am the only member of my department to have taught a course over 250 or so students.  For two years I taught a 400 student Intro to Ancient Rome class.  It was a huge production and logistical challenge.  It's not just like teaching 200 student classes times two (which is the common assumption of those who have never done it).  It's about ten times the work and complexity.  The current instructor dealt with this in Fall 2014, when we had over 300 students in the course.  It was his first experience of such a set-up, which included managing graders; it was a real challenge and he learned a lot by trial and error.  My concern is that, now, the Classics Department is asking someone with no experience in any of the key components required to run the course to step in and a. run a huge, multi-section course; b. know how to manage, support, and train the instructors of the individual sections.

Thankfully, LAITS had a kind of back-up plan in place.  They have hired the current instructor to teach the Extended Campus version of the course in the Summer and Fall 2015.  In the Summer, he will be focused entirely on converting the 15 week course into a five week course while a graduate student from the department is the instructor.  He will work closely with that grad student and mentor him as much as possible on the challenges and tricks of online instruction.  Likewise, he will work with the course coordinator in the fall semester.  I was relieved to learn of this plan and I think it will do a lot to keep things from being a total disaster in the fall.  But it depends on the fact that the current instructor is extremely gracious--far more than I would be in a similar circumstance.  It also leaves unaddressed the larger issue that the course coordinator is far from a content expert or online teaching expert, and yet is going to need to mentor several graduate instructors each semester.  I would understand if there had been no other option but that just wasn't the case.

I'm a bit disappointed that the article's melodramatic (and oddly gendered) tone encourages comments about me instead of about the issues at stake here.  This isn't about me or my course design.  In fact, a key component of the project from the start was figuring out a sustainable model at scale--all without overly compromising on the quality.  It was a huge challenge but we did it.  The course is an excellent example of a quality online course that is "efficient"--or at least more efficient than our current campus version; and works at a 1/100 scale.  It also, ideally, provides graduate students the opportunity to teach online under the supervision of an experienced course coordinator.  Finally, it created two lecturer positions for my department for next year, with money from my college as well as LAITS.

In return for taking on this project, I received nothing.  I was not paid a stipend.  My time was paid for, including summer funding; but, for instance, the 20 hours/week this spring didn't come close to paying for the 60 hour weeks the project required.  It's not at all clear that the creation of this course will "count" for anything professionally.  After all, it's not a monograph.  I get no royalties, which I would get if this were a published textbook.  I didn't even get a thank you from anyone.  In essence, I donated two potentially productive years of my career to a project that I cared deeply about and was intellectually challenging to me; but which benefited my career and bank account not at all.  I'd have been better off to design courses for online programs for a negotiated stipend (if it was the money that I cared about).

The real lesson of my experience is not that your baby can be taken away (ugh!); or that instructors can be separated from their courses (duh!);  but that, at the moment, institutions are still playing catch-up in terms of policy and infrastructure around the delivery of online courses to campus-based students.  What I found, over and over, was that we were inventing policy and procedures as we went. A big part of the problem is the fact that the development of these courses is done in non-academic units, but then the courses are handed over to academic units to manage.  Sometimes this is ok.  But if the department isn't up to date on the latest best practices of online education; or if a chair decides to make unilateral decisions and not collaborate with the people who are familiar with the course and its design, it can lead to real problems.

The reason I have made such a sensitive issue public is to try to encourage more conversation (and action) on the policy front.  If administrators are going to ask faculty to take risks, to sacrifice years of their time to projects like these, then there need to be clear and rational policies.  For instance, despite the tradition that departments control all staffing decisions, with online courses (which are very expensive to develop and keep updated), this is probably not the best procedure, especially if the main concern is ensuring a quality course and learning experience for our students.  I want to encourage my smart and talented colleagues to take on a project like this instead of focusing entirely on work that is great for their career but doesn't do much for the larger community (i.e. writing monographs and articles).  At the same time, for this to happen on a larger scale, we need to get policies in place.  Faculty want to take risks, they want to take on projects like this.  I think most of us have no problem handing our courses over to others.  What we want are some assurance that the course we spent two years building will be treated well and, hopefully, run as it was designed to run.  This seems like a reasonable expectation to me vis-a-vis Online Rome, given that I have essentially donated two years of work to my department and have received no extra compensation.

As we venture forward into this brave new world, we also need to make sure that all decision-makers are educated in the areas about which they are making important decisions; and ensure that there is campus support for all online instructors/course coordinators.  This brave new world of online course design and delivery is, potentially, a fun, stimulating, and surprising one.  It is also one that doesn't quite work like our traditional campus education.  Online teaching requires a lot of different skills than does classroom teaching.   I'm confident that, eventually, all these pieces will get into place on our campuses.  My aim in discussing my experience is entirely to provide a kind of "case study" for why it matters that we get these policies in place sooner rather than later.

Monday, May 18, 2015

Administering Exams in Online Classes


One of the persistent challenges in teaching online is designing assessments that are credible, reliable, and resistant to academic dishonesty.  In smaller courses, this can be done by avoiding exams and turning to papers, blogs, creative writing, and other kinds of activities that can be gathered into a final portfolio.  As much as I would love to assess students using something other than quizzes and exams in Online Rome, it just isn't feasible given the large enrollments that we are required to maintain.  We do make a point of making written work some part of the grade, in the form of essays and short answer exam questions.  But, ultimately, about 50% of the final grade is determined by the students' performance on quizzes and exams. 

I've been waiting for a reliable and affordable online proctoring service to come on the market.  With today's announcement that Instructure (Canvas) has partnered with Verificient, an online proctoring service, it looks like we are making some progress on this front.  This was an obvious next move for Instructure, now that their LMS is being used as a platform for online course delivery of all sorts, including MOOCs.  The absence of a proctoring service was a serious gap that, it seems, is now being filled.  We'll see how effective it is--I never underestimate the creativity of students and their ability to outsmart any monitoring system.  But, at least, it is a good start.  As online courses become more common, we need a way to protect the integrity of the grade.  The system doesn't need to be perfect.  After all, academic dishonesty happens all the time in face to face classes.  But instructors need to feel reasonably confident that the grades they are awarding were, in fact, earned.  This is especially true as universities move towards a policy of not distinguishing between online and classroom-based courses on transcripts.

The quizzes, which appear at the end of each module, are not proctored.  We tried to disincentivize dishonesty by writing questions that are application of facts rather than regurgitation of facts (and therefore difficult to Google or find in a textbook).  The quizzes are timed and each question has three variations, so it is unlikely that students working together will have very many questions in common.  The quizzes are also not worth all that much of the final grade.  Ultimately, even if a student cheats on 1-2 questions/quiz, it is unlikely to make any difference in the final course grade. 

Still, I would have preferred for the quizzes to be mastery quizzes rather than graded quizzes.  When we tried that approach in the Fall 2014 version of the course, however, students didn't study for them and, ultimately, were not learning the material as well as they needed to in order to perform well on midterm exams.  By switching to graded quizzes, we were able to get the students to take them seriously.  The performance data looked almost identical to the data produced by my classroom-based students, which suggests to me that there was not much cheating going on.  Likewise, we saw significant improvements in the exam scores--another indication that the students were studying for the quizzes.  Finally, there were no obvious cases where a student had high quiz scores and low exam scores--not a sure sign of academic honesty, but a decent indication that any cheating was small-scale.

Online Rome also had three midterm exams that were administered on campus, in a proctored environment.  We included these, in part, as a way to ensure the course's credibility.  I suspect that, at some point, these midterms can be eliminated.  I suspect that the quiz data will demonstrate that it is a reliable indicator of a student's mastery of the content; and that the instructor can design other, more engaging activities that require students to make connections across the course.  A key skill in the study of Ancient Rome is the ability to see patterns and connections across time.  Studying for midterm exams is one way that students begin to see broader patterns--but there are many other ways that this could happen.

For now--and, I suspect, the next several years, midterm exams will be a part of the course.  For all sorts of reasons, it would be easier to administer these exams via Canvas.  This spring, we required that all students--even non-UT students--take the exam on campus or in an accredited testing center.  We had to resort to this after several problems with academic dishonesty in the fall semester, when we did allow distance students to take the exams online (there was an oral component to the exam, which was largely just a pain for everyone and did not work especially well).  Since the majority of the students who currently take Online Rome are Austin-based, it has been fairly easy to administer the exams on campus.  As we make a real effort to expand the enrollment to non-Austin based students, however, it would be great to have an effective proctoring service built into Canvas.

Sunday, May 17, 2015

It's about the Relationships, Always

Image source: https://notegraphy.com/mdvfunes/note/1750710
I can't emphasize enough how true this statement is.  One of the major reasons why the delivery of effective online education presents such a challenge to institutions is the failure to understand this basic principle.  Online courses need to be well-designed and accessible; they require a different set of logistical skills to manage than does a classroom course.  They do not especially require a lot of bells and whistles.  It turns out that students don't much care if the video is high quality so long as it is acceptable quality.  While it is convenient to have the resources of a production studio, as I did when I built Online Rome, it is certainly not a sine qua non.  There are plenty of tools available gratis on the internet that can be used to record video and/or audio and even live stream video/audio.

Without an instructor who is a. capable of creating and nurturing relationships with the students; and b. capable of creating a learning environment that provides opportunities for students to connect to and learn from one another, an online course is very unlikely to be truly successful.  It certainly will never rival the classroom experience.  Faculty who oppose online education generally rationalize their opposition by claiming that the online medium cannot replicate the intimacy of the classroom, the ability of the classroom space to nurture the kinds of relationships that support learning.  This is a dangerous failure of imagination.  In fact, a skilled and experienced online instructor knows exactly how to nurture these crucial relationships--and often can do so much more deeply and with many more students than can a classroom instructor.  That's the irony in all of this.

These days, most of the attention of institutions is on the courses. There is a tendency to get wowed by fancy videos, animations, and the like.  None of these things matter much for student learning if the course lacks an instructor who knows how to build relationships and also provide space (and incentivizes) for the students to build relationships with each other.  This latter task is especially challenging.  Discussion boards are the old fallback but, in my experience, they don't actually do much to encourage student to student interaction.

One of the areas where Online Rome could use more development is in the area of supporting peer to peer learning. Steve, the course instructor, and I had planned to do some of this in the upcoming semester.  I hope that we will have the chance to implement our plans for non-UT Austin students at some point.  For campus-based students, however, it should not be very challenging to design some activities that require or at least strongly encourage peer learning.  I know that there was a fair amount of this happening on an informal level (e.g. students worked together on the modules; they studied for exams in small groups).  We captured information about some of this through course surveys.  But I suspect that a well-designed class activity (or series of activities) that puts students into small groups for the semester would improve the student learning and general experience even more.

Until then, though, I won't stop saying: teaching is about relationships.  Period.  The medium of instruction influences how we construct those relationships, what tools are available to us, but the process of teaching and learning is always about relationships and will always be about relationships.  Further, it is incredibly shortsighted (and uninformed) for faculty to believe that face to face relationships are inherently superior to other kinds of relationships.  This prejudice for presence is at least as old as Socrates and Plato; but it has been debunked over and over again.

P.S. So, TechCrunch asks why the university is still here.  An idiotic question, but a pretty simple answer: in part, because we haven't figured out how to support social learning online.  MOOCs are the opposite of the right answer.

Saturday, May 16, 2015

Learning Outcomes and Backward Design



 A few weeks ago, my department chair sent an email to the faculty list-serv with the news that UT Austin had once again failed to pass its SACS Accreditation.  The problem, it seems, hinged on the failure of departments/colleges to define and measure learning outcomes in a way that was acceptable to SACS..  The chair expressed her view that this was a nonsensical process, anticipating the general moaning and groaning that was sure to emerge from faculty who believe that things like learning outcomes are silly.   At the start of the academic year, our CTL had sent around a very helpful sample syllabus, with the various UT policies included as well as things like a place for the instructor to list the course's learning outcomes.  Two senior colleagues ridiculed this document as useless and stupid--after all, they have been teaching for decades and don't need such guidelines.  Sadly, I'm fairly certain that if they were asked to define the learning outcomes for their course and explain how their course was going to support students in achieving those outcomes, neither colleague would be able to do so. 

Writing learning outcomes is very difficult for faculty who were never trained to think about their teaching in such terms.  We are great at describing what content our course will cover; we are pretty good at knowing that we expect our students to master a certain amount of content or skill set by the end of the semester.  We are terrible at framing our expectations for student learning in terms of learning outcomes, with all of our learning activities in the course aligned to those learning outcomes.  We are even worse at measuring learning outcomes.  We conflate grades with learning outcomes on the regular.

Every professor will tell you that students learned in their course--they will insist on it, despite never having measured how much students knew at the start or measuring their knowledge using a standardized instrument rather than an instructor-written exam.  In fact, we generally have no real way of measuring what they did or didn't learn, and how well.  This is especially true for one-off courses, like the lower division general education courses many of us teach.  So a student of mine doesn't really know anything about Roman culture or history at the end of the semester...  This is likely to have little impact on their educational career unless they decide to be a Classics major.

To my mind, one of the great boons of digitizing our teaching is precisely that it requires faculty to finally learn about things like learning outcomes and backwards design.  These are not difficult concepts but they require some intentionality--and sometimes some assistance--to implement.  The activity of building a hybrid or online class encourages faculty to think hard about how all the pieces contribute to student learning.  The existence of the course on a digital platform means that the experience of taking or teaching a course is now preserved as an artifact that can be examined by a third party.  It is no longer an ephemeral experience in which we depend on the reports of instructors and their students to evaluate teaching efficacy.

I worry about "Big Data" intruding on academic freedom, both in terms of what we teach and how we teach.  Will we all be forced to teach in 3-5 minute blocks because someone decided that this was the average attention span?  At present, online courses are closely scrutinized and monitored by our campus administrators.  To what extent will this monitoring expand as our ability conduct this monitoring with computers grows?  Currently, we are required by law to post our course syllabus.  At what point will we be required to use the Campus LMS for all graded activities, so that student data can be captured more easily?  I suspect that the refusal of most faculty to understand that they are accountable for demonstrating student learning will make it all the easier for university administrators to start tracking students and imposing ever more restrictions on what and how we teach.

I make it a point to identify and articulate course learning outcomes for all my courses, including graduate seminars.  I also include a "map" that illustrates how the different learning activities in the course will help students reach these outcomes.  It took a bit of practice to learn to think about teaching and course design in these terms but, after two years, it's become second nature.  These are the learning outcomes for Online Rome.  The least important of them is mastery of the course content.  I am much more interested in helping students develop crucial "soft skills":

·         become active, “self-regulated” learners
·         learn and become more skilled at good time management techniques
·         learn basic skills of “reading” ancient texts/art/architecture
·         develop and practice ability to think make connections between different parts of course (i.e. think analytically)
·         develop and practice ability to evaluate competing explanations or theories
·         master basic narrative of Ancient Roman cultural history from Iron Age-2nd Century CE


Enaging Undergraduate Students in Course Design and Instruction


There is a lot of conversation on college and university campuses about getting undergraduates involved in research as soon as possible.  It is clear that the close contact with faculty as well as more experienced students is a high impact experience and is positively correlated with graduation and shorter time to degree.  At UT Austin, this emphasis on undergraduate research seems to be a main plank in our new president's platform.  It has been a central topic in the Campus Conversations, which the president sponsored over this past year in his role as the Provost.  There are many different initiatives on campus, particularly in STEM fields, which already focus on this effort, including the Freshman Research Initiative in the College of Natural Sciences.  My own home, the College of Liberal Arts, briefly had a program for freshmen and sophomores, whereby students could work with a faculty member on a project connected to the faculty's research program.  I worked with a couple of students through this program but, ultimately, found it to be an exercise in futility.  The students who were eligible had no language training and my research absolutely requires knowledge of Latin as a sine qua non.  Even the task of constructing a bibliography requires more than a knowledge of English.  I struggled to devise interested but manageable tasks for the students; and I think they left the experience wondering why anyone would want to do research in Classics!

In liberal arts, and especially in the field of Classics, the major obstacle to involving undergraduates--especially early undergraduates--is that they lack the specific skills that are necessary to do even the most basic research.  Classics training at the undergraduate level focuses on language instruction, and it typically takes a student most of their undergraduate years to even master Latin and/or Greek to a level where they can begin to make sense of an ancient text.  Very often, research is postponed to graduate training, even the PhD these days.  Given this, it is a real challenge to think about how to bring inexperienced undergraduates into the faculty research process.  In truth, it can't really happen for a scholar who produces the kind of scholarship I do.

This is not to say that I cannot engage in productive conversations about research with undergraduates; or have them learning some of the "skills of the trade" in my company.  As I learned during the production of Online Rome, though, the best place for this to happen--for me, given the kind of research I do--is around teaching.  It is my sense that the value for students in engaging in research early in their undergraduate career is not research qua research so much as it is, first, structured contact with faculty as well as, ideally, more senior graduate and undergraduate students who can act as peer mentors; second, an opportunity to see the application of knowledge (I don't think it's particularly relevant whether the application occurs in a research or a teaching environment); and third, an opportunity to see "behind the curtain" of academia, to see what it is that professors do and what it means to be a professor.  This third element is especially important for students in liberal arts who think they might want to continue their study of the field as graduate students.

When I was putting together a team of students to work with me on building Online Rome last summer, I included an advanced undergraduate student (a double major in Classics and Religion).  In retrospect, I wish I'd included more.  It was fascinating to see how, in giving her the task of creating a first draft for one of the modules, we were able to have deep and engaged conversations about the content, how best to present it, what sorts of trends it connected to, etc.  Whereas I feel like any effort I have made to involve undergraduates in my research has ultimately meant that they are doing uninteresting "grunt" work, this project allowed for genuine intellectual growth.  The key was the fact that it was project-based for the student.  She worked on it and then we discussed the work.  In the future, I hope very much to involve a larger team of undergraduates in a course-building project.  In liberal arts, and especially in Classics, it is an ideal way to connect with undergraduates on intellectual grounds, but at a level that is accessible to them (because the task is not the production of original research but, instead, how to teach complicated content to their peers); and which results in the production of something tangible and useful.

I've never been much of a fan of the "hack the syllabus" approach to undergraduate teaching, primarily because it doesn't work terribly well with the particular audiences I teach.  What does work is involving a group of students in the design and build of a course from the very start.  I can imagine a very interesting learning experience in which, for one semester, a group of students works with me to design and build a course.  The following semester, we would teach that course, with the student builders acting as peer mentors to the enrolled students.  In this way, it would be possible to put more experienced students together with less experienced students--something that happens to great benefit in graduate seminars but is nearly impossible to do in undergraduate classics courses (in part because, through an odd system of incentives, nearly all students in upper division "seminar" courses are non-majors).

At present, I have only involved undergraduates in the delivery of Intro to Ancient Rome/Online Rome as graders.  It's not a bad job and it is enormously helpful to the instructional team, especially when we are teaching very large numbers of students and need to turn around the exams quickly.  The course instructor graded 10-15 short answer sections on the exams, to establish a rubric.  The instructor then met with the graders, reviewed the rubric and scoring of points (with attention to common errors and breakdown of points for each question).  Without exception, the undergraduates did an excellent job and were very attentive and punctual.  They more than earned their stipend.  The job offered an opportunity to engage with the instructor and to see a bit how large courses are run.

I am a much bigger fan of peer mentors in the classroom.  I saw them used to excellent effect in my colleague Cynthia LaBrake's Intro to Chemistry course.  I wish that I had had the funds to create a more elaborate network of peer mentors for Online Rome. This is one aspect of the Online Rome course that could be developed.  Especially in the online class, putting current students in contact with former, successful students would go a long way towards helping the new students figure out how to take an online class and, in particular, how to do well in Online Rome.  Online bulletin boards/Rate My Professor-like sites can offer some information, but it tends to reflect narrow points of view.  It would be much more effective to have a group of former, successful students who act as mentors and graders.  If some effort were made to meet weekly with these peer mentors, it would also be an excellent way to sharpen and expand the mentors' content knowledge as well as their ability to explain complicated concepts to their peers.  There is really a great opportunity here, especially for Classics and related majors, if the funding can be found and if someone is willing to spend the time developing and running such a program.