More Thoughts on the Small Pilot vs Immediate Scale-Up Dilemma

I love social media, especially Twitter and blogging, for many reasons but especially for the way that it facilitates the creation of communities dislocated in space.  When I first started to experiment with the flipped class model three years ago, I really had no idea what I was doing.  I didn't even know that this thing that I was wanting to do actually had a name and that there were communities of teachers out there who knew a lot about how it worked (mostly in K-12 and in math and sciences, but still).  I don't remember exactly why I got on Twitter, but I immediately discovered all sorts of interesting and helpful conversations that I could insert myself into (I also learned, rather quickly, to not be a shy lurker).  Since that time, I have benefited enormously from the connections and conversations I've made via Twitter.  I can honestly say that I learn something new every single day thanks to the plethora of smart people, through their comments or through their curation of interesting essays, articles, and blog posts.  This has been an especially critical outlet for me because my own university is still very much in the preliminary stages of thinking about the kinds of questions that interest me.  And, since I'm not an administrator, I'm generally not part of the conversation even at my own university.  Twitter is where I've found people who inspire me, engage with me, and help me think through the nuances of different important issues.

Over the weekend I wrote a post about my own experience of "jumping into the deep end" with course development projects.  In both cases, I had planned to do a pilot of the new course and then, over time, scale up the enrollments.  In both cases, for different reasons, this isn't how things played out. The first iteration of each project ended up being taught at scale (400 students for the flipped class; about 330 for the online class).  I've become convinced that one of the reasons our development process was successful--despite the enormous stress caused by the immediate scale up--is that we were able to troubleshoot very effectively.  By the second iteration, we had a class that ran smoothly and produced high levels of learning.  Thus, when I watched the e-Literate TV case study of ASU and saw that they were embracing a "jump into the deep end" approach, it resonated with me.  Likewise, when I read Phil Hill's comments on the "to pilot or not to pilot" question, I was struck by his observation that many pilots on academic campuses end up in what he calls "pilot purgatory" (a fabulous turn of phrase!).

I was really pleased to see that Matt Reed (aka @deandad) had taken up the issue in his regular column for Inside Higher Education.  He made a number of really important points about this important question.  Perhaps most usefully, he reminded readers that context matters.  When talking about a single course that is largely self-contained and not especially resource-intensive, the immediate scale-up makes more sense (in my case, the additional resources required were negligible); but if the project is costly and depends on the collaboration of multiple units, taking it a bit slower often makes sense.  Likewise, if total failure is a real possibility and if people can be irreversibly harmed by that failure, then common sense and basic morality dictates a slower development process.  The main dilemma is distinguishing between projects that would benefit from a faster scale-up; and projects that, for various reasons, need to be developed more slowly and under less risky and potentially chaotic circumstances.

It is great to see a nice conversation developing from e-Literate's ASU case study.  The question of how to develop projects, and especially how to do first iterations, is a major one on campuses these days.  It seems to me that this conversation has gone a long way towards helping experimenters make informed choices. 
One thing that came up in a Twitter exchange between me and Phil Hill: industry is much more willing to take risks in developing new projects.  Academia tends to be extremely conservative and, in general, fails to understand the cycle of iteration.  Academics (and administrators) see any kind of failure as a bug instead of a feature.  Failure is how we learn. 

Yet it is difficult for professors to put something out there that is not, in their mind, perfect.  It is even more difficult for them to deal with the inevitable setbacks and obstacles that arise during the first iteration of a new course design (including basic student resistance to change).  Thus, a lot of money is spent to develop projects and get them to the testing phase.  Yet very few ever reach the "developed" stage.  It is challenging for faculty to change their mindset, to understand that everything we do--including our teaching--is a work in progress and should be constantly evolving.  One of the great gifts of my experience with these course development process is an increased tolerance for error; and decreased perfectionism.  It can be incredibly difficult to put something out that you know isn't as good as it could be--but to accept that the fastest way to improve it is to just put it out there and let people go at it.  But this gets easier, as one understands that it is just part of the process.

Matt is absolutely correct about the risks of running a project at scale that requires lots of collaboration across different parts of the institution.  With my flipped class, everything was very self-contained.  I could handle problems myself.  With the online class, I was suddenly trying to collaborate with a number of units across campus to keep things running: the registrar, the deans in my College, IT/Canvas support, my department, our Extended Campus school; our Liberal Arts Development Studio staff, and my instructor.  It was nothing short of a circus, especially because the first iteration of the online class happened to coincide with a big increase in the number of faculty using Canvas as their LMS.  Canvas crashed a few times during the semester and had a number of odd but critical glitches. 

It was not easy to try to keep everything running and ensure that everyone was doing what they needed to be doing.  Balls definitely got dropped.  It didn't help that I ended up having unplanned surgery in early November and was out of commission for about three weeks.  One thing I noticed, however, was that it was easier to get these different units to stop dragging their feet and take action precisely because we were live with over 300 students.  They knew that if they didn't do their job, a large number of students were going to be adversely affected.  So, while it can be a massive challenge to get all these different partners pulling their weight, it was actually a bit easier to pressure them when so many students were involved.  It certainly put an enormous amount of pressure on me, as project leader, to ensure that everything got done and no major balls got dropped; and I spent a lot of time nagging people (and, I'm sure, annoyed many administrators!).  Still, in the end, I was able to get things done--largely, I think, because so much was at stake.

There are absolutely times when it makes sense to hold back and develop ideas more incrementally.  At the same time, it seems that too often, the default is to start with the small pilot with the plan of figuring out how to scale up someday.  I suspect that more of these pilots would take hold if a more aggressive experimental process was followed; and if faculty were more willing to embrace a tolerance for failure (or, rather, see failure as an essential part of the process).