The Problem of Pilots

Last week I read yet another report compiling the top reasons for the high rate of teacher turnover.  While about half of the reasons look very much like why anyone would leave any job (poor leadership, lack of professional development, few advancement opportunities) and the other half were very specific to teachers (emphasis and time spent on standardize tests, lack of respect for the profession), there was one reason that really jumped out at me – “no input into classroom materials.”

I was puzzled.   Everywhere I go, I see pilots.  From pre-school to graduate programs, pilots are running every semester in every conceivable space in education.   I have read a thousand surveys, hundreds of case studies, and attended dozens of focus groups dedicated to understanding the teacher’s perspective on everything from new textbooks to adaptive platforms.  So, how come teachers don’t feel like they are being heard?  I believe the cause of the disconnect stems from both sides.

Pilots as sales events:  “We give it to them free for a semester.  And then when they love it, they will beg us to sell it to them.”    Every sales rep knows that inertia is your biggest friend.  Once a teacher uses something, it is hard to get that something replaced; even IF it doesn’t meet all of their needs.  So, get it in use and then worry about the rest later.  Sure, there are surveys for the teacher to fill out, but these “pilots” aren’t about getting customer feedback; they are about a seemingly easy way for a rep to sell a product.  The problem is, of course, none of that feedback makes it into a new iteration of the product, OR if it actually does, no one is closing the loop with the people who provided that feedback.  “Thanks.  Here is your reviewer stipend.” On to the next pilot.

Pilots as “stretching the budget”: And sometimes it works in the opposite way.  Schools and districts are strapped for cash.  This is a pretty easy way to get educational products into the school at no cost.  Just ask.  I have seen schools (both P-12 and Higher Ed) ask for tens of thousands of dollars’ worth of product only to switch at the end of the year because the next company has come in and offered another free pilot.  Again, no one is asking the teachers for their feedback because the next free option is already on the horizon. 

So, how can we make better use of the feedback and insight we gather from pilots?  How can we ensure teachers that their voices are not only heard, but acted upon?

 The best way to start is to be realistic and agree upon goals and measurements up front and make sure that feedback happens during the pilot, not just at the end.  Most pilots last one semester or less making it nearly impossible to measure success through improvement in standardized tests.  So what can we measure?  Student engagement for one.  Teacher engagement for another.  What tasks could the teacher eliminate because of your product?  Did she?  Were both advanced students and remedial students able to benefit from the product?  Did the pilot teachers convince other teachers to try the product?  Were their experiences consistent?  Did students talk about the product?  Teachers don’t need to wait for a company to ask them what they think; they can just tell them.  Good companies will want to know. 

Successful pilots can take many forms as the product iterates and the teachers become more comfortable.  Being clear about the key metrics, the expectations on both sides, and the implications of the results as well as establishing regular and open communications throughout the pilot will increase the benefits of pilots for both teachers and companies.  And ensure that your customers are being heard.