In a world where understanding how products and services can improve how we educate our students is of the utmost importance, we are still doing a poor job of understanding the impact of ed tech tools on student learning. The body of research is growing, but it is slow and inconclusive. “Efficacy” of ed tech products is touted in case studies, white papers, indexes, and testimonials, but for many instructors that boils down to the same tired website claims like “My students were more engaged” (a.k.a. the instructor assigned students to watch videos on the concept, they watched them, and now they are “engaged”) and “Students said that the practice helped them focus on their weaknesses” (Hmmm, specific practice helped students learn troubling concepts? Is that a revelation?) Still others show that grades increased by 10%. BUT, in all of the cases, what we seldom know is what else changed to increase engagement, increase practice, increase grades by 10%? Was it really just the product?
If teachers assigned new content types, or made practice a part of student grades, or changed when, where, or how they introduced content, they changed their behavior. They introduced a new pedagogical strategy. Maybe not a huge shift like a flipped classroom but maybe a little shift—like reducing lecture time for coaching time or providing more low stakes opportunities for students to affect their grade. Where teachers learned about the pedagogy, how they prepared to try it, and why they felt confident enough to move ahead is where impact starts.
Like many, I have struggled for years with how to demonstrate that my products work for students and for teachers. I have run pilots at all levels, arranged case studies, and implemented surveys. But trying to prove efficacy of learning by focusing on a product is a fool’s errand. The product is one leg of the stool. The other two legs are pedagogy and professional development. What I should have focused on then (and what I do focus on now) is how all three work together. For instance, an instructor who supports a blended learning initiative by first selecting a digital product will fall far short of the goal of transforming learning for her students. Because along with a commitment to blended learning comes a host of new instructional frameworks, a new understanding of the role of the instructor and the responsibilities of the student, and a number of strategies for changing how teaching (and learning) happens. The innovation is blended learning; the tool is there to implement the pedagogy. The pedagogy isn’t created to support the tool. When you understand that impact is a combination of tools, pedagogy and skills, you can begin to understand, and improve, the impact your product is having.
That is not to say that an ed tech company cannot be catalysts in assisting the shift. I often hear that a company-sponsored workshop or webinar help moved teachers to consider a shift. But product training should not be confused with professional development. Professional development can and should involve the tools that instructors will be using, but mostly it should give teachers the competencies and confidence to introduce new strategies with their students. But without a solid pedagogical reason, the competencies to execute, and the time to integrate it into an instructor’s workflow, your product won’t deliver any lasting impact.