Simply providing feedback on practice profiles and prescribing data has so far shown to have little impact on clinical practice. From our work with the Alberta Physician Learning Program (PLP), several projects have shown that clinicians need more context and comparators, not just infographics. In a number of projects, we have made use of virtual scenarios and OpenLabyrinth as an education research platform to provide such context-based data-informed feedback. Adopting this approach first arose out of our DISECKT project (Data Informed Self-Evaluation and Collaborative Knowledge Translation) and its initial findings. Sadly, the DISCEKT project itself was not able to move beyond the prototype stage but it drove the learning designs in several of our other projects.

For example, in our DynIA project, we used virtual scenarios and webinars, along with data-informed feedback, to explore how we could improve the uptake of CPGs on chronic pain, specifically low back pain and headaches. The webinars were conducted with a number of rural teaching sites. The learning design successfully incorporated a flipped classroom approach, where experienced practitioners showed that, contrary to other projects with experienced learners, they were keen to tackle challenging cases both before and during the webinar series. We designed a novel 4R feedback , in a participatory action research approach, actively engaging the rural sites in the formatting and utility of the 4R graphics. The subject matter expert presenters were able to view how the participating sites tackled the virtual scenarios and guide their advice appropriately. The learning designs were also set up to capture actual changes in practice across the webinar and virtual scenario series.

In our ODT project with CaMH, we established a series of virtual scenarios depicting some key challenges in opioid dependency treatment strategies. We used a variety of feedback mechanisms from these cases, both to measure learners’ performance and to improve the functionality of the virtual scenarios. Because tracking of individual participant data was important to CaMH, we were able to directly link from our OpenLabyrinth research platform to the Desire2Learn LMS at CaMH, using a secure IMS-LTI authentication link, providing data interoperability between the two systems.

Based on the success of our webinars in the DynIA project, we secured funding to extend this approach and also to incorporate the integration of short teaching videos into webinars, in our CURIOS project. The CURIOS funding allowed us to create the CURIOS video mashup tool which supports the segmentation of existing YouTube videos, both ours and other authors’, using a video linkage mechanism that minimizes copyright issues. We created 285 video snippets and 20 virtual scenarios for use in 22 webinars in this series. We were able to track participant progress in the virtual scenarios and also their use of the video mashups, which in turn led to course improvements and scenario modifications, closing that feedback loop. The CURIOS video mashup service has been designed in such a way that it can be used independently and we have successfully integrated that service into other modalities such as classroom presentations, PowerPoints, WordPress web pages, Moodle courses and others. See here for info on how to incorporate this service.

One of the common challenges with virtual scenarios and virtual patients is that they are not good at handling natural language responses – usually, participants have to choose from preset options. In our interprofessional education outreach collaborations with the University of Calgary School of Nursing, we have been exploring a new technique that we call Turk Talk. This adaptation of the OpenLabyrinth research platform provides participants with the opportunity to engage in natural language, text-based chats with scenario facilitators. This has revolutionized the learning designs that we can employ for all levels of learner. Because all data is captured in real-time, we are also able to analyze both participant progress through the virtual scenarios (as we were before) and also the actual narrative and the therapeutic language used. This has really opened up the kinds of analytics that can be used. We welcome potential collaborations with other researchers who are exploring this area.

All of these projects have essentially focused on activity data: using actual performance metrics to track what learners really do, not just their impressions and their teachers’ impressions of how they performed. We have since become very active in the whole area of activity metrics – our group is now heavily involved in promoting collaborative data standards, for sharing of such analytics, via the good offices of Medbiquitous. In particular, we have been very active in driving the adoption of the Experience API (xAPI), which affords collation of such activity metrics from a number of independent sources into federated Learning Record Stores. This whole API-based approach is much more flexible and promising than previous attempts at data integration using SCORM, because the data coupling is less tight, and less prone to breakage from system updates on the linked LMS, CMS etc. This new design approach is also being adopted in OLab4, our new education research platform, that is being co-developed by our consortium of medical schools and educational organizations.

This technical report was generated to summarize the multiple ways in which we are using data-informed feedback in continuing and professional education at the University of Calgary. Further details on each project are available in the linked documents. We welcome enquiries from other scholarly groups who are exploring how activity metrics can be used to improve teaching and learning processes, content and outcomes.

Dr David Topps, Medical Director, Office of Health & Medical Education Scholarship
Dr Heather Armson, Assistant Dean, Office of CME & PD
Dr Lara Cooke, previous Associate Dean, Office of CME & PD
Dr Kelly Burak, Associate Dean, Office of Continuing Medical Education & Professional Development, Cumming School of Medicine, University of Calgary.