David Topps, Medical Director, OHMES, University of Calgary
Ellen Meiselman, Senior Analyst, Learning Systems, University of Michigan

The focus on Competency-Based Medical Education (CBME) has shone welcome attention on the need for good assessment: how do you know when a learner is competent? We need data. All of us are happy when the learner is doing well. Some have suggested¹ that we are becoming too focused on checklists and that gestalt ratings are just as good. But what about those learners on the lower shoulder of the curve?

We need better data for earlier detection of those who need help. We need better data for earlier detection of those who need help. Over the years, several have used various approaches in intelligent tutoring systems to examine learner
engagement.^2-4 However, these have looked at engagement but, for a skilled learner, the materials may not meet their needs. Struggling learners often remain undetected, the failure to fail problem.^5-7 We need stronger data to support decisions to terminate, lest we spend weeks in litigation.

We have seen several efforts to support competency assessment: EPAs, milestones, pre-EPAs, each looking at lower levels of complexity on the spectrum of learning activities. But too many of these depend on observer-b(i)ased checklists, and we are already encountering survey fatigue at all levels of learner assessment. Yet, much of what we do is now captured online in one form or another: electronic medical records, learning management systems, simulation systems, enterprise calendars.

Activity metrics take a ground up, rather than top-down, approach to tracking what learners actually do rather than what they, or their teachers, say they do. This already happens in limited individual systems but we need an extensible approach if we are to garner a more comprehensive view. The Experience API (xAPI)⁸ is a technically simple approach that is easy to integrate into existing systems. Data is captured into a Learning Record Store (LRS)⁹.

xAPI statements follow a very simple actor-verb-object structure: Bob Did This. And yet this simplicity belies great power through a very extensible set of vocabularies. The LRS structure is designed to swallow such xAPI statements from multiple sources, in true Big Data fashion, and can be securely federated so that a wide range of analytic and data visualization tools can be employed.

xAPI is technically well established. Groups, such as Medbiquitous¹⁰, are standardizing profiles of activities for medical education and health system outcomes. Now is the perfect time to engage medical educators in the power of these metrics. Their assessment needs should drive how and where these activities are measured.

References:

1. Hodges B. Scylla or charybdis: navigating between excessive examination andnaïve reliance on self-assessment. Nurs Inq. 2007;14(3):177. http://onlinelibrary.wiley.com/doi/10.1111/j.1440-1800.2007.00376.x/full. Accessed January 21, 2017.
2. D’Mello S, Graesser A. Automatic Detection of Learner’s Affect from Gross Body Language. Appl Artif Intell. 2009;23(2):123-150. doi:10.1080/08839510802631745.
3. Qu L, Wang N, Johnson WL. Using Learner Focus of Attention to Detect Learner Motivation Factors. In: Springer, Berlin, Heidelberg; 2005:70-73. doi:10.1007/11527886_10.
4. Soldato T. Detecting and reacting to the learner’s motivational state. In: Springer, Berlin, Heidelberg; 1992:567-574. doi:10.1007/3-540-55606-0_66.
5. Nixon LJ, Gladding SP, Duffy BL. Describing Failure in a Clinical Clerkship: Implications for Identification, Assessment and Remediation for Struggling Learners. J Gen Intern Med. 2016;31(10):1172-1179. doi:10.1007/s11606- 016-3758-3.
6. Wang FY, Degnan KO, Goren EN. Describing Failure in a Clinical Clerkship. J Gen Intern Med. 2017;32(4):378-378. doi:10.1007/s11606-016-3979-5.
7. McColl T. MEdIC Series: Case of the Failure to Fail – Expert Review and Curated Community Commentary. AliEM. https://www.aliem.com/2017/06/medic-case-failure-to-fail-expert-review- curated-community-commentary/. Published 2017. Accessed April 29, 2019.
8. Haag J. xAPI Overview – ADL Net. ADL Net. https://www.adlnet.gov/xAPI. Published 2015. Accessed May 29, 2017.
9. Downes A. Learning Record Store – Tin Can API. Rustici LLC web site. http://tincanapi.com/learning-record-store/. Published 2015. Accessed May 29, 2017.
10. Greene P, Smother V. MedBiquitous Consortium | Advancing the Health Professions Through Technology Standards. http://www.medbiq.org/. Published 2001. Accessed November 1, 2016.