The Challenges of CULA

Authors:

  • Professor David Topps, Medical Director, OHMES, University of Calgary
  • Tamsin Treasure-Jones, Director, LIME Centre, Leeds University
  • Sebastian Dennerlein, KNOW Center, Graz, Austria
  • Matthias Traub, KNOW Centre, Graz, Austria

In the spring of 2018, the Cross-University Learning Analytics (CULA) Project was launched. This project initially included:

  • KNOW Center, Graz, Austria
  • OHMES, University of Calgary, Canada
  • LIME Centre, Leeds University, UK
  • Aristotle University, Thessaloniki, Greece
  • Innsbruck University, Austria
  • Graz University Medical School, Austria
  • Liverpool University Medical School, UK

The purpose of the project is to explore whether data from existing educational software systems, across a broad variety of schools, would be of sufficient volume and quality to be able to act as seed data for the Scalable Recommender system (ScaR), iKNOW, that is being tuned for such purposes by the KNOW Center in Graz.

The recommender system has been well proven in other areas of academic and commercial interest and has proven flexible in its ability to work with a wide range of data types. Previous brief studies with a single medical school showed that there was insufficient volume of data for the recommender system to make useful recommendations. However, it is hoped that by aggregating data across multiple universities a sufficient quantity of useful data could be aggregated.

Two approaches were envisioned for this project. Some groups planned to provide data logs for the past five years from existing platforms, such as their Learning Management Systems. Other schools were skeptical of the volume and quality of data available from their own systems and were more interested in generating a common approach to producing prospective data, tuned to the purposes of seeding a recommender system. Theoretically, these two approaches could be conducted in parallel, with the findings from existing systems being used to fine tune and better inform the types of data being prospectively collected by the second approach.

The expertise of the KNOW Center in being able to translate the many and disparate data formats, into a set of algorithms and rules for the recommender system represented a unique opportunity to the CULA Project. You cannot just toss raw data at a recommender system and expect it to create wisdom or information all by itself. Because of this, we were grateful to the Austrian Research Promotion Agency, supported by the Austrian government, which made the KNOW Center expertise and fine tuning available to CULA at a subsidized rate.

The project group met on several occasions throughout 2018 and into 2019, exploring various data sources and formats, looking for common areas of data and for ways to aggregate data from the various systems at their disposal.

Some progress was made on the prospective data aggregation approach. The group decided to use a Learning Record Store and the xAPI protocol for its infrastructure. The xAPI (formerly known as Tin Can) protocol is an open-standards approach in activity metrics. Some in the group had already been able to integrate xAPI tracking into some of their existing educational tools such as WordPress, OpenLabyrinth and Moodle. Because the xAPI protocol is very flexible and provides very detailed data, the group was quite optimistic that this approach would prove feasible and powerful enough to feed data in sufficient quantity and quality to be useful to the recommender system.

The supposed advantage of using the Learning Record Store as the intermediary is that the KNOW Center team would only need to handle a single format for incoming data, compared to multiple CSV formats from existing sources.

However, much greater challenges were faced by the schools who intended to use log scraping of historical data from existing systems. Disappointingly, there was very little data commonality between the systems, and even the quantity of data available from most LMSs was disappointing. Some of this reflects the relatively low level of use of LMSs by medical schools. While their host universities all supported an LMS of one kind or another, the curriculum structure of medical schools is typically not well aligned with the atomic unit and architecture of the LMS. In short, medical students go through rotations, not courses.

At this point, the project is on hold while the group attempts to find better quality existing data sources to work from.