E-Learning On Steroids
A Case Study of the Use of AI in Learning for Agile

An issue for organisations seeking greater Business Agility is how to scale the learning required to get staff on board with Agile product development. The training of Product Owners, Scrum Masters and the like in either physical or virtual classrooms is a must, as is their coaching and mentoring, but may be prohibitively expensive to extend to the whole organisation, depending on its size. But without sufficient understanding, people in the wider organisation may well turn into resisters of the change. E-learning on proprietary Learning Management Systems (LMS) is the go-to response, especially for big organisations, but success is limited as one-size fits all online courses fail to engage learners. Often they are forced to wade through material that bores them rigid because they feel they already know a lot of what is being presented.
Personalised E-learning
In May of last year Emerald Hill Limited launched what we believe to be the world’s first Adaptive Learning course for Agile development: Scrum Fundamentals. The course content, including quizzes and surveys, are all created by humans. It is in the presentation of that content that AI algorithms kick in. The learning is ‘adaptive’ because the learners’ understanding of the subject is being continually probed, and the platform ensures that they are only presented with content they do not yet fully understand. The result is a highly personalized, engaging experience that quickly leads learners to 100% competency. We’ve deployed the Scrum Fundamentals course to four different companies since then. One of them, an organisation undergoing a digital transformation enabled by Scrum and other Agile approaches, was enthusiastic enough to install the course on its LMS for a rollout to over 100 staff as a trial. The results are in. They are staggering.
Full Proficiency in Less Than Half the Time
The entire cohort has achieved 100% competency in their understanding of terminology and meaning of Scrum as defined in The Scrum Guide. The platform continually probes the learner with quizzes and questions, returning to topics until they demonstrate full understanding. Learners typically take the course in bite-size chunks at their own pace. End-to-end, there is 2 hours and 45 minutes of material. In the trial, the mean time to achieving proficiency was 1 hour and 19 minutes of engagement with the course. That’s 100% competency in less than 48% of the time! The minimum time taken to progress through the course to a successful completion was 25 minutes. The differences in completion times between the different learners partly reflected their varying experience and degrees of understanding of Scrum before starting the course. Only a tiny number needed to see all of the content.
The Power of AI
The platform we use is Area9 Lyceum’s RhapsodeTM . Area9 is a Danish company that has been developing AI algorithms for adaptive learning for more than twenty-five years - originally to help train doctors in the medical profession – and has helped more than 30 million learners worldwide. Our Scrum Fundamentals course is, we believe, the first to use the technology in the Agile space. Adaptive learning delivers a very close approximation to the personalisation of one-to-one training. Essentially, the platform flips the traditional training model. Instead of presenting information and then assessing it, probes are seeded throughout the course to identify what the learner already knows; what they don’t know; and, crucially, what they think they know but really don’t. Training content is then selected by the platform for presentation depending on the data from the probes. There are one hundred and forty seven probes covering the five modules and eighty five learning objectives of the Scrum Fundamentals course. The client company requested that we include a module about how they scale their Agile teams. In the public version of the course, there are 68 learning objectives and 114 probes.
Capturing Learner Data
Corporate learning and development (L&D) seeks meaningful return on investment (ROI) measures which go beyond the transactional data of the number of people taking a course or its completion rates. Metacognition, or the process of thinking about one’s own thinking and learning, plays a crucial role in adaptive learning. Data points gathered from learner’ individual responses can be measured to provide a high-level feel for how challenging a course is and whether the learners had high or low misconceptions prior to starting it. The platform we use generates information automatically which allows the client to assess improvements in proficiency, competence and confidence.
We can demonstrate, for example, that there was a 91% improvement in understanding of Scrum across the trial cohort taken as a whole. The company concerned has invested in developing an understanding of Agile amongst its staff for a number of years, so it was no big surprise to find that across all the topics 52.6% of the material was understood and the trial group were confident in that knowledge. In fact, there was a small amount of content (4.6%) where they knew more than they thought they did. The big gains were in the areas where they realised that they didn’t know material (12.8% of it) and where they had important misconceptions about it. This last category included no less than 30.7% of the course content. As a general rule, a figure of less than 10% in this category (called ‘unconscious incompetence’) is considered low and anything above 30% is considered very high. However, there are good reasons to expect high levels of unconscious incompetence when the topic is the Scrum framework. It has evolved considerably since first being made public in 1995 and the current version of 'The Scrum Guide' is the sixth update. The bottom line is that the trial revealed that for all its prior investment in both instructor-led and e-learning in Scrum, there is a huge amount of misconception that needs to be addressed. But it also showed that the Scrum Fundamentals course had, with a laser-like focus, fixed that issue in the trial cohort – and in record time.
We can also see from the data collected where in the course individuals struggled and where they achieved mastery more easily. The client’s LMS quite rightly protects the identity of those individuals from Emerald Hill and from Area9. Their single sign-on system generates a unique ID number for enrolees, but the client company can, of course, identify those individuals itself and follow up with more focussed learning and development if needed. From our point of view as trainers, we have been able to use aggregate data at a fine level of detail (for example, how long it took on average to answer a question or complete a quiz) to identify places where we can improve the course. Changes are automatically updated and are made available not only to new learners, but also to those who have already completed the course. Once enrolled a learner can access the materials for a refresh at any time.
Benefits of Adaptive Learning for Agile
Adaptive learning – like Agile product development itself - is geared for an age of accelerating, often seismic change. Skills gaps come and go, and companies have to fill them with new hires and/or the reskilling of existing staff. Compared to last generation e-learning it cuts training time in half and creates higher proficiency at lower cost. The personalisation of material means no-one is left behind. Adaptive Learning is scalable, personalized learning.
From the learner’s perspective, boredom is largely eliminated and their engagement is intensified. Retention and reinforcement of key ideas are vastly improved.
Adaptive learning is especially strong in targeting ‘unconscious incompetence’ or what the learner thinks they know but really don’t. One immediate payoff in the Scrum Fundamentals course is a common vocabulary across the cohort that is taken straight into workplace practice. Confusion about terminology and what the Scrum framework is, and what it isn’t, is eliminated.
Learners have used our course in different ways:
· As a standalone course increasing their knowledge of Scrum and agile practice
· As a preparation for in-class training (for our Certified Scrum Master and Certified Scrum Product Owner courses, for example)
· As a tool for revision in preparation for taking Scrum Alliance’s Certified ScrumMaster (CSM) or scrum.org’s Professional Scrum Master (PSM) examinations.
Based on the extraordinary success of these trials we, at Emerald Hill, will be expanding our portfiolio of Adaptive Learning courses for Agile product development and Business Agility. Keep an eye out for new announcements in the near future.
To learn more, contact Maria maria@emerald-hill.co.uk







