E-Learning On Steroids

A O'CALLAGHAN • February 26, 2024

A Case Study of the Use of AI in Learning for Agile

An issue for organisations seeking greater Business Agility is how to scale the learning required to get staff on board with Agile product development. The training of Product Owners, Scrum Masters and the like in either physical or virtual classrooms is a must, as is their coaching and mentoring, but may be prohibitively expensive to extend to the whole organisation, depending on its size. But without sufficient understanding, people in the wider organisation may well turn into resisters of the change. E-learning on proprietary Learning Management Systems (LMS) is the go-to response, especially for big organisations, but success is limited as one-size-fits-all online courses fail to engage learners. Often they are forced to wade through material that bores them rigid because they feel they already know a lot of what is being presented.

 

Personalised E-learning

In May of last year Emerald Hill Limited launched what we believe to be the world’s first Adaptive Learning course for Agile development: Scrum Fundamentals (1). The course content, including quizzes and surveys, are all created by humans. It is in the presentation of that content that AI algorithms kick in. The learning is  ‘adaptive’ because the learners’ understanding of the subject is being continually probed, and the platform ensures that they are only presented with content they do not yet fully understand. The result is a highly personalised, engaging experience that quickly leads learners to 100% competency. We’ve deployed the Scrum Fundamentals course to four different companies since then. One of them, an organisation undergoing a digital transformation enabled by Scrum and other Agile approaches, was enthusiastic enough to install the course on its LMS for a rollout to over 100 staff as a trial. The results are in. They are staggering.


Full Proficiency in Less Than Half the Time

The entire cohort has achieved 100% competency in their understanding of terminology and meaning of Scrum as defined in The Scrum Guide. (2) The platform continually probes the learner with quizzes and questions, returning to topics until they demonstrate full understanding. Learners typically take the course in bite-size chunks at their own pace. End-to-end, there is 2 hours and 45 minutes of material. In the trial, the mean time to achieving proficiency was 1 hour and 19 minutes of engagement with the course. That’s 100% competency in less than 48% of the time! The minimum time taken to progress through the course to a successful completion was 25 minutes. The differences in completion times between the different learners partly reflected their varying experience and degrees of understanding of Scrum before starting the course. Only a tiny number needed to see all of the content.


The Power of AI

The platform we use is Area9 Lyceum’s RhapsodeTM (3).  Area9 is a Danish company that has been developing AI algorithms for adaptive learning for more than twenty-five years - originally to help train doctors in the medical profession – and has helped more than 30 million learners worldwide. Our Scrum Fundamentals course is, we believe, the first to use the technology in the Agile space. Adaptive learning delivers a very close approximation to the personalisation of one-to-one training. Essentially, the platform flips the traditional training model. Instead of presenting information and then assessing it, probes are seeded throughout the course to identify what the learner already knows; what they don’t know; and, crucially, what they think they know but really don’t. Training content is then selected by the platform for presentation depending on the data from the probes. There were one hundred and forty seven probes covering the five modules and eighty five learning objectives of the Scrum Fundamentals trial. The client company requested that we include a module about how they scale their Agile teams. In the public version of the course, there are four modules, 68 learning objectives and 114 probes.


Capturing Learner Data

Corporate learning and development (L&D) seeks meaningful return on investment (ROI) measures which go beyond the transactional data of the number of people taking a course or its completion rates. Metacognition, or the process of thinking about one’s own thinking and learning, plays a crucial role in adaptive learning. Data points gathered from learners’ individual responses can be measured to provide a high-level feel for how challenging a course is and whether the learners had high or low misconceptions prior to starting it. Rhapsode generates information automatically which allows the client to assess improvements in proficiency, competence and confidence.


We can demonstrate, for example, that there was a 91% improvement in understanding of Scrum across the trial cohort taken as a whole. The company concerned has invested in developing an understanding of Agile amongst its staff for a number of years, so it was no big surprise to find that across all the topics 52.6% of the material was understood and the trial group were confident in that knowledge. In fact, there was a small amount of content (4.6%) where they knew more than they thought they did. The big gains were in the areas where they realised that they didn’t know material (12.8% of it) and where they had important misconceptions about it. This last category included no less than 30.7% of the course content. As a general rule, a figure of less than 10% in this category (called ‘unconscious incompetence’) is considered low and anything above 30% is considered very high. However, there are good reasons to expect high levels of unconscious incompetence when the topic is the Scrum framework. It has evolved considerably since first being made public in 1995 and the current version of The Scrum Guide'is the sixth update. The bottom line is that the trial revealed that for all its prior investment in both instructor-led and e-learning in Scrum, there is a huge amount of misconception that needs to be addressed. But it also showed that the Scrum Fundamentals course had, with a laser-like focus, fixed that issue in the trial cohort – and in record time.


We can also see from the data collected where in the course individuals struggled and where they achieved mastery more easily. The client’s LMS quite rightly protects the identity of those individuals from Emerald Hill and from Area9. Their single sign-on system generates a unique ID number for enrolees, but the client company can, of course, identify those individuals itself and follow up with more focussed learning and development if needed. From our point of view as trainers, we have been able to use aggregate data at a fine level of detail (for example, how long it took on average to answer a question or complete a quiz) to identify places where we can improve the course. Changes are automatically updated and are made available not only to new learners, but also to those who have already completed the course. Once enrolled a learner can access the materials for a refresh at any time.


Benefits of Adaptive Learning for Agile

Adaptive learning – like Agile product development itself - is geared for an age of accelerating, often seismic change. Skills gaps come and go, and companies have to fill them with new hires and/or the reskilling of existing staff. Compared to last generation e-learning it cuts training time in half and creates higher proficiency at lower cost. The personalisation of material means no-one is left behind. Adaptive Learning is scalable, personalized learning.


From the learner’s perspective, boredom is largely eliminated and their engagement is intensified. Retention and reinforcement of key ideas are vastly improved.


Adaptive learning is especially strong in targeting ‘unconscious incompetence’ or what the learner thinks they know but really don’t. One immediate payoff in the Scrum Fundamentals course is a common vocabulary across the cohort that is taken straight into workplace practice. Confusion about terminology and what the Scrum framework is, and what it isn’t, is eliminated.


Learners have used our course in different ways:

·        As a standalone course increasing their knowledge of Scrum and agile practice

·        As a preparation for in-class training (for our Certified Scrum Master and Certified Scrum Product Owner courses, for example)

·        As a tool for revision in preparation for taking Scrum Alliance’s Certified ScrumMaster (CSM) or scrum.org’s Professional Scrum Master (PSM) examinations.


Based on the extraordinary success of these trials we, at Emerald Hill, will be expanding our portfiolio of Adaptive Learning courses for Agile product development and Business Agility. Keep an eye out for new announcements in the near future.

To learn more, contact maria@emerald-hill.co.uk


(1) https://www.emerald-hill.co.uk/adaptive-learning22909ce1

(2) https://scrumguides.org/scrum-guide.html

(3) https://area9lyceum.com

By A O'CALLAGHAN November 7, 2025
By Alan O'Callaghan
By A O'CALLAGHAN November 7, 2025
By Alan O'Callaghan
By A O'CALLAGHAN November 7, 2025
By Alan O'Callaghan
By A O'CALLAGHAN November 6, 2025
By Alan O'Callaghan
By A O'CALLAGHAN November 6, 2025
By Alan O'Callaghan
By A O'CALLAGHAN November 6, 2025
By Alan O'Callaghan
By A O'CALLAGHAN November 6, 2025
By Alan O'Callaghan
By A O'CALLAGHAN November 6, 2025
by Alan O'Callaghan
By A O'CALLAGHAN November 6, 2025
by Alan O'Callaghan
By A O'CALLAGHAN May 24, 2022
In 1897 Mark Twain famously complained that reports of his death were exaggerated. A hundred and twenty five years later the reports of the death of Agile similarly abound, and are just as far off the mark. In fact, according to the Business Agility Institute, Business agility – defined by the BAI as “…a set of organizational capabilities, behaviors, and ways of working that affords your business the freedom, flexibility, and resilience to achieve its purpose. No matter what the future brings.” – has increased globally in a very significant way since the start of the pandemic. Other respected authorities, McKinsey, for example, have noted a rush towards Agile ways of working as a consequence of organizations finding flaws and dysfunctions in their existing structures, cultures and processes that hinder their ability to respond to unanticipated change. One wellspring of the rumours of the death of Agile has actually been from well-known Agilists such as Dave Thomas who were actually pointing out that big consultancies have jumped on the Agile bandwagon and corrupted its meaning and intent. Way too many organizations have swallowed the snake oil that suggests Agility is about adopting a few practices in the product development area to become “faster, better, cheaper” while changing nothing else. Agile has always been about responsiveness to change while delivering products that delight customers: “faster, better, cheaper” are second order effects of lowering the cost of change. These are not obituaries, but passionate warnings about the need to defend the values and principles of Agile. A Post-Agile World? A different, but insidious and dangerous perspective comes from those who contrast Business Agility with Agile product development. To the extent that the term “Business Agility” focuses on the need for organizational-wide cultural and structural change, it reflects a welcome shift in an understanding of the challenges businesses face in every sector of the world economy. On the other hand, these are not new problems. The pandemic, supply chain disruption, the ‘Great Resignation’ and even the war in Ukraine have only exacerbated and revealed more clearly the need equip organizations to respond to change and to innovate more quickly. Changing the world of work has always been at the heart of genuine Agile thinking and practice. Yet we are welcomed to the ‘Post-Agile World’ by the authors of one acclaimed book. An Agile Straw Man The book concerned, by Fin Goulding and Haydn Shaughnessy is called Flow . First off, it’s a good book. I recommend it. It is full of interesting ideas to promote cultural change throughout a business. I love their concept of extreme visualization, of work as a learning model, of the need to co-create processes and so on. I see these as valuable additions to our understanding of Agile. But they do not. At one point in the book, they post a table that compares Flow with Lean Startup and with Agile. Agile is summarized in eight bullet points. But their ‘Agile’ is a straw man – easy to knock over because it’s not the real deal. Let’s look at those eight bullet points: “A structure for software programming”. While it is true that Agile took off in software development, it didn’t start there, and it won’t finish there. Most commentators trace Agile thought and practice back to the Shewhart cycle first promoted in 1929. Today, Saab build fighter aircraft, Tesla make autonomous cars, GKS develop pharmaceuticals and the EduScrum movement is transforming education using Agile approaches. “An a priori project-planning method”. I’m not sure where that one came from. One of the four values of the Agile Manifesto is about valuing “responding to change over following a plan.” Agile planning is data-driven. We plan on what we know rather than speculate. But in a complex world we don’t have all the information up front and some of that will change in any case. Scrum, for example has a planning event at the beginning of every iteration precisely because a priori planning doesn’t work. “Risk reduction”. Quite true. But as the authors contrast this with ‘Management of uncertainty’ in their own framework I think they have misrepresented ‘risk reduction’ in Agile as meaning ‘risk avoidance’. In reality, risk is identified and promoted in importance so that it can be dealt with early in Agile development. “Fixed process”. Just a second while I pick myself up from the floor. I’ve spent 25 years explaining to those I have coached or trained in Agile that ‘the process’ is merely the sum total of all the decisions a self-organizing team makes. As such it is adaptive and will be different for each and every team. Predefined processes are a death march in the face of uncertainty and risk. They are anathema to Agile. “Emphasis on avoiding big failures”. Again, I see here an attempt to draw Agile as a conservative risk-aversive approach compared to Flow. Once again, this is a nonsense. Agilists have always advocated “fail fast; embed the learning” as the only way to succeed in a volatile, uncertain, complex, and ambiguous world. ‘Big failures’ are avoided by the emergent behaviour of running safe-to-fail experiments that either validate assumptions or prove them wrong. Agile teams are empowered to take whatever actions the results of such experiments suggest to them. “More supervisory roles”. What are they then? A Scrum Master for example is a servant leader – the very opposite of a supervisory role. A Product Owner has some independent authority but is a peer member of a Scrum team. There is no rank amongst a Scrum team’s developers. As the eleventh Principle Behind the Agile Manifesto says, “The best architectures, requirements, and designs emerge from self-organizing teams” (emphasis added). Self-organization is defined as order which arises through frequent, local interactions rather than being directed externally. You need a peculiar form of mental gymnastics – standing on your head and turning your insides out maybe - to translate that into the idea that Agile requires more supervisory roles “Fulfills the plan”. See the discussion on ‘a priori planning’ above. Finally: “Execution method”. No! To be truly self-organizing, Agile teams have to take ownership of business goals and objectives. The very reason for having a Product Owner in a Scrum team, for example, is so that her business domain knowledge and experience is available to the team as a whole. A Scrum team’s purpose is to deliver business value. That makes it a genuine business unit, not just an execution/implementation unit. How the team achieves its goals and objectives is entirely up to the team itself. Agile Teams vs Business Agility In their blatant mischaracterization of Agile, Goulding and Shaughnessy have taken the caricature of Agile dreamed up by the “faster, better, cheaper” school of thought and put it on steroids. I suppose when you launch any new product you’ll feel the need to differentiate it from ‘competitors’, but in this case the authors of Flow are in danger of undermining the drive towards Business Agility. Many organizations need to heed the words of Nancy Sinatra: ”You keep saming when you oughta be changing”(These Boots Are Made for Walking). The question is what to change? And more specifically, what to change next? Some people like to think of the recent history of Agile as consisting of three waves: first, team Agile (single team product development); second, scaled Agile (multi-team product development) and third, Business Agility. Karim Harbott in his book, The 6 Enablers of Business Agility, describes it that way. That’s fine and good, provided you don’t think that in this so-called third wave the need to invest in Agile teams is somehow in the past. Karim doesn’t think that by the way. In general, it is clear that organizational and cultural change require leadership. But successful transformations cannot be purely top down, and often do not start that way. My friend Jim Coplien often says of empowerment that power is never given, it is always taken. Most Agile transformation drives to date have been bottom-up. Development teams sit in the heart of the value stream of the organization. The products or services they create are what deliver value to the customer. No-one is in a better position to see how existing hierarchies, processes and compliance demands get in the way. No-one is in a better position to understand which impediments and dysfunctions are the most important to remove next. It is not as if, in the so-called ‘first wave’, the vision of empowered self-organizing teams was universally achieved (even in many companies who thought they had adopted Agile). If anything, the influence of frameworks like Safe and DAD in the so-called ‘second wave’ took us all a step backwards. Scrum Masters and Agile coaches will continue to have to fight to increase the space within which self-organizing teams can operate. The measurement of success with Business Agility can only be the value delivered to customers. Business Agility cannot be achieved without continuous investment in self-organizing Agile development teams. Let no-one tell you anything different.