“What’s the next big thing in Agile?” is a question I’ve been asked a lot over the years. Answering used to make me a feel a little awkward. It was almost as though the questioner is treating Agile as a fashion, one that includes many fads. My usual answer has been, “Whatever your team decides to make it” reflecting my deeply-held conviction that Agile in general, and Scrum in particular, is fundamentally about self-managing teams. More recently though the tone of the question has been different, and has reflected the fact that there are people now with real track records in applying Agile approaches to software development, and they have a real thirst for more learning. So, when the question was asked of me for the umpteenth time recently, I found myself answering “The next big thing is scaling Agile”.
What Does It Mean to Scale Agile?
Scaling Agile usually involves two things: developing products that require more resources than one small development team can supply, and making organizational changes to accommodate the approach. Let’s be clear, Scrum and eXtreme Programming (XP) both started with enterprise applications. There have always been big projects that have used Agile, albeit a small minority of them, but now the software development community has reached a stage – or at least is rapidly approaching it- when the use of Agile for big projects will be the norm.
The eleventh State of Agile survey has just been published by VersionOne. These reports are invaluable in helping agile practitioners understand where their practices, problems and challenges fit the context of the wider world.
As with all VersionOne’s previous reports, the eleventh survey paints a picture of the onward march of agile. Its progress is rarely in a straight line, however, and this latest survey has revealed a very interesting contradiction.
Increased Focus on Business Value?
When respondents were asked how the success of agile initiatives were measured, the second ranked answer after “on-time delivery” was “business value.” In the previous report, “business value” was the fourth-ranked answer. This time, some 46 percent of respondents chose it ahead of “customer/user satisfaction” (44 percent), “product quality” (42percent) and “product scope” (40 percent). No other answer was given by more than a quarter of respondents.
The definition of done (DoD) is one of the most important and least-understood elements of the Scrum Framework. It is specifically called out in “ The Scrum Guide ” in what is probably its biggest section, and yet, I’ve seen so-called ‘definitions’ of Scrum that fail to mention it at all.
In this post, we’ll be talking about why, exactly, the DoD is so important.
So, what is the definition of done? Fundamentally, it is the Scrum team’s agreement about the standard of quality that it will apply across the product. This concept is closely related to that of the Potentially Shippable Increment that must be created at the end of each and every sprint. The two words in that phrase that the DoD concerns are “potentially” and “increment."
While all agile approaches – Scrum included – aspire to “deliver early and deliver often,” this does not mean that a product must be handed over to the customer at the end of every sprint. Whether enough useful value has been accumulated to warrant a product’s release is a business decision, and one that is the product owner’s responsibility to make.
Governance seems to be one of those frightening words that threatens to stop an Agile transformation effort dead in its tracks. I’ve been hearing it whispered, and even screamed once or twice, quite a lot recently. There’s no big surprise here. As the big corporations and Government agencies get increasingly fascinated by frameworks like Scrum, they are mandating their IT departments to “Go agile” and then, sooner or later…governance!
Types of Governance
There is operational governance represented in the defined processes that organizations and teams are expected to follow when software is in production. There is project management governance, perhaps dictated by PRINCE2 or similar, while the product is in development. PRINCE, by the way, is an acronym standing for ‘PRojects In a Controlled Environment’.
Project management governance is often a subset of a wider IT governance. According to the TOGAF version 9.1 , IT governance supposedly provides the framework and structure that links IT resources and information to enterprise goals and strategies. “IT governance institutionalizes best practices for planning, acquiring, implementing and monitoring IT performance…” Standards like COBIT, which stands for Control OBjectives for Information and related Technology, might be in place. And then, of course, there is a range of issues to do with compliance to the requirements of external regulators in all public bodies, as well as commercial institutions in sectors such as insurance and banking. So, is this a case of an irresistible force (Agile) meeting an unmovable object (governance)?
Published on 23/3/17 on Learning Tree page. Click here to read more
“We need to get better at estimating,” an experienced member of a Scrum development team once told me. “Management is getting concerned that we keep coming up short on our commitment.”
“Really?” I responded. “What have you been committing to?”
“Thirty story points” she said. “We get there about 50% of the time. In a couple of recent sprints, we’ve even exceeded thirty points, but the last sprint marked the third time this quarter that we fell short of our target.”
“Why was your forecast off target, do you think?” I asked.
“Well, things come out of left field occasionally. You know, stuff that couldn’t be anticipated in sprint planning,” she answered.
“So, why are you estimating effort if you can’t predict what will happen in the sprint?” I said.
Now, at this point I want to make clear that I am not one of those who say that development teams should not estimate effort. For me, the ability to estimate independently is an important part of the autonomy of teams. What I was trying to do in this particular conversation was to get the team member to consider why teams estimate in the first place, and what “commitment” means in that context.
It is not just a matter of those doing the work being the only ones who are able to estimate effort, which I believe to be true. I recently participated in a large multi-team project of effort being estimated by a centralized systems analysis unit. In that project, the development teams were involved in estimating, but it was the systems analysts’ forecast that was being communicated to the customer. The result was massively inflated customer expectations, and retrospection revealed that the teams’ estimates were about four to five times larger—and thus far more accurate--than those of the analysts.
“Ready? Steady! Go!”. That phrase echoes from my youth. I used to twitch waiting for the starter to shout that phrase at the beginning of the 800m races I ran, representing my school. It was also the title of a ground-breaking pop music show which caused me to fall in love with Dusty Springfield, Sandy Shaw and every other female singer who appeared (I was a very impressionable teenager). It resonates today for me in Scrum.
When Development teams pull Product Backlog Items (PBIs), or user stories, from the Product Backlog into a Sprint they need to be ready to hit the ground running immediately after the Sprint Planning meeting. For that to happen the PBIs must be in a state that allows them to do that.
The pushmi-pullyu (pronounced “push me-pull you”) is a fictional creature in “ The Story of Doctor Dolittle ,” which is described as a cross between a gazelle and a unicorn.
Recently, I was talking with a fellow software professional about some issues he was having implementing Kanban, and the pushmi-pullyu came to mind. Allow me to explain why.
Kanban is the most popular agile approach after Scrum, according to the most recently published State of Agile Survey . Scrum is, of course, massively dominant.
Only 5 percent of respondents described their teams as Kanban teams, while three-quarters used either Scrum, a Scrum hybrid or Scrumban. However, when asked what techniques they used, nearly four in every 10 teams said that they use Kanban boards.
This is might be an exaggerated figure since, in my experience, many people confuse Kanban boards with Scrum boards. They look similar, but have very different purposes.
While a Kanban board maps a value stream for the lifetime of the product, a Scrum board is a visualization of a sprint backlog, and is reset at the beginning of every new sprint.
Additionally, the Scrum board is owned by a single Scrum development team, while Kanban is agnostic about who owns the board. These differences turned out to be at the heart of the issue that my colleague raised with me.
“What is the difference between an up-front requirements document and a fish?” said Gollum, hoping to catch the Hobbit out with his riddle. But Bilbo Baggins, though a Halfling, was an experienced Agilist. He knew the answer. “A fish rots from the head down, “ he said, “but a requirements documents rots from the bottom up”. “Curses!”, screamed Gollum, “You cheats us”. But Bilbo wasn’t cheating. He was just drawing on experience.
Detailing all of the requirements out up front is a colossal waste of effort in the development of software products. The average churn after the document has been signed off is said to be about 35%, although most software professionals when I ask them to give an estimate, tend to give me a much higher figure . In any case the point is, that by the time the requirement is reached, there’s a good chance it will have either significantly changed or disappeared completely from the To-Do list. The effort spent in eliciting needs, analysing them and documenting the system requirements that meets those needs has been wasted. That in turn entails a cost: the salaries of the professionals involved for a start. And now at least part of that effort and cost has to be spent again in understanding the new situation.
(Originally posted 2014)
I have recently been reading The 7th Annual State of Agile Survey (1). It has some interesting statistics. Not surprisingly, Scrum is still the dominant approach in Agile projects by a country mile. 54% of respondents use Scrum and another 11% use a Scrum/XP hybrid. The named approach with the highest reported penetration after that is Scrumban with just 7% of 'the market'. The results concerning the role of the Product Owner were much more concerning, only 51% of respondents reported that they have a dedicated Product Owner and, most worryingly of all, Product Owners are listed at the bottom of those who are most knowledgeable about Agile in their companies (only 1% of respondents chose the Product Owner).
These figures just don't add up. The Product Owner function is one of the three key roles in Scrum. You cannot truly be using the Scrum framework if there is no Product Owner in your Scrum Team, and you certainly will not be as high performing as you can be if the Product Owner does not understand Agile (let alone Scrum). So what's the problem?
Ensuring the Value of the Product
Perhaps a clue to at least part of the explanation can be found in a comment made to me recently, "The Product Owner is a kind of a waiter, yes?". For the commentator - at a client's site - the customer raises requirements just like the customer in a cafe or a restaurant. The development team, like the kitchen staff, create the solution. The Product Owner is, like the waiter, a go-between who informs the kitchen (development team) of the order and then delivers the meal (product) to the customer's table. If that was all the Product Owner had to do then there might be a good reason for either having no-one in the job at all, or rebadging some existing function (a Business Analyst, perhaps) without developing in them any special skills or knowledge about Agile.
But in Scrum, the Product Owner is responsible for the Return on Investment (ROI) in the product, and the Total Cost of its Ownership (TCO). She has to ensure the value of the product to the customer, and the value of the work done by the development team. This is a heavy responsibility: Ken Schwaber once described the Product Owner as "the one wringable neck" in the Scrum Team (2).
For sure, when the customer is not directly accessible by the developers, the Product Owner acts as a customer proxy, but to play even that aspect of the role properly requires the Product Owner to be proactive in creating an intensive collaboration with the customer to understand their Business Value Model. This is so that the relative value of the different features being requested can be understood and communicated to the developers.
Ordering Product Backlog Items vs ‘Prioritising’ Them
Many organisations understand that at least, but there is an even deeper aspect to the role of the Product Owner that has not been grasped so well. When Schwaber and Jeff Sutherland revised The Scrum Guide in October 2011 (3) they removed the references to 'prioritisation' in relation to Product Backlog Items (PBIs) and the Product Owner's responsibility for them. In too many projects this had been interpreted as the Product Owner categorising PBIs by value ('High, Medium, Low or using MoSCoW rules or something similar) and then leaving it up to the Development Team to decide what the build order of the PBIs should be in any given iteration or release. The Guide now talks about the Product Owner's responsibility for ordering the PBIs in the Product Backlog 'top to bottom'.
What is the significance of this change? Of course, the Product Owner is still seeking to get higher valued features delivered earlier than lower value ones. This typically will involve some categorisation of PBIs by value. But the Product Owner also has to take on, amongst other things, the push back from the Development Team about technical dependencies between those features, and from other stakeholders, such as System Architects, who might be pointing out some architectural issues, especially in legacy systems. Another consideration is that what the customer will expect to be seeing in any Sprint Review is not a collection of discrete "done" elements, but a small and skinny version of the product as a whole to which more 'flesh' can be added in later iterations. Any one of these concerns could lead to lower value features being bundled together with higher value ones in any given Sprint Goal.
The point is that the intense collaboration that goes on between the customer and the Product Owner is married with an equally intense collaboration between the Product Owner and the Development Team. The Product Owner is right at the centre of some of the key relationships that make the whole thing work. It is a hugely demanding role, and one which is strategically important to the success of any Scrum-driven project.
The intensity of the collaborations does not diminish, by the way, as the project progresses. The Product Backlog is not ordered top-to-bottom once and then left alone. The weight of any of the factors that led to the initial decisions can change, and other unforeseen ones emerge, as the Scrum Team surfaces new information through the development process itself. The ordering of the Product Backlog is therefore being constantly revisited, and constantly changed as the Team, and the project's stakeholders, learn more. But remember, only the Product Owner has the authority to make those changes to the Product Backlog.
The Importance of the Product Owner
If you think about it, the ordering of the Product Backlog in Scrum is the equivalent of the 'critical path' in traditional, plan-driven management of software projects. Project Managers in those projects spend time and effort sequencing activities in such a way that the dependencies between them lead to the shortest possible sequence - the critical path. This in turn determines the ordering of the work by the developers and the scheduling of other resources. In Scrum the Development Team is cross-functional, self-organising and self-managing. No-one can tell it how to do its work. The need for managing sequences based on different activities carried out by different specialists in those activities is gone. Instead the developers self-organise around the order of the PBIs in the Product Backlog by carving out a Sprint Goal from the top that is within their capacity for development in a single Sprint. And the Product Owner 'owns' the Product Backlog. That's why the Product Owner role is so important.
It is also the Product Owner who determines when enough value has accumulated in the product for it to be released, and the Product Owner who, alone in a Scrum Team, has the authority to cancel a Sprint for whatever reason. In addition, of course, it is the Product Owner who forms an agreement about the quality standards of the project with the developers in the shape of the Definition of Done
Now, what waiter tells the kitchen when there's enough of a meal on the plate for it to be served? Or negotiates with the chef and the cooks about the quality standards of the food? In what restaurant does the waiter dictate the order in which the food items will be created?
We will talk more about the role of the Product Owner and the Product Backlog in future articles, but hopefully even in this short space we have established that the Product Owner function is one that deserves respect and attention. Product Owners in Agile teams need the authority and responsibility to do the job properly, and that means their skills have to be developed - through training, coaching, mentoring and experience - to the appropriate level. Organisations that ignore or neglect the role of the Product Owner are just storing up problems for themselves in the future.
Emerald Hill Limited
(1) 7th Annual State of Agile Development Survey. VersionOne. www.versionone.com . 2013
(2) Schwaber,K. The Enterprise and Scrum. Microsoft Press. 2007
(3) Schwaber K., and J. Sutherland. The Scrum Guide. www.scrum.org . October, 2011
You might not have noticed, but The Scrum Guide was updated recently (July 2016) for the first time in three years. The changes aren’t big in terms of word count, but they are significant. So I decided I needed to update my own description (1, 2) of the Guide’s evolution.
Cumulative Flow Diagrams
Let’s deal with the smallest change first. Cumulative Flow diagrams (CFDs) have been added to the list of example ‘information radiators’ that a Scrum team might use to track its progress towards its Sprint Goal. CFDs are associated popularly with Kanban – though I think they started life in Feature Driven Development – so it might be a bit of a surprise to some to see them mentioned alongside burndown charts and task boards. Don’t worry though, it is just that – a mention. You don’t have to use any specific charts or diagrams to use Scrum, so there’s nothing fundamental being indicated here.
The big change is a paragraph that reintroduces the five values of Scrum. They’ve been around since 2002 at least when they were described in a book by Ken Schwaber and Mike Beedle (3). They’ve never really gone away. Although they have not previously been mentioned in The Scrum Guide they have always been included in the curriculum for the Scrum Alliance’s Certified Scrum Master and Certified Scrum Product Owner courses, for example.
My grandson, Finnlay, aged nearly 3, wants to know how Santa manages to make and deliver all the toys to all the children in all the world for Christmas. The answer “Christmas Magic” may be beginning to wear thin.
So here’s my new version. Santa does a lot of work through the rest of the year, long before we get to Christmas. He gathers all the information about which boys and girls are being naughty and which ones are being nice, for a start. Then he has to find out where they live, the size of their chimneys, and what toys they would like. Based on this information he has a list – which, of course, changes a lot as new information is discovered or older information goes out of date. The most important deliveries (to the children who have been most good) go to the top of the list. Santa wants to guarantee that the good kids (“Like you, Finn”, I say, fingers crossed behind my back) get their presents on time.