Last week I was reading „The 2015 State of Scrum Report“ from Scrum Alliance. In February 2015, Scrum Alliance surveyed almost 5000 people about their use of Scrum. The survey respondents make up a diverse group, representing 108 countries and 14 industries. They reflect a range of functional areas, including IT software development, product development, operations, human resources, executives and sales and marketing (you can download the full report here).
Whilst reading, two questions and their responses caught my attention.
1) What were some of the challenges faced by your organisation in achieving its goals with Scrum?
The number one answer with 52% was “We did not have clearly identified metrics to identify and measure the success of Scrum projects and delivery”.
2) Considering all the projects in your organization that were managed using Scrum, what percent of the time would you estimate they were delivered successfully?
The answers were as following:
a. 75%+ of time: 42%
b. 60-75% of time: 32%
c. 25-50% of time: 14%
d. 0-25% of time: 12%
Regarding the first question it is hard for me to understand that organizations are not able to measure the success of a project/delivery. For me this is independent of Scrum, cause the same measurement can be used by any other project methodology.
Yes the sentence “full scope, on time, in budget and high quality” will not work with Scrum, but it does not for Waterfall either. This only states that your planning was correct (or that your project adapted to the planning). It does not say anything about the successful outcome of the project itself.
Depending on your project you can use a set of metrics like:
– Employee / Project Member satisfaction
– Stakeholder satisfaction
– Customer satisfaction
– Quality delivered to customer
– Time to market
– Return on Investment
– Meeting business case objectives
– Customer adaption
– Meeting governance criteria
– Benefits realization
When you look at this list of metrics you will notice that none of the typical Scrum Project metrics like Techncial Debt, Retrospective Items or Velocity are listed. This is because those metrics measure the quality of the Scrum process and are focused on improving Scrum teams. They do not say anything about the success of the project itself. When defining your own organization’s metrics it very important to make this difference.
What you can also see is that none of the metrics is Scrum specific. Nor specific for Waterfall or any other method. So when the answer is “We did not have clearly identified metrics to identify and measure the success of Scrum projects and delivery” this is the same as saying «I have no metrics in place to identify success of any project or delivery in my organization«. Which is pretty scary.
The reason why the second question caught my attention is the relation with the first. When you have no project success definition how can you answer the question regarding project success rates?
