Actionable Agile Metrics for Predictability by Daniel S. Vacanti, Re-read Saturday Week 14: Service Level Agreements

A New Copy!

Chapter 12 of Daniel S. Vacanti’s Actionable Agile Metrics for Predictability: An Introduction (buy a copy today) is titled Service Level Agreements. Service Level Agreements (SLA) are an agreement to perform within certain limits based on past performance. SLAs can be derived based on data visualization. As with previous chapters, Vacanti begins with a reminder. In this case, the reminder concerns the percentile lines that were drawn on scatter plots in chapter 10. The percentile lines tell us the probability of a story completed in an amount of time up to a specific cycle time. In our discussion of chapter 10 we drew a line on a scatter plot where half of the observations were above the line and half below. The percentile line can be used to define an SLA.

Vacanti suggests that he would rather call service level agreements ‘service level expectations’ or ‘cycle time targets’. The term SLA has been used to in contractual agreements to create absolute expectations that often include payments and penalties. Using terms like expectations and targets helps the people using the data to remember that the SLA will never be absolute. For example, if we use an 85 percentile line we know that experience shows that 15% of the stories in the past have exceeded whatever the number of days the 85% line indicates.

The visualized data from the scatterplot allows everyone in the value stream of work to be able to talk to when things will be done within certain degrees of confidence. We need (as Vacanti did in the book) remind ourselves that missing an SLA is an opportunity to learn. This is typically at odds with how SLAs are used in most contracts.

Another topic tackled in the chapter is how data points do you need to set a service level expectation. The number of observations required is an unknown. The exact number will be a balance of how much confidence is needed, risk tolerance and the variability of the performance being measured. I personally have set expectations with just four data points. It is dependent on the quality of the data. The lower the quality, the more data you need to be predictable. Many organizations and teams wait until they oodles of data trying to attain statistical significance. Arguably, one of the best pieces of advice I got many years ago was to start collecting and using data and then inspected and adapt to make it better. In essence, I was being told that if you waited for perfection, you’d be waiting a long time.

Vacanti suggests that there are three mistakes often made when setting SLAs or expectations. They are:

Setting an SLA without regard to the performance of the team or organization.
Allowing someone external to the work to set the SLA.
Setting an SLA without collaborating with your stakeholders.

Each of these three mistakes will cause poor behaviors and will negatively impact predictability. It is hard to conceive how any of these three mistakes are ever considered a good idea. For example, how an SLA can set if the team’s data is not used. Ignoring perfectly good data is at BEST wishful thinking.

Chapter 12 includes a discussion of whether it makes sense to set different levels of expectations for different types of work. Vacanti suggests this is an advanced topic and that different classes of service will be tackled in Chapter 13. The practical advice delivered is to achieve a level of consistency and predictability for the entire process before considering slicing and dicing work into thinner and thinner slices.

Part of setting an expectation is not just the data but rather it should include the work that goes towards meeting that expectation. The section on right-sizing talks about the need to validate whether a team believes an item can be completed within the expectation set by the SLA. Vacanti assures the readers that he is not suggesting heavy duty planning and estimating. An approach to right-sizing can provide significant benefits, in addition, to help ensuring the piece of work is the right size. My experience is that a team-based sizing process generally evokes significant conversation that fills in the gaps and spreads information and knowledge across the team.

Vacanti completes the chapter with a discussion of using the percentile lines as a trigger for intervention. Triggers for intervention help teams to decide when they are getting close to missing an expectation so they can do something before things get out of hand. The statement, “the older a work item gets, the greater it has of aging still more” drives home the point that teams need to act as items get older and before they start a runaway aging process which will cause the team to blow by as the expectation (or SLA). The point is that once we have passed that line it becomes difficult to communicate credibility when the item will be done. As items get older and begin to approach the SLA the percentile lines can be used as a trigger to focus attention and action. I would suggest that the last two sections of the chapter, right-sizing and triggering intervention, are part of becoming and staying predictable.

The major theme of the book has been on providing a platform for teams to become predictable; being predictable grants teams special powers with their stakeholders. SLAs are one such power. We can now communicate with our stakeholders in a way that is more than a guess and an apology packaged in the same conversation.

Previous Installments

Introduction and Game Plan

Week 2: Flow, Flow Metrics, and Predictability

Week 3: The Basics of Flow Metrics

Week 4: An Introduction to Little’s Law

Week 5: Introduction to CFDs

Week 6: Workflow Metrics and CFDs

Week 7: Flow Metrics and CFSs

Week 8: Conservation of Flow, Part I

Week 9: Conservation of Flow, Part II

Week 10: Flow Debt

Week 11: Introduction to Cycle Time Scatterplots