Book Cover

Why does leadership bring a release home to great adulation only to have the next release crash and burn? Did the leader’s skill change between releases or were other random factors, such as luck, involved. Kahneman suggests a simple formula as a thought experiment.  Success = skill + luck. Chapter 17 of Thinking, Fast and Slow, Regression To The Mean, discusses correlation and causal interpretation.  

The chapter begins with a story of Kahneman training a group from the Isreali Air Force about the psychology of effective training.  When pointing out that praise for a good job was more effective in improving performance than punishing mistakes, a senior trainer disagreed.  The trainer’s said that his experience was the opposite. Both Kahneman and the trainer were correct, reflecting a reversion to the mean.

There are several reasons why understanding that all processes have some degree of natural variation. The most pressing reason can be traced back to our almost single-minded need to assign a causal interpretation to the fluctuations of process.  As most of this blog’s readers know I am a runner, not a great runner, but pretty consistent. Nearly every weekend I run the same course (flat and very few street corners). My time varies, sometimes I am faster and sometimes I am slower. When I get home and compare my time to the previous week, I almost always fall prey to telling myself a story (creating a causal interpretation).  The sad part is that I know better even though the right “explanation” is that I am experiencing the combination of regression to the mean and the randomness. There is not always a causal story to tell.

Going back to our re-read of How to Measure Anything, Chapter 10 describes Bayesian Statistics.  One of the takeaways from that chapter was the myth that correlation is evidence of causation.  Kahneman makes a further point that is easy to lose sight of; whenever the correlation between two scores isn’t perfect, there will be a regression to the mean.  An example of a perfect correlation is the relationship between dawn and sunrise. There are very few perfect correlations in software development which argues against assigning cause to specific events. Dr. Deming describes the distinction between common cause (variance inherent in the process subject to regression to the mean) and special cause (an outcome not explainable through natural variation).   

Understanding the concept of regression to the mean is critical to anyone that thinks that inspect and adapt and/or experimentation are important tools for process improvement.  Kahneman reminds the reader that the only way to ensure the is a relationship between any input and outcome is to have a control group. Realistically, most agile teams are not going to have the wherewithal for experiments with control groups. In that case, they will need to measure the impact of changes by monitoring the long term performance trend of the team. 

Remember, if you do not have a favorite, dog-eared copy of Thinking, Fast and Slow, please buy a copy.  Using the links in this blog entry helps support the blog and its alter-ego, The Software Process and Measurement Cast. Buy a copy on Amazon,  It’s time to get reading!  

The installments:

Week 1: Logistics and Introduction

Week 2: The Characters Of The Story

Week 3: Attention and Effort

Week 4: The Lazy Controller

Week 5: The Associative Machine

Week 6: Cognitive Ease

Week 7: Norms, Surprises, and Causes

Week 8: A Machine for Jumping to Conclusions 

Week 9: How Judgement Happens and Answering An Easier Question 

Week 10:  Law of Small Numbers 

Week 11: Anchors 

Week 12: The Science of Availability 

Week 13: Availability, Emotion, and Risk 

Week 14: Tom W’s Speciality 

Week 15: Linda: Less Is More 

Week 16: Causes Trump Statistics