Experimentation is a dirty word in many organizations, however, it can be a powerful tool to drive and guide change.  Tammy Gretz shares how she adopted experiments in her practice and how she uses them to change the behavior and culture of people and organizations.

I am occasionally asked why I enjoy meet-ups so much. Meeting people like Tammy is a major part of the reason.  Thanks to the Women in Agile, Cleveland for the great introduction which lead to this interview!  


Direct Playback
Subscribe: Apple Podcast
Check out the podcast on Google Play Music

SPaMCAST 581 features a discussion on whether most agile transformations have provided teams with the technical skills to be successful with agile. Kim Pries, the Software Sensei, Jeremy Berriault, QA Corner, and I had a wide-ranging discussion covering experimentation, learning and both personal and management responsibility. 

Business Agility Conference is sponsoring this Podcast!  

Dates: March 11-12, 2020

Location: New York City, 117 West 46th Street (more…)

Safe to fail experiments are about education.

Safe to fail experiments are about learning

In IT, ‘safe to fail’ describes activities that are used to generate knowledge. Additional terms that might be used are probes, spikes, R&D and prototypes. The outcome of a safe to fail experiment can be innovation, the information necessary to make decisions or to discover alternatives.  All these scenarios are appropriate for a safe to fail experiment, however sometimes safe to fail experiment are not safe nor are they appropriate.

Safe to fail experiments are appropriate in situations where learning is required to deliver a project. You might need an experiment or a bit of prototyping to learn whether something is possible or if there is a more effective way to perform a repetitive task. For example, earlier in my career my team converted credit card portfolios from one provider to another. We used a standard process for most of the hard work. Yet, we were always looking for a way to make the process more efficient using mock conversions as experiments. The goal was to find better processes and techniques.  These experiments did not always yield process changes, but they always added to our knowledge of what would and wouldn’t work. Another opportunity for safe to fail experiments is in situations where several possible alternatives exist. You need an experiment to determine which (if any) of the alternatives make sense. A test or an experiment is considered successful if it proves either that an alternative is a good choice OR if the results can be used to cross an alternative off the list. Organizations that judge results that can disqualify an alternative as a failure are apt sort out alternatives in the market place rather than before they go into production because their experiments will not truly be safe. Finally, safe to fail experiments are useful tools in formal decision-making frameworks. For example, frameworks built to comply with the CMMI process area, Decision Analysis and Resolution (DAR), often leverage safe to fail experiments to generate data for formal, structured decision-making.

Safe to fail experiments are not always appropriate.  Many projects exist with a both fairly well understood solution and a hard deadline. In some portion of these projects the solution may not be perfectly optimal or the coolest possible solution, but it is doable in the timeframe. Experimenting to find a more optimal solution when taking the time and effort could cause you to miss the due date is not appropriate. In the credit card example mentioned earlier, we explored new ideas and optimization between projects to avoid experiments that would put the conversion at risk or cause the team to miss more of their weekend than was already necessary. Experiments that imperil the commitments a team has made or inhibits their ability to effectively deliver are not experiments and are safe to fail. Experimenting is also not appropriate where negative results are thought of as a black mark on a career. In this case straying from the tried and true approaches are generally a bad idea. This scenario tends to occur in highly politicized or overly competitive organizations. In this type of scenario if you can’t change the environment, I would suggest experimenting with other job opportunities. A third scenario is more questionable than absolutely inappropriate. In scenarios where there is little potential upside to the experiment experimentation is probably not a good idea. In most organizations, time and attention are always in short supply. Every team needs to judge what it needs to learn and ration resources that are in short supply when there is little or no perceived upside.

Safe to fail experiments are scenarios where an organization or team truly wants to sift through possible alternatives before using them in a project or in production. Safe to fail experimentation is an application of the scientific method to generate knowledge and decisions so that teams and projects deliver more value.

Experimentation in Process Improvement Programs
Thomas M Cagley, Jr

Audio Version in SPaMCAST 189

Is the use of experiments part of your process improvement culture? When I’m at conferences or networking events, I get asked whether or not an idea or technique will work in a specific organization or situation.  In many cases, the answer I give is that in reality, it depends. I know that sounds like common consultant speak but what “it depends” means is that to really understand if something could work, you need to determine how to balance organizational culture, people, skills and the assets that can be deployed. Not all tools are good.  The only way to increase the knowledge needed to forecast success of any specific change is through experimentation in a real-world environment.

Using the term experiment will send many people running for the exit.  Wikipedia defines the term experiment as a methodological trial and error procedure carried out with the goal of verifying, falsifying or establishing the validity of a hypothesis.  Restating the formal definition in terms of a software process improvement program; experiments are designed to determine if an approach will fail or to create an approach that can be scaled and rolled out.

I continually see two issues blocking process improvement programs from using experiments:  The first is the word failure.  The second is that what we call an experiment many times is no more rigorous than winging it.

In some environments experiments that don’t provide an affirmative answer can be viewed as failures.  Labeling something as a failure can be injurious to the careers of those involved.  Experiments are critical for understanding whether a process change can work in any particular environment.  Experiments lead to innovation by providing a methodical mechanism to try ideas and gauge results. The personnel involved in experiments should be provided the “high cover” required to allow them to use experiments safely.

It would be great if all experiments succeeded.  Successes in process improvement experiments generally fall into three categories.  Proving a process change will work and scale, work but won’t scale or won’t work at all.  By definition an experiment can result in verifying or falsifying a hypothesis; an experiment is only a failure if it returns the wrong result or is actively misinterpreted and if we don’t actively learn from the process of running the experiment.  That said, experimentation is not just winging it then declaring it a failure as a learning experience and an experiment.

Experiments, whether in the world of the hard sciences, or the social sciences need to follow a disciplined process.  The scientific method in it’s simplest form has five steps:

  • Formulate a question
  • Develop a hypothesis
  • Make a prediction
  • Run the test
  • Analyze the results

All experiments need to follow this process or something similar.  Determine what you are trying to prove, predict how you think the process will behave, run the experiment (listen, observe and measure,) then analyze the results. Declaring a change is an experiment after it fails misses many of the learning opportunities, as well as reduces your credibility (someone will notice).

In an upcoming Software Process and Measurement Cast, Raja Bavani of Mindtree states that process improvement can’t come just through the accident of learning from failures.  W. Edwards Deming and Walter Shewhart famously defined a methodical approach to  process of improvement through the cycle of “Plan – Do – Check – Act.”  Experimenting with process changes and ideas in a methodical and controlled manner is not only a time path, but truthfully is the only way to improve the impact and success rate of your process improvement program.