A lecture I attended by Jared Diamond in 2007 changed how I think of IT departments.  His lecture was in support of his book. Collapse: How Societies Choose to Fail or Succeed (2005).  While he wrote about societies, his research is equally relevant to why process improvement programs succeed or fail.  Two of the ideas Dr. Diamond put forth o are instantly germane to SPI programs.  They were:

  • Elites isolate themselves, and
  • An inability to reassess core values

Organizations that leverage the factory model of IT create stovepipes, then assign groups and roles (like SPI) to the stovepipe. This causes isolation.  How many cube farms have you been in where a person in one group may sit next to a person in an another group yet neither know each other’s name (unless is stuck to the outside of the cube)?   The stovepipe/factory model of IT has supported the creation of insular specialists, such as development methodologists, process improvement specialists, project managers and DBAs.  This isolation cuts off decision making groups from those they effect, especially when it comes to support groups like process improvement groups, PPQA and project management.  One management theory suggests that specialization in a production environment is more productive.   I would suggest that in scenarios where there are more knowns than unknowns this type of model can be very efficient.  The model of specialization tends to breakdown in scenarios where the unknowns outweigh the knows and a solution requires interdisciplinary knowledge and interaction.  Using a specialist model in the later scenario reduces efficiency, creates unrest and requires management to expend effort to control work rather than on motivation (a directive versus leadership model).   One of the unintended consequences of putting up barriers is the creation of fertile grounds for grassroots movements (such as teams deciding to use Agile methods on their own or not to use a specific tool). The creation of barriers makes work more difficult leading to a feeling of repression that can only be cured by sweeping away the barriers to getting work done and success.  These cabals may lead to behavior that may or may not be good for the organization. Turning aside the possibility of a popular revolute for a future essay, the isolation of the process improvement ‘elites’ from the general IT population will lead to inefficient solutions.  Involvement allows collaboration; it helps channel change even though overall change uncontrollable but it can dampen it, making it easier for change to generate value.   There many standard methods to try to facilitate a linkage (which different from involvement and collaboration) between groups such as programmed communication, meetings and the like.  The problem is that those solutions tend to feature multiple unidirectional flows of data, people talk at each other rather than to each other.  The standard methods such presentations and deliverable reviews provide spin controlled information but not dialogs and interaction.  In the long run (measured in months not years) the gap between the elites and the general IT population will broaden as the process developers get further away from the real work of an IT organization.

One, tried and true method for making sure the wall stays as thin as possible is to staff process improvement groups primarily from other disciplines (use process improvement as a stepping stone on a career ladder).  Adding open collaboration tools such as WIKISs, RSS feeds and blogs intensifies the interaction and create an intimacy that builds value and community. The goal is to ensure that the decisions made when designing and implementing new processes are tailored toward facilitating work and delivering value.  Involvement requires that those designing and developing processes have to ‘eat their own cooking’, in other words, live by the consequences of their actions in the crucible of software development and maintenance.

The second idea I gleaned from the lecture, the inability to reassess core values, is perhaps a more insidious problem.  Societies, organizations and even teams develop patterns of behavior to guide them.  In IT, we call these patterns frameworks, methodologies and processes.  Without a periodic reassessment of how the organization’s values are supported by implemented methods and processes it is impossible to be sure that the most efficient and effective path will be followed.  If a significant shock to the environment goes unrecognized (a change in technology or globalization) efficiency might be the last thing you worry about as your process choke output.  In short, processes and methods need to be reassessed periodically based on current conditions, organizational needs and values.  Methods and processes should not be elevated to belief structures (e.g. the church of the CMMI, cult of the eXtreme or the commune of Scrum).  What has served you in the past may not serve you in the future.

Involvement and the removal of barriers are important to delivering change that lasts, but aren’t all that is needed to generate continuous value (it is a good start though).  The problem is that change that lasts does not necessarily equate to value. What if it the change starts to impede delivering value as the environment changes?  Transparency, involvement and reassessment of software processes and methods based on current values and needs are the prescription needed to keep the ship of IT pointed towards delivering value.


A version of this essay was originally podcast at www.spamcast.libsyn.com.  Please read and listen.  Comments and suggestions would be appreciated.