Mythical Man-Month, No Silver Bullet, Refired, Part 17

The Mythical Man-Month

In the essay No Silver Bullet, Refired, Brooks reexamines his essay No Silver Bullet (aka NSB or last week’s re-read) nine years after its original publication date. Both essays were additions to original 1974 The Mythical Man-Month as Brooks sought to project the course of the software development industry.  As sadam510 pointed out on the blog last week, we have passed out of the traditional Re-Read territory and are now cutting through ‘fresh’ snow.  NSB was originally published in 1986, and NSB, Refired was published nine years afterwards both to address the criticisms and to examine how the world had progressed. 

Brooks begins the essay by addressing some of the misunderstandings that were generated by his particular brand of prose (several readers have commented that these later essays are a reflection of Brook’s involvement in academia, that is, the straight-on writing style of the earlier essays had become more difficult to understand). The first of the misunderstandings he tacked was the distinction between essence and accident. Brooks reminds the readers that the word ‘accident’ was used to mean incidental. The essence of a software application is the logic needed to solve the business problem, while the coding language or development environment are incidental to solving the business problem. The second point of misunderstanding focuses on the how much of the effort or complexity of creating an application was caused by incidental factors versus the essence. Brooks argued in NSB that the essence was the most significant factor, and in NSB Refired he reiterated that stance. While both essence and accidental are meaningful, in order to generate an order of magnitude change, the complexity of the essence must be reduced.

The second thread in the essay featured a rebuttal of David Harel‘s analysis/criticism. Brooks felt that Harel’s criticism was one of the most well-formed, therefore he spent significant effort to rebut it. Harel criticism begins by arguing that Brooks’s conclusions were pessimistic and gloomy.  On the pessimistic side Brooks defended his perspective by suggesting he is at worst he was realistic based on his developer’s background. Developers tend to be optimistic about the ability to solve ANY problem (note Dr. Ricardo Valerdi in his interview on SPaMCAST 84 provided evidence to support this position). Brooks stated that taking an overly rosy view as suggested by Harel of the potential for change was a pipe dream.  In Brooks’s opinion, pipe dreams do not create the pressure needed for progress. On the topic of gloomy, Harel suggested that Brooks perspective grew out of three sources:

1. The sharp delineation between essence and accident, which he believed was artificial.
2. The 10-year time horizon used by Brooks was overly short.
3. Analyzing each silver bullet separately ignored the relationships and potential synergy between the potential silver bullets.

As a thought experiment in the critique, Harel applied the essence and accident analysis Brooks used in the NSB to the development world of 1952. Harel suggests the structure used by Brooks would only fit in a very specific timeframe, and therefore, would not necessary hold in the future. Brooks rejects the thought experiment by pointing that as soon as large-scale systems began to be addressed (approximately 1954), the analysis approach works.

Harel completed his criticism by proffering his own silver bullet based on a system of modeling the concepts of the application (in Brooks’s language, the essence). There was an undercurrent in the essay that many of those criticizing NSB were doing so to suggest a silver bullet. In this case, while Brooks agreed with the idea of leveraging models to identify and develop the conceptual basis of the application, he noted that most developers develop their own highly individualized conceptual models of what they are working on and that they are not typically generalizable.

Another class of criticisms suggests that Brooks’s focus on productivity in NSB was misplaced. Rather, as noted by Caper Jones (former boss, friend, and mentor), the focus should be on quality and then productivity will follow. Brooks suggests whether focusing on quality or productivity the conclusions of the NSB don’t change. (I think I will ask Capers his opinion…)

After reviewing the criticisms, Brooks circles back to review what has happened to productivity in the years between NSB and NSB, Refired.  Capers Jones’ data suggests that for comparable programming languages it had productivity had increased three-fold; while Ed Yourdon had evidence of up to five-fold increases. Large changes, but as Tom DeMarco indicated, not an order of magnitude increase.  In order to understand where the productivity improvement had come from Brooks re-examined some the major categories from the NSB. 

Build versus buy had become a powerful tool (and still is in 2015).  The ability to customize software packages easily has continued to increase the power of commercial off-the-shelf software.  Packages like SAP and PeopleSoft are perfect examples of the increased productivity configurable/customizable packages deliver.

Brooks suggests that object oriented development (OO) represents at least a brass bullet. OO has delivered value and continues to show promise.  Even today OO continues to grow in use and value (even though OO has not fundamentally changed productivity by an order of magnitude and is therefore, no silver bullet). In hindsight from 2015, OO has continued to penetrate and deliver value, but it has not changed the world . . . yet.

Reuse Brooks found was being done in pockets but that it had not spread to the degree he had suggested in NSB  would be needed to actually be a silver bullet. The advent of natural language programming languages may have inadvertently added complexity to adopting reuse programs. Natural languages make finding patterns to reuse harder. While not an insurmountable problem, they add complexity, effort and the killer  – cost, which has slowed the adoption of reuse. My personal observation from 2015 suggests that reuse is still a marginal practice due to the overhead required to identify and package generalized functions.

In the end, despite all of the criticisms and critiques, Brooks’s position is unchanged.  He states that “complexity is the business we are in and complexity is what limits us.” Until complexity is conquered there will be no silver bullets, only incremental improvements. Not that incremental improvement are anything to sneeze at!

Programming Notes:  My version of the Mythical Man-Month includes one large essay, The Mythical Man-Month after 20 Years and an epilogue.  I suspected we will complete the Re-Read in two weeks.  Suggestions for the next book in the series?

Previous installments of the Re-read of The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

Aristocracy, Democracy and System Design

The Second-System Effect

Passing the Word

Why did the Tower of Babel fall?

Calling the Shot

Ten Pounds in a Five–Pound Package

The Documentary Hypothesis

Plan to Throw One Away

Sharp Tools

The Whole and the Parts

Hatching a Catastrophe

The Other Face

No Silver Bullet – Essence and Accident In Software Engineering

 

The Other Face, Re-Read Saturday: The Mythical Man-Month, Part 15

The Mythical Man-Month

The Other Face is the 15^th installment of the Re-Read Saturday of the The Mythical Man-Month by Fred P. Brooks.  In this essay Brooks tackles the thorny issue of documentation. Both code and documentation are the manifestations of a message.  Code is the tool though which the programmer communicates with the machine. Documentation is the tool through which the programmer communicates with world outside the operating systems. Both communication paths are critical to delivering the most value possible. While documentation is widely acknowledged as need, the downside is that, from time immemorial, documentation has been viewed as a scourge, a pain, and sometimes as a punishment; it is designed to keep developers from doing real work – creating code. Leaders and teachers need to find a means to surmount this seemingly natural hesitancy. Brooks’ solution is to show them how to do the documentation needed, rather than relying solely on exhortation.

 

Why did the Tower of Babel fall? Re-Read Saturday: The Mythical Man-Month, Part 7

The Mythical Man-Month

In the seventh essay of The Mythical Man-Month, Brooks returns to the topics of organization and communication using story of the Tower of Babel as a metaphor for many of the engineering fiascoes seen in software development. In the story of the Tower of Babel, when communication broke down, coordination failed and the project ground to a halt. As projects become larger and more complex, communication and organization always becomes more difficult and more critical.

The essay Why Did the Tower of Babel Fall? presents three mechanisms to facilitate communication.

1. Informal Communication, which includes all of the telephone calls and conversations that occur in and among teams. Healthy teams and teams of teams talk, interact and share information and ideas.
2. Meetings are more formal and include reviews, technical briefings, demonstrations, sprint planning and release planning events. Done well formal meetings, whether part of an Agile framework or not, can find problems and share information and knowledge.
3. Project Workbooks are the repository of the evolving intellectual property that defines any significant endeavor.

Brooks goes into depth on the definition and structure of the project workbook. While the workbook that Brooks describes is a physical construct we should consider that the idea of workbook is less of a document then a structure for grouping information. All the project documents need to fit into the workbook and be part of the overall structure of project information so that it can be reused and inherited from. The ultimate goal of the workbook is ensure that everyone who needs information has access to the information and gets the information.

Workbook Mechanics: In order to meet the goal of ensuring of everyone access to the information they need, the workbook needs to be maintained and organized. For example, Brooks suggests ordering and numbering all project memorandums (we would call these emails now) so that people can look through the list and actually see if they need information. Brooks points out that as size of the effort, project or program goes up, the complexity of maintaining and indexing the project communications increases in a non-linear fashion. Today’s technical tools like wikis, SharePoint and hypertext markup make curating and sharing information easier than paper and microfiche (if microfiche is before your time, Google it). As a reminder, capturing and reusing IP is required in all types of work. As a coder, it is easy to fall prey to the idea that the code and a functional system is the only documentation required; however, this is false. For most large efforts, capturing decisions, designs, architectures and even the occasional user manual is an option that can be ignored only if you want to court a communication failure. Remember however that over-reliance on documentation is not the goal.

Brooks uses the formula (n²-n)/2, where n is the number of people working on an effort, to define the possible communication interfaces. As the number of people increase the number of interfaces increases very quickly. Organization is a tool to reduce the amount of interfaces. This is one of the reasons why small teams are more effective than large teams. Scaling Agile is often accomplished by assembling teams of small teams. We use the metaphor of trees or flowers are often to describe the visualizations of team of teams. Communications spreading from a central point can look like a tree or flower. Organizations leverage techniques like the specialization and division of labor to reduce the number of communication interfaces.

There is often push back on division of labor and specialization in Agile. This push back is often misplaced. Within a team the concept of T Shaped people, which eschews specialization, increases capabilities; however, teams often specialize thus becoming capability teams. Capability teams often tackle specific types of work more effectively and efficiently by reducing learning curves. Capability teams fit into Brooks’ ideas on organizing to reduce the communication overhead.

Brooks proposes that each grouping of work have six essential needs.

1. A Mission,
2. A Producer,
3. A Technical Director or An Architect,
4. A Schedule,
5. A Division of Labor, and
6. Interface Definitions Among the Parts.

Four of these essentials are easily understood; however, the roles of the producer and technical broke new ground and extended the ideas Brooks began in The Surgical Team. The role of the producer is to establish the team, establish the schedule and continually acquire the resources needed to deliver the project. The role of the producer includes many of the classic project and program management activities, but also much of the outside communication and coordination activities. When small effort with a single team the roles of the producer and technical director can be held by the same person. As efforts become larger and more complex the roles are generally held by separate people. The role of the producer in the Scaled Agile Framework (SAFe) is handled by the Release Train Engineer. The technical director focuses on the design and the technical components (inside focus). In SAFe this role is held by the System Architect.

Brooks leverages the metaphor of the Tower of Babel to focus on the need for communication and coordination. Projects, team and organizations need to take steps to ensure communication happens effectively. Techniques like meetings and project workbooks are merely tools to make sure information is available and organized. Once we have information organization provides a framework to ensure information is communicated effectively. Without communication any significant engineering effort will end up just like the Tower of Babel.

Are there other mechanisms for organization, communication and coordination that have worked for you? Please share and lets discuss!

Previous installments of the Re-read of The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

Aristocracy, Democracy and System Design

The Second-System Effect

Passing the Word

 

Re-Read Saturday: The Mythical Man-Month, Part 4 – Aristocracy, Democracy and System Design

The Mythical Man-Month

Today we re-read the fourth essay in The Mythical Man-Month titled: Aristocracy, Democracy and System Design. In this essay, Brooks deals with the role of conceptual integrity in building systems and the impact organizationally of getting to an appropriate level of conceptual integrity. According UC San Diego, conceptual integrity is the principle that anywhere you look in your system, you can tell that the design is part of the same overall design. Brooks suggests that conceptual integrity is most important consideration in system design. He begins the essay by building the case that conceptual integrity leads to high ease of use, and therefore higher value. Systems with conceptual integrity have one set of design ideas and that ANY idea or concept that violates conceptual integrity must be excluded. Conceptual integrity is important because systems that are based on any different architectural concepts are both hard to work with and maintain.  For example, many of the houses in my neighborhood began life as lake cottages. Many of these cottages have been added to over the years in a fairly haphazard manner. Over the last few years many have been knocked down due to high cost of upkeep that their complexity generates. Software systems are no different; complexity leads to increased support costs, bugs and systems that are hard to use. In Aristocracy, Democracy and System Design, Brooks addresses three questions (he posits four, but postpones the discussion of the fourth until the next essay):

1. How is conceptual integrity achieved?
   Brooks addresses this question from the point of view of a programing system. The goal of a programing system is to making using a computer easy. You could easily substitute any other type of system or application for the term “programing system”. Brooks argues that the goal of ease of use dictates unity of design, therefore conceptual integrity.  Ease of use can’t exist if the design consists of disparate ideas that make the system both less simple and less straightforward.
2. Doesn’t unity of design imply an aristocracy of architects?
   Brooks suggests that conceptual integrity in design comes from one or the collaboration of a small set of coordinated minds. Unfortunately constraints like schedule pressure require organizations to use many hands to develop, design and build a system. There are two fixes to the schedule constraint. The first is the separation of design and build resources. Architects develop the design in advance of the developers. The architect acts as the user’s agent, developing a description of what is to happen. The developers in an implementation mode define how the whats will be constructed. The second solution is to use a surgical team. The surgeon defines how the operation will be done and then directs the operation.  The surgeon is the single person that controls the flow work and is responsible for the outcome. In any project the number of architects will always be less than the developers or testers therefore they will be viewed as aristocrats.
3. What do the implementers do while waiting for the design?
   When organizations separate design and implementation, there are complaints because of the perception that the process yields:
   1. A scenario in which an overly rich architecture that can’t be implemented in the constraints most projects operate within. Brooks admits that this complaint is true and indicates that he will address it in the essay: The Second System Effect. In my experience, many organizations use standards, peer reviews and time boxing to combat this problem.
   2. A scenario in which developers have no outlet for their creativity.
      Brooks says this is a false argument. In my experience, even in organizations with the strictest architectural and design constrains, developers have wide latitude to be creative. That creativity is applied in determining how to implement the design within the boundaries and constraints they are given.
   3. A scenario in which developers will have to sit around waiting for the design to be developed.
      While Brooks says that this is a false argument, in my experience it is partially true. In scenarios where designers and developers are separated there will be some timing considerations. The Scaled Agile Framework Enterprise (SAFe) addresses this issue by developing an architectural runway for the developers. Just enough design and architecture is developed ahead of the development teams to provide the guidance they need just before they need it.

Conceptual integrity is an important concept that affects how any system will be developed. Brooks links higher levels of conceptual integrity to improved ease of use, productivity and maintainability. However the old bugaboo of schedule pressure, along with the perception that scenarios in which developers are sitting on their hands or are stripped of their creativitiy makes the pursuit of conceptual integrity difficult, but not impossible.

Previous installments of Re-Read Saturday for the The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

 

Re-read Saturday: The Mythical Man-Month, Part 3 The Surgical Team

The Mythical Man-Month

When we began the re-read of The Mythical Man-Month my plan was to go through two essays every week. To date the planned cadence of two essays has been out of reach. Each of the essays are full of incredibly rich ideas that need to be shared therefore I am amending the plan to one essay per week. Today we re-read The Surgical Team. In this essay, Brooks addresses the impact of team size and team composition on the ability to deliver large projects.

The concept of a small team did not jump into popular discussion with the Agile Manifesto of 2001. Even before The Mythical Man-Month was published in 1975 the software development industry was beginning to coalesce around the idea that smaller teams were more efficient. Smaller teams are easier to coordinate and information sharing is easier because there are fewer communication paths with less people. The problem that Brooks postulated was that big systems, which are needed. can’t be built by a single, small team in any reasonable period of time. Efficiency does not always translate to effectiveness. Paraphrasing Brooks, the question we have to ask is if having an efficient single small team of first class people focused on a problem is great, how do you build the large systems that are needed? If small teams can’t build big systems efficiently the solution is either to not build large solutions all at once, find a mechanism to scale smaller teams or revert back to using brute force (which is often the method of choice).

The brute force method of developing large systems has been the most often leveraged answer to build large systems before (and after) the publication of The Mythical Man-Month. Brute force is a large project team typically coordinated by a team of program and project managers. Earlier in my career I was involved in systems side of bank mergers. During one of the larger mergers I worked on over 300 coders, business analysts, testers, project managers and others worked together to meet a merger date. Unquestionably the approach taken was brute force and as the date got closer the amount of force being brought to bear became more obvious. Brute force is problematic based on lack of efficiency and predictability. Brute force methods are affected by the variability of an individual’s capability and productivity. The goals of the systems portion of the bank mergers were not the efficiency of the process, but rather making the date given to the regulators for cut over to a single system without messing people’s money up and ending up on the front page of The Plain Dealer.

If brute force is an anathema (and it should be), a second solution is to only use small single purpose teams. Products would evolve as small pieces of functionality are conceived, built and then integrated into a larger whole. Scrum, at the team level, uses small teams to achieve a goal. Team level Agile embraces the effectiveness of small teams discussed in The Surgical Team, however does not address bringing large quantities of tightly integrated functionality of market quickly.

Agile has recognized the need to get large pieces of functionality to market faster than incremental evolution without abandoning the use of small teams by adding scaling techniques. A Scrum of Scrums is a technique to scale Scrum and other team-level Agile frameworks. Other scaling frameworks include DSDM, SAFe and Scaled Scrum. All of these frameworks leverage semi-autonomous small teams with some form coordination to keep teams moving in the same direction. Scaling adds some overhead to the process which reduces the efficiency gains from small teams, but allows larger pieces of work to be addressed.

In The Mythical Man-Month, Brooks leverages the metaphor of the surgical team to describe a highly effective AND highly efficient model of a team. In a surgical team the surgeon (responsible party) delivers value with a team that supports him or her. Transferring the surgical team metaphor to a development team, the surgeon writes the code and is responsible for the code and the backup surgeon (Brooks uses the term co-pilot) is the surgeons helper and is typically is less experienced. The backup is not responsible for the code. The rest of the team supports the surgeon and backup. The goal of the support team is to administer, test, remove road blocks and document the operation or project. While we might dicker about the definition of specific roles and what they are called, the concept of the small, goal-oriented team is not out of line with many Scrum teams in today’s Agile environment.

Scrum and most other Agile techniques move past the concept of teams of individuals with specific solo roles towards a teams with more cross functional individuals. Cross-functional teams tend to yield more of a peer relationship than the hierarchy seen in the surgical team. The flatter team will require more complex communication patterns which can be overcome in Scrum with techniques like the stand-up meeting to address communication. The concept of the Scrum team is a natural evolution of the concepts in The Surgical Team.  Scrum tunes the small team concept to software development where a number of coordinated hands can be in the patient simultaneously, if coordinated through the techniques of common goal, stand-up meeting, reviews and continuous integration.

 

Previous installments of Re-read Saturday for the The Mythical Man-Month

Intro and Tar Pit

The Mythical Man-month Part 2

 

Daily Process Thoughts, Don’t Ask? April 17,2013

Denial of a problem doesn’t resolve the problem. How many project status reports report a green status, right up until the day they suddenly are red? Recently I heard of project that was 80% complete for five years before having to be rescued. Both examples might be extreme, however they are not unique. Every time we close our eyes and deny some even minor fact in hope that it will go away, rather than tackling it head on, we postpone our day of reckoning.

In this lecture, Slaying The Dragon Within Us, Jordan Peterson says “if we don’t deal with our dragons they will continue to grow.” Covey says “act or be acted upon.” Projects usually don’t get in behind over night. It is easy to ignore the day here and the day there because it is easier than telling your sponsor or boss that you will be late however by delaying we may foreclose many of the good options for getting back on track. IN THE END both Peterson and Covey exhort us to tackle our problems before they get larger and easier to solve.

Note this may have posted earlier this week.