The Documentary Hypothesis: Re-Read Saturday: The Mythical Man-Month, Part 10

The Mythical Man-Month

The Documentary Hypothesis is the tenth essay of The Mythical Man-Month by Fred P. Brooks. This essay tackles the documents that a manager needs to focus and run his or her project. The topic of documentation in today’s project management environment often generates a very animated discussion, but as with many of the earlier essays, Brooks’ approach foreshadows lean or Agile approaches. The approach Brooks uses to identify the core documents is to identify the documents/data used to manage work in two non-software environments, perform a comparison and then to determine if there is a unifying thread. The final step in Brooks’ evaluated whether a common thread could be found and determined whether the information was required or useful in software projects.

The first type of work that Brooks used to identify the critical project management documents is the building of a piece commercial computer hardware.  The critical items identified were:

1. Objectives
2. Specifications
3. Schedule
4. Budget
5. Organization chart
6. Space allocations
7. Market estimates, forecasts and prices

The relationships between forecasts, estimates and prices are used to show how changes can ripple through a project. For example, a change in the planned product price can affect the sales forecast; which then will impact the estimates of required materials and labor, the budget and might trigger a rethink of the specifications. Quoting Brooks:

“To generate a market forecast, one needs performance specifications and postulated prices the quantities from the forecast combine with the component counts from the design to determine the manufacturing cost estimate and they determine the per-unit share of development and fix cause these costs then determine price.”

These critical project management documents let a manager monitor changes in the project so the don’t become a vicious cycle.

The second type of work Brooks uses to illustrate his point is a university department. Brooks points out that the critical list of documents a manager uses to manage this type of work is very similar.

1. Course descriptions
2. Requirements for research proposals
3. Plans for research when funded
4. Class schedules and teaching assignments
5. Budgets
6. Space allocation
7. Assignment of staff and graduate students

From these two lists, Brooks identified the common threads that a software project needs to start developing. They include an objective, product specifications, a schedule, a budget, space and an organization chart. These items define the who, what, when, why and with what for any effort. Regardless of methodology or framework, these “documents” are needed by every team, leader or project manager. The quotes around the word document reflect that the team doesn’t actually need to generate paper or a .docx file. For example, a fixed team and fixed duration is synonymous with a budget for effort.

The inclusion of an organization chart is an interesting twist. Brooks used the organization chart to introduce the idea of Conway’s Law into the Mythical Man-Month. Conway’s Law states law states that how an organization is structured influences the systems they develop. The organization chart reflects how an organization formally communicates which directly influences the design and development of functionality. This is due to the need to communicate to define, develop and maintain interfaces. Interestingly, in most IT organizations, you can use the existing interfaces between applications to identify boundaries between teams and physical departments. Conway’s Law is alive and well in the corporate world.

Brooks uses a computer product and university department to abstract what he believes is the informational core needed by a manager of a software project. But, why make these documents formal, when formal is shorthand for written down and sharable? Brooks provides three reasons.

1. Writing decisions down exposes gaps and inconsistencies. The process of writing helps the writer to generate definitions that are clearer and more exact. As a writer, I concur (you never see my first drafts . . . and you don’t want to!).
2. Documents (define this in any way you like, from wiki pages to an audio file) are an effective and efficient means of communicating decision. Documents create common knowledge quicker than relying on verbal communication paths alone. Documentation also reduces rehashing by providing a basis for common historical knowledge base.
3. Documents provide a baseline that, when periodically reviewed, provides a manager with a method of identifying changes. All changes provide managers and teams with information that can be used to tailor behavior.

Many might bridle at the idea of documentation on general principle; however, all teams need to capture some core truths. What matters is less tools we use to capture the core information about any effort. Regardless of whether we use paper or something leaner or more agile, we need to document our core information. such as goals, objectives, backlog items, team norms or even a calendar of sprint activities.  The act of capturing information is an admission that we need structure to remember and refine a core set of information! The goal of documentation in any form is to guide behavior through communication.

Previous installments of the Re-read of The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

Aristocracy, Democracy and System Design

The Second-System Effect

Passing the Word

Why did the Tower of Babel fall?

Calling the Shot

Ten Pounds in a Five–Pound Package

 

Re-Read Saturday: The Mythical Man-Month, Part 5 – The Second-System Effect

The Mythical Man-Month

In the fifth essay of The Mythical Man-Month, titled The Second-System Effect, Brooks circles back to question he left unanswered in the essay Aristocracy, Democracy and System Design. The question was: If functional specialties are split, what bounds are left to constrain the possibility of a runaway architecture and design? The thought is that is that without the pressure of implementation an architect does not have to consider constraints.

Brooks begins the essay by establishing a context to consider the second-system effect with a section titled, “Interactive discipline for the architect”. All architects work within a set of constraints typically established by project stakeholders. Operating within these constraints requires self-discipline. In order to drive home the point, Brooks uses the analogy of a building architect. When designing a building an architect works against a budget and other constraints, such as a location and local ordinances. Implementation falls to the general contractor and subcontractors. In order to test the design, the architect will ask for estimates from the contractors (analogous to the teams in a software environment). Estimates provide the architect with the feedback needed to test ideas and assumptions and to assure the project’s stakeholders that their build can be completed within the constraints. When estimates come in too high, the architect will need to either alter the design or challenge the estimates.

When an architect challenges an estimate her or she could be seen as leveraging the power hierarchy established by separating functions (see last week’s essay). However the architect needs to recognize that in order to successfully challenge the estimate they need to remember four points.

1. The contractors (development personnel in software) are responsible for implementation. The architect can only suggest, not dictate, changes in implementation. Force will generate a power imbalance that will generate poor behaviors.
2. When challenging an estimate be prepared to suggest a means of implementation, but be willing to accept other ways to implement to achieve the same goal. Recognize that if you make a suggestion before being asked that you will establish an anchor bias and may not end up with an optimal solution.
3. When making suggestions, make them discretely. Quiet leadership is often most effective.
4. The architect should be prepared to forego credit for the changes generated as estimates and constraints are negotiated. Brooks pointed out that in the end it is not about architect, but rather about the solution.

The first part of the essay established both the context and framework for developing the self-discipline needed to control runaway design. Brooks concludes the essay by exposing the exception he observed. Brooks called this exception the second-system effect. In a nutshell, the second-system effect reflects the observation that a first work is apt to be spare, where as second attempts tend to be over designed as frills and signature embellishments start to creep in. Brooks points out this behavior can often be seen in scenarios in which a favorite function or widget is continually refined even as it becomes obsolete. For example, why are designers spending precious design time on the steering wheel for the Google self-driving car? (It is should be noted that recently the steering wheel was removed from the self-driving car then put back in  . . . with a brake).

How can you avoid the second-system effect? The simplest would be to never hire an architect with only one design job under their belt therefore avoiding the second-system effect. Unfortunately that solution over time is a non-starter. Who would replace the system architects that retire or move to other careers. Other techniques, like ensuring everyone is aware of the effect or stealing an idea from Extreme Programming and paring architects with other more seasoned architects or business analysts, are far more effective and scalable.

Brooks provides the punch line for the essay in the first two paragraphs. A project or organization must establish an environment in which self-discipline and communication exist to reduce the potential for runaway design.

Previous installments of Re-Read Saturday for the The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

Aristocracy, Democracy and System Design

 

Re-Read Saturday: The Mythical Man-Month, Part 4 – Aristocracy, Democracy and System Design

The Mythical Man-Month

Today we re-read the fourth essay in The Mythical Man-Month titled: Aristocracy, Democracy and System Design. In this essay, Brooks deals with the role of conceptual integrity in building systems and the impact organizationally of getting to an appropriate level of conceptual integrity. According UC San Diego, conceptual integrity is the principle that anywhere you look in your system, you can tell that the design is part of the same overall design. Brooks suggests that conceptual integrity is most important consideration in system design. He begins the essay by building the case that conceptual integrity leads to high ease of use, and therefore higher value. Systems with conceptual integrity have one set of design ideas and that ANY idea or concept that violates conceptual integrity must be excluded. Conceptual integrity is important because systems that are based on any different architectural concepts are both hard to work with and maintain.  For example, many of the houses in my neighborhood began life as lake cottages. Many of these cottages have been added to over the years in a fairly haphazard manner. Over the last few years many have been knocked down due to high cost of upkeep that their complexity generates. Software systems are no different; complexity leads to increased support costs, bugs and systems that are hard to use. In Aristocracy, Democracy and System Design, Brooks addresses three questions (he posits four, but postpones the discussion of the fourth until the next essay):

1. How is conceptual integrity achieved?
   Brooks addresses this question from the point of view of a programing system. The goal of a programing system is to making using a computer easy. You could easily substitute any other type of system or application for the term “programing system”. Brooks argues that the goal of ease of use dictates unity of design, therefore conceptual integrity.  Ease of use can’t exist if the design consists of disparate ideas that make the system both less simple and less straightforward.
2. Doesn’t unity of design imply an aristocracy of architects?
   Brooks suggests that conceptual integrity in design comes from one or the collaboration of a small set of coordinated minds. Unfortunately constraints like schedule pressure require organizations to use many hands to develop, design and build a system. There are two fixes to the schedule constraint. The first is the separation of design and build resources. Architects develop the design in advance of the developers. The architect acts as the user’s agent, developing a description of what is to happen. The developers in an implementation mode define how the whats will be constructed. The second solution is to use a surgical team. The surgeon defines how the operation will be done and then directs the operation.  The surgeon is the single person that controls the flow work and is responsible for the outcome. In any project the number of architects will always be less than the developers or testers therefore they will be viewed as aristocrats.
3. What do the implementers do while waiting for the design?
   When organizations separate design and implementation, there are complaints because of the perception that the process yields:
   1. A scenario in which an overly rich architecture that can’t be implemented in the constraints most projects operate within. Brooks admits that this complaint is true and indicates that he will address it in the essay: The Second System Effect. In my experience, many organizations use standards, peer reviews and time boxing to combat this problem.
   2. A scenario in which developers have no outlet for their creativity.
      Brooks says this is a false argument. In my experience, even in organizations with the strictest architectural and design constrains, developers have wide latitude to be creative. That creativity is applied in determining how to implement the design within the boundaries and constraints they are given.
   3. A scenario in which developers will have to sit around waiting for the design to be developed.
      While Brooks says that this is a false argument, in my experience it is partially true. In scenarios where designers and developers are separated there will be some timing considerations. The Scaled Agile Framework Enterprise (SAFe) addresses this issue by developing an architectural runway for the developers. Just enough design and architecture is developed ahead of the development teams to provide the guidance they need just before they need it.

Conceptual integrity is an important concept that affects how any system will be developed. Brooks links higher levels of conceptual integrity to improved ease of use, productivity and maintainability. However the old bugaboo of schedule pressure, along with the perception that scenarios in which developers are sitting on their hands or are stripped of their creativitiy makes the pursuit of conceptual integrity difficult, but not impossible.

Previous installments of Re-Read Saturday for the The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

 

Re-Read Saturday: Mythical Man-Month, Part 1

The Mythical Man-Month

The Mythical Man-Month: Essays on Software Engineering was originally published in 1975. That was the year I took my first course in programming – FORTRAN, at Memphis State University. Mythical Man-Month has been a must read for every professional involved in delivering value since it was published. The core themes in the book, which include Brook’s Law (adding people to a late project just makes it later – more in a future installment), are just as important today as they were when they were published because they still true for all types of software development and maintenance.   The Mythical Man-Month, I think on reflection you will agree, is still important because we are still trying to come to grips with most if not all of the concepts that Brooks exposed.

Here is the game plan for the Re-Read. We will be reading from Anniversary Edition of the Mythical Man-Month (Addison Wesley 1995). My copy is the 14^th printing from 2014. Part of this re-read will actually be a first read for me. I believe I originally read Man-Month in the early 1980’s, and this edition has a new preface and four new chapters. My intent during the re-read is to cover two essays/chapters per week so the re-read takes ten weeks “ish.”  Today will be the exception to the rule. We will tackle the first essay, The Tar Pit (I already used the ideas in the essay to make a point in the essay on Scrum of Scrum membership).

Preface(s):
The original preface provides background for Brooks’ perspective. Brooks came to his conclusions based on a history as a hardware architect then as the manager of the OS/360 project (IBM’s highly successful mainframe operating system). In the preface Brooks telegraphs one of his central theorems – large projects are different than small projects. It took a few years and the Agile Manifesto (2001) to recognize this fact and begin to act on it.

The anniversary edition includes a new preface in addition to the original. There were two notes in the new preface that caused me pause. The first was the observation by Brooks was that few of the central thoughts in the book had been critiqued, proven, and disproven by research and experience. I would like your feedback as we review the essays. I will explicitly point those points out during the re-read and if miss any skip ahead to Chapter 18 to keep me honest. The second point, based on my interest in the publishing industry,  from the new preface was that Brooks was able to publicly thank Addison Wesley staff that he found instrumental when preparing the first edition. The publishing world has changed. The question I pose to the Software Process and Measurement Cast blog readers is has our profession changed as much?

Chapter 1: The Tar Pit

In the Tar Pit Brooks sets the context for the system programming as a craft that is more than just the lone coder sitting in his or her office creating a stand alone app but rather a member of a team building large scale systems. The Tar Pit describes why the complexity and level of effort is different and then exposes the joy and woes of the profession.

The first takeaway from the Tar Pit is that as the product of a piece of work progresses from the delivery of a single program to programming product or system then to a programming systems product, the complexity of managing and understanding the work increases. As size and complexity increase so do tasks and activities to ensure the product works together. Brook suggests that the difference between level of effort between the creation of stand-alone program and a systems product is 9 times! The complexity of coordinating all of these additional tasks and activities while solving complex business problems impacts our ability to deliver on-time, on-budget and on-scope. At times, it also impact our ability to deliver WOW.

The second takeaway are the joys of programming.  Joys motivate programmers to put forth the effort needed to deliver value while trying to navigate the tar pit.  Brooks points out that the rewards of programming are at least four fold:

1. The joy of making things.
2. The joy of making things that are useful to others.
3. The joy of problem solving.
4. The joy of learning.

These joys are offset by several woes.  These woes are the part of the tar pit that make programming hard.

1. Expectation of perfection. Unless the code is correct and solves the requirements, it is wrong. There is little to no gray area.
2. Dependence on others. Others set your objectives, provide resources and often control the source of information. This often leads to the tension caused when individual or teams have the responsibility to deliver without commensurate authority. Agile principles, when applied to teams, are a start to addressing these woes.
3. Finding the bugs in the big picture.  Designing the big picture is significantly more fun than the tedious effort required to find bugs.
4. Dead on arrival projects.  Long running projects are occasionally dead on arrival or the market has changed and what has been delivered is not what is needed.

In 1975 when this essay was published Agile was not a word one would attach to building applications or systems. However I think we can see the seeds of the Agile revolution in this essay.