photoAn Agile team is comprised of a product owner, team members (from all the disciplines needed to deliver the project) and the scrum master. Delivering on the team’s commitment is the ultimate measure of value. The scrum master helps to create an environment for the team to work together. Over the life of a project, everyone on the team has to lead and facilitate for the team to effectively deliver value. (more…)

The front cover of The Mythical Man-Month

The Mythical Man-Month

Airports and driving have conspired to keep me from finishing this week’s essay from The Mythical Man-Month, therefore I thought I would seek your advice.  The Re-Read of The Mythical Man-Month by Fred P. Brooks is nearly complete.  We have one very long retrospective essay and an epilogue left in the book and a summary essay from me. I anticipate one or two more weeks of posts. The burning question is: what is next?  We have re-read Stephen Covey’s Seven Habits, The Goal by Goldratt and now The Mythical Man-Month.  I have two questions for you. 1) Do you want me to continue with this feature?  And, 2) which book should be next?   It is time for a poll.  If you would like to suggest a book that is not on the list, feel free to “write-in” a candidate in the comments for this essay.  I have included the top entries from the poll I did a few years ago and a few suggestions readers and listeners have made along the way (or books I have rediscovered as I poked around my library).  (more…)

No Silver Bullet is the 16th installment of the Re-Read Saturday of the The Mythical Man-Month by Fred P. Brooks. No Silver Bullet is the longest of the essays, and even includes an abstract and introduction. In this essay, Brooks discusses hard parts of software development and how most of the productivity gains of the previous decades were focused around improving the processes around software development rather than addressing the really hard parts. The hard parts are capturing and translating requirements into a design. (more…)

The front cover of The Mythical Man-Month

The Mythical Man-Month

Hatching a Catastrophe is one of most quoted of essays from The Mythical Man-Month by Fred P. Brooks. In this essay, Brooks makes the point that projects get behind one day at a time. Rarely, if ever, does a project go from being on time to substantially behind schedule overnight; even though we have all seen projects go from green to red at the blink of an eye. The slow erosion of schedule performance is often very hard to recognize even when a catastrophe is in progress. We often use the example of the metaphor of the frog and the pot of water to illustrate this problem. If a frog is placed in a pot of water that is slowly brought to boil, they will not try to hop out because they don’t recognize the problem until it too late. Whether true or not, the metaphor illustrates one of the main points that Brooks makes in this essay. However, as with most catastrophes, there are mechanisms to avoid disaster with a bit of structure and discipline.

Milestones or Millstones: Brooks proposes that big project on a tight schedule require control. I suggest that any project that is important enough to fund (even in blood, sweat or tears) requires control. Milestones are an action or event marking a significant stage or event in development. For example, the completion of testing or acceptance in a demo is a milestone. Milestones, when defined correctly, provide a tool to help evaluate whether a project is drifting off course. The fifty dollar phrase in the last statement is “when defined correctly.” In order to be useful, milestones must be concrete, specific and measurable in order to be crisply defined.

Milestones that are unambiguously defined with a crisp definition of done provide a mechanism for measuring whether a task or a set of tasks are complete. The ability to know that a piece of work is done is empowering. Who does not like to check something off as done? Failing to define a crisp definition of done does a disservice to everyone involved in a project. The most precise definition of done for a milestone is that the tasks that are linked to the milestone are ALL 100% complete (not just really close). The crisper the definition of done is for the milestone, the less chance that we can deceive ourselves or rationalize that not being exactly done is close enough. The process of deception or rationalization is demotivating to the team and the organization.

Milestones are a tool to establish discipline within the team. Without the discipline provided by milestones, it is easy to rationalize slipping the schedule just a little bit (slipping the schedule is the same as not meeting commitments in Agile). One classic rationalization that Brooks points out is using the excuse that another piece of work is late, so we do not need to be on time. While one would not expect this type of behavior amongst professionals, it occurs. I once sat in a meeting where a hardware manufacturer announced they were over a year late, but since another firm was even later, it did not matter. Interestingly the other firm turned out not to be as late as it was perceived, and significant problems ensued.

As noted earlier, projects become late a day at a time. No one gets overly excited about getting behind a few hours or a day. We can easily rationalize that we can make up the time. Software engineers are trained problem solvers who can easily rationalize the slip until it is too late. How often have you seen a task or story that is nearly completed day after day? It is easy to sweep being late under the rug until it can’t be hidden.

Brooks describes several solutions. One example is generating PERT diagrams that identify critical paths and the slack any task might possess. More interestingly, Brooks describes how the team member/manager tension can be defused. For example, not having the leader act on issues that the person having the problem can solve, thus empowering the team member. Concepts in use today that help prevent hatching a catastrophe include servant leadership, self-managing teams, short time box and daily stand-ups. Including techniques that generate early warning signals that illuminate the true status of work and prevent problems from getting larger.

Previous installments of the Re-read of The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

Aristocracy, Democracy and System Design

The Second-System Effect

Passing the Word

Why did the Tower of Babel fall?

Calling the Shot

Ten Pounds in a Five–Pound Package

The Documentary Hypothesis

Plan to Throw One Away

Sharp Tools

The Whole and the Parts

The Mythical Man-Month

The Mythical Man-Month

In the seventh essay of The Mythical Man-Month, Fred P. Brooks begins to tackle the concept of estimating. While there are many estimating techniques, Brooks’ approach is a history/data-based approach, which we would understand as today as parametric estimation. Parametric estimation is generally a technique that generates a prediction of the effort needed to deliver a project based on historical data of productivity, staffing and quality. Estimating is not a straightforward extrapolation of has happened in the past to what will happen in the future, and mistaking it as such is fraught with potential issues. Brooks identified two potentially significant estimating errors that can occur when you use the past to predict the future without interpretation.

Often the only data available is the information about one part of the project’s life cycle. The first issue Brooks identified was that you cannot estimate the entire job or project by just estimating the coding and inferring the rest. There are many variables that might affect the relationship between development and testing. For example, some changes can impact more of the code than others, requiring more or less regression testing. The link between the effort required to deliver different types of work is not linear. The ability to estimate based on history requires a knowledge of project specific practices and attributes including competency, complexity and technical constraints.

Not all projects are the same. The second issue Brooks identified was that one type of project is not applicable for predicting another. Brooks used the differences between small projects and programming systems products to illustrate his point. Each type of work requires different activities, not just scaled up versions of the same tasks. Similarly, consider the differences in the tasks and activities required for building a smart phone app compared to building a large data warehouse application. Simply put, they are radically different. Brooks drove the point home using the analogy of extrapolating the record time for 100-yard dash (9.07 seconds according to Wikipedia) to the time to run a mile. The linear extrapolation would mean that a mile could be run in 2.40 (ish) minutes (a mile is 1760 yards) the current record is 3.43.13.

A significant portion of this essay is a review of a number of studies that illustrated the relationship between work done and the estimate. Brooks used these studies to highlight different factors that could impact the ability to extrapolate what has happened in the past to an estimate of the future (note: I infer from the descriptions that these studies dealt with the issue of completeness and relevance. The surveys, listed by  the person that generated the data, and the conclusions we can draw from an understanding of the data included:

  1. Charles Portman’s Data – Slippages occurred primarily because only half the time available was productive. Unrelated jobs meetings, paperwork, downtime, vacations and other non-productive tasks used the remainder.
  2. Joel Aron’s Data – Productivity was negatively related to the number of interactions among programmers. As the number of interactions goes up, productivity goes down.
  3. John Harr’s Data- The variation between estimates and actuals tend to be affected by the size of workgroups, length of time and number of modules. Complexity of the program being worked on could also be a contributor.
  4. OS/360 Data- Confirmed the striking differences in productivity driven by the complexity and difficulty of the task.
  5. Corbatoó’s Data – Programming languages affect productivity. Higher-level languages are more productive. Said a little differently, writing a function in Ruby on Rails requires less time than writing the same function in macro assembler language.

I believe that the surveys and data discussed are less important that the statistical recognition that there are many factors that must be addressed when trying to predict the future. In the end, estimation requires relevant historical data regardless of method, but the data must be relevant. Relevance is short hand for accounting for the factors that affect the type work you are doing. In homogeneous environments, complexity and language may not be as big a determinant of productivity as the number of interactions driven by team size or the amount of non-productive time teams have to absorb. The problem with historical data is that gathering the data requires effort, time and/or money.  The need to expend resources to generate, collect or purchase historical data is often used as a bugaboo to resist collecting the data and as a tool to avoid using parametric or historical estimating techniques.

Recognize that the the term historical data should not scare you away.  Historical data can be as simple as a Scrum team collecting their velocity or productivity every sprint and using it to calculate an average for planning and estimating. Historical data can be as complex as a pallet of information including project effort, size, duration, team capabilities and project context.

Previous installments of the Re-read of The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

Aristocracy, Democracy and System Design

The Second-System Effect

Passing the Word

Why did the Tower of Babel fall?

The Mythical Man-Month

The Mythical Man-Month

In the seventh essay of The Mythical Man-Month, Brooks returns to the topics of organization and communication using story of the Tower of Babel as a metaphor for many of the engineering fiascoes seen in software development. In the story of the Tower of Babel, when communication broke down, coordination failed and the project ground to a halt. As projects become larger and more complex, communication and organization always becomes more difficult and more critical.

The essay Why Did the Tower of Babel Fall? presents three mechanisms to facilitate communication.

  1. Informal Communication, which includes all of the telephone calls and conversations that occur in and among teams. Healthy teams and teams of teams talk, interact and share information and ideas.
  2. Meetings are more formal and include reviews, technical briefings, demonstrations, sprint planning and release planning events. Done well formal meetings, whether part of an Agile framework or not, can find problems and share information and knowledge.
  3. Project Workbooks are the repository of the evolving intellectual property that defines any significant endeavor.

Brooks goes into depth on the definition and structure of the project workbook. While the workbook that Brooks describes is a physical construct we should consider that the idea of workbook is less of a document then a structure for grouping information. All the project documents need to fit into the workbook and be part of the overall structure of project information so that it can be reused and inherited from. The ultimate goal of the workbook is ensure that everyone who needs information has access to the information and gets the information.

Workbook Mechanics: In order to meet the goal of ensuring of everyone access to the information they need, the workbook needs to be maintained and organized. For example, Brooks suggests ordering and numbering all project memorandums (we would call these emails now) so that people can look through the list and actually see if they need information. Brooks points out that as size of the effort, project or program goes up, the complexity of maintaining and indexing the project communications increases in a non-linear fashion. Today’s technical tools like wikis, SharePoint and hypertext markup make curating and sharing information easier than paper and microfiche (if microfiche is before your time, Google it). As a reminder, capturing and reusing IP is required in all types of work. As a coder, it is easy to fall prey to the idea that the code and a functional system is the only documentation required; however, this is false. For most large efforts, capturing decisions, designs, architectures and even the occasional user manual is an option that can be ignored only if you want to court a communication failure. Remember however that over-reliance on documentation is not the goal.

Brooks uses the formula (n2-n)/2, where n is the number of people working on an effort, to define the possible communication interfaces. As the number of people increase the number of interfaces increases very quickly. Organization is a tool to reduce the amount of interfaces. This is one of the reasons why small teams are more effective than large teams. Scaling Agile is often accomplished by assembling teams of small teams. We use the metaphor of trees or flowers are often to describe the visualizations of team of teams. Communications spreading from a central point can look like a tree or flower. Organizations leverage techniques like the specialization and division of labor to reduce the number of communication interfaces.

There is often push back on division of labor and specialization in Agile. This push back is often misplaced. Within a team the concept of T Shaped people, which eschews specialization, increases capabilities; however, teams often specialize thus becoming capability teams. Capability teams often tackle specific types of work more effectively and efficiently by reducing learning curves. Capability teams fit into Brooks’ ideas on organizing to reduce the communication overhead.

Brooks proposes that each grouping of work have six essential needs.

  1. A Mission,
  2. A Producer,
  3. A Technical Director or An Architect,
  4. A Schedule,
  5. A Division of Labor, and
  6. Interface Definitions Among the Parts.

Four of these essentials are easily understood; however, the roles of the producer and technical broke new ground and extended the ideas Brooks began in The Surgical Team. The role of the producer is to establish the team, establish the schedule and continually acquire the resources needed to deliver the project. The role of the producer includes many of the classic project and program management activities, but also much of the outside communication and coordination activities. When small effort with a single team the roles of the producer and technical director can be held by the same person. As efforts become larger and more complex the roles are generally held by separate people. The role of the producer in the Scaled Agile Framework (SAFe) is handled by the Release Train Engineer. The technical director focuses on the design and the technical components (inside focus). In SAFe this role is held by the System Architect.

Brooks leverages the metaphor of the Tower of Babel to focus on the need for communication and coordination. Projects, team and organizations need to take steps to ensure communication happens effectively. Techniques like meetings and project workbooks are merely tools to make sure information is available and organized. Once we have information organization provides a framework to ensure information is communicated effectively. Without communication any significant engineering effort will end up just like the Tower of Babel.

Are there other mechanisms for organization, communication and coordination that have worked for you? Please share and lets discuss!

Previous installments of the Re-read of The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

Aristocracy, Democracy and System Design

The Second-System Effect

Passing the Word

The Mythical Man-Month

The Mythical Man-Month

Today we re-read the fourth essay in The Mythical Man-Month titled: Aristocracy, Democracy and System Design. In this essay, Brooks deals with the role of conceptual integrity in building systems and the impact organizationally of getting to an appropriate level of conceptual integrity. According UC San Diego, conceptual integrity is the principle that anywhere you look in your system, you can tell that the design is part of the same overall design. Brooks suggests that conceptual integrity is most important consideration in system design. He begins the essay by building the case that conceptual integrity leads to high ease of use, and therefore higher value. Systems with conceptual integrity have one set of design ideas and that ANY idea or concept that violates conceptual integrity must be excluded. Conceptual integrity is important because systems that are based on any different architectural concepts are both hard to work with and maintain.  For example, many of the houses in my neighborhood began life as lake cottages. Many of these cottages have been added to over the years in a fairly haphazard manner. Over the last few years many have been knocked down due to high cost of upkeep that their complexity generates. Software systems are no different; complexity leads to increased support costs, bugs and systems that are hard to use. In Aristocracy, Democracy and System Design, Brooks addresses three questions (he posits four, but postpones the discussion of the fourth until the next essay):

  1. How is conceptual integrity achieved?
    Brooks addresses this question from the point of view of a programing system. The goal of a programing system is to making using a computer easy. You could easily substitute any other type of system or application for the term “programing system”. Brooks argues that the goal of ease of use dictates unity of design, therefore conceptual integrity.  Ease of use can’t exist if the design consists of disparate ideas that make the system both less simple and less straightforward.
  2. Doesn’t unity of design imply an aristocracy of architects?
    Brooks suggests that conceptual integrity in design comes from one or the collaboration of a small set of coordinated minds. Unfortunately constraints like schedule pressure require organizations to use many hands to develop, design and build a system. There are two fixes to the schedule constraint. The first is the separation of design and build resources. Architects develop the design in advance of the developers. The architect acts as the user’s agent, developing a description of what is to happen. The developers in an implementation mode define how the whats will be constructed. The second solution is to use a surgical team. The surgeon defines how the operation will be done and then directs the operation.  The surgeon is the single person that controls the flow work and is responsible for the outcome. In any project the number of architects will always be less than the developers or testers therefore they will be viewed as aristocrats.
  3. What do the implementers do while waiting for the design?
    When organizations separate design and implementation, there are complaints because of the perception that the process yields:

    1. A scenario in which an overly rich architecture that can’t be implemented in the constraints most projects operate within. Brooks admits that this complaint is true and indicates that he will address it in the essay: The Second System Effect. In my experience, many organizations use standards, peer reviews and time boxing to combat this problem.
    2. A scenario in which developers have no outlet for their creativity.
      Brooks says this is a false argument. In my experience, even in organizations with the strictest architectural and design constrains, developers have wide latitude to be creative. That creativity is applied in determining how to implement the design within the boundaries and constraints they are given.
    3. A scenario in which developers will have to sit around waiting for the design to be developed.
      While Brooks says that this is a false argument, in my experience it is partially true. In scenarios where designers and developers are separated there will be some timing considerations. The Scaled Agile Framework Enterprise (SAFe) addresses this issue by developing an architectural runway for the developers. Just enough design and architecture is developed ahead of the development teams to provide the guidance they need just before they need it.

Conceptual integrity is an important concept that affects how any system will be developed. Brooks links higher levels of conceptual integrity to improved ease of use, productivity and maintainability. However the old bugaboo of schedule pressure, along with the perception that scenarios in which developers are sitting on their hands or are stripped of their creativitiy makes the pursuit of conceptual integrity difficult, but not impossible.

Previous installments of Re-Read Saturday for the The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team