Rachel Lovinger of Razorfish began a conversation recently about adopting Agile/Scrum software development practices for content strategy, and as synchronicity would have it, I am, e’en now, working on helping my team explore Scrum for our migration to Drupal. Rachel’s articulation—and subsequent posters’ agreements—of the challenges of working with CMS builders in Agile mode to craft content and system simultaneously has raised for me a few thoughts.

Waterfall Content: Must we adopt the monolith?

Traditionally, software systems have been built by the “waterfall” method. An elder child of “scientific management,” the waterfall sees development as a long, linear sequence of requirements gathering, followed by design, followed by building and coding, and wrapped up with testing before release. Without going into a complete review of the difficulties with this method, its underlying principles suppose that it is possible to foresee all the features and details of a system in advance, so that no change to the design should be required (or indeed allowed) after development has begun.

All the laborers in the process, from the “overseer” project manager to the lowliest coder, must build their part of the plan, and then hand it off to the next phase of the waterfall. Any change to the “master plan” is considered a threat to the success of the project, so a serious set of blinders must be kept in place to preserve the sense of moving forward. By the time the system is delivered, even if it conforms perfectly to the original specifications, it may not turn out to serve its users very well. The waterfall makes no allowance for learning, for creative discovery, or for unforeseen circumstances.

Now, since I’ve become acquainted with content strategy (now about two years, and still a noob!), I have got the impression that large content projects are often done in the same way: Gather requirements, design content models, build templates, create content, and deliver.

And why not? When you’re embroiled in a monolithic CMS project, if any thought has been given to content at all, the system won’t be ready for it until late, late, late in the waterfall. Content types will have been fixed in data capture templates. Layouts will have fused with information architecture. Workflows will have been encoded deep in the CMS DNA. At best, content strategy becomes a parallel cascade that will merge (eventually) with the technical cascade, as the whole project rushes to the sea.

Challenges to Content Agility

Let me draw out some of the themes from the e-mail conversation to which I referred earlier.

I can’t design content in iterative cycles without the “big picture.”

It is a natural panic when abandoning the waterfall, that if we can’t know all the requirements, all the specifications up front, then ipso facto, someone must be expecting us to work without any of them. I’ve heard this cry regularly from waterfall practitioners in my own organization: “But you have to have some idea up front of what you’re trying build!” Yes, absolutely. You have to have “some idea.” You have to have a very clear idea, a rich idea, in fact, a vivid idea of the final product. We don’t, however, need a perfectly detailed idea of it.

The starting point for the Agile/Scrum is an enormous, rich “product backlog,” which includes all the features and attributes that could possibly be included in the final product, captured as “user stories.” These user stories come directly out of strategy: business strategy, market strategy, content strategy, and technology strategy. The whole Team has access to that prioritized backlog from the beginning. Even if the Team is working on specific stories during a sprint, they can infer a general design from any important perspective from the entire backlog in order to guide their work during the sprint.

We use “agile” within the context of a larger “waterfall.”

It is tempting in a disciplined field like technology development to think that methods, like software components, can be disassembled and reconfigured in other combinations, thereby achieving the best of all possible worlds. Of course, “Agile” was first a set of principles set out in a manifesto before it collected any specific techniques. (agilemanifesto.org)

The Agile Manifesto and its extensions stand in direct opposition to the waterfall:

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

These principles were articulated in this way because the latter of each pair was considered a principal impediment to the former: Process deadens individual interactions. Documentation delays completed work. Contract negotiation undermines collaboration. Rigid plans shatter in the face of change.

So in my view, the fundamental philosophies of the two approaches are direct opponents. I don’t think you can do both to any real benefit.

Specialists sit idle if they aren’t involved in the stories in this sprint.

Agile refuses to believe that work must be sequential. Each sprint is planned according to the prioritized backlog, the needed skills to fulfill the user stories, and the available resources. A fully-utilized Team should be able from the full range of stories to select enough work to keep everyone fully engaged during the sprint, and it’s the responsibility of each member to estimate his or her capacity. But at the same time, the Team doesn’t have to limit itself to the current sprint backlog.

In addition, a sprint is simply a “timebox” into which the Team plans its work. Saying that you have “potentially shippable software” by sprint’s end doesn’t mean that you don’t work on anything but the stories to be completed during the sprint. In fact, the Team must always be looking ahead to future sprints, so that if there is “prework” to do before future stories are selected, the Team needs to plan when and how that work will be done within the sprints.

Because some work is dependent on other work, someone will get left at the end with too many tasks and not enough time.

Part of the Scrum philosophy is that the whole Team is responsible for the success of the whole. It’s a team issue, rather than a methodology issue if the work is not evenly distributed, or if the Team is in danger of missing its commitments. It is also important to expect that over time the Team get a better sense of its own process and relationships. One sprint or two of someone getting left holding the bag should be enough to raise a flag to the Scrummaster: Something’s not working, and that’s not the fault of Agile/Scrum, but of the Team’s ability to select and complete an appropriate number of user stories for a given sprint.

Agile is too loose to “manage” large content projects successfully.

I have also heard it said that Agile development would work if human beings and their projects were more predictable. Also, I have heard that the larger the project the more important hierarchy and sequence becomes. These are exactly the myths of “scientific management” and its child, the “waterfall.”

• A large system can be controlled with the proper structure and oversight.
• If they work hard enough, people can be predictable.
• Error and inefficiency are the results of carelessness and poor planning.

It is counterintuitive, but true nonetheless, that waterfall approach cannot work as long as any human beings are involved. We are inherently unpredictable. We learn. We communicate ineffectively and incompletely. We make mistakes. We fail. We discover. We invent. We play.

Agile and Scrum accept the ultimate truth that no system can be controlled or predicted as long as human beings are involved. Instead, by breaking the work into short bursts, emphasizing conversation and regular inspection over the course of iterative and incremental work, and accepting change as inevitable, it becomes possible to:

• Minimize the long-term impact of inevitable mistakes.
• Take full advantage of humans’ creativity and inventiveness.
• Move forward without being paralyzed by uncertainty.

Your Platform Needs to be Agile, Too

As a final comment, as I posted to the e-mail conversation, whether you have the freedom to be Agile depends a lot on the CMS you’re using. A monolithic platform that was designed to be implemented according to the waterfall—get all the requirements together, build it, launch it, and don’t make any changes—can absolutely kill any flexibility that an Agile approach might have afforded. It’s well and good to say that your CMS developers are using Agile techniques, but if it requires tons of rework to change the content templates because you learned something later on that shifted your modeling, then you’re really still working under the waterfall.

On the other hand, a system like Drupal, can handle lots of global change as the sprints progress, so it actually supports the Agile process better. For more idea of how that works, look for presentations by Rob Purdie at The Economist.