In 2001, seventeen American, British and Canadian software engineers and IT managers met at a ski resort in Snowbird, Utah, to start a movement to remake the way software is built. Over the previous decade, the attendees had independently created similar processes for organizing and managing software engineering projects that broke with tradition and promised to make software development better, cheaper, and more innovative. Their methods were diverse and went by many brand names: Extreme Programming, Crystal Clear, Scrum, Adaptive Software Development, Test Driven Development, and many others. But they also shared many goals and ideals in common.

The outcome of the Snowbird meeting was the coining of a new term that brought all these approaches under the same roof: Agile. They published a manifesto outlining twelve core principles, and founded the Agile Alliance, a non-profit advocacy organization to promote their new methods to the software engineering community.

Over the last decade, the Agile movement has been extremely successful. Its founders have become influential “thought leaders” who give lectures and keynotes at programming conferences around the world. A large corpus of books have been written about Agile methodologies—many are software engineering best-sellers. And the Agile Alliance is now a global organization, hosting conferences and training sessions and boasting affiliate user groups on six continents, with the support of large corporate sponsors like Lockheed Martin and IBM.

One way to understand what the Agile movement stands for is by understanding what it stands against: the traditional software development methodology known within Agile as Waterfall.

Although not formally defined, the Waterfall methodology is understood in the Agile community as a way of organizing software work into discrete stages that are oriented around producing documents that specify aspects of the system to be built. Derived from manufacturing and construction fields, a typical Waterfall process goes through approximately four stages: gathering requirements from customers; design (which in this case means specifying the general organization of the system’s components); writing code; and finally, verification that the system meets the requirements. Like a river with a tiered series of waterfalls, each stage is completed before the next one begins and it only moves forward, never backwards. After the process is complete, the system is deployed, ready to be used.

For the signatories of the Agile manifesto and a large community of software engineers who adhere to its principles with near-religious devotion, the Waterfall process simply doesn’t work. They reference a large body of academic literature and case studies by private analysts that reveals the astounding failure rates of IT projects initiated inside companies. The Standish Group’s appropriately named annual CHAOS Report is often cited. In its inaugural edition, the report found that 31% of IT projects are cancelled before they are completed, 53% exceed their initial cost estimates by more than 180%, and only 16% that succeed.

For Agilists, the blame for these failures lies squarely on the shoulders of the Waterfall model, and they offer their family of methodologies as the solution.

One major criticism leveled against the Waterfall model is that it assumes that the system’s requirements can be fully and exhaustively documented up front and that they will be correctly implemented. Although there’s a verification stage, if the new system does not live up to the client’s expectations, the process does not allow for significant changes. The Waterfall process delivers the new system at the end of a long, possibly multi-year process, which increases the risk of failure because the needs of the client sponsoring the work may have changed in the interval.

In contrast, Agile methodologies are incremental, delivering new features continuously throughout the project lifecycle. Work is organized into short cycles (sometimes called ‘sprints’) that may be up to 4 weeks long in which new features are specified, designed, implemented and deployed to users.

Agile stresses continuous iteration and feedback. Rather than documenting the complete system before work as an architect would produce detailed blueprints of every feature of a new house, Agile instead delivers new features to the client at the end of every sprint, allowing for evaluation of the system by the client, and continuous modification of the requirements in response to their ideas, feedback, newly discovered needs and changing priorities.

Extending the home construction analogy, one can think of Agile as rejecting the model of delivering a completed house to the client. Instead, a homebuilder working with an Agile process would deliver a small, working part of a whole house—maybe just the kitchen—and have the client to move in, begin using what’s there to the extent possible, and start requesting new rooms or modifications to existing rooms.

This style of construction is rooted in belief that it is essentially impossible to specify everything in advance because the needs of the customer are too complicated and change too quickly. Agile developers use continuous adaptation and modification as the alternative.

The primary difference between Waterfall and Agile that I’ve been describing are about the sequence of activities of software development, i.e., about time. Of the twelve principles in the Agile manifesto, half are concerned with time, especially as it concerns speed and change. This is partly due to the influence of “new economy” rhetoric that permeates throughout the Agile movement.

A term that gained wide currency in the mid 1990s, the new economy was a utopian idea that Western economies would be able to outsource their manufacturing industries to the developing world, and transition to economies based on services and knowledge industries. The result would be an age of prosperity, economic growth and technological innovation for all. The new economy would bring dramatic changes in the nature of capitalism: dynamic and spontaneous rather than static, old-fashioned and controlling; overturning old, corrupt hierarchies with an egalitarian meritocracies; fun and creative instead of boring and mechanical; opting for collaborative, self-organizing networked organizations over top-down bureaucracies and militaristic command-and-control forms. Companies that embraced these new principles would flourish, and those that didn’t would find themselves in dire straits.

Agile is deeply rooted in these ideas, positioning its methodologies as the secret to unlocking the innovation, dynamism and competitiveness required of companies, taking advantage of new economy ideas to provide business imperatives for adopting the methodologies. In his book Agile Software Development Ecosystems, Jim Highsmith, a software consultant who attended the Snowbird meeting, dedicates three chapters to the task of situating Agile methodologies in new economy ideas, taking this quote from a New Economy management text:

Agility is dynamic, context-specific, aggressively change-embracing, and growth-oriented. It is not about improving efficiency, cutting costs, or battening down the hatches to ride out fearsome competitive “storms”. It is about succeeding and about winning: about succeeding in emerging competitive arenas, and about winning profits, market share and customers in the very center of the competitive storms many companies now fear.

This quote comes from Agile Competitors and Virtual Organizations, a book that contains a foreword by Chrysler tycoon Lee Iacocca and has a cover that promises “Over 100 Examples of Agility!”, in order that you know you’re getting your money’s worth. Although aimed at managers in manufacturing industries, the Agile software community drew inspiration from this and other New Economy texts, adopting many of the ideas, the narrative of economic upheaval forcing organizations to adapt, and even the name itself, which displaced the previous terms light or lightweight methodologies due to their negative connotations.

But Agilists brought many of their own ideas. Beyond the time-oriented ideas of speed, flexibility and change, the Agile movement also refashioned the relationships among clients, managers and workers. Five of the manifesto’s principles aim at this goal. These are:

• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Business people and developers must work together daily throughout the project.

What’s most notable about the Sustainable Development principle is that it also concerns time but deviates from the theme of speed and flexibility to mandate a constant pace. It’s not immediately clear what this refers to, but looking closely at Agile literature, it becomes apparent that it’s a way to prevent software engineers from being overworked. Kent Beck, the creator of the once-dominant ‘Extreme Programming’ variant of Agile, argues in his first edition of Extreme Programming Explained for prohibition on asking software engineers to exceed 40 hours of work per week.

This is a reaction to a trend in some parts of the software industry, most notably in video game companies, where crunch time is widespread. Crunch time refers to the practice of mandating extended hours for programmers, who may be expected to work up to 12 hour days, 7 days a week for months at a time, especially in the run up to a major deadline. Agile methodologies rightly reject this as inhumane and unsustainable.

The second and third principles—trusting individuals, self-organizing teams—offer software engineers more autonomy and influence over their daily activities and reflect what some in the Agile movement regard as a vigorous anti-management attitude often summarized as the “We don’t need no stinking managers” attitude. The concept of servant leadership reigns as the dominant role envisioned for managers, a more nurturing and empathetic style that eschews control and authority and focuses on empowering and enabling the worker.

Here again, a close examination of Agile literature is illuminating. The Scrum methodology—today, one of the most popular variants—embraces the self-organizing team, but also advocates using what it calls “subtle control” to influence the team behind the scenes. For those who can master this technique, their power is made all the more absolute because it is wielded under the imprimatur and moral legitimacy of the ideal of self-organization. In 2007, Jim Highsmith took a more direct approach in a blog post titled No More Self-Organizing Teams:

I’ve been thinking recently that the term “self-organizing” has outlived its usefulness in the agile community and needs to be replaced. While self-organizing is a good term, it has, unfortunately, become confused with anarchy in the minds of many. Why has this occurred? Because there is a contingent within the agile community that is fundamentally anarchist at heart and it has latched onto the term self-organizing because it sounds better than anarchy.

Highsmith proposes replacing the principle of self-organization with what he calls Light-Touch Leadership and defined as “the right mix of delegation of decision making to teams while retaining appropriate decision-making authority with the leader or in other parts of the organization.” The exact meaning of “right mix” and “appropriate authority” is not made clear, but it is obvious that it means less autonomy for the non-managerial worker than in what Highsmith would call anarchy.

The forth and fifth principles mandate co-location of the engineering team and daily interaction with business people. The rationale for these principles is invariably rooted in observations about the inefficiency of distributed teams communicating across time zones and the necessity of face-to-face contact as a catalyst for the innovative and creative spirit of the team. But practically speaking, these two principles function as a prohibition against outsourcing software engineering work to low cost IT consultancies in the developing world.

From this subset of principles, it’s clear that although Agile positions itself as a software development methodology, a closer inspection reveals clues to a greater ambition: to protect the interests of software engineers at work. It does this by limiting the number of working hours, demanding autonomy and self-management, rejecting the arbitrary dominating power of the boss, preventing their jobs from being shipped overseas, and ensuring that software engineers are involved in their workplace as decision makers.

With this agenda, it is possible to characterize the Agile movement as a labour union.

In areas of the internet where programmers gather, the question of unionization returns with regularity; the reaction is mostly dismissive when it’s not outright hostile. Unions are seen to violate the values of programmer communities: they’re accused of opposing innovative technologies, being anti-meritocratic by protecting the jobs of the incompetent, and increasing costs and reducing profitability. Pro-capitalist libertarian and ‘objectivist’ views that laud the free market are emphatically endorsed.

The most significant way that Agile is covertly operating as a union is not immediately obvious from its formalized organizing principles, only becoming apparent after looking at the structure of the work itself. As previously mentioned, in an Agile environment the interactions between the client and the workers should be collaborative and iterative, incrementally building and delivering features—unlike the Waterfall method of specifying requirements in a document that is completed and turned over to the coding team for implementation.

The Waterfall model involves a greater division of labor and specialization of roles. The four discrete stages of requirements— gathering, design, coding and verification—generally correspond to distinct job titles: product manager or business analyst, systems architect, software developer and quality assurance engineer. So, from a certain perspective, the Waterfall process can be seen as a factory production line, and Agile methodologies as undermining the role of specialization, which enables deskilling. Specialization reduces software engineers to mere “code monkeys” who simply crank out code in highly repetitive and routinized ways, and lack any decision making power over the higher order architectural and requirements work that is reserved for managers and the most senior developers.

Some Agile methodologies advocate for generalists and cross-functional teams. According to the Scrum Guide,

Scrum Teams are self-organizing and cross-functional. Self-organizing teams choose how best to accomplish their work, rather than being directed by others outside the team. Cross-functional teams have all competencies needed to accomplish the work without depending on others not part of the team. The team model in Scrum is designed to optimize flexibility, creativity, and productivity.

But most Agile methodologies combat role specialization by rejecting documentation. Traditional software projects are largely run through the production of lengthy and detailed specification documents completed in isolation by managers and then “thrown over the wall” to workers at the next stage. The Agile manifesto calls instead for “working software over comprehensive documentation,” collaboration and conveying information through face-to-face conversation. These ideas tend to be founded on the assertion that excessive documentation inhibits the flexibility and agility that companies in the New Economy are supposed to need, but this also protects software engineers’ autonomy and status as professional high-skill knowledge workers who shouldn’t be reduced to mere “cogs in the machine”.

Many agilists mount aggressive challenges to what has come to be known as Taylorism, an efficiency-oriented management theory that originated in the late 19th century to improve the productivity of workers through time- and motion-based measurement, standardization and rationalization of work, and division of labor. Agilists sometimes view Taylorism in terms of a concern with creating metrics that measure software engineers’ work, and a desire to optimize those metrics. By rejecting the discrete stages of the Waterfall model and reducing documentation to a minimum, Agile methodologies resist another key aspect of Taylorism, a principle known as the separation of conception from execution.

This phrase comes from the American Marxist economist Harry Braverman in his 1974 book Labor and Monopoly Capital: The Degradation of Work in the Twentieth Century, a wide-ranging critique of 20th century capitalism and Taylorism in particular. With small modifications, Braverman’s critique of the separation of conception and execution could fit comfortably in any Agile promotional text:

In the human, as we have seen, the essential feature that makes for a labor capacity superior to that of the animal is the combination of execution with a conception of the thing to be done. But as human labor becomes a social rather than an individual phenomenon, it is possible—unlike in the instance of the animal where the motive force, instinct, is inseparable from action—to divorce conception from execution. This dehumanization of the labor process, in which workers are reduced almost to the level of labor in its animal form, while purposeless and unthinkable in the case of the self-organized and self-motivated social labor of a community of producers, because crucial for the management of purchased labor. For if the workers’ execution is guided by their own conception, it is not possible, as we have seen, to enforce upon them either the methodological efficiency or the working pace desired by capital. The capitalist therefore learns from the start to take advantage of this aspect of human labor power, and to break the unity of the labor process.

Braverman differs from the Agile understanding in that he views the unity of the labor process as essential to human labor as such. Agilists often claim that this unity inheres only in knowledge work. Martin Fowler’s 2000 introductory essay The New Methodology is typical in this regard. The essay includes a section called “Separation of Design and Construction” and takes on this principle of Taylorism (also known as scientific management) directly:

Deciding that people come first is a big decision, one that requires a lot determination to push through. The notion of people as resources is deeply ingrained in business thinking, its roots going back to the impact of Frederick Taylor’s Scientific Management approach. In running a factory, this Taylorist approach makes sense. But for the highly creative and professional work, which I believe software development to be, this does not hold.

Kent Beck also included a chapter in his second edition of Extreme Programming Explained where he describes Taylorism in general and then says:

No one walks around a development shop with a clipboard and a stopwatch. The problem for software development is that Taylorism implies a social structure of work. While we’ve lost the rituals and trappings of time-and-motion studies, we have unconsciously inherited this social structure and it is bizarrely unsuited to software development. The first step of social engineering in Taylorism is the separation of planning from execution. It is the educated engineers who decide how work is to be done and how long it will take. The workers must faithfully execute the assigned task in the assigned way in the allotted time and all will be well. Workers are cogs in a machine. No one wants to think of himself as a cog.

Thus the Agile worldview is a synthesis, addressing the new business imperatives of speed, flexibility, innovation and adaption to changing needs and linking them to worker protections that are more commonly represented by unions. It’s possible to view the various methodologies that fall under the Agile umbrella as union contracts that represent a negotiation between the interests of labor and management.

By most accounts, Agile has been highly successful. The 2011 edition of the previously cited Standish Group report compared the success rate of IT projects from 2002 to 2010 by which development methodology they used, and came down emphatically on the side of Agile:

The agile process is the universal remedy for software development project failure. Software applications developed through the agile process have three times the success rate of the traditional waterfall method and a much lower percentage of time and cost overruns.

The victory may be short-lived. Agile faces two challenges that could limit its influence, one external and one internal, which may even roll back some of the protections it has won for software workers.

The Internal Challenge from Taylorism

In recent years, product development processes inspired from industrial manufacturing practices have gained more interest in the Agile community, including Lean Software Development and Kanban. These new methodologies focus intensively on Taylorist concerns of maximizing efficiency, eliminating waste, standardizing work and continuously monitoring, measuring and optimizing the work of software engineers.

These techniques are gaining ground, but not without controversy. One prominent Agilist, Alistair Cockburn, recently expressed deep concern in a blog post titled Taylorism strikes software development:

Although we software people have spurned Taylorism loudly ever since Kent Beck introduced Extreme Programming, nonetheless we are moving heavily in that direction. I am sitting here at the Scandinavia Developer Conference and it is striking more and more forcibly each passing month and with each talk I listen to… The trigger for me writing this little note was sitting and watching yet another person describe how the introduction of visual flow (kanban-style views) and Work-in-Progress limits allows teams to develop metrics and post graphs of productivity, variance, and improvement progress on the wall for management to see. I don’t mind these things, at some level —- it is called “getting better”, and there are indeed ways of getting better. I find it ironic, however, that the very cluster of people who started and are driving agile development as the center of self-organization and craft are pushing the Taylorist agenda of scientific management.

Cockburn links approvingly to another essay that argues quite effectively that Lean and Agile are fundamentally different worldviews. But this leaves a something to be explained: if Taylorism is so antithetical to Agile, what accounts for its sudden popularity?

This might be explained by work that the word “agile” does in the movement. The word itself encapsulates the attempted synthesis that ensures worker protection become a precondition for delivering on New Economy business goals. In order to have innovation, flexibility and speed, management must permit greater autonomy for workers, reduce job specialization and reunite conception and execution. The genius of Agile is the way that it took advantage of New Economy rhetoric to advance a successful critique against the inefficiency of top-down controls, ultimately liberating software engineers from at least some measure of management’s power.

The main flaw—potentially a fatal one—was in framing the purpose of the movement as wholly in service of business goals. Agile protects workers’ interests, but it does this covertly, after obfuscating the antagonism between labor and capital. Agilists found a way to make a business case for protecting workers, and abandoned the old-fashioned moral and political framework of socialists like Braverman. But having done that, they are now vulnerable to the infiltration of dehumanizing Taylorism in service of those same business goals. The demand for agility, once turned against management, now returns in its more conventional guise as a focus on worker efficiency and waste.

Some Agilists like Cockburn would push back against these incursions, but they’re hampered by the presence of ideas drawn from complexity theory that have existed in Agile thinking from the beginning. Using terms like emergence, adaptation, evolution and fitness landscape, Agilists portray Agile organizations in highly mechanistic terms.

Complexity theory is used to explain how natural organisms are able to evolve, adapt and reproduce without any planning or theoretical understanding of the environment. Applied to a traditional Taylorist workplace, that idea implies that management, with its monopoly on conception, planning and understanding, is unnecessary. Workers simply execute an algorithm, performing an iterative walk through the possible versions of the product until they achieve optimal fitness with the “environment,” i.e. the needs of the customer or market.

This challenges the basis for separating conception from execution by suggesting that conception per se is superfluous—one can simply iterate without any planning. However, it also contradicts the humanistic assumption that creativity and innovation are undermined by mechanistic, dehumanizing workplaces. This creates strong rhetorical support for the Tayloristic impulses present in Lean methodologies.

The External Challenge from User-Centered Design

User-centered design (UCD) is a movement and a philosophy of digital product development that became popular and widely adopted in approximately the same time period as Agile (early 1990s to the present). UCD is rooted in the Cooperative Design movement of Scandinavian trade unions in the 1970s, who sought to challenge the power of management by investing workers with more decision-making power over the design of new technologies in the workplace. This move was rooted in the insight that the design of a new tool reflects (and when put into use, enforces) a specific vision of the nature of the work and the role of the worker. Designed by managers, technology could be a dehumanizing curse, leading to routinization, standardization and deskilling. In the hands of workers, it could be a blessing, extending their capacities and supporting their goals, responsibilities and interests.

Writing in his 1991 book Design for Success: A Human-Centered Approach to Designing Successful Products and Systems, William Rouse explicitly positions his design philosophy against Taylorism:

[the proliferation of complex informations systems in society] could lead one to feel that people will increasingly become like “cogs in machines,” subjugated to the economy and technological necessities of squeezing the last ounce of performance, and hence competitiveness, out of our systems. There is an element of truth in this perspective, but this vision is no longer a necessity—an alternative design philosophy is possible. The design philosophy presented in this book has three important attributes. First, this philosophy focuses on the role of humans in complex systems. Second, design objectives are elaborated in terms of roles of humans. Third, this philosophy specifies design issues that follow from these objectives. Notice that none of the three elements of this design philosophy focuses on performance, productivity, economics or related metrics. While these concerns are very important, they subserve human-centered design issues…

Rouse goes on to argue that no matter how much automation replaces human activity, humans will always be ethically, legally and socially responsible for the acts of automated systems, so technology must always be thought of as subordinate and in service to human goals and needs. Rouse continues:

This simple assertion has profound implications. From this perspective, the purpose of a pilot is not to fly the airplane that takes people from A to B—instead, the purpose of the airplane is to support the pilot who is responsible for taking people from A to B. Similarly, the purpose of factory workers is not to staff the factory designed to achieve particular productivity goals—rather, the purpose of the factory is to support the workers who are responsible for achieving productivity goals.

These same values animate the Agile movement, but here they are applied quite differently. Where Agile is concerned with protecting the interests of knowledge workers who build software, Cooperative Design and its UCD and HCD descendants want to protect the interest of workers who use these systems. When it was introduced in the United States, managers were uncomfortable with the idea of collaborating directly with workers, so it was modified so that workers would engage in design independently of managers, and renamed “participatory design” instead.

UCD further abstracts, circumscribes and disempowers the end user by the use of research methods to discover their needs and evaluate the system against their goals. In contrast to cooperative design methods which involve users directly in design, in UCD, users are involved in mediated form, through documented research activities that represent the users to the decision makers. In practice, this means that business goals almost always trump users’ goals when they come into conflict, but nonetheless, the presence of UCD practitioners in the development process introduces that antagonism—admittedly minimal in many cases—and prevents it from being completely papered over.

Agile has some important similarities and differences with UCD. Agile also emphasizes the value of engaging with people outside of the development team, but due to its focus on the interests of internal workers, places more importance on bringing software engineers directly into contact with the customer (or their representatives) to learn about what users need. From the UCD perspective, Agile is naïve in assuming that a new system’s customer (who is often drawn from the ranks of management) can adequately represent the goals, needs, context and interests of its users. Agilists often collapse the distinction, defining “the customer” as the people who are paying for the software and who are going to use it, as if they are simply the same.

In most Agile methodologies, the role of the customer is both highly demanding and almost completely unspecified. They’re expected to have detailed and comprehensive information of every requirement for the system, as well as deep knowledge of the problem space and the needs of the users. It is simply assumed that the customer has this information, or can easily get it, but in practice, this is almost never true. UCD addresses this problem by offering a wide array of research methods drawn from academia and adapted for product development: interviews, surveys, diary studies, contextual inquiry, ethnographies, focus groups, task analysis, user behavior metrics. But these cannot be used to full advantage by Agile teams. These methods fall outside of the education, training, job descriptions and identity of most software engineers, so they can only be used by employing specialists outside of the engineering team, which separates conception from execution, threatening to de-emphasize the centrality of that job function.

Agile faces more challenges from the rise of UCD in software development work. In comparison with the Waterfall mode, Agile offers much faster iterative feedback cycles, which are measured in weeks rather than months or years. In a 30-day sprint, Agile developers are able to define requirements, design and build a feature, and then ask the customer to evaluate it. This is considered very fast by the standards of conventional software development, but designers working in a UCD process are able to work much faster, producing concepts for a new feature that can be evaluated by users in days rather than weeks. This is because designers typically don’t write code—instead, they create design artifacts called prototypes, wireframes and mockups which simulate the interface and the end user’s experience with just enough realism so that the proposed feature can be evaluated by users.

The process of evaluation often takes the form of usability testing, where researchers observe users who are asked to complete tasks using software prototypes to verify that the design makes sense to users and they are able to complete the tasks successfully and with a minimum of unnecessary effort.

This poses two other critical problems for Agile developers. First, UCD enables designers and researchers are able to perform iterative feedback cycles without engaging software engineers for anything except the most advanced features that can only be simulated by writing code, effectively reducing the importance of software engineers in the creation process. And second, much like the problem of researching user needs, UCD’s usability testing methodology for evaluating software with users is a specialized skill that again exceeds the expertise of software engineers.

These three problems can be simplified into variations of a single problem. The rise of UCD signals a shift from thinking of a software technology as a technical system towards thinking of it as a socio-technical system. This expanded understanding translates into a need for a new set of competencies beyond traditional software engineering concerns, new methods of evaluating success that emphasize the user’s experience (going beyond delivering software on-time and under-budget), and a new interest in the qualities of the user interface—a crucial point of integration between the human and the machine.

The Agile movement is undoubtably aware of this shift, but only dimly. Although many authors emphasize the importance of user participation in the design and evaluation of software, the methods for doing these are under-developed in the Agile literature and so adoption of UCD techniques among teams is uneven and difficult at best. User-centered designers often report difficulty working in Agile development environments, typically finding that software engineers want to start programming as quickly as possible, and take little interest in user research, design ideation and exploration and validation of prototypes with users. In many Agile teams, activities that do not involve writing code are viewed as unnecessary impediments to progress, a kind of software engineering chauvinism that reflects a technical mindset rather than a socio-technical one.

Agile processes are founded on protecting software engineers’ autonomy, but the lines are drawn in terms of programmers vs. non-programmers, not labor vs. management as in traditional unions. This limits the influence of everyone outside of the software engineering team, not just managers, but also user researchers, interface designers, content strategists, and information architects, pushing these new, crucial functions outside of the team.

Effectively, a condition has been created where the only way to take advantage of these disciplines is through a Waterfall model. The widely accepted best practice for integrating UCD into Agile is the staggered-sprints model: while software engineers are working on features in one sprint, designers and researchers are preparing specifications for the next one. This model preserves the speed and flexibility of Agile, but restores the separation of conception from execution. Designers and researchers conceptualize, prototype and test, and then turn over their work to software engineers for implementation.

Agile processes are optimized for constructing technical systems conceived and built by programmers, but poorly suited to building socio-technical systems that seek an integration between humans and technology—a model that requires software engineers to embrace a wider range of disciplines and skills as equals, and adopt a more comprehensive vision of what it means to successfully build software.

Failing to make this transition is very likely a death sentence for the Agile movement. Except for the most senior software engineers, most of the creativity and innovation in software will move to specialized design and research activities using techniques that exceed the reductionist thinking that dominates computer science departments. Simultaneously, Agile will be hollowed out from within, as the infiltration of lean production techniques standardizes and routinizes the creative work of software engineering.

While software engineering adopts the strategies of factory production lines to become more like blue collar labor, design techniques and ways of thinking are working their way into management. In North America and Europe, MBA programs have begun to include courses on how managers can think and innovate like designers and practicums where students learn problem-solving techniques in a style traditionally found in design departments.

A notable example is the University of Toronto Rotman School of Management, which offers MBA students elective courses through the Rotman DesignWorks studio. Their mission:

We teach students how to tackle complex business challenges using Business Design, a human-centered, creative problem solving methodology. We draw on the designer’s way of working and apply it to the way we build businesses – from innovative user experiences to creative business strategies and models.

In the United States, Stanford University’s Hasso Plattner Institute of Design (the d.school) provides similar training in design thinking to executives and corporate leaders alongside its German sister organization, the Hasso Plattner Institute School of Design Thinking. Many universities have begun to offer joint design and business degrees: at California College of the Arts, an MBA in Design Strategy; Johns Hopkins University Carey Business School has an MBA/MA in Design Leadership offered in conjunction with Maryland Institute College of Art; an MBA/MDes dual degree from Illinois Institute of Technology; and the prestigious Kellogg School of Management offers a graduate degree with a triple focus of business, design and engineering.

The next generation of corporate leaders are learning to apply the lessons of designers to the problem of innovation, but how does this new orientation fit alongside the efficiency, productivity and measurement mindset that was once the cutting edge of management thinking and now is coming to dominate Agile software engineering? In an article for the New York Times, Sara Beckman of UC Berkeley’s Haas School of Business and advocate of design thinking suggests an analogy: design thinking is like quantum physics transcending the Newtonian physics of analytical management. But there is no need to choose one or the other:

Both worlds — the quantum one where designers push boundaries to surprise and delight, and the Newtonian one where workers meet deadlines and margins — are meaningful. The most successful companies will learn to build bridges between them and leverage them both.

In light of the fact that design is slowly transforming into a management discourse, the difference articulated here between designers and workers should be understood as the same as managers and workers. The synthesis of these two ideas is clear: design thinking offers a flexible and open-ended framework that loosely structures the strategically important business of innovation, while the tools of efficiency, productivity and standardization are applied outwards, to tightly regulate and discipline the activities on the shop floor.

An influential segment of the Agile movement has inexplicably adopted the lean production rhetoric of efficiency and waste to govern their own work. We need only look to at the impact of these ideas on assembly line workers to understand the possible impacts on working conditions. In Alternatives to Lean Production: Work Organization in the Swedish Auto Industry, Christian Berggren writes about the importation of Japanese lean production into American auto plants:

If anything, the rhythm and pace of the work on the assembly line is more inexorable under the Japanese management system than it ever was before. Off-line jobs, such as those in subassembly (the senior workers’ favorite positions, in which a personal work pace and “banking” were possible) have been outsourced or are geared strictly to the main line by means of just-in-time (JIT) control. Idle time is squeezed out of each work station through the application of kaizen techniques, while work pressure has been intensified and staffing drastically reduced in the name of eliminating all “waste” (muda)… [A]uto workers resented the fact that the speed of the belt made it difficult for workers to detatch themselves from the repetitive work. These difficulties have been aggravated by the combined JIT and quality pressure (zero defect) of the modern Japanized lines, which demand a high degree of mental concentration on work that is still very standardized. … [Lean production’s] factory regime is rigorous with demands for perfect attendance, mandatory company uniforms, the prohibition of all personal articles, strict rules of conduct, and an elaborate system of sanctions.

These lean factory conditions contrast sharply with the lavishly appointed offices, flexible work environments and perks of Silicon Valley, and it’s difficult to imagine a future where software engineers face anything like this. At the same time, the Agile union may be faltering, and although it’s hard to predict the impact on the day-to-day lives of software engineers, it is unlikely to be a positive one.