The Shifted Complexity: From Waterfall through Agile to AI

Dieser Blogpost ist auch auf Deutsch verfügbar

TL;DR

Complexity in software development has kept shifting over time: from project management and requirements engineering towards individual developers.
Many waterfall projects did not fail because of the role separation, but because of the illusion that software development could be planned deterministically instead of being complex by nature.
Agile methods bring part of this complexity back to the team, by working in iterations and joint refinement between PO and team.
AI-driven, agentic development shifts developers' role again in a fundamental way: spec-driven development and orchestration move it closer to requirements engineering and therefore to the chaotic stakeholder environment.
Because of AI tools, tasks seem simpler than they actually are. This apparent simplification actually requires deeper engagement with the domain and therefore leads to higher mental load.
Junior developers are especially affected: decreasing demand for their capacity meets a lack of the competencies needed to handle the increased complexity. The learning pipeline for the next generation of senior developers breaks down.

The Increased Mental Load

“I am totally exhausted after work!” I have been hearing this sentence over and over lately when I talk to software developers about their everyday work. They talk about a noticeably increased mental load: more context, more responsibility, more decisions that used to be made elsewhere. The AI-driven development process seems to be the cause of this. Despite faster delivery, less manual coding, and higher output, the use of AI seems to bring an entirely new kind of complexity into developers' everyday work.

I’m convinced that this phenomenon is actually the culmination of a development that began long before AI entered software development: the complexity of software development has never really been reduced by new methods and tools. It has only been shifted, again and again: from project management and requirements engineering towards the team, and from there towards individual developers. But AI is not the cause of this shift, but only its most radical accelerator so far.

To understand this, it’s worth looking through the lens of the Cynefin framework, which helps distinguish problem spaces by their degree of complexity. The Cynefin framework provides the vocabulary to describe what has actually happened over the last two decades. It only becomes clear why developers in particular are affected by this shift, and why this is especially a problem for juniors, once we understand which kind of complexity shifted, and to where.

Cynefin in a Nutshell

The Cynefin framework, introduced by David Snowden and Mary Boone, distinguishes tasks and systems by how easily cause and effect can be identified, and derives from that which type of approach is most appropriate in a given context. The Cynefin framework distinguishes four different domains:

Simple / Clear: Cause and effect are obvious, and there are well-established standard solutions. A bug with a known cause and a known fix, a standard CRUD feature following an established pattern. In a domain like this, applying the right best practice is what’s needed. This is especially the domain where junior developers can gain their first experience, building the skill set needed to tackle bigger challenges.
Complicated: Cause and effect can be identified, but analyzing them requires expertise. A performance optimization that requires profiling and analysis to identify the bottleneck, or the integration of a complex third-party system based solely on documentation. Here, good practices, methodology, and experience help find a solution to the problem at hand. According to Snowden, this is the problem area where experts can shine. In software development, this could be the specialist with deep know-how who needs clear structure, prefers to work alone, and thrives in this environment, developing outstanding solutions.
Complex: Cause and effect cannot be identified in advance, only in hindsight. An architectural decision for a new, unknown problem area, a product experiment with uncertain user behavior. No blueprint helps here, only trying, observing, and adapting (emergent practice). This is the domain with the highest conceptual uncertainty. Here, we work most successfully with agile, cross-functional teams, pooling our knowledge and advancing through hypotheses and experiments.
Chaotic: At first, there is no identifiable relationship between cause and effect. We have to act urgently and establish some structure before we can analyze anything at all. In software development, the typical example is usually not the system itself but its surrounding environment: conflicting stakeholder interests, political infighting between departments, unpredictably changing requirements, stakeholders with varying degrees of willingness to cooperate and hidden agendas. Here, no methodology and no analysis process will help. The first thing to do is establish order, communication, and trust (novel practice) before anything resembling a solvable problem can even be formulated. This is exactly the kind of environment that has a destructive effect on a development team, and it’s the product owner’s responsibility to shield the team from this chaos, giving them the freedom to tackle the complex, domain-related and technical problems at hand.

There is one important effect that the Cynefin framework describes as the complacency cliff: The transition from Simple to Chaotic is fluid and treacherous. A simple-looking task can carry far more uncertainty than is apparent at first glance. And if that uncertainty goes unnoticed, the situation tips abruptly into chaos. We’ll encounter this effect again later, when we talk about AI-driven development.

We will use these four domains not as a diagnostic tool for everyday work, but as a vocabulary: they help us to name which types of complexity have moved where during the last two decades.

How Complexity Shifted from the Environment to the Team

Illusion of Predictability in the Waterfall

In classical software development, there was a strict separation between roles: project management, requirements engineering, and development were separate functions, sometimes even carried out by distinct people or departments. This separation was not accidental. It actually served a Cynefin function: project management was meant to absorb the chaotic environment of stakeholder interests, political constraints, and shifting expectations, while requirements engineering was supposed to distill a solvable, complicated problem from it.

The waterfall model had a critical design flaw: It assumed that a software project as a whole could be planned deterministically. Speaking in Cynefin terms, a larger software project is complicated at best, but never complex. This is exactly where many waterfall projects failed. Larger software projects are complex by nature, sometimes even chaotic: requirements change, assumptions turn out to be wrong, user behavior can’t be fully predicted. An approach based on a single, sequential process of analysis, design, and implementation cannot cope with this kind of system behavior, regardless of how cleanly the roles are separated. What was missing was exactly the iterative approach that Cynefin proposes for complex domains: probe, observe, adapt, instead of planning everything upfront, once and for all.

This is exactly the design flaw that agile methods address, by a tight temporal coupling of conception and implementation instead of a sequence of phases. This comes with a second, independent shift: the roles themselves blend together more than before.

A central mechanism for this is the refinement process: requirements are no longer thought through solely by an analyst and handed over to the team, but are worked out, questioned, and sharpened together by the product owner and the team. The PO still serves as a buffer for the team: effective stakeholder communication is meant to protect the team from the chaotic environment. But because of shared refinement, this buffer has become more permeable than it was under strict role separation. A good deal of the conceptual, partly complex work that used to be done outside the development team now remains within the team permanently.

Tackling Complexity Requires the Right People in the Right Place

This increased complexity within the team cannot be handled by process alone. It also requires distributing it appropriately among the people in the team. Not all developers are equally able or willing to engage with high complexity, be it due to a lack of experience or due to personality. Experienced developers tend to take on the more complex, uncertain parts, while tasks ranging from simple to complicated provide a space in which less experienced developers can gradually work their way up to greater complexity.

A strong team is characterized by different personalities working together constructively, with tasks sliced so that each team member can contribute their skills and preferences in the best possible way. This deliberate composition and distribution of work is essentially a distribution of complexity among the people who are both able and willing to deal with it best. This principle holds true independently of the chosen approach, but it has gained importance through the agile shift, since there is more complexity to distribute within the team itself.

The Chaos Arrives at the Team

Over the past few years, software development has changed significantly through AI-driven and increasingly agentic tools. Where developers used to write code line by line themselves, AI agents now generate entire functions, modules, or even architectural skeletons based on specifications and prompts.

The actual work shifts from writing code to formulating, directing, evaluating, and fixing it. Developers increasingly orchestrate the development process instead of carrying it out themselves. This results in a huge productivity gain, but it also changes which skills are needed and where in the process the actual work happens.

The pattern from the previous section continues: Complexity keeps moving further towards the development team. What’s changing is not only the speed of this shift, but also a new factor that didn’t exist before: the incentive to shrink teams, since using AI seems to require fewer people for the same output. These effects (an accelerating shift, shrinking teams, and a growing scope of what individual developers have to handle) intensify what was already underway.

From Coder to Requirements Engineer

Approaches like spec-driven development shift developers' role once again, fundamentally: it’s no longer enough to question and sharpen requirements together with the PO during refinement. Developers are now responsible for writing the specification from which AI generates the code. As a result, they now have to bear a significant share of what used to be PO and requirements engineering work. That’s a qualitative leap: the role moves structurally closer to that of PM and PO, and therefore closer to exactly the chaotic environment that the earlier role separation was meant to protect against.

A Mismatch with One’s Own Career Choice

This shift runs up against an uncomfortable reality: Many people chose software development precisely because of its clear structures and low ambiguity. Besides enthusiasm for technology, this was a legitimate, often central factor in their career choice. Now, the new role requires exactly the skills that matter in a chaotic, politically charged environment: tolerance for ambiguity, negotiation skills, and communication with stakeholders who vary in their willingness to cooperate. What used to be a question of distribution within the team (not every member needed every skill, the team balanced out differences) is now increasingly becoming a requirement for each individual.

When Complexity Masquerades as Simplicity

This is where the complacency cliff mentioned earlier becomes concrete: AI tools make tasks look simple (prompt in, code out), while the underlying task is in reality complex or even chaotic, precisely because it now carries elements of requirements engineering and stakeholder management. The reduction to orchestration suggests a simplification of the work, but actually requires deeper engagement with the domain than writing code ever did. This discrepancy between apparent simplification and an actual increase in cognitive burden is likely a key reason why so many developers report significantly higher mental load. Not because the tools provide less support than promised, but because the manual work they save resurfaces elsewhere, as cognitive and accountability work.

Smaller Teams, Missing Learning Pipeline

The team reduction mentioned above exacerbates the situation: the previous organizational response to complexity, distributing that complexity among different personalities and skills, becomes harder as teams get smaller.

Junior developers are affected most, through two compounding effects: First, the demand for their capacity is shrinking, since AI now handles exactly the simple-to-complicated tasks that junior developers used to learn from. Second, they often still lack the skills needed for the now larger share of complex and chaotic work. This is not only a short-term problem: if junior developers no longer have the space to practice in the complicated domain before facing complex or chaotic situations, the learning pipeline that is supposed to produce the next generation of senior developers collapses.

Larger Scope, More Responsibility

Besides the increased complexity, this same development also simply increases the scope of what developers have to oversee. If you orchestrate the development process instead of just coding, you have to understand more of the domain, oversee more code than you would have written yourself, and make more decisions that used to be distributed across several roles. This is a burden quite independent of how complex or chaotic individual tasks are, in Cynefin terms. Just more context, more threads to keep in mind simultaneously, more responsibility per person.

Combined with the often invisible complexity described earlier, this creates a double burden: the work hasn’t just become more difficult, it has also become more extensive, and these two effects reinforce each other.

Now What?

Looking at how software development has changed over time through the Cynefin lens, we can identify two fundamental shifts:

The clear role separation of classical software development was an attempt to shield the team from the chaotic and complex parts of a software project. Project and requirements management were meant to translate the surrounding chaos into clearly structured requirements, so that a complicated, solvable problem remained for development. Agile methods have loosened this separation: through iteration and joint refinement, a good deal of conceptual, complex work moved back into the team, while the PO remained as a buffer against the chaotic environment.
AI-driven development doesn’t just intensify this shift gradually, it moves the domain boundary itself, while at the same time concealing the resulting increase in complexity. Tasks that look simple through spec-driven development and orchestration in reality carry a significant share of what used to be requirements engineering and stakeholder management, and with it exactly the complex and chaotic parts that the original role separation was meant to protect against. Junior developers are hit hardest: not only is demand for their capacity declining, but the learning pipeline in which they used to gradually work their way up to higher complexity is also collapsing.

Against this background, the increased mental load that developers report is not a surprise, but an expected symptom: the scope and complexity of the work are growing at the same time, while the organizational buffers that used to absorb both are shrinking.

This leaves us facing entirely new questions about how we organize software development:

How do we design teams and roles so that this newly emerging complexity can be distributed again, instead of being dumped solely on individual developers?
What does this mean for training junior developers, once their previous practice space disappears?
How can we ensure that junior developers can still work their way up to more complex tasks step by step?
How do we make sure that those who deliberately chose clear structures don’t simply get left behind in this shift?

Source

David J. Snowden, Mary E. Boone: A Leader’s Framework for Decision Making