Dieser Blogpost ist auch auf Deutsch verfügbar
TL;DR:
- Whether a technology is feasible can be checked quickly and cheaply with a spike built by agents, even for several alternatives at once.
- But an “it works” without understanding is a weak signal. And if the agent fails, the fault may lie with the agent or the prompt.
- If a technology is actually going to be adopted, the developer needs to build a spike personally. A randomized study by Anthropic shows that learning with AI assistance works worse when the task is largely delegated.
- This leads to a two-stage approach: agent-built spikes to screen options, then a hand-built spike for the candidates. The code can be discarded; the understanding should stay.
This post is part of a series.
- Part 1: Building to Learn and Building to Deliver
- Part 2: Building to Understand (this post)
One of the four types of assumption tests that Teresa Torres distinguishes is the research spike. She defines it as an engineering activity that assesses feasibility risk, often in the form of a technical prototype. In practice, such a spike usually serves a second function as well: the team gets to know the technology along the way. How do these two functions change with agentic development?
Feasibility
Whether something is possible at all can be tested with a spike developed entirely by agents. It is fast and cheap, and in the time a hand-written spike takes, agents can produce several alternatives. This fits Torres’ advice to compare several solution ideas rather than deciding the fate of a single idea.
Note, however, that an “it works” without understanding is a weak signal, because you cannot see why it runs and where the hidden limits are. A failure of the agent is also no reliable sign that the technology is unsuitable, because the fault may lie with the agent or the prompt.
Understanding cannot be delegated
If you actually want to adopt a technology, it becomes part of what the developer’s theory of the program has to cover (Thesis 9). The developer has to build a deep understanding of the newly introduced technology, diagnose errors, and be able to judge generated code. For that, we consider a spike that the developer builds personally to be necessary.
In a randomized study (paper), Anthropic examined how AI assistance affects learning a new library. 52 mostly junior developers were asked to learn a Python library unfamiliar to them using a tutorial-style task. In the subsequent test, the group with an AI assistant scored 50 percent on average, the group without 67 percent. The largest gap showed up in debugging.
How someone used the AI mattered. Those who largely delegated the task did poorly. Good results came mostly from those who asked conceptual questions and wrote the code themselves. The study has its limits, though. The groups for the individual usage patterns are quite small, and understanding was measured shortly after the task was completed. In addition, a chat assistant was tested, not agentic development. The authors, however, expect stronger effects with agentic tools.
Two stages
This leads to a two-stage approach. First, a spike built by agents checks whether a technology comes into question as a candidate. Success makes it a candidate, but not yet the choice. A failure of the agent, by contrast, is only a weak argument against the technology. If there are several alternatives, you can still set it aside, because losing a single option costs little. If there is only one, or it looks particularly promising, you should find out the cause before discarding it.
For candidates you actually want to adopt, a spike that the developer builds personally follows. The AI can explain concepts and answer questions along the way, but the developer writes the core code. This matches the patterns with the best results in the study. That way, the effort for understanding is limited to the technologies that remain as candidates.
A third purpose of building
Alongside building to test and building to deliver, there is thus a third purpose: building to understand. The code of this spike can be discarded. The understanding should stay.