Dieser Blogpost ist auch auf Deutsch verfügbar

TL;DR

  • As soon as model output becomes part of the product logic, good prompts are no longer enough. You need contracts, evals and observability.
  • Contract-first instead of prompt-first: first define the (JSON) schema that the rest of the application expects from the model’s answer, then the prompt.
  • Evals instead of gut feeling: the contract guarantees the structure, not the substantive quality. A dataset plus grader shows whether the output is reliable enough.
  • Observability and the UI are part of it: nondeterministic features need telemetry (model, latency, validity, retries, cost) and a real interaction flow, not just text output.

The setup: our example

To make all of this a bit more concrete, let me start with an example: a mini web app that turns a GitHub issue, or any ticket text (for example from Jira or GitLab), into a structured implementation plan. To do that, I start with a target structure that I’ll use to explain an AI FullStack feature.

{
  "summary": "string",
  "affectedAreas": ["frontend", "backend", "database", "infra"],
  "risks": ["string"],
  "implementationSteps": ["string"],
  "testIdeas": ["string"],
  "requiresApproval": true
}

Based on this example, it’s easy to have a web app generated that uses this blueprint to present different implementation plans sensibly. In this case, Claude Code built it for me.

Website (in dark mode) for displaying a GitHub issue in a structured way
Website (in dark mode) for displaying a GitHub issue in a structured way

In the generated web app, the user can paste an issue into the text box or fetch it from GitHub, and have it displayed in a structured way. Sure, this is a really simple and somewhat contrived use case, but the principles and concepts described here apply to more sophisticated UIs too, it’s just easier to follow in this one.

Contract-first instead of prompt-first

With the example above, I had Claude build everything in one go, and really, we’re already done here. The app works (using a prompt created by ChatGPT).

Generated prompt for building the demo
You are a senior TypeScript engineer. Build a small but polished demo web app that showcases schema-first AI output using the Vercel AI SDK.

Do not ask follow-up questions. Make reasonable choices and implement the full app end-to-end. Do not leave placeholders or stubs. Keep the code compact, readable, and suitable for a public demo repo / blog post.

Goal

Build a mini web app that turns either:

- freeform story / ticket text
- a GitHub issue

into a structured implementation plan.

The point of the app is to demonstrate that the AI feature is contract-first, not chat-first.

Stack

Use:

- Next.js App Router
- TypeScript
- AI SDK
- AI SDK UI
- Zod
- @ai-sdk/openai
- minimal styling only (simple CSS or Tailwind is fine)
- no database
- no authentication system
- no server-side persistence

Non-negotiable AI SDK architecture

Use these exact patterns:

- shared Zod schema imported by both client and server
- streamText + Output.object(…) in the route handler
- experimental_useObject on the client
- createOpenAI(…) to instantiate the provider with a request-specific API key
- custom request headers from the client to the analyze route
- no plain-text fallback mode for the main analysis flow
- do not use useChat for the main experience

This app is object-generation first, not chat first.

Core JSON contract

The app must use this exact top-level JSON shape and these exact property names:

{ "summary": "string", "affectedAreas": ["frontend", "backend", "database", "infra"], "risks": ["string"], "implementationSteps": ["string"], "testIdeas": ["string"], "requiresApproval": true }

Implement it as a shared Zod schema and export the inferred TypeScript type.

Schema requirements:

- summary: non-empty string
- affectedAreas: non-empty array of unique enum values, restricted to: frontend, backend, database, infra
- risks: array of strings
- implementationSteps: non-empty array of strings
- testIdeas: non-empty array of strings
- requiresApproval: boolean

Do not parse model text with regex. Do not accept "almost matching" output. The UI and server must rely on schema-validated structured output.

But let’s take a few steps back. Normally I don’t build something only to fill it with AI afterwards. Instead, my data should be prepared and integrated intelligently into the UI. There are already countless articles about this online, and AI already does a great job helping me here.

The key point, though: the contract comes first, then the prompt. I need to know which data I have to extract in a structured way before I move on to writing and perfecting my prompt. This means doing the kind of engineering work that should have been familiar even in the pre-AI era.

  • Which fields do I want to extract from the unstructured data?
  • Which fields are required? Which are optional?
  • Which value ranges apply to numbers and strings? Are there enums?
  • Where should I constrain the output so that it stays usable in the UI or pipeline?

Depending on the programming language, different frameworks help me here. If we think of one of the more modern development environments (TypeScript), ai-sdk could be an option, for example. For Java there’s Spring AI too, and other languages surely come with similar tooling. These frameworks help unify the communication with different model providers, so I don’t have to tie myself directly to a vendor and its APIs. In this article I focus on TypeScript and ai-sdk.[1]

Such frameworks have the big advantage of unifying the different providers' APIs and keeping common features easy and uniformly configurable, structured outputs for instance, which is exactly what we need.[2]

To build this, you usually need a contract, above all a JSON schema. JSON schemas describe the contract a JSON has to satisfy: which fields are required? Which regex must these values match? Number ranges, and so on. To create or validate this JSON schema, we can use tools like https://json.ophir.dev/. Or ask an AI to create a suitable JSON schema for our needs.

The most important part of my AI feature is not the prompt, but the contract that the rest of the application expects from the model’s answer.

With the generated or hand-written contract in hand, we can set out to formulate a suitable prompt. The answer will then either match the contract, or be an error contract that we have to handle separately, for example OpenAI’s refusal.[3]

Evals instead of gut feeling

Anyone who builds an AI feature without evals is testing on the user.

As described above, Claude built me a working web app in a single shot. If I enter, as an example, the issue https://github.com/rubenvitt/bluelight-hub/issues/420, I get structured output:

{
  "summary": "Implement a Verpflegungsmanagement feature under the Betreuung area to manage feeding locations, meal planning, and distribution tracking for Einsatzkräfte and Betroffene. The change includes frontend views on the specified route and backend CRUD plus integrations for planning quantities and recording consumption.",
  "affectedAreas": [
    "frontend",
    "backend"
  ],
  "risks": [
    "Integration dependencies on Kräfte-, Betroffene-, Logistik-Modul, and Lagekarte may block parts of the feature or require fallback handling.",
    "Quantity estimation can become inconsistent if personnel and affected-person counts are unavailable, delayed, or separated differently than expected.",
    "Tracking meal status and portion counts across multiple Ausgabestellen introduces risk of incorrect state transitions or double counting.",
    "Capturing Sonderbedürfnisse must stay usable in the UI while still being detailed enough for planning."
  ],
  "implementationSteps": [
    "Add backend CRUD support for Verpflegungsstellen with fields for name, location, type, capacity, opening/output times, and responsible personnel.",
    "Add backend CRUD support for Mahlzeiten including time, type, estimated quantity, assigned Ausgabe-/Verpflegungsstelle, separated counts for Einsatzkräfte and Betroffene, Sonderbedürfnisse, and status values Geplant, In Zubereitung, Ausgabe, Abgeschlossen.",
    "Implement backend logic for quantity estimation based on the provided formula using counts from Kräfte and Betroffene where available.",
    "Add backend endpoints or service integration for recording consumption data per Mahlzeit and Ausgabe-/Verpflegungsstelle: issued portions, beverage consumption, remaining quantities, and material consumption for Logistik handoff.",
    "Build the frontend page at /app/einsatz/$einsatzId/betreuung/verpflegung with management views for Verpflegungsstellen and Mahlzeiten.",
    "Implement Verpflegungsstellen cards showing key details and current status, with create/edit/delete flows.",
    "Implement a Mahlzeiten timeline that separates planned, current, and past meals and supports status updates through the defined lifecycle.",
    "Add meal planning UI for time, type, quantity estimate, separated person counts, special needs, and assigned distribution point.",
    "Implement quick-entry controls for portion output using +/- counters and forms for beverage usage and remaining quantities.",
    "Add a dashboard view showing supply coverage as x% versorgt based on planned vs. served data.",
    "Integrate map marking of Verpflegungsstellen on the Lagekarte from the route workflow.",
    "Connect quantity planning to personnel counts from Kräfte and affected-person counts from Betroffene, and send recorded material consumption to Logistik for replenishment workflows."
  ],
  "testIdeas": [
    "Create, update, list, and delete Verpflegungsstellen with all required fields and verify they appear correctly in the management UI.",
    "Plan a Mahlzeit with separated counts for Einsatzkräfte and Betroffene and verify the estimated quantity follows the specified calculation.",
    "Verify meal status transitions from Geplant to In Zubereitung to Ausgabe to Abgeschlossen are reflected correctly in the timeline and detail views.",
    "Record portion output for a meal at a specific Verpflegungsstelle with +/- quick entry and confirm totals persist correctly.",
    "Record beverage consumption, remaining quantities, and material consumption and verify the data is stored and associated with the correct meal and location.",
    "Enter Sonderbedürfnisse such as allergies, vegetarian, halal, and confirm they are retained and visible in planning views.",
    "Verify Verpflegungsstellen are shown on the Lagekarte with the configured location.",
    "Test integration behavior when Kräfte or Betroffene counts are available versus unavailable, ensuring quantity estimation degrades predictably.",
    "Verify the dashboard coverage percentage updates as portions are issued.",
    "Confirm material consumption data is handed off correctly to the Logistik integration path."
  ],
  "requiresApproval": false
}

40 lines of output. LGTM. Done.

Or… not?

The question is not whether the answer sounds “smart”, but whether it is reliable enough for my workflow.

How do I know that the result is a good result? How do I know I can trust it? The contract gives me a structural guarantee. Whether the content holds up in substance is not answered by that. So above all the substantive questions need clarifying: Are the affected areas identified sensibly? Are the risks useful? Can I really make use of the test ideas, or is it pointless nonsense? Is the approval flag plausible? One option to answer all of these in full is to go through every single response and judge these questions for myself. Then I can adjust the prompt and rate again. But my honest opinion on that? I don’t want to, no! I don’t feel like it.

And yes, there is an alternative. Several, in fact. OpenAI has its own optimizer, but there are also numerous open source tools. One example would be https://github.com/promptfoo/promptfoo.[4]

For simplicity I chose the OpenAI Platform here. An evaluation is quickly set up there: create a prompt (important: with a variable!), create a new dataset in the Evaluation tab, then add test data. I simply take some of my issues from the bluelight-hub project, then click play, and the prompt runs against the list of inputs (careful: this costs money).

OpenAI helps us a lot here, almost everything can be reworked automatically right there, and our JSON schema can be generated too. I simply pasted the zod schema from the example project, and the matching JSON schema was generated from it…. perfect.

With the prompt, make sure to set the text format to json_schema so you can specify a fixed data contract
With the prompt, make sure to set the text format to json_schema so you can specify a fixed data contract (which can then also be generated automatically according to my specifications)

The advantage? We now have several columns in our eval view and can read off the individually produced values directly and decide how we want to handle them.

Several columns in the view once we have provided the matching JSON schema
Several columns in the view once we have provided the matching JSON schema

Now we can begin the actual evaluation: create new columns either for manual review (annotation), or we let the AI review the individual answers (grader), checking and rating them according to our criteria.

After a first evaluation against two criteria, it becomes clear: our prompt needs some rework. What to do? In this tool a single click on Optimize is enough, and OpenAI takes care of the rest, based on the rating from the prompt evaluation.

After a first prompt test run: seems it's not that good after all
After a first prompt test run: seems it's not that good after all

After a bit more optimization and a manual adjustment (via prompt), the result got a little better. Sure, this is a really small set for evaluation, I’m more interested in the principle here; I also simply left out manual feedback, even though normally that’s part of it, to get meaningful results in the end.

Eval result after prompt optimization
Eval result after prompt optimization

With that great result, I go straight to Claude and ask it to update the prompt with the one I created.

Now we have an improved prompt that covers our requirements better. With the structured output we build UI, instead of plain text output. So much for the mini glimpse. There is so much more to discover, optimize and refine here.

Still, the basic principle of the eval should have become clear: define goals, collect a dataset from examples and possibly expected results, define metrics, compare runs, have them evaluated.

Observability instead of black box

When we implement nondeterministic systems in our applications, it is all the more important to understand what happened when something breaks, so that we can find a way to mitigate it. This includes a few important data points that we ideally want to collect:

  • which model was used?
  • how long did the LLM take for an answer?
  • was the returned JSON valid against the contract?
  • were retries necessary?
  • if applicable: which tools were used?
  • where did humans have to intervene?
  • how many tokens were used and how high were the associated costs?

OpenTelemetry now has conventions for GenAI applications covering events, metrics and spans. That means the AI functions can be integrated into observability too. The AI SDK mentioned above also has a connector to automatically capture and send corresponding data through the framework.[5][6]

The FullStack part is the UI, not just the model

It only becomes a FullStack topic once I’m not just calling a model, but designing an interaction flow for people.

Now that we’ve settled on a reasonable contract with the LLM, communicated it over the API, and built a really great prompt, what’s missing? Right, the UI. At the start of the post I already hinted at this: just showing a bit of text, anyone can do that. Even after the one-shot attempt with Claude it wasn’t a chat interface, but still just a sequence of texts. We’re moving into somewhat dry territory here, an implementation plan for stories, but at least for the presentation we can surely come up with something great, right?

Display the data a bit more nicely
Option 1: simply display the data a bit more nicely

As a first step, I just wanted the data presented a bit more nicely. So far so good, so how about an implementation canvas?

Implementation canvas with editable content
Implementation canvas with editable content

Another way of displaying it, with the option to edit the content, but how can we create real value? If the task involves frontend work… shouldn’t we generate a mockup right away? No sooner said than done…

Ohh, I love Agentic Engineering.

Automatically generated UI mockup based on the implementation plan
Automatically generated UI mockup based on the implementation plan
  1. Spring AI — Level up your Java code and explore what Spring can do for you. (spring.io)  ↩︎

  2. Building Agents | AI SDK — Complete guide to creating agents with the ToolLoopAgent. (ai-sdk.dev)  ↩︎

  3. Structured model outputs | OpenAI API — Understand how to ensure model responses follow specific JSON Schema you define. (developers.openai.com)  ↩︎

  4. Prompt optimizer | OpenAI API — Learn how to use your dataset to automatically improve your prompts. (developers.openai.com)  ↩︎

  5. Generative AI semantic conventions — Important GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository. (opentelemetry.io)  ↩︎

  6. Telemetry | AI SDK — Using OpenTelemetry with AI SDK Core. (ai-sdk.dev)  ↩︎

Conclusion

Prompt engineering remains useful. But as soon as model output carries real product logic, I mainly need three things: contracts, evals and telemetry. That’s exactly where AI Systems Engineering begins for me. And in discovering new interaction patterns.

Repo: https://github.com/rubenvitt/plan-from-issue Finished web app: https://plan-from-issue.pages.dev/

Plan from Issue — Schema-first AI structured output demo — turn text or GitHub issues into implementation plans. (plan-from-issue.pages.dev)