The Data Dilemma, Part 2

Dieser Blogpost ist auch auf Deutsch verfügbar

TL;DR

• Part 2 makes the Minimum Viable Data Mesh concrete: four layers, from an honest inventory through curated views to agent access, illustrated throughout with a purchasing agent.
• It all starts with a small but business-relevant use case; the four layers build on one another but do not have to be run through strictly in sequence.
• The core is Layer 2: curated views as composed data products with ODCS contracts, explicit usage semantics (suitable_for / not_suitable_for), and testable quality rules that keep the business side and engineering aligned via sign-off in the pull request.
• Access is bundled by an MCP server (analytical via views, operational via narrowly scoped tools), and a feedback loop lets the system grow along real usage; the real hurdle is discipline, not tooling.

This post is part of a series.

• Part 1: The Data Dilemma, Part 1
• Part 2: The Data Dilemma, Part 2 (this post)

The first part of this article described the dilemma: agents need data with documented meaning. That is exactly what is missing in most data landscapes, because the knowledge about semantics and special cases lives in the heads of the domain experts. Data Mesh would be the fitting answer, but it presupposes domain teams and an organizational maturity that many mid-sized companies do not have. As a way out, Part 1 sketched a Minimum Viable Data Mesh (MVDM): Data as a Product, business ownership, and explicit contracts are preserved; domain teams, polyglot persistence, and the platform as a product are replaced by simpler means for now. This second part shows the approach in practice, in four layers and with a running example.

Start with a concrete use case

Before anything happens with data, you need a clear use case. The entry point should be small but business-relevant: a clear task with clear boundaries and a benefit that the business unit can measure. Without this anchor, every data initiative quickly drifts into “we first have to model everything cleanly” mode. Use cases give data work direction and priority.

A concrete example that will accompany us through the rest of the article: a purchasing agent is supposed to compare providers for certain products, check technical requirements against supplier specifications, and take into account historical prices, discounts, and earlier purchasing terms. The case is business-relevant, has clear data sources such as the supplier master, order history, and product data, and can be measured: for example through time saved per purchasing transaction, better terms, or fewer follow-up questions.

A good first use case is business-relevant enough that someone in the business unit really wants it, and technically small enough that it can be implemented in a few weeks. And it touches three to five data sources, not twenty.

Four layers, from minimal to mature

From this, an approach in four layers emerges. The layers build on one another, but do not have to be run through strictly in sequence. Layer 3 often emerges in parallel with Layer 2.

Layer 1: Inventory

At the beginning comes the overview: what is actually there? A simple repository holds the most important information per data source: name, system, business owner, technical contact, purpose, most important entities, update frequency, rough quality assessment, known problems, sensitive data classes, historical peculiarities. No schema auto-discovery, no complete lineage. To start, the following statement is enough: “We have these 30 to 80 tables and endpoints, and these are the 15 most important ones.” For the purchasing agent, that concretely means: supplier master from the ERP, order history, item master with technical specifications, maybe a grown Access database with framework contracts that only purchasing knows about. Four sources, four entries in the inventory, and already the first insight: for the framework contracts there is no longer a technical contact.

Such an inventory emerges in two or three workshops with the departments. The output is a catalog that is honest about gaps. A Git repository with Markdown files is enough for that. If you want, you can use DataHub or OpenMetadata, but for the start that is not decisive.

Honesty is important. A good inventory may show gaps: unclear owner, questionable data quality, historical special cases, missing documentation. It is precisely this information that is valuable. For agents, an incomplete but honest inventory is better than a complete catalog that suggests cleanliness.

Honesty also concerns the origin of the knowledge itself. Not every definition that lands in the inventory is confirmed by the business side. Some the central team has to reconstruct from the data: “This field is 40 percent empty before 2019, presumably a migration.” The business side can often only make such hypotheses plausible, not authorize them, because the original knowledge is no longer available. Then exactly that should be in the entry: definition reconstructed from code and data, not confirmed by the business side.

The value of this layer is not only technical. It forces the organization to jointly get an overview of its own data landscape. The gaps that become visible in the process are often the actual insight gained.

Layer 2: Curated views for the top use cases

Instead of curating all data, you choose three to five concrete agent use cases and build clean views for them. Not everything is a view. Before building the views, a simple distinction is needed. Is it about historical, analytical data such as price history, order volume of the last months, or reporting questions? Or is it about operational real-time data such as current stock levels, latest measurements, or open tickets from today? Curated views are suitable for the former. For current data, it is better to access the operational system directly; more on that in Layer 3. The agent later sees both as equal tools, but under the hood they are two different paths with different requirements for refresh, consistency, and performance.

For the purchasing agent, the price history clearly falls on the analytical side, the current stock level of a supplier on the operational side.

The path into the views. For the analytical data, an ingestion strategy per source is needed. Four patterns have proven themselves in practice:

• Direct pull from the source. The curated view is a SQL view on a database that is queryable anyway, for example an operational Postgres, a replica of the Salesforce DB, or SAP BW. This is the simplest case and works when performance and load are uncritical. But beware: the information in the table is not always identical to what you see in the UI. Often the process or the UI aggregates different values.
• ELT replication to a central place. Tools like Airbyte, Fivetran, or Meltano, in a pinch also a Python script with cron, pull data from the source systems into a central DB. dbt models then build the curated views on top of it.
• Native export or API interfaces. Many standard systems, especially ERPs, come with official export or integration interfaces, for example SAP OData, Salesforce Bulk API, REST or CSV exports from the application. Where they exist, they are the clean path: documented, versioned, often incremental, and with clear permissions. A job pulls through the interface into the central DB, dbt builds the views on top. This is more robust than a self-built export and usually the first choice when the source system offers such an interface. The correct process logic is also applied to the data.
• Federation via file exports. When the legacy system has neither DB access nor a usable interface, the nightly dump remains: a job puts CSV into a bucket, DuckDB reads the files directly, dbt builds the views on top. This is unsexy, but it works for data that changes once a day. The Access database with the framework contracts from our example lands exactly here.

Heterogeneity on the ingestion side is acceptable. One source, one pattern. What matters is that the curated view layer looks uniform on top of it. That is where the decoupling lies: the view is the contract with the agent, the path to it is an implementation detail and may change later.

Views as composition. It would be a misunderstanding to see curated views as documented copies of existing tables. The actual value lies deeper. Many values in operational systems are not interpretable on their own, but only in interplay with others. A supplier rating of 5.0 stars says little if it is based on a single review. An average price is misleading if it mixes tiny quantities and framework-contract terms. In such cases, the interpretation rule lies neither in a single field nor in a single table. It lies in a join relationship plus a business rule: “Ratings with fewer than ten votes are not reliable.”

People apply such rules implicitly when they know them. An agent does not know them, and field-by-field documentation has no natural place for them. The curated view is this place. It delivers bewertung together with anzahl_bewertungen and a derived field like bewertung_belastbar, whose threshold the domain declares and engineering encodes. From this follows a simple design criterion: A curated view should be business-interpretable without further joins. If a consumer needs additional tables to classify a value correctly, the view is not fully composed.

What matters about the views themselves. Every view needs three things: columns in business language (lieferanten_id instead of lfrnr, bestellt_am instead of dt_best), a description per column with meaning, unit, value range, example values, and known problems, and a view description with purpose, grain (that is, the granularity: what one row represents in business terms), refresh frequency, and owner of the pipeline.

At its core, this is one data contract per view. There are existing standards for this: the Open Data Contract Standard (ODCS) is an established, open format under the umbrella of the Linux Foundation that provides exactly these fields, from the schema description through classification to quality and SLA. dbt builds the view, the ODCS contract describes and checks it. The actual investment lies in this metadata, not in the SQL logic behind it.

Usage semantics: what it is suitable for, what it is not. Classic data documentation describes what a table contains. For agents, that is not enough. They additionally need usage semantics: for which questions is this view suitable, for which not? Many misinterpretations arise at exactly this point: an agent finds a technically fitting view and uses it for a business-inappropriate question.

A concrete example for our purchasing agent as an ODCS contract, abbreviated:

apiVersion: v3.1.0
kind: DataContract
id: 5a9b2c14-1e7d-4f0a-9d3e-7b2c1f8e4a10
name: lieferanten_preishistorie
version: 1.0.0
status: active
domain: einkauf
dataProduct: lieferantenvergleich

description:
  purpose: Historical net purchase prices per supplier and article from January 2021.
  usage: Price comparisons between suppliers and analysis of price development.
  limitations: Not for current quoted prices, ongoing negotiations, or availability.

tags: ['einkauf']

schema:
  - name: lieferanten_preishistorie
    logicalType: object
    physicalType: view
    physicalName: v_lieferanten_preishistorie
    dataGranularityDescription: One row per supplier, article, and order month
    properties:
      - name: lieferanten_id
        businessName: Supplier number
        logicalType: string
        physicalType: varchar(10)
        primaryKey: true
        primaryKeyPosition: 1
        classification: internal
        description: Unique supplier ID from the ERP
        examples: ["L-30017", "L-40233"]
      - name: artikel_id
        businessName: Article number
        logicalType: string
        physicalType: varchar(12)
        primaryKey: true
        primaryKeyPosition: 2
        examples: ["A-778102"]
      - name: monat
        businessName: Order month
        logicalType: date
        physicalType: date
        primaryKey: true
        primaryKeyPosition: 3
        examples: ["2024-01", "2024-02"]
      - name: netto_preis_eur
        businessName: Net purchase price
        logicalType: number
        physicalType: decimal(12,2)
        classification: confidential
        description: Average net purchase price per unit in the month, excluding framework-contract discounts
      - name: bestell_anzahl
        businessName: Number of orders
        logicalType: integer
        physicalType: int
        description: >
          Number of order transactions the average price is based on.
          Prices from fewer than 3 orders are considered not reliable.
customProperties:
  - property: suitable_for
    value:
      - "Price development of an article at a supplier over the last 12 months"
      - "Comparison of historical purchase prices between suppliers"
  - property: not_suitable_for
    value:
      - "Current quoted prices or daily prices"
      - "Overall terms including framework-contract discounts (see rahmenvertrag_konditionen)"
      - "Deliverability or current availability"

Quality rules, freshness, and retention have their own sections in ODCS (quality, sla) and are left out here for the sake of overview. suitable_for and not_suitable_for are not known to ODCS as standard fields, which is why they sit under customProperties. For agents they are important nonetheless: they make the business intent explicit and give the agent a signal for when it should look for a different source. The reference to rahmenvertrag_konditionen at the same time shows the composition principle from above: the pure order price without framework-contract context would be exactly the kind of value that misleads on its own.

Note also the bestell_anzahl field: it is in the view even though the agent primarily asks about prices. Without the number of underlying orders, the average price cannot be classified; the view carries its basis for interpretation with it.

Two honest limitations regarding not_suitable_for. First, a negative list is never complete; agents find misuses that no one anticipated. It therefore grows best out of observed missteps, not out of upfront brainstorming; more on that in Layer 4. Second, with a growing catalog a retrieval question arises: with five views, an agent reads all contracts into context, with fifty the selection of the right view itself becomes the problem. For getting started this is not an obstacle, but you should know that this limit exists.

Timeliness as part of the data product. People often implicitly know that a report is from last night or that certain figures are only reliable after month-end closing. Agents only know that if it is described explicitly. Every curated view should therefore contain information about freshness: last update time, normal latency, business cut-off date, from when data counts as outdated. A simple last_refresh_at column or a corresponding metadata entry prevents many misunderstandings. The agent can make visible whether it is talking about data from this morning, last night, or last week.

On top of that come quality rules that sit directly in the contract: uniqueness of keys, mandatory fields, accepted values for status fields, referential integrity, simple plausibility checks. The Data Contract CLI initiated by INNOQ reads the ODCS contract, connects to the source, and runs exactly these checks, in CI or scheduled. If a view breaks, that is noticed before an agent answers on that basis.

Access rights: metadata describes, the database enforces. Access rights are part of the description of a data product. A view should make visible what kind of data it contains and for what use it is intended: usable company-wide, department-internal, confidential, personal, especially worthy of protection. For people it is often clear from context that a source contains terms or confidential supplier data. For agents the classification has to be explicit, otherwise they treat sensitive data like public data.

You should not fool yourself here: a classification field is a hint, not enforcement. An agent respects it only as long as nothing goes wrong. The actual access control belongs where it cannot be circumvented: in the database and the access path. Concretely, that means the later MCP server should not access all views with an all-powerful service account. Otherwise a classic confused-deputy problem arises: the purchasing agent of a clerk could suddenly aggregate terms that this clerk should never see. Instead: access in the name of the respective user, grants per view, row-level security where necessary. The protection level is in the metadata. It is enforced in the database.

The seam between business and technology. Data Mesh assumes that the domain owns the whole data product: meaning and implementation are in one hand. In the MVDM we have centralized engineering for the reasons described in Part 1: a mid-sized company does not have ten domain teams. This creates an interface: the domain knows the meaning, the central team builds the data product.

This separation is feasible if the responsibilities are explicit. If you go through an ODCS contract field by field, almost every field falls clearly on one side. ODCS is suitable for carrying business representation and physical implementation in one artifact.

The domain declares what should apply, engineering encodes and enforces it. What remains shared is above all the agreement on the grain and fields like limitations, which can be business, legal, and technical at the same time.

It stays one document. The ODCS contract carries business and technical fields together. The separation runs via ownership per field. Two files are not needed for that. The domain is responsible for the business fields, engineering for the technical ones, both in the same contract, separated by the sign-off in the pull request.

This also lets the role question be solved pragmatically. No data product owner per domain is needed, a role that in the mid-market often no one fills. The domain does not have to write the whole contract, it has to confirm the business statements: the central team drafts a field from the data, a named person from the business unit confirms it in the pull request. Whoever is responsible for the business statement does not have to have written it themselves. Because MVDM is use-case-driven, this responsibility lies with the units that want to use the respective agent. And where even the business side can only reconstruct, the principle from Layer 1 applies: the uncertainty becomes part of the contract, not documented away.

A dedicated glossary only pays off once the same definition appears across multiple views. As long as “purchase price” appears only in one view, the definition lives inline. As soon as several contracts need the same term, you pull the canonical definition into a separate artifact and link it via authoritativeDefinitions. The glossary thus grows out of the need. Making drift testable. The price is discipline at this interface. If the business definition says “reliable means at least three orders” and the SQL filters on >= 2, definition and implementation drift apart silently. Then the agent again answers convincingly but wrong.

Appeals help little against this. It takes checks that test the business statement directly. The most effective means are example-based tests that the business side formulates and signs off: “The price of supplier L-30017 for article A-778102 in March 2024 is based on two orders and must be marked as not reliable.” Such cases sit as quality rules in the contract, the Data Contract CLI runs them in CI, and a failure blocks the deployment of the view. The sign-off in the pull request thereby confirms concrete, verifiable statements about the data. Business definition and technical encoding can still diverge, but no longer unnoticed.

Layer 3: Unified access via MCP

An MCP server, self-built or existing for Postgres, Snowflake, and similar systems, bundles access for the agent. It provides two kinds of tools.

Analytical MCP tools on curated views. For historical and aggregated questions. The agent can search the catalog (list_datasets, describe_dataset), run read-only SQL on the curated views, and optionally fetch example rows. The views' metadata, including suitable_for and not_suitable_for, has to be accessible to the agent, not just in a documentation page for humans. Only then can it decide which view is suitable for a task. Our purchasing agent thus finds lieferanten_preishistorie for the price comparison and is forwarded by its not_suitable_for to rahmenvertrag_konditionen when it comes to overall terms.

Operational MCP tools (data) on live systems. For current data, you rely on specific, narrowly scoped tools instead of free SQL. get_current_inventory(article_id) accesses the inventory management system directly. get_supplier_lead_time(supplier_id, article_id) queries the lead time in the ERP. get_open_orders(supplier_id) returns open orders. These tools are more narrowly scoped than the analytical ones: point lookups instead of aggregations, clear parameters, no open queries. This protects the operational systems from load and keeps the agent interaction predictable. It is at the same time a security argument: a parameterized tool with a clear scope offers no formulation, however creative, the chance to reach through to productive systems via a free query.

In both cases: no direct, unthrottled access to raw systems. Performance, security, and data quality remain controllable in one place. Access happens, as described in Layer 2, with the identity of the respective user, not with a central collective account. The agent sees a uniform tool inventory, regardless of whether a view or a live API sits behind a tool. Therein lies the value of MCP as an abstraction.

Layer 4: Feedback loop

You log which queries agents ask, where they fail, and which datasets are never used. From this, hints about the next curated views emerge. This way the system can grow without having to model everything in advance.

Agents deliver useful signals for this. If the purchasing agent repeatedly pulls price data from the wrong view, a better not_suitable_for description is probably missing; the negative list thus grows along real missteps instead of upfront speculation. If users regularly ask for data that does not yet exist, for example supplier certifications, that is a hint about the next data product. If a view is never used, it is perhaps irrelevant, poorly described, or redundant.

Feedback also includes the question of whether the agent answers correctly, not only whether it finds data. Quality rules test the data; they do not test the interplay of data, metadata, and agent. For that, a small set of reference questions with known answers is needed: “What did article A-778102 cost on average at supplier L-30017 in the first quarter of 2024?” with the expected result next to it. These questions run against every change to views or contracts. Ten such questions per use case are enough to start and catch the cases where technically everything is green but the answer is still wrong.

A word on the logging itself: agent queries and their results may contain exactly the confidential or personal data whose classification you carefully maintained in Layer 2. The log is thereby itself a data set worthy of protection, for which the classification from Layer 2 applies equally: it needs a legal basis, a retention period, and restricted access. The system thus grows along real usage. One use case provides the entry point, usage shows the next need, the data products get better step by step.

Pragmatic tooling

A concise tool selection for getting started:

• Contracts and catalog: a Git repo of ODCS contracts. The contract is at the same time the catalog entry; a separate catalog tool like DataHub or OpenMetadata is optional and comes later, when it is needed.
• Views: dbt Core, because the transformations live versioned in Git and can be kept cleanly against the contracts.
• Storage: whatever is there anyway. Postgres, SQL Server, BigQuery, Snowflake. In a pinch, DuckDB as a federation layer over CSV exports.
• Agent access: MCP server, often ready-made for Postgres, dbt, Snowflake, and others.
• Contracts and quality checks: the Data Contract CLI from datacontract.com, works natively with ODCS, validates contracts and runs schema and quality tests against the curated views. With it, format, catalog, and tests fall into a single, open tool.

More important than the tool selection is the discipline with which you use the tools and maintain the data contracts.

The bitter data lesson

The decisive hurdle is not primarily technical. It lies in discipline on two levels. First, you have to stay with the use case and avoid falling into “let’s first model all the data cleanly” mode. Second, the interface between business and technology has to be maintained so that business definition and technical encoding do not drift apart. Both are the price for the pragmatic deviation from the Data Mesh ideal.

Simply unleashing agents on existing systems creates access without understanding. The problem is known from BI projects. With agents it becomes riskier, because missing understanding disappears into convincingly formulated answers. An agent that knows the not_suitable_for hints of a view is more valuable than one that reaches ten systems but has to guess their semantics. The title of this section is a deliberate reversal. Richard Sutton’s Bitter Lesson states that scaling compute ultimately beats every knowledge structure painstakingly encoded by humans. For enterprise data, the opposite applies: no model, however large, can guess that “active” in this company means “order in the last twelve months” or that a rating with fewer than ten votes is worthless. This semantics does not sit in the data, it sits in the organization. And it only comes into being when someone makes it explicit. The bitter lesson of data is that here there is no shortcut via the next model.

The result is not a textbook Data Mesh. It is a Minimum Viable Data Mesh: a few curated views with clear contracts, explicit usage semantics, a unified access point for agents, and a feedback mechanism for organic growth. The basic idea is preserved: explicit contracts and responsibility close to the data. The organizational complexity decreases without giving up the claim to clean semantics.

A few productively usable views are often more valuable than a comprehensive AI strategy without a reliable data basis. Agent-ready data comes into being through concrete data products with clear business value. The dilemma with the data is not solved by a better model. It only resolves when organizations make explicit what their data means.