Data incidents in AI products often do not fail loudly. The field still exists. The type has not changed. The enum still has all its values. Dashboards run, training jobs run, and pipelines stay green.

What breaks is semantics.

This article uses a tool-calling platform as the example. A success field originally meant “the third-party tool truly executed successfully.” Later, to improve frontend experience and operational reporting, it was changed to mean “the platform accepted the request and started processing.” The field name did not change. The boolean type did not change. But dashboard success rate became inflated, rerank training labels were polluted, and quality alerts were misled.

A data contract is not a copy of table schema in a wiki page. It must put field meaning, counterexamples, quality rules, downstream usage, and release process into the engineering system. Otherwise, one of the most dangerous AI-product failures happens quietly: models, dashboards, and operational decisions all become confident on the same dirty data.

The Incident Started with a Good-Looking Success-Rate Dashboard

Suppose we have a tool-calling platform. A user asks a question, the system retrieves candidate tools, reranks them, and calls one of them. The tool might query flights, company records, documents, exchange rates, or an internal knowledge base. No real company or customer name is needed here. Treat it as a generic platform.

The core log table is tool_call_events. At first it looks like this:

Field	Example	Original meaning
event_id	evt_01H…	ID of one tool-call event
trace_id	tr_9f2…	One user-request trace
tool_name	company_lookup	Called tool
provider	provider_a	Provider behind the tool
request_valid	true	Whether platform parameter validation passed
success	true	Whether the third-party tool returned business success
provider_status	OK	Raw third-party status
error_type	null	Failure attribution
latency_ms	842	Time from third-party request to response
created_at	2026-06-15T09:12:03Z	Event time

Originally, success=true had a narrow meaning: the platform actually sent a request to the third-party provider, the provider returned business success, and the result could be used by the user. Validation passing but provider timeout was not success. A cache hit with expired content was not success. Merely putting a task into a queue was not success.

Then the product changed. To reduce red failure messages, the platform split tool calling into two stages: the frontend first showed “processing started,” and the backend waited asynchronously for the provider result. At the same time, the operations dashboard wanted to know whether a request was successfully accepted by the platform. An engineer saw the old success field and reused it:

Old logic: success = provider_status == "OK" && result_usable == true
New logic: success = request_valid == true
             && dispatch_state in ("queued", "sent", "completed")

The table schema did not change. success was still a boolean. Enums did not change. Old SQL did not fail. The dashboard even looked better: success rate rose from 82% to 96%. The problem is that what rose was not the real tool success rate. It was the platform acceptance rate.

More Than One Dashboard Broke

If this field served only one operations dashboard, the incident would be easier to find. The hard part is that AI-product logs often have many invisible downstream consumers.

In this platform, success is used by at least four systems:

Downstream	Usage	Consequence after pollution
Operations dashboard	Tool-call success rate and provider health	Inflated success rate hides provider failures
Alert rules	Alert when success rate stays low	Alerts are delayed or never fire
Rerank training	Treat successful calls as positive samples	Third-party failures, empty results, and timeouts enter positives
Evaluation sampling	Sample failure calls as boundary cases	Failure samples shrink and eval set becomes too clean

Rerank pollution is especially subtle. Training does not fail just because labels are dirty. The model simply learns the wrong thing over time. Suppose a tool often times out for “company registration lookup.” It should become a negative case. Now, because the request was valid and the job was dispatched, these calls are labeled success=true. The model learns the wrong signal: this tool seems good for these queries.

Weeks later, online ranking gets worse. Users do not see “system error.” They see strange tool choices. A document-search question is routed to an external tool that often returns empty results. A provider has become unstable, yet rerank still puts it near the top. In meetings, people first suspect the model, prompts, training parameters, or retrieval strategy. Few people first suspect that the semantics of one boolean field changed.

That is why AI data incidents are hard to debug. The field did not disappear. The pipeline did not fail. All systems are running normally. They are just running on the wrong definition.

The Field Name Was Already Dangerous

success is a dangerous field name. It sounds like a conclusion, but does not say who succeeded, at which layer, and for whom.

In a tool-calling platform, there are at least five kinds of success:

Success layer	Better field name	Meaning
Parameter success	request_valid	Platform validation passed required fields and format
Dispatch success	dispatch_accepted	Task entered execution queue or was sent to provider
Provider success	provider_success	Third party returned business success
Usable result	result_usable	Content is non-empty, parseable, and passes basic quality checks
User success	user_resolved	User adopted the result, stopped asking, or explicitly found it useful

Calling all five success saves time in the short term and creates incidents in the long term. Analytics engineers understand success as call success. Training engineers understand it as positive label. Product managers may understand it as user resolution. Nobody is malicious. The same word is simply overloaded across contexts.

The first job of a data contract is to suppress this ambiguity. Field names can be short. Field semantics cannot be short.

A Useful Contract Must Include Counterexamples

If the contract says only “success: whether successful,” it is effectively empty. A useful contract includes positive examples, negative examples, and boundary cases.

I would define the event contract like this:

dataset: tool_call_events
version: 2.1.0
owner: tool-platform-data
grain: one row per attempted provider call
time_field: created_at
freshness_sla: 15 minutes

fields:
  provider_success:
    type: boolean
    nullable: false
    meaning: >
      True only when the provider returns a business-level successful response
      and the platform receives a parseable payload for this call.
    positive_examples:
      - provider_status is OK and response body contains at least one usable result
      - provider_status is OK and empty result is a valid business answer for this tool
    negative_examples:
      - request validation passed but provider request was not sent
      - provider timed out
      - provider returned HTTP 200 with business error code
      - provider returned malformed payload
      - platform queued an async job but final provider result is unknown
    allowed_transitions:
      - from: null
        to: true
        when: final provider response is observed
      - from: null
        to: false
        when: provider failure or unusable result is observed

  dispatch_accepted:
    type: boolean
    nullable: false
    meaning: True when the platform accepted the request and created an execution attempt.

  result_usable:
    type: boolean
    nullable: false
    meaning: True when returned content passes parser and minimum result quality checks.

  label_for_rerank:
    type: enum
    values: [positive, negative, exclude]
    meaning: Stable training label derived from provider_success, result_usable, user feedback, and sampling rules.

Notice that bare success is no longer used. If historical compatibility requires it, demote it to a compatibility field:

  success:
    type: boolean
    deprecated: true
    meaning: Legacy field. Do not use for new metrics, labels, alerts, or exports.
    replacement:
      metrics: provider_success
      dispatch_dashboard: dispatch_accepted
      rerank_training: label_for_rerank

Counterexamples matter. Many field incidents do not happen because nobody knew the correct definition. They happen because boundary cases were never written down. When an engineer sees “HTTP 200, but the body contains a business error code,” and the contract says nothing, they will use their own judgment. Once counterexamples are explicit, code review has a basis.

Table Design Should Support Semantic Layers

A contract is not only YAML. The table itself should leave room for semantics. In the incident above, all success concepts were squeezed into one success field. Downstream misuse became almost inevitable.

I would change the raw event table to this:

create table tool_call_events (
  event_id text primary key,
  trace_id text not null,
  user_request_id text not null,
  tool_name text not null,
  provider text not null,
  request_valid boolean not null,
  dispatch_accepted boolean not null,
  provider_success boolean,
  result_usable boolean,
  provider_status text,
  provider_error_code text,
  error_type text,
  latency_ms integer,
  created_at timestamp not null,
  finalized_at timestamp,
  contract_version text not null
);

Then add a separate training-label table:

create table rerank_training_labels (
  label_id text primary key,
  event_id text not null,
  trace_id text not null,
  query_hash text not null,
  tool_name text not null,
  label text not null check (label in ('positive', 'negative', 'exclude')),
  label_reason text not null,
  source_contract_version text not null,
  generated_at timestamp not null
);

This has two benefits.

First, raw facts and derived labels are separated. provider_success=false does not automatically mean a negative training example. If user parameters were obviously invalid, the sample may need to be exclude; otherwise the model learns “do not choose this tool” when the real issue was incomplete input. If a provider had a temporary outage, that is a provider-health negative, but may not be a long-term rerank negative.

Second, label generation is versioned. Model training can know exactly which contract and label rules were used. If a rule is later found wrong, labels can be replayed instead of guessing what a historical field meant at the time.

Quality Rules Cannot Stop at Non-Null Checks

Many data-quality checks remain at the schema level: field exists, type is correct, value is not null. They are useful, but they would not catch this incident. success stayed boolean, and its non-null rate looked good.

Semantic drift needs cross-field, cross-table, and time-window rules.

For a tool-calling platform, I would add checks such as:

quality_rules:
  - name: provider_success_requires_final_state
    severity: blocking
    expression: provider_success is null or finalized_at is not null
    description: provider_success cannot be finalized before provider result is observed

  - name: provider_success_not_equal_dispatch
    severity: warning
    expression: corr(dispatch_accepted, provider_success) < 0.98 over 7 days
    description: provider_success suspiciously tracks dispatch_accepted too closely

  - name: success_rate_matches_error_distribution
    severity: warning
    expression: >
      if provider_success_rate increases by more than 10pp day over day,
      provider_error_rate must decrease or result_usable_rate must increase
    description: success rate jump without error/result movement is suspicious

  - name: rerank_label_positive_requires_usable_result
    severity: blocking
    expression: label != 'positive' or result_usable = true
    description: positive training labels require usable results

  - name: timeout_cannot_be_provider_success
    severity: blocking
    expression: error_type != 'timeout' or provider_success = false
    description: timeout is never provider success

The second rule may look unusual, but it is practical. If provider_success and dispatch_accepted are almost identical over seven days, someone may have rewritten provider success as dispatch success. Semantic rules do not need to prove every error. They need to push suspicious cases in front of humans.

Training labels also need distribution checks:

Rule	Purpose
Positive/negative ratio per tool should not jump sharply in one day	Prevent sudden tracking or label-generation changes
label=positive must trace back to usable result or user adoption	Prevent platform success from becoming user success
Samples during provider outage windows should default to exclude or be separated	Prevent temporary failures from polluting long-term ranking preference
Sampled query distribution should stay close to online exposure	Prevent training set drift

In AI products, data quality is not only “did the data arrive?” The more important question is whether the data still represents the fact we think it represents.

Downstream Usage Must Be Registered

When an incident happens, the upstream team often says, “I did not know this field was used for training.” This sentence is common and often true. Relying on people to remember every downstream dependency is not reliable.

A contract should register field-level usage:

consumers:
  - name: ops_provider_health_dashboard
    type: dashboard
    fields:
      - provider_success
      - error_type
      - latency_ms
    owner: analytics
    criticality: high

  - name: rerank_daily_training_set
    type: training_pipeline
    fields:
      - provider_success
      - result_usable
      - error_type
      - tool_name
    owner: search-ml
    criticality: high

  - name: failure_case_eval_sampler
    type: evaluation_dataset
    fields:
      - provider_success
      - error_type
      - provider_status
    owner: eval-platform
    criticality: medium

Then a field change can automatically show who is affected. Without this, notification becomes someone saying in a chat group, “I am changing this field,” and someone will inevitably miss it.

Downstream teams also have responsibility. A training pipeline should not quietly use a field in SQL and expect upstream to never change it. If a field is used for a core metric, training label, evaluation sampler, billing basis, or alert rule, its usage and owner should be registered. A data contract is not upstream serving downstream unilaterally. It makes dependency relationships explicit on both sides.

Releases Must Distinguish Additions, Renames, and Semantic Changes

Teams often fear data contracts because they sound heavy. They do not have to be. The process can be lightweight, but risk levels must be clear.

For this platform, I would split changes into four classes:

Change type	Example	Process
Compatible addition	Add dispatch_accepted	Owner review and CI contract-format check
Field rename	Split success into provider_success	Downstream notification and dual-write period
Semantic narrowing or widening	Whether empty result can count as provider_success	Downstream owner approval and historical impact replay
Deletion or deprecation	Remove old success	Release window, migration check, rollback plan

The incident did not happen because no one reviewed code. It happened because review did not treat “semantic change” as a high-risk change. A PR may contain only this diff:

- success = provider_status == "OK"
+ success = request_valid && dispatch_state != "rejected"

It is easy to view that as a product-metric adjustment. A contract process should surface the impact automatically:

Contract impact:
- Field `success` is used by 3 critical consumers.
- Semantic source changes from provider result to dispatch state.
- This is a breaking semantic change.
- Required action: create new field `dispatch_accepted`,
  keep `provider_success`, mark `success` deprecated.

Not every field needs a meeting. But fields that affect core metrics and training labels must be reviewed seriously. In AI products, semantic changes are interface changes.

Compatibility Migration Is More Than Renaming

After discovering that success was overloaded, the direct fix is to add fields and dual-write:

Stage	Action
T0	Add dispatch_accepted, provider_success, and result_usable
T1	Keep writing old success, but mark it deprecated
T2	Migrate dashboards to provider_success or dispatch_accepted
T3	Generate rerank labels through label_for_rerank table
T4	Replay historical data and rebuild training labels in the polluted window
T5	Contract prevents new downstream consumers from using success

The easiest stage to miss is T4. If only future data is fixed while historical pollution remains, model training may still consume dirty labels. Whether replay is necessary depends on whether the polluted field entered training sets, evaluation sets, or core reports.

Replay also requires caution. Not every historical event can recover true provider_success. If the original provider_status, raw error code, and result summary were not saved, the only honest label may be unknown or exclude. Do not fill a deterministic label from current guesses just to make the dataset look complete. Fewer samples are better than false certainty.

A Minimal Contract Database

If there is no data-governance platform, you do not need to wait for one. A few tables are enough to start.

A field-contract table:

create table data_field_contracts (
  dataset_name text not null,
  field_name text not null,
  contract_version text not null,
  data_type text not null,
  nullable boolean not null,
  semantic_owner text not null,
  meaning text not null,
  positive_examples text not null,
  negative_examples text not null,
  allowed_values text,
  freshness_sla_minutes integer,
  deprecated boolean not null default false,
  replacement_field text,
  updated_at timestamp not null,
  primary key (dataset_name, field_name, contract_version)
);

A downstream dependency table:

create table data_contract_consumers (
  dataset_name text not null,
  field_name text not null,
  consumer_name text not null,
  consumer_type text not null,
  owner text not null,
  criticality text not null,
  usage_note text not null,
  created_at timestamp not null,
  primary key (dataset_name, field_name, consumer_name)
);

A quality-rule table:

create table data_quality_rules (
  rule_name text primary key,
  dataset_name text not null,
  severity text not null,
  rule_sql text not null,
  owner text not null,
  run_frequency text not null,
  enabled boolean not null default true
);

These tables are not fancy, but they pull contracts out of a wiki and into the engineering workflow. When a PR changes a field, query the dependency table. After a scheduled job runs, execute quality rules. When an alert fires, include owner and contract version. That is already much stronger than “the field description lives somewhere in a document.”

SQL Should Look Like It Uses a Contract

A contract is not written for others to admire. Downstream SQL should reflect contract awareness.

Before the pollution, a dashboard might have written:

select
  date(created_at) as dt,
  provider,
  avg(case when success then 1 else 0 end) as success_rate
from tool_call_events
group by 1, 2;

After the fix, the success layer should be explicit:

select
  date(created_at) as dt,
  provider,
  avg(case when provider_success then 1 else 0 end) as provider_success_rate,
  avg(case when dispatch_accepted then 1 else 0 end) as dispatch_accept_rate,
  avg(case when result_usable then 1 else 0 end) as usable_result_rate
from tool_call_events
where contract_version >= '2.1.0'
group by 1, 2;

Training labels should not directly use provider_success as the label:

insert into rerank_training_labels
select
  generate_label_id(event_id) as label_id,
  event_id,
  trace_id,
  query_hash,
  tool_name,
  case
    when result_usable = true and user_feedback in ('accepted', 'copied', 'no_followup') then 'positive'
    when error_type in ('tool_not_applicable', 'empty_unhelpful_result') then 'negative'
    when error_type in ('provider_timeout', 'rate_limited', 'temporary_provider_error') then 'exclude'
    else 'exclude'
  end as label,
  build_label_reason(provider_success, result_usable, error_type, user_feedback) as label_reason,
  contract_version as source_contract_version,
  current_timestamp as generated_at
from tool_call_events
where created_at >= current_date - interval '7 days';

The point is not the function names. The point is the attitude: a training label is an independent data product with its own rules and counterexamples. It can use raw fields, but it should not equate one raw “success” concept with the model’s “good sample.”

Evaluation Sets Can Be Polluted Too

Many teams watch training data and ignore evaluation sets. Evaluation sets are just as vulnerable to field-semantics drift.

Suppose the evaluation sampler says:

Each day, sample 200 tool calls from success=false and ask humans to label failure reasons.

When success is changed to dispatch success, the failure pool suddenly shrinks. Real provider failures, empty results, and timeouts no longer enter the sample. The eval set looks cleaner, and model scores may even improve. The system did not improve. The hard cases were filtered out by the sampler.

Evaluation-set contracts should define:

Item	Example
Sample source	Provider-finalized events in tool_call_events
Failure definition	provider_success=false or result_usable=false
Exclusion rule	Temporary provider outage is separated, not mixed with tool-applicability negatives
Coverage requirement	At least 30 failure candidates per high-frequency tool per day
Label definition	Tool not applicable, insufficient parameters, provider failure, unusable result, ambiguous user request
Version	eval_tool_failure_v3

Model-evaluation credibility depends on whether the eval set still represents real problems. When field semantics drift, the eval set drifts. Once the eval set drifts, model comparison is like measuring height with a moving ruler.

Do Not Overwrite Raw Facts with Derived Definitions

Another lesson from this incident: preserve raw facts as much as possible. Do not overwrite them with derived definitions.

If the table has only success, but not provider_status, provider_error_code, result_usable, or finalized_at, recovery becomes hard. You know the system wrote true at some time, but not why. For reports, manual correction may still be possible. For training data, it is much worse because you cannot know which positives were real positives.

A more robust design keeps several layers of fact:

Layer	Example fields	Can be overwritten?
Raw provider response	provider_status, provider_error_code, response-summary hash	No overwrite, append only
Platform processing state	dispatch_accepted, finalized_at	State may update, but history is preserved
Quality judgment	result_usable, parse_error_type	Recomputable, with rule version
Business metric	provider_success_rate	Derived, not written back to raw facts
Training label	label_for_rerank	Derived, versioned with reason

Privacy and compliance still matter. Full raw responses may not be stored indefinitely. You can store desensitized summaries, error codes, structured metadata, or short-lived isolated payloads. But do not erase the only source of truth. Without raw facts, a contract becomes a wish.

What Happens When a Contract Fails

What should the system do after a data-quality rule fires? If it only sends a message, the model may continue training. If it blocks everything, it may disrupt the business. The contract should define failure behavior.

For this platform, I would use:

Failure scenario	Behavior
provider_success violates a blocking rule	Block release or roll back write logic
Unusable results appear in positive training labels	Stop the day’s training job
Dashboard freshness exceeds SLA	Show freshness warning and exclude from weekly conclusion
Provider outage causes negative samples to spike	Mark samples in that window as exclude
Contract version mismatch	Downstream task fails with migration guidance

This step is critical. Many data platforms can detect problems, but detection without system behavior still leaves people watching alerts at midnight. AI-product training, evaluation, indexing, and reporting pipelines are automated. Contract failure should enter the same control plane.

Someone Must Own Semantics

Field semantics are not purely technical. Whether provider_success allows an empty result to count as success requires product, engineering, data, and model teams to decide together. In a company-query tool, “no company found” may be a valid answer. In document search, an empty list may mean the tool did not help the user.

So owner cannot simply be “data team.” I would split ownership into at least three roles:

Owner	Responsibility
Producer owner	Where the event is produced and how write logic is guaranteed
Semantic owner	Business meaning, counterexamples, and definition changes
Consumer owner	Downstream usage, migration, and validation

When a field’s semantics change, the producer owner evaluates implementation, the semantic owner judges definition, and consumer owners evaluate impact. Without this split, everyone can honestly say after an incident: “I only changed one line according to the requirement.”

Small Teams Can Start Small

Small teams do not need a full data-contract platform on day one. Start from the most painful part of this incident.

I would first do five things:

Action	Purpose
Ban new code from using bare success	Stop the bleeding
Add provider_success, dispatch_accepted, and result_usable	Split semantics
Create a separate table for training labels	Avoid direct consumption of raw success fields
Write counterexamples for core fields	Give review a decision basis
Automatically list downstream consumers in PRs	Prevent missed notifications

These five actions do not require an expensive platform. They are mostly engineering discipline. Once they work, you can add lineage, quality dashboards, a contract repository, and release gates later.

Data governance goes wrong when it starts as a giant project. Giant projects often produce abstract rules that never enter real release flow. Start with one field, one table, and one training pipeline. That is how a team learns that contracts are not ceremony.

Reviewing the Success Incident

The whole incident can be summarized as:

Over-broad field name
  -> product requirement writes "platform accepted" into original success
  -> schema unchanged, CI does not block
  -> dashboard success rate inflates
  -> alert rules fail
  -> rerank positives include failed calls
  -> evaluation failure sampling becomes too clean
  -> online ranking gets worse and debugging goes in circles

The contract fix is:

Split success layers
  -> write counterexamples and boundary cases
  -> register downstream usage
  -> add cross-field quality rules
  -> put semantic change into PR gates
  -> separate raw facts and training labels
  -> replay polluted windows or mark samples exclude

This is not about writing more documents. It is about letting the system detect a dangerous field change before it becomes both a model incident and a reporting incident.

My Current View on Data Contracts

Data contracts are not data-team perfectionism. They are not governance theater for large companies only. Once a field feeds reports, training, evaluation, recommendation, and alerts, it is already a product interface. An interface deserves semantics, versions, owners, counterexamples, and a release process.

AI products amplify data impact. If a dashboard definition is wrong, a meeting may argue about it. If a training label is wrong, the model absorbs the mistake. If an evaluation set is wrong, the team may believe the model improved. If an alert is wrong, incidents are found later. Put together, these are not “small data issues.” They are product-quality issues.

I do not like making data contracts sound mystical. Their simplest value is to force a team to ask concrete questions before changing a field:

Which layer of success does this field represent? Which cases definitely do not count? Who uses it for model training? Does the dashboard mean provider success or dispatch success? Will evaluation sampling change? Can quality rules detect semantic drift? If historical data is already polluted, should it be replayed or excluded?

Answering these questions is much cheaper than tuning model parameters for a week after the data has already drifted.

References

• OpenLineage documentation
• OpenLineage Object Model
• Atlan Data Contract Spec

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode