Source data

Playback, payments and catalog data arrive from other systems. Each time a source is captured, a snapshot records the source system, when it was captured, the period it covers, the number of records and a hash of the content. Its freshness is marked Fresh, Stale or Failed. Metric observations point to the snapshot they came from, so a number can always be traced to the capture behind it.

Source mappings describe how a field in a source system becomes a field here: the source and target field, their types, the transform rule, the version and the date it takes effect. Mappings are reviewed before use and versioned when a source changes its format.

• Snapshot with coverage. Every snapshot states the start and end of the period it covers and how many records it holds. Gaps in coverage are visible.
• Freshness before use. Only fresh snapshots should feed results. A stale or failed snapshot is a reason to hold or invalidate a result.
• Versioned mappings. A changed source format gets a new mapping version with its effective date. Earlier results keep the mapping they used.

Agree the source systems and how often each is captured. Agree what makes a snapshot stale, for example more than a day old for playback. Agree how record counts are checked against the source system’s own totals. Decide who reviews new mappings and how transform rules are written so they can be read by an analyst.

The playback snapshot for March 3 is captured at 06:00 the next morning and covers March 3 in full, with 1.2 million records. The Data Steward compares the record count with the player’s own daily total and marks it Fresh. On March 5 the capture fails partway; the snapshot is marked Failed and no results are calculated from it.

In April the player renames its watched-time field. The steward adds a new mapping version effective April 1 that maps the new field to watched minutes. March results keep the earlier mapping, and April results use the new one.

• Source Snapshot Freshness — Snapshots by source system with capture date, coverage start and end, record count and freshness.
• Invalid Metric Results — Results invalidated by a stale or failed source, for the steward to resolve.

• Metric definition to results: Use fresh snapshots and approved definitions to calculate results.

Plan your integrations before the first capture.