Snapshots and runs

Every number in the app starts with a capture from a source system. A source snapshot records that capture: its name, the source system, when it was captured, the coverage start and end, the record count, a source hash and a freshness status of Fresh, Stale or Failed. Metric observations point at the snapshot they came from, so a result can be traced back to the exact extract.

Workflow work is recorded as operation runs. Each run has a run number, an operation key, the source revision it acted on, start and completion dates, an input hash, a result reference and a status of Queued, Running, Succeeded, Failed or ReconciliationRequired. Its numbered steps record the action, the input and output references, and any receipt from an external system with the receipt ID, source, date and evidence file.

• Deduplicated by operation key. Starting the same operation twice for the same revision returns the existing run.
• Read-back before success. A publication run reaches Succeeded only after it reads back the fields it wrote.
• Unknown steps are checked. A step left Unknown is confirmed against the external source before the run is retried.

Source mappings sit alongside the snapshots. Each mapping records the source and target fields, their types, the transform rule, a version and the date it takes effect, linked to the outcome definition it serves.

List the source systems you will capture from, such as an EHR extract or a clinical data warehouse, and agree how often each is captured. Agree the coverage window each capture must span and the age after which a snapshot counts as stale. Decide how the source hash is computed so a repeated extract can be recognized. Agree who reviews mapping changes and how a new mapping version is tested before it takes effect.

Capture a monthly extract and record its coverage start, coverage end and record count. Build a cohort from it and confirm that the observations point at this snapshot. Now mark the snapshot Stale and try to build another cohort from it. The Outcomes Analyst should be stopped until the Data Engineer records a fresh capture.

Next, interrupt a publication run after the write step and before the read-back. The run should end as ReconciliationRequired or Failed with a failure reason, and the “Operation runs needing attention” view should list it. The Data Engineer should be able to see which step was last confirmed and retry without writing a second result.

• Source snapshot coverage — each capture with its source system, capture date, coverage start and end, record count and freshness.
• Operation runs needing attention — failed and reconciliation-required runs with their operation key, revision and failure reason.
• Run step log — steps still pending, running, failed or unknown, with their external source and receipt ID.

• Run reconciliation and correction: Resolve failed runs and correct a published revision without losing the original.
• Cohort construction to validation: Build a cohort revision, load its facts and accept or reject each one.

Review the implementation plan and the data model before preparing your import files.