Vergence: Automated Data Quality and Reconciliation Pipeline

Overview

Designed, built and published Vergence, a controls-first pipeline that prepares financial ledger data for analysis by measuring where two independent sources disagree. A General Ledger and a Trial Balance arrive from providers whose formats and column names differ; the pipeline resolves them to a canonical schema, validates, deduplicates, reconciles the two against each other and emits an exception register. The name is the design brief. Vergence is how far two lines are from meeting, so a difference between sources is a quantity to be sized and evidenced rather than a fault to be quietly closed.

One hard line separates what a model may decide from what it may not. Schema resolution is the only stage permitted to call a language model, and it reaches that tier only for the columns two deterministic tiers could not settle: an exact match against a registry of approved mappings, then synonym and fuzzy matching, then a proposal. Proposals are constrained to the canonical field enumeration by their type rather than by a prompt, so a field that does not exist cannot be constructed at all, and anything below the confidence gate waits for a person. The resolver defaults to no model attached, in which case the pipeline still reaches a correct result and simply routes more columns to review.

Seven controls then run over the population exactly as delivered. Money is held as Decimal rather than floating point, so binary rounding cannot manufacture a break that looks genuine. Validation quarantines a failing row with its rule and raw values intact instead of coercing it, and nothing is deleted: removing a duplicate would silently change what three other controls report. On the published run the seven controls raised nine exceptions against five deliberately planted defects, and the same two files pushed through three different mapping paths produced a byte-identical register. Mapping decides how columns are read, not what the controls find. It is a scoped prototype against synthetic data, with two designed stages left unbuilt, and the repository states those limits before it states anything else.