Duplicate Detection and Conflicts

The same person arrives as "Rob Smith", "Robert Smith" and "R Smith, 14 High St". A system treating them as three people holds three partial histories and no complete one.

Where to find it

Architect Panel → Data:

Match Policies — the console — define a policy and test it against real data
Match Policy Fields — the fields compared, their comparators and weights
Merges — what has been merged, and the snapshots to unmerge from

Duplicate detection and conflict checking are the same engine

"Is this person already on the system?" and "does this person conflict with an existing party?" are the same question with different consequences. Both take a set of field values, score them against existing records, and act on the score.

So there is one scoring engine and a blocking threshold. A separate conflict module would have meant two implementations that eventually disagreed about whether "A. Smith" matches "Alexandra Smith" — and the one used for conflicts is the one that must not be wrong.

Crucially, the check can score values that are not yet a record. That is what a conflict check needs: the answer has to arrive before the matter is opened, not after it has been created.

How the score works

Each field in a policy contributes a value between 0 and 1 from its comparator, weighted by how much that field matters. The score is the weighted mean across fields where both records have a value, expressed as a percentage.

Empty fields are skipped, not counted as disagreement

If either side is missing a value, the field is left out of the calculation entirely.

This is deliberate and important. A missing date of birth is an absence of evidence, not evidence of difference. Scoring it as a mismatch would make every sparse record unmatchable — and sparse records are exactly the duplicates people create when they are in a hurry.

Required fields veto

A field marked required can veto a match outright: if both sides have a value and the comparator says they differ, the pair scores zero regardless of everything else.

This is what stops a policy matching two different people who share a surname and a town. Make date of birth required, and a disagreement ends the comparison. Without a veto, high-weight agreement on three other fields drowns out the one field that actually settles it.

Setting up a policy

Open Data → Match Policies and create a policy for the datastore.
Add the fields to compare, choosing a comparator suited to each — a name comparator tolerates spelling and word order; a date comparator does not.
Weight them by how much each contributes to identity. Surname matters more than postcode; postcode matters more than title.
Mark the genuinely decisive fields required so they can veto.
Set the review threshold, and — for conflict checking — the blocking threshold.
Test against real data before enabling. This is the step people skip and regret.

Tuning the thresholds

Set the review threshold so that reviewing is realistic. Too loose and the queue fills with pairs that are obviously different, and people stop reading it. Too tight and you only catch duplicates you would have spotted anyway.

Start conservative, review what it produces for a fortnight, and loosen once you trust it. The blocking threshold for conflicts should be higher — blocking a matter is disruptive, so it should happen only on a strong match.

Worked example — a legal practice conflict check

Before a matter opens, the prospective client and the opposing party are scored against every existing party. Date of birth and company number are required fields, so a difference in either ends the comparison. A score above the blocking threshold stops the matter being opened and routes it to the COLP. A score above review, below blocking, opens the matter with a flag for the fee earner to clear.

Worked example — a council resident record

A nightly sweep scores newly created residents against existing ones. Name, date of birth and address contribute; date of birth is required. Candidates go to a review queue worked by the data team, who merge the genuine duplicates and dismiss the rest — a dismissal being recorded so the same pair is not re-presented every night.

The sweep

The Duplicate Detection Sweep task looks for new candidates in the background. It ships disabled; daily is usually right, because duplicates accumulate steadily rather than suddenly.

Recommendations

Always mark at least one field required. A policy with no veto will eventually match two different people.
Never auto-merge on score. Scoring is evidence; merging is a decision.
Record dismissals, or your reviewers will see the same non-duplicate every day until they stop looking.
Run conflict checks before opening, not after. A conflict found three weeks in is a professional problem, not an administrative one.