Parties, Duplicates & Conflicts

Duplicate Detection and Conflicts

The same person arrives as "Rob Smith", "Robert Smith" and "R Smith, 14 High St". A system treating them as three people holds three partial histories and no complete one.

Where to find it

Architect Panel → Data:

• Match Policies — the console — define a policy and test it against real data
• Match Policy Fields — the fields compared, their comparators and weights
• Merges — what has been merged, and the snapshots to unmerge from

Duplicate detection and conflict checking are the same engine

"Is this person already on the system?" and "does this person conflict with an existing party?" are the same question with different consequences. Both take a set of field values, score them against existing records, and act on the score.

So there is one scoring engine and a blocking threshold. A separate conflict module would have meant two implementations that eventually disagreed about whether "A. Smith" matches "Alexandra Smith" — and the one used for conflicts is the one that must not be wrong.

Crucially, the check can score values that are not yet a record. That is what a conflict check needs: the answer has to arrive before the matter is opened, not after it has been created.

How the score works

Each field in a policy contributes a value between 0 and 1 from its comparator, weighted by how much that field matters. The score is the weighted mean across fields where both records have a value, expressed as a percentage.

Empty fields are skipped, not counted as disagreement

If either side is missing a value, the field is left out of the calculation entirely.

This is deliberate and important. A missing date of birth is an absence of evidence, not evidence of difference. Scoring it as a mismatch would make every sparse record unmatchable — and sparse records are exactly the duplicates people create when they are in a hurry.

Required fields veto

A field marked required can veto a match outright: if both sides have a value and the comparator says they differ, the pair scores zero regardless of everything else.

This is what stops a policy matching two different people who share a surname and a town. Make date of birth required, and a disagreement ends the comparison. Without a veto, high-weight agreement on three other fields drowns out the one field that actually settles it.

Setting up a policy

1. Open Data → Match Policies and create a policy for the datastore.
2. Add the fields to compare, choosing a comparator suited to each — a name comparator tolerates spelling and word order; a date comparator does not.
3. Weight them by how much each contributes to identity. Surname matters more than postcode; postcode matters more than title.
4. Mark the genuinely decisive fields required so they can veto.
5. Set the review threshold, and — for conflict checking — the blocking threshold.
6. Test against real data before enabling. This is the step people skip and regret.

Tuning the thresholds

Set the review threshold so that reviewing is realistic. Too loose and the queue fills with pairs that are obviously different, and people stop reading it. Too tight and you only catch duplicates you would have spotted anyway.

Start conservative, review what it produces for a fortnight, and loosen once you trust it. The blocking threshold for conflicts should be higher — blocking a matter is disruptive, so it should happen only on a strong match.

Worked example — a legal practice conflict check

Before a matter opens, the prospective client and the opposing party are scored against every existing party. Date of birth and company number are required fields, so a difference in either ends the comparison. A score above the blocking threshold stops the matter being opened and routes it to the COLP. A score above review, below blocking, opens the matter with a flag for the fee earner to clear.

Worked example — a council resident record

A nightly sweep scores newly created residents against existing ones. Name, date of birth and address contribute; date of birth is required. Candidates go to a review queue worked by the data team, who merge the genuine duplicates and dismiss the rest — a dismissal being recorded so the same pair is not re-presented every night.

The sweep

The Duplicate Detection Sweep task looks for new candidates in the background. It ships disabled; daily is usually right, because duplicates accumulate steadily rather than suddenly.

Recommendations

• Always mark at least one field required. A policy with no veto will eventually match two different people.
• Never auto-merge on score. Scoring is evidence; merging is a decision.
• Record dismissals, or your reviewers will see the same non-duplicate every day until they stop looking.
• Run conflict checks before opening, not after. A conflict found three weeks in is a professional problem, not an administrative one.

Merging and Unmerging

A merge combines duplicates into one record. The property that makes it safe is that it is genuinely reversible.

Where to find it

Architect Panel → Data:

• Merges — every merge, its snapshot, and the unmerge action
• Match Policies — where merge candidates come from

Architect Panel → Security:

• Dual Authorisation — optionally require two people to merge

What a merge does

One record survives and the others are folded into it. Their cases, correspondence, documents and history reattach to the survivor. Nothing is discarded — it moves, and the fact that it moved is recorded.

Unmerge is the constraint that shapes the merge

Before anything is written, a complete snapshot of both rows is taken — not a diff of what the merge intends to change, but the whole of each record as it stood.

That distinction is what makes unmerge trustworthy. A field the merge never touched can still be edited afterwards, and an unmerge built from a diff would silently revert those later edits or fail to restore them. A full snapshot means the engine knows exactly what each record was, independently of what happened next.

Choosing the survivor

Usually the record with the most complete history, not the most recent one. Where one record carries a reference that has been quoted to the outside world — a client number, a case reference on a letter, an account number — that is a strong reason to keep it, because the outside world will carry on using it.

Review before merging

Merging is not the place to be efficient. Look at both records properly, particularly dates of birth, addresses and any identifier.

Two people genuinely can share a name and a street. A merge that joins them creates a single record containing two people's data — which is a personal data breach, not an untidiness problem, and it is reportable. This is the single strongest argument for requiring dual authorisation on merges in a system holding sensitive casework.

Unmerging

Unmerge separates the records again and returns what belonged to each. It is not magic: anything created after the merge has to be attributed by a person, because the system genuinely cannot know which of the two originals a new case belongs to. But it means a mistaken merge is recoverable rather than permanent, which changes the risk of merging from severe to manageable.

After a merge

Check anything that referenced the merged record by identifier, particularly outbound integrations and any system you export to. A merge that is clean internally can still leave an external system pointing at a record that is now a redirect, and that failure surfaces later and elsewhere.

Worked example

Two resident records for the same person, one created by a housing application and one by a council tax registration. The housing record has the longer history and the reference quoted on correspondence, so it survives. Before merging, the reviewer confirms the dates of birth match and the addresses are consistent. After the merge, the council tax integration is checked because it holds the other record's identifier.

Recommendations

• Require two people to merge party records in any system holding sensitive data.
• Merge deliberately and in small batches. A bulk merge run is how two people become one at scale.
• Record why, briefly. "Same person, DOB and NI match" is enough and answers the question later.
• Treat a wrong merge as an incident, not a correction — unmerge it, then work out how the review missed it.