Duplicates & Merging

Match Policies

A match policy defines what "the same person" means for one datastore. It is the most valuable thing you can configure in a CRM and the thing most often left until the data is already a mess.

Where to find it

Architect Panel → Data:

• Match Policies — the policies and their thresholds
• Match Policy Fields — the fields compared and their weights
• Datastores — the datastore a policy applies to

Architect Panel → Automation:

• Tasks — Duplicate Detection Sweep, daily

What a policy holds

• A key and label.
• The datastore it applies to.
• A review threshold and a block threshold.
• Criteria limiting which records are considered.
• An enabled flag.

Fields are weighted

Each field in the policy carries a comparison method, a weight, and whether it is required. Comparing two records produces a total score from the fields that agree.

Weighting is what makes this work. A matching e-mail address is strong evidence; a matching first name is almost none. Giving them equal weight produces either a flood of false matches or a policy that misses real ones.

Weight by how identifying a field is

• High — e-mail address, mobile number, customer reference, national identifier.
• Medium — surname with postcode, organisation with full name.
• Low — first name, town, job title.

The question to ask of each field is: if two records agree on this and nothing else, are they the same person? If the answer is clearly no, the weight is low.

Required fields

Marking a field required means no match is considered without it. Useful for narrowing an expensive comparison, and dangerous if the field is often blank — a required e-mail address means two records for somebody with no e-mail will never be matched.

The two thresholds

• Review threshold — at or above this score, the pair is raised as a candidate for a person to look at.
• Block threshold — a higher score, at which the match is treated as certain.

The gap between them is your uncertainty band, and it should be wide enough to catch genuine ambiguity without burying reviewers in obvious cases.

Set the block threshold high

Whatever happens automatically at the block threshold should only happen when you are genuinely certain. Two people at the same company with similar names are a real and common case, and treating them as one is a worse outcome than a duplicate.

When in doubt, lower the review threshold rather than the block threshold. More review is cheap; a wrong automatic decision is not.

Tune with real data

Run the policy against your existing records and look at what it produces before enabling anything automatic. Check both directions: pairs it raised that are not duplicates, and known duplicates it missed.

Configure it before loading data

Duplicates are far cheaper to prevent at entry than to merge later, and every integration you connect will create them faster than people do.

Worked example

A policy on People weights e-mail at 60, mobile at 50, surname plus postcode at 30 and first name at 5, with a review threshold of 60 and a block threshold of 100. Matching e-mail alone reaches review; e-mail plus mobile is treated as certain. Running it over 40,000 existing records raises 900 candidates, of which about 700 turn out to be genuine.

Recommendations

• Weight by how identifying each field is.
• Set the block threshold conservatively high.
• Tune against real data before enabling anything.
• Configure it before loading, not after.

Reviewing Candidates

Pairs scoring at or above the review threshold are raised as candidates. Working that queue is what keeps the data clean.

Where to find it

Architect Panel → Data:

• Match Policies — the policies producing candidates
• Merges — the record of what was merged

Architect Panel → Automation:

• Tasks — Duplicate Detection Sweep, daily

The sweep

The Duplicate Detection Sweep task runs daily and compares records under each enabled policy. It ships disabled — enable it under Automation → Tasks, or candidates are only ever produced when something else triggers a comparison.

Work it regularly

Duplicates get harder to resolve with age. Two records created yesterday differ by a typo; two records a year old have separate activity, separate correspondence and possibly separate transactions, and merging them means reconciling all of it.

A short weekly session on a small queue is far less work than an annual clean-up, and produces better decisions.

What to check on a pair

1. Which fields matched, and what they scored. A high score from one strong field is different from a high score accumulated from several weak ones.
2. The fields that did not match. Two records agreeing on name and address but with different dates of birth are probably two people.
3. The activity on each. Records with genuinely separate histories deserve more scrutiny.
4. When each was created, and by what route. Two records created minutes apart through a form are almost certainly one submission twice.

Family members are the classic false match

Same surname, same address, similar or shared contact details. Two people at one household are not a duplicate, and merging them is both a data error and a data protection one — you have combined two individuals' records.

Where a policy raises these repeatedly, it needs a distinguishing field weighted higher rather than a reviewer being more careful.

When you cannot tell

Leave it. An unresolved candidate is a small annoyance; an incorrect merge combines two people's data.

Where it matters, find out — contact them, or check another source. Where it does not, leaving it is a legitimate outcome.

Feed the queue back into the policy

Patterns in what you dismiss are telling you the policy is wrong. Repeatedly rejecting the same shape of pair means a weight or a threshold needs adjusting, not that reviewers should keep rejecting them.

Review the policy after the first few weeks of real use.

Decide who owns the queue

It needs to be somebody's job, with enough knowledge of the data to judge an ambiguous pair. An unowned queue simply grows, and a growing queue eventually gets cleared carelessly.

Worked example

A weekly review of about twenty candidates resolves fifteen as clear duplicates from a web form submitted twice. Three are family members at one address and are dismissed. Two are genuinely unclear and left. After a month, the recurring family-member pattern prompts adding date of birth to the policy with a high weight, and those stop appearing.

Recommendations

• Enable the sweep and work the queue weekly.
• Look at what did not match, not only what did.
• Leave genuinely ambiguous pairs alone.
• Adjust the policy when you dismiss the same pattern repeatedly.

Merging and Unmerging

Merging combines two records into one. The design decision that matters: both original records are preserved in full, which is what makes a merge reversible.

Where to find it

Architect Panel → Data:

• Merges — every merge, its snapshots and its reason
• Match Policies — the policy a merge was made under

Architect Panel → Activity:

• Activity Log — the resulting changes

What a merge records

• The survivor and the merged record.
• The policy and the score.
• A full snapshot of both records as they were.
• A reason.
• Who merged them and when.
• Whether and when it was unmerged, and by whom.

Why snapshots matter

Most systems treat merging as destructive: fields are copied across and one record disappears. If the merge was wrong, the information is gone and the only recovery is a database restore.

Keeping both snapshots means a merge is a reversible decision. That changes how it feels to make one — reviewers can act on a confident judgement without the paralysis of an irreversible action.

Choosing the survivor

The survivor keeps the identity everything else points at. Choose:

• The record with the most activity and history.
• The one other systems reference.
• The one with the better reference number, where one is customer-facing.

Field-by-field, the best value can be taken from either — the survivor is about identity, not about which record had better data.

Field resolution

Where the two disagree, each field's outcome is recorded. Take the more recent or more complete value, but look rather than accepting a default — a merge that quietly discards the only working phone number is a bad merge that looks fine.

Always write the reason

It is what makes the merge reviewable a year later. "Duplicate created by web form on 3 March, confirmed same person by e-mail address and order history" is a decision somebody can check. "Duplicate" is not.

Unmerging

A merge can be reversed, restoring both records from their snapshots and recording who reversed it and when.

Two caveats worth being honest about. First, activity recorded since the merge sits against the survivor and will not automatically redistribute. Second, unmerging is a recovery from an error rather than a routine operation — the reason to unmerge is that the merge was wrong, not that circumstances changed.

Merging two people is a data protection matter

If you combine records for two different individuals, you have disclosed one person's information into the other's record — visible to anybody who can see it, and likely to surface in a subject access request.

Unmerge promptly, and treat it as an incident rather than a correction.

Merge early

The best time is immediately after the duplicate is created, when neither record has accumulated much. A merge decision at that point is easy and low-consequence; the same decision two years later involves reconciling two histories.

Worked example

A reviewer merges two customer records created a week apart, keeping the older as survivor, taking the newer mobile number, and recording the reason. Three months later it emerges they were father and son sharing an address and a landline. The merge is unmerged from its snapshots, both records return, and the two orders placed since are reassigned by hand — recoverable precisely because nothing had been thrown away.

Recommendations

• Merge as soon as a duplicate appears.
• Choose the survivor by identity, then take the best field values from either.
• Always record a specific reason.
• Treat a wrong merge of two people as an incident.