Persian contact enrichment and name resolution

1. Introduction and scope

Large Persian-language datasets contain multiple surface forms of the same real-world entity. Person names, organisation titles and place names appear with inconsistent character encodings, optional honorifics, variable compound spacing, phonetic letter substitutions and inverted token order. When these variations exist across independent source systems, exact string matching becomes insufficient.

This blueprint is deliberately layered: inexpensive deterministic transformations execute first, followed by blocking keys that sharply reduce the candidate space, then lightweight string metrics, and only on residual hard pairs more expensive embedding models. The same stages can run in Python (Spark, DuckDB, Pandas), SQL engines, and NoSQL platforms such as Elasticsearch.

2. Structural challenges of Persian text at scale

Persian orthography creates several interacting sources of variation that must be addressed before reliable matching is possible.

3. Pipeline architecture

Five sequential stages. Each is independently executable so intermediate results can be materialised, audited and re-processed.

1. Ingestion and basic validation — schema checks, null handling, source tagging.
2. Deterministic normalisation — character mapping, diacritic removal, ZWNJ standardisation, honorific isolation.
3. Blocking-key generation — phonetic skeleton and sorted-token fingerprint.
4. Candidate generation — records sharing both keys form small comparison groups.
5. Similarity scoring and decision — rapid token-based metrics first; residual pairs may receive embedding-based scores.

Figure 1. End-to-end Persian name cleaning and entity-resolution pipeline. Stages progress from raw ingestion through deterministic normalisation, blocking-key generation, candidate-pair formation and final similarity scoring.

4. Deterministic normalisation layers

The first transformation layer must be fully deterministic and reversible for audit purposes.

5. Blocking keys for candidate generation

After normalisation, exact string equality remains too strict. Blocking constructs inexpensive keys that place likely matches into the same partition so expensive pairwise comparison occurs only inside small groups.

Figure 2. Multiple observed surface forms of a single identity converge, after normalisation, onto a shared phonetic key and sorted-token fingerprint.

6. Scoring residual pairs

Inside each blocking bucket the remaining pairs are scored with lightweight string metrics. Token-sort ratio and token-set ratio provide a strong first signal on cleaned Persian text. A weighted edit distance that assigns lower cost to substitutions inside the same phonetic group further improves ranking. Only the highest-ambiguity residual pairs need embedding models — this keeps cost acceptable at scale.

7. ParsBERT and residual neural scoring

ParsBERT remains the strongest monolingual foundation for Persian tasks that require deep understanding of orthographic variation and compound structure. In production the cross-encoder is reserved for the final ranking stage after aggressive blocking. Bi-encoders scale better when embeddings can be pre-computed. Parameter-efficient fine-tuning (LoRA) recovers nearly the same F1 as full fine-tuning while staying within consumer-GPU budgets.

8. Location and city name handling

Place names show a more restricted vocabulary than personal names yet still display Arabic/Persian letter variation, optional administrative prefixes and Latin transliterations. Maintain a canonical dictionary of principal cities and provinces with their frequent variants; apply the same character and spacing normalisation; look up the normalised string; unmatched values fall back to phonetic blocking and fuzzy scoring.

Figure 3. Example of city-name normalisation. Surface variants in Persian, Arabic-influenced and Latin scripts collapse to a single canonical identifier after cleaning.

9. Operational considerations and quality gates

• Materialised intermediate tables — each major stage writes Parquet so any step can be re-run independently.
• Quality metrics — track the percentage of records that receive a blocking key, average candidate-bucket size, and similarity-score distribution.
• Threshold calibration — decision thresholds are domain-specific (person vs organisation vs location) and should be tuned on a stratified sample.
• Auditability — original raw values, intermediate normalised forms and final match decisions are retained.
• Dual-script indexing — when Latin transliterations coexist with Persian script, maintain parallel normalised keys that both resolve to the same master entity identifier.

The objective of an enterprise pipeline is to maximise the proportion of records that can be linked automatically with high confidence, while producing a compact, ranked residual set that can be handled by downstream processes or controlled review.