Technical White Paper
PII Masking Architecture
Detection, redaction, and tokenization design for privacy-preserving conversational AI in production.
Executive summary
The masking pipeline uses layered detection (rule-based + model-assisted) before retrieval and generation. This minimizes PII exposure while keeping enough semantic context for accurate responses and clean human handoff behavior.
Entity detection method
- Deterministic pattern detectors for high-confidence identifiers (email, phone, account IDs, card-like patterns).
- Named-entity model pass for person, location, organization, and free-form sensitive fields.
- Domain policy rules for tenant-specific entities (membership IDs, policy numbers, custom schema fields).
- Confidence-weighted merge stage to reduce false positives before applying masking actions.
Supported entity coverage
| Entity class | Examples | Language coverage |
|---|---|---|
| Direct identifiers | Email, phone, account ID, passport-like patterns | Pattern-based, locale-aware variants |
| Personal attributes | Name, address, date of birth, employer hints | English, Spanish, French, German, Portuguese |
| Payment-sensitive fields | Card-like strings, billing identifiers | Pattern + checksum validation where applicable |
| Tenant custom entities | Policy ID, loyalty member ID, claim references | Configured per workspace policy profile |
Redaction vs tokenization
Redaction mode
Replaces detected entities with neutral placeholders before model calls.
- Maximizes privacy for third-party inference paths
- Best for support and FAQ intents
- Lowest risk of accidental sensitive output
Tokenization mode
Substitutes entities with reversible secure tokens for downstream systems that require continuity.
- Maintains workflow traceability across systems
- Supports controlled de-tokenization in trusted zones
- Requires strict key access governance
Validation metrics
| Metric | Current benchmark | Evaluation notes |
|---|---|---|
| PII precision | 0.97 | Weighted across deterministic and model-detected entities |
| PII recall | 0.94 | Measured on multilingual synthetic + anonymized production samples |
| False positive rate | 1.9% | Monitored with monthly audit sampling and exception review |
| Added median latency | +120ms | Includes detect + transform stages before inference |
Tenant configuration options
- Per-channel masking policy (web widget, email, messaging adapters).
- Entity class toggles by workspace and environment.
- Strict mode for always-redact, balanced mode for confidence-gated handling.
- Audit export mode with masked text + detector confidence metadata.