개인 데이터 찾기
읽는 데 약 77분분 걸린다, 13절에 11개 예제이(가) 있음.
AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식
영어판 2026.11.31판을 번역한 것이다. 원문 읽기
이 페이지의 내용
- 스키마로 열을 분류하기
- Run Presidio offline
- 열의 값을 매 실행마다 같은 방식으로 샘플링하기
- 값으로 열 분류하기
- Measure a detector against labelled spans
- 인식기를 정답에 맞춰 조정한다
- See where patterns fail
- Compare a PII model
- Read the model's misses
- Count what a wrong label costs
- Classify again when the schema changes
- What goes wrong in production
- What this chapter decided
이 장을 마치면 데이터 플랫폼에 있는 개인 데이터를 자동으로 찾고, 얼마나 놓치는지도 말할 수 있다. 열의 이름, 자료형, 그리고 재현 가능한 값 표본으로 모든 열의 분류를 제안한다. 자유 형식 텍스트 안의 개인 데이터를 구간 단위로 찾는다. 그리고 어떤 탐지기든 누군가 의존하기 전에 레이블이 붙은 데이터로 성능을 측정한다. 그 과정에서 Presidio의 패턴 인식기를 오프라인으로 실행하되 임곗값은 측정으로 정한다. 이 결과를 개인 데이터를 찾도록 학습한 모델이 기록해 둔 탐지 결과와 비교하고, 스키마 변경으로 추가된 열만 다시 분류한다.
플랫폼의 모든 열에는 이미 분류가 있다. 2장에서 사람들이 검토하는 파일인 governance/metadata.toml에 분류를 넣었는데, 검토를 거친 파일은 검토자들이 그 열에 무엇이 들어 있다고 믿는지를 기록할 뿐이다. 이 책에서 거버넌스를 다루는 웨어하우스를 가진 온라인 학습 플랫폼에는 그런 믿음 가운데 세 가지가 한 번도 확인된 적이 없다. billing.payments.card_number은 설명에 카드 번호 전체가 들어 있다고 적힌 레거시 열인데, 아무도 들여다보지 않았다. crm.learners.notes는 지원 담당자가 직접 입력하는 자유 형식 텍스트라서 열 이름만으로는 내용을 알 수 없다. 학습자는 지원 티켓과 강좌 후기에 하고 싶은 말을 무엇이든 적는다. 열 전체를 하나로 분류해서는 이런 텍스트를 보호할 수 없다. 텍스트를 마스킹하려면 먼저 그 안의 이름, 이메일, 전화번호를 구간 단위로 하나씩 찾아야 한다. 9장에서 강좌 어시스턴트가 후기를 학습하기 전에 후기에 대해 하는 일이 바로 이것이다.
이 장에서는 이 작업을 위한 책 전체의 공용 모듈 lib.pii을 만든다. 이 모듈은 열 이름에 대한 규칙, 열 값의 시드 고정 표본, 오프라인으로 실행하도록 구성한 Presidio 탐지기, 그리고 어떤 탐지기든 레이블이 붙은 데이터로 점수를 매기는 코드로 이루어진다. 이 장의 모든 탐지기는 정답이라고 부르는 하나의 파일 data/platform/pii_spans.parquet을 기준으로 측정한다. 플랫폼의 데이터 생성기는 이 파일을 티켓 및 후기와 나란히 쓰는데, 파일에는 생성기가 텍스트에 넣은 모든 개인 데이터와 그 정확한 시작 위치 및 끝 위치가 담겨 있다. 송장 번호, 주문 번호, 문제가 발생한 날짜는 플랫폼이 개인 데이터로 보지 않으므로 레이블이 붙지 않은 채로 남는다.
이 책의 전화번호, 카드 번호, IBAN은 데이터 카드에 적힌 대로 무작위로 만든 값이며, 어떤 예제도 이를 전부 출력하지 않는다. 마지막 네 자리를 제외한 나머지는 모두 가린다.
스키마로 열을 분류하기
열에 대한 가장 저렴한 근거는 스키마, 곧 열의 이름과 타입이다. 이름에 대한 규칙은 데이터를 읽지 않으므로 풀 리퀘스트 안에서 카탈로그 전체에 대해 실행할 수 있고, email이나 date_of_birth처럼 이름만으로 담고 있는 내용을 알 수 있는 열을 잡아낸다. 카탈로그 도구도 같은 방식으로 시작한다. OpenMetadata의 자동 분류는 열 이름을 정규 표현식과 대조하는 열 이름 스캐너로 시작한다(OpenMetadata docs). 다음 단계는 타입이 결정한다. 텍스트 열의 값은 샘플링하여 스캔할 수 있지만, 타임스탬프에는 이메일이 들어 있을 수 없으므로 날짜 패턴으로 스캔하면 모든 행이 걸린다. 그래서 숫자, 날짜, 타임스탬프, 불리언은 이름만으로 분류한다.
플랫폼은 규칙을 governance/pii.toml에 보관한다. 이 파일에는 탐지기의 모든 설정이 들어 있으며 메타데이터와 똑같이 검토를 거친다. 규칙은 소문자로 바꾼 열 이름에 적용하는 8개의 정규 표현식이며, 각 규칙은 자신이 시사하는 엔터티를 지정하고 처음 일치한 규칙이 적용된다. 같은 파일이 각 엔터티를 분류에 대응시킨다. 전화번호는 personal로, 카드 번호나 IBAN은 sensitive으로 대응시킨다. lib.pii.by_name가 이 규칙을 적용한다. 규칙은 카탈로그의 모든 열을 대상으로 하며, 결과는 검토를 거친 메타데이터 옆에 놓인다.
1"""Runs the name rules over every column of the catalog and compares each column they mark with the reviewed metadata.2The prose rests on the count of marked columns that agree, and on the personal columns no rule marks."""3from pyiceberg.types import StringType45from lib import metadata, pii6from lib.catalog import open_catalog78catalog = open_catalog()9described = metadata.load() # the reviewed metadata, keyed by table and then by field ID10# every column as (table, field): the field carries the ID the metadata is keyed by, so a renamed column still matches11columns = [(table, f) for (namespace,) in sorted(catalog.list_namespaces())12 for _, name in sorted(catalog.list_tables(namespace))13 for table in [f"{namespace}.{name}"] for f in catalog.load_table(table).schema().fields]14print("columns in the catalog:", len(columns))15print("name rules:", len(pii.NAME_RULES))1617named = [(table, f, pii.by_name(f.name)) for table, f in columns if pii.by_name(f.name)]18print("columns a name rule marks:", len(named))19for table, f, entity in named:20 print(f" {table}.{f.name}: {entity}, so {pii.CLASSIFICATION[entity]}")21agree = sum(pii.CLASSIFICATION[entity] == described[table].columns[f.field_id].classification22 for table, f, entity in named)23print("of them, agreeing with the metadata:", agree)2425# Only text is sampled: numbers, dates, timestamps and booleans are classified by name alone.26text = [(table, f) for table, f in columns if isinstance(f.field_type, StringType)]27print("columns of text, whose values can be sampled:", len(text))2829# Columns the metadata, reviewed by people, calls personal or sensitive, that no name rule marks.30missed = [f"{table}.{f.name}" for table, f in columns if not pii.by_name(f.name)31 and described[table].columns[f.field_id].classification in ("personal", "sensitive")]32print("personal columns no name rule marks:", ", ".join(missed))
python column-name-rules.pycolumns in the catalog: 64
name rules: 8
columns a name rule marks: 11
billing.payments.card_number: CREDIT_CARD, so sensitive
crm.instructors.full_name: PERSON, so personal
crm.instructors.email: EMAIL_ADDRESS, so personal
crm.instructors.payout_iban: IBAN_CODE, so sensitive
crm.learners.full_name: PERSON, so personal
crm.learners.email: EMAIL_ADDRESS, so personal
crm.learners.phone: PHONE_NUMBER, so personal
crm.learners.city: LOCATION, so personal
crm.learners.street_address: LOCATION, so personal
crm.learners.date_of_birth: DATE_TIME, so personal
learning.events.ip_address: IP_ADDRESS, so personal
of them, agreeing with the metadata: 11
columns of text, whose values can be sampled: 29
personal columns no name rule marks: crm.learners.notes, learning.reviews.body, support.tickets.body
규칙은 작동하는 곳에서는 메타데이터와 일치하지만, 자유 텍스트는 놓친다. 카탈로그의 64개 열 가운데 규칙이 표시한 열은 11개이며, 그중 11개 전부가 사람이 부여한 분류와 일치한다. 검토자가 personal로 분류한 열 세 개는 규칙이 놓친다. crm.learners.notes, learning.reviews.body, support.tickets.body이다. 세 열 모두 자유 텍스트이며, 이런 열에서 이름은 내용물이 아니라 그릇을 설명한다. 플랫폼이 가장 우려하는 표시된 열인 billing.payments.card_number은 무엇이 들어 있는지 아무도 알아보지 않은 채 이름만으로 sensitive을 받는다. 카드 번호 전체든, 마지막 네 자리든, 결제 제공자의 토큰이든 모두 card_?(number|num|no)에 일치할 것이다.
이름 규칙은 알려진 방식으로 취약하다. 열 타입을 탐지하는 모델인 Sherlock의 저자들은 이름 매칭이 지저분한 데이터에 강건하지 않고 소수의 타입만 탐지한다고 보고한다(Hulsebos et al., 2019). 그래서 플랫폼은 카탈로그에 있는 29개의 텍스트 열에서 값도 읽는다.
작업 규칙: 이름 규칙은 비용이 들지 않으므로 먼저 모든 열에 실행하되, 분류를 올리는 데만 쓰고 낮추는 데는 쓰지 않는다. 규칙은 짧게 유지하고, 코드로 작성하며, 다른 규칙과 똑같이 검토한다. 이름이 무엇이든 자유 텍스트 열은 값을 샘플링해야 하는 열로 취급하고, 다른 타입의 열은 이름만으로 분류한다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Run Presidio offline
To read values, the platform uses Presidio, an open-source library for finding and masking personal data. Presidio's analyzer runs recognizers: regular expressions with checksums, phone-number parsing, and a named-entity model reached through an NLP engine. The NLP engine does two jobs, named-entity recognition and feature extraction for rules, such as the context words that raise a score. Its default model is spaCy's en_core_web_lg (Presidio docs). At 2.2.366, AnalyzerEngine() with no arguments downloads that model when it isn't installed (source). So a pipeline without network access fails on its first line, and one with access installs a model of about 400 MB as a side effect (en_core_web_lg 3.8.0).
lib.pii.detector configures the engine itself, with NoOpNlpEngine, which loads no language model (source). What that costs is context. With a model, Presidio raises a pattern's score when a word such as "phone" appears near the match, by comparing the lemma of each nearby word with each recognizer's context terms (Presidio's tutorial on context). With no model there are no lemmas, so the engine can't use the text's context words at all. The engine has one more way onto the network. Its email recognizer checks domains with tldextract, which fetches the Public Suffix List the first time it's used (tldextract 5.4.0's README). Importing Presidio imports tldextract, whose README says an empty TLDEXTRACT_PUBLIC_SUFFIX_LIST_URLS turns the fetch off if it's set before the import. So lib/__init__.py, which Python runs before any module of the book's lib, empties the variable. If an example imported Presidio first, the same file replaces tldextract's shared extractor with one that has no URLs. lib.pii refuses to load if any remain, and tldextract uses the copy of the list it ships with, so the same text gives the same answer on every machine.
The defaults are tested on real text: load every predefined recognizer on one of the platform's support tickets, which holds a name, an email, a phone number and a date:
1"""Loads Presidio's predefined recognizers on an engine that loads no language model, and runs them on one support2ticket. The prose's point rests on the detections printed under the ticket, and on the assert that the model isn't3installed."""4import spacy56from lib import pii7from lib.platform import load89# AnalyzerEngine() with no arguments loads spaCy's en_core_web_lg, and downloads it when it isn't installed.10installed = spacy.util.is_package("en_core_web_lg")11assert installed is False # the pipeline assumes no model: a machine that has one fails here, not silently12print("en_core_web_lg installed:", installed)13print("URLs tldextract would fetch the Public Suffix List from:", len(pii.suffix_list_urls()))1415detector = pii.detector() # every predefined recognizer, on an NLP engine that loads no model16names = sorted(r.name for r in detector.registry.recognizers)17print("recognizers loaded:", len(names))18print("recognizers:", ", ".join(names))19print("default score threshold:", detector.default_score_threshold)2021# The ticket holds a phone number, which may be in service: masked() prints it with only its last four digits.22ticket = pii.texts("tickets")[1421] # one ticket with a name, an email, a phone number and a date of the problem23found = sorted(pii.detect(detector, ticket))24print("ticket 1421:", pii.masked(ticket))25for start, end, entity, score in found:26 print(f" {entity} on {pii.masked(ticket[start:end])}: {score:.2f}")2728# Where is the phone number valid? The recognizer reads it in each of its regions.29phone = next(ticket[start:end] for start, end, entity, _ in found if entity == "PHONE_NUMBER")30[learner] = [row["learner_id"] for row in load("support.tickets").to_pylist() if row["ticket_id"] == 1421]31[country] = [row["country"] for row in load("crm.learners").to_pylist() if row["learner_id"] == learner]32print("the learner's country:", country)33# The recognizer's built-in regions, not the learner's country.34print("default regions where it's valid:",35 ", ".join(r for r in pii.PhoneRecognizer.DEFAULT_SUPPORTED_REGIONS if pii.valid_phone(phone, r)))
python presidio-offline-engine.pyen_core_web_lg installed: False
URLs tldextract would fetch the Public Suffix List from: 0
recognizers loaded: 17
recognizers: CreditCardRecognizer, CryptoRecognizer, DateRecognizer, EmailRecognizer, IbanRecognizer, IpRecognizer, MacAddressRecognizer, MedicalLicenseRecognizer, NhsRecognizer, PhoneRecognizer, UrlRecognizer, UsBankRecognizer, UsItinRecognizer, UsLicenseRecognizer, UsPassportRecognizer, UsSsnRecognizer, UuidRecognizer
default score threshold: 0
ticket 1421: I can't open the videos in Introduction to Design Systems, they stop after a few seconds. It happened on 12/03 around 9pm. My colleague Jennifer Abbott (jennifer.abbott@example.net) wants to enrol as well. You can reach me on •••••••0440 after 6pm.
DATE_TIME on 12/03: 0.10
EMAIL_ADDRESS on jennifer.abbott@example.net: 1.00
URL on example.net: 0.50
PHONE_NUMBER on •••••••0440: 0.40
US_BANK_NUMBER on •••••••0440: 0.05
US_DRIVER_LICENSE on •••••••0440: 0.01
the learner's country: US
default regions where it's valid: BR
Presidio's defaults on this engine see the email and the phone number, and miss the colleague's name. en_core_web_lg isn't installed, so a bare AnalyzerEngine() would try to download it, and tldextract has 0 URLs to fetch its list from. The no-model engine loads 17 recognizers, and none of them looks for names, so Jennifer Abbott, the colleague in the ticket, goes unnoticed. Her email is found with a score of 1.00, and its domain is found again as a URL, at 0.50. The learner's phone number, an 11-digit number typed without separators, is found three times: as a phone number at 0.40, a US bank number at 0.05 and a US driver's licence at 0.01. Even the phone number is found by chance. The learner's country is US, but of Presidio's default regions, the US among them, the digits are valid only in BR. The date of the problem, 12/03, is read as a date too, which says nothing about anyone.
Those scores aren't probabilities. Each pattern carries a constant score, and weak patterns get low ones, as Presidio's tutorial explains for a five-digit zip code (Presidio docs). A US bank number is any run of 8 to 17 digits (supported entities), and the FAQ admits that a driver's licence "could be any 9-digit number" (FAQ). The analyzer's default threshold is 0, so every weak match comes back.
Working rule: Configure Presidio's engine explicitly: NoOpNlpEngine when you run patterns only, or a pinned model you install yourself when you need its names. Never call AnalyzerEngine() bare in a pipeline, and pin anything a recognizer would download, such as the Public Suffix List. Read each score as a constant set for its pattern, never as a probability.
열의 값을 매 실행마다 같은 방식으로 샘플링하기
모든 텍스트 열의 모든 값을 검사하는 일은 어떤 웨어하우스에서도 너무 느리므로, 도구들은 샘플링을 한다. OpenMetadata의 워크플로는 행을 샘플링하여 개체 인식 엔진으로 검사한다(OpenMetadata docs). 샘플에는 세 가지 결정이 필요하다. 어떤 행인가: 처음 몇 행이나 무작위 시작점부터의 연속 블록이 아니라 무작위 샘플이어야 한다. Google의 Sensitive Data Protection은 무작위 행에서 시작하여 계속 읽어 나가는 무작위 시작 모드인 RANDOM_START를 제공한다(Google Cloud docs). 가입일 순으로 정렬된 테이블에서는 그 블록이 학습자 한 코호트에 해당한다. 매 실행마다 같은 행인가: 그래야 분류를 재현할 수 있고, 분류가 달라졌다면 데이터가 바뀌었다는 뜻이 된다. 그리고 몇 개인가: 반드시 찾아내야 하는 데이터가 얼마나 드문지에 따라 정해진다.
DuckDB의 저수지 샘플은 시드를 받지만, DuckDB 2.0.0 docs가 경고하듯 시드가 재현 가능한 샘플을 보장하는 것은 DuckDB가 스레드 하나로 실행될 때뿐이다. lib.pii.sample는 스레드를 하나로 설정하고, NULL을 걸러 낸 다음, 시드를 지정한 저수지 샘플을 추출한다.
드문 값에는 더 큰 샘플이 필요하며, 얼마나 더 커야 하는지는 산술로 알 수 있다. 개인정보를 담은 값의 비율을 p라 하자. 샘플링한 값 가운데 그런 값이 하나 이상 포함될 확률은 다음과 같다.
chance = 1 − (1 − p)ⁿ
99% 확률을 주는 n은 ln(0.01) / ln(1 − p)이다. 이는 경험칙이다. 추출을 서로 독립으로 취급하는데, 수천 행에서 비복원으로 추출하는 샘플은 거의 독립에 가깝다.
이 공식을 실제로 적용해 보자. crm.learners에서 메모 20개를 두 번 추출한 다음, 모든 행을 기준으로 한 확률과 메모만을 기준으로 한 확률을 비교한다.
1"""Draws the same twenty notes twice from a seeded reservoir sample, then counts the notes that hold personal data, to see2how many sampled values a rare value needs. The prose rests on the assert that the draws match, the two chances and the sample sizes."""3import math45from lib import pii6from lib.catalog import open_catalog7from lib.warehouse import connect89con = connect(open_catalog(["crm.learners"]))10first = pii.sample(con, "crm.learners", "notes", rows=20)11second = pii.sample(con, "crm.learners", "notes", rows=20) # the same call: the seed and one thread (set in lib.pii.sample) make the draws match12assert second == first13print("second draw, same notes as the first:", "yes" if second == first else "no")14# Notes can hold a phone number, which may be in service: masked() prints it with only its last four digits.15print("first sampled notes:", " | ".join(pii.masked(note) for note in first[:3]))1617# How rare is personal data in the column? Run the detector over every note, once, to know.18detector = pii.platform_detector()19[(rows, notes)] = con.sql("select count(*), count(notes) from crm.learners").fetchall() # count(notes) skips NULLs20every_note = [note for (note,) in con.sql("select notes from crm.learners where notes is not null").fetchall()]21holding = sum(bool(pii.detect(detector, note)) for note in every_note)22print("learners:", f"{rows:,}")23print("learners with notes:", notes)24print("notes holding personal data the detector finds:", holding)2526# The chance that n sampled values hold at least one of them is 1 - (1 - p)^n, for a share p of the values.27for unit, p in (("rows", holding / rows), ("notes", holding / notes)): # p counted over all learners, then over the notes28 print(f"chance 20 sampled {unit} catch one:", f"{1 - (1 - p) ** 20:.1%}")29 print(f"sampled {unit} needed for a 99% chance:", math.ceil(math.log(0.01) / math.log(1 - p))) # solves for n30print("values the platform samples per text column:", pii.ROWS)31# The rarest share p that a sample of ROWS values still catches 99% of the time, printed as 1 in a whole number.32print("rarest share that sample catches 99% of the time:", f"1 in {math.floor(1 / (1 - 0.01 ** (1 / pii.ROWS)))}")
python seeded-column-sample.pysecond draw, same notes as the first: yes
first sampled notes: VIP: bought a team licence. | Asked about invoices for their employer. | Called about a refund, call back on (•••)•••-1366x•••.
learners: 5,000
learners with notes: 527
notes holding personal data the detector finds: 138
chance 20 sampled rows catch one: 42.9%
sampled rows needed for a 99% chance: 165
chance 20 sampled notes catch one: 99.8%
sampled notes needed for a 99% chance: 16
values the platform samples per text column: 200
rarest share that sample catches 99% of the time: 1 in 43
추출이 반복되므로 샘플을 감사할 수 있다. 시드가 같으면 두 번째 추출도 같은 메모 20개를 반환하며, 세 번째 메모에는 이미 전화번호가 들어 있는데 출력에서는 이를 가린다.
학습자 5,000명 중 527명에게 메모가 있고, 그 가운데 138에는 플랫폼의 탐지기가 찾아내는 개인정보가 들어 있다. 이 탐지기는 이 장의 뒤에서 조정한다. NULL은 개인정보를 담을 수 없으므로 샘플을 희석한다. 행 20개를 샘플링하면 42.9%의 확률로 개인정보를 잡아내고, 99% 확률에는 행 165개가 필요하다. 존재하는 메모만 기준으로 세면 20개로 99.8%의 확률로 잡아내고, 99%에는 16개면 충분하다.
작업 규칙: NULL이 아닌 값을 고정된 시드로 스레드 하나에서 샘플링하고, 시드를 결과와 함께 보관한다. 반드시 찾아내야 하는 가장 드문 개인정보를 기준으로 샘플 크기를 정하며, 99%에는 n = ln(0.01) / ln(1 − p)를 쓴다. 플랫폼은 텍스트 열마다 200개의 값을 샘플링하며, 이는 governance/pii.toml에 설정되어 있다. 이 크기로는 개인정보를 담은 값의 비율이 1 in 43 이상일 때 99%의 확률로 그 개인정보를 잡아낸다. 그보다 드문 경우에는 열 전체를 검사하거나 규칙으로 분류한다.
값으로 열 분류하기
이름 규칙과 재현 가능한 표본을 결합하면 열마다 하나의 제안이 나온다. lib.pii.classify은 두 근거 중 어느 쪽이든 뒷받침하는 가장 엄격한 분류를 택한다. 두 근거란 이름이 시사하는 엔터티와 표본 추출한 값 중 어느 하나에서라도 발견된 모든 엔터티를 말한다. 근거가 없는 열에는 internal를 부여한다. 아무것도 찾지 못한 탐지기는 아무것도 보여 준 것이 아니며, 열이 public임을 판단하는 일은 사람만 하기 때문이다.
가장 엄격한 쪽을 택하는 것은 하나의 선택이다. Presidio가 표를 위해 제공하는 도구인 Presidio structured는 열마다 엔터티를 하나만 부여하며, 기본값인 Most Common은 가장 자주 나타나는 엔터티를 고른다(Presidio 문서). 이 방식은 열을 어떻게 마스킹할지 정하는 데는 적합하다. 그러나 분류에서는 200개 값 중 카드 번호가 하나만 들어 있어도 그 열에는 여전히 카드 번호가 들어 있는 것이다.
플랫폼의 탐지기로 카탈로그의 모든 열에 대한 제안을 만들고, 각 제안이 검토된 메타데이터와 어디에서 다른지 살펴본다. 탐지기가 쓰는 인식기는 뒤의 절에서 선택한다.
1"""Proposes a classification for every column, the strictest of what its name and its sampled values support, and compares2each proposal with the reviewed metadata. The prose's table rests on the proposed and found lines of each column, and3its counts on the totals printed after them."""4from lib import metadata, pii5from lib.catalog import open_catalog6from lib.warehouse import connect78catalog = open_catalog()9findings = pii.classify(catalog, connect(catalog), pii.platform_detector()) # the platform's tuned detector, not the defaults10print("columns classified:", len(findings))11print("columns sampled:", sum(f.sampled > 0 for f in findings))1213holding = [f for f in findings if f.found] # columns where sampled values hold personal data14print("columns whose sampled values hold personal data:", len(holding))15# A column counts when its values hold an entity other than the one its name suggests (named is None when no rule matches).16print("of them, holding something their name doesn't suggest:", sum(set(f.found) != {f.named} for f in holding))17for f in holding:18 print(f"{f.table}.{f.column}, proposed: {f.classification}")19 print(f"{f.table}.{f.column}, found: " + ", ".join(f"{e} in {n} of {f.sampled}" for e, n in f.found.most_common()))20[cards] = [f for f in findings if f.column == "card_number"] # the one card_number column in the catalog21# The card recognizer's count: values that match its pattern and pass its Luhn check, out of the values sampled.22print("sampled card numbers the card recognizer accepts:", f"{cards.found['CREDIT_CARD']} of {cards.sampled}")2324# Classifications the values raise above what the name alone gives: the floor is used for a name no rule matches.25named_only = [f for f in findings if pii.CLASSIFICATION.get(f.named, pii.FLOOR) != f.classification]26print("columns the values classify more strictly than the name:", ", ".join(f"{f.table}.{f.column}" for f in named_only))2728# Compare with the metadata people reviewed. A column the detector found nothing in proposes only the floor, internal.29described = metadata.load()30evidenced = [f for f in findings if f.named or f.found]31print("columns with evidence:", len(evidenced))32print("of them, equal to the metadata:",33 sum(f.classification == described[f.table].columns[f.field_id].classification for f in evidenced))34for f in evidenced:35 reviewed = described[f.table].columns[f.field_id].classification36 if metadata.stricter(f.classification, reviewed) != reviewed: # the stricter of the two: a lower proposal prints nothing37 print(f"stricter than the metadata, {f.table}.{f.column}: {f.classification}, reviewed as {reviewed}")
python classify-sampled-columns.pycolumns classified: 64
columns sampled: 29
columns whose sampled values hold personal data: 9
of them, holding something their name doesn't suggest: 5
billing.payments.card_number, proposed: sensitive
billing.payments.card_number, found: CREDIT_CARD in 151 of 200, PHONE_NUMBER in 3 of 200
crm.instructors.email, proposed: personal
crm.instructors.email, found: EMAIL_ADDRESS in 50 of 50
crm.instructors.payout_iban, proposed: sensitive
crm.instructors.payout_iban, found: IBAN_CODE in 50 of 50
crm.learners.email, proposed: personal
crm.learners.email, found: EMAIL_ADDRESS in 200 of 200
crm.learners.phone, proposed: personal
crm.learners.phone, found: PHONE_NUMBER in 175 of 200
crm.learners.notes, proposed: personal
crm.learners.notes, found: PHONE_NUMBER in 29 of 200, EMAIL_ADDRESS in 18 of 200
learning.events.ip_address, proposed: personal
learning.events.ip_address, found: IP_ADDRESS in 200 of 200, PHONE_NUMBER in 39 of 200
learning.reviews.body, proposed: personal
learning.reviews.body, found: EMAIL_ADDRESS in 4 of 200
support.tickets.body, proposed: sensitive
support.tickets.body, found: EMAIL_ADDRESS in 47 of 200, PHONE_NUMBER in 37 of 200, DATE_TIME in 29 of 200, IBAN_CODE in 21 of 200, CREDIT_CARD in 12 of 200
sampled card numbers the card recognizer accepts: 151 of 200
columns the values classify more strictly than the name: crm.learners.notes, learning.reviews.body, support.tickets.body
columns with evidence: 14
of them, equal to the metadata: 13
stricter than the metadata, support.tickets.body: sensitive, reviewed as personal
이름만으로 정해지지 않는 부분은 값이 해결해 준다. 표본에 개인정보가 들어 있는 9개 열 중 일부는 crm.learners.email처럼 이름이 말하는 내용만 담고 있다. 나머지 5개 열은 이름이 시사하지 않는 내용을 담고 있거나, 아무것도 시사하지 않는 이름을 가지고 있다.
| 열 | 제안 | 표본에서 발견된 것 |
|---|---|---|
billing.payments.card_number |
sensitive | CREDIT_CARD in 151 of 200, PHONE_NUMBER in 3 of 200 |
crm.learners.notes |
personal | PHONE_NUMBER in 29 of 200, EMAIL_ADDRESS in 18 of 200 |
support.tickets.body |
sensitive | EMAIL_ADDRESS in 47 of 200, PHONE_NUMBER in 37 of 200, DATE_TIME in 29 of 200, IBAN_CODE in 21 of 200, CREDIT_CARD in 12 of 200 |
learning.reviews.body |
personal | EMAIL_ADDRESS in 4 of 200 |
learning.events.ip_address |
personal | IP_ADDRESS in 200 of 200, PHONE_NUMBER in 39 of 200 |
레거시 카드 열의 설명에는 "아예 저장해서는 안 된다"라고 되어 있는데, 표본을 보면 이 열에는 전체 카드 번호가 들어 있다. 이 표의 소유자인 team-billing가 조치할 수 있는 사안이다. 표본 추출한 카드 번호 가운데 151 of 200개는 Presidio의 카드 패턴과 일치하고 그 패턴이 확인하는 Luhn 체크섬도 통과하며(소스), 나머지는 패턴이 다루지 않는 형식이다(패턴이 실패하는 지점 보기). 메모 열은 값만으로 personal으로 분류된다. 티켓 본문은 sensitive으로 분류된다. 학습자가 티켓에 카드 번호와 IBAN을 붙여 넣기 때문이며, 사람이 personal으로 검토한 메타데이터보다 더 엄격한 유일한 제안이다. 근거가 하나라도 있는 14개 열 중 13개는 검토된 분류와 일치한다.
판정만 보지 말고 근거를 읽어야 한다. 일부 IP 주소는 전화번호로도 인식된다. IP 주소는 어차피 personal이므로 여기서는 달라지는 것이 없지만, 탐지기의 레이블은 값이 무엇처럼 보이는지를 말해 줄 뿐 값이 무엇인지를 말해 주지는 않는다.
작업 규칙: 이름이나 표본 추출한 값 중 어느 하나라도 뒷받침하는 가장 엄격한 분류를 제안하되 internal를 하한으로 삼고, 제안과 함께 근거를 보여 준다. 검토된 메타데이터보다 엄격한 제안은 표의 소유자에게 전달하는 발견 사항이며, 소유자는 분류를 높이거나 그러지 않는 이유를 기록한다. 탐지기가 분류를 낮추게 해서는 안 된다. OpenMetadata는 이미 PII 태그가 달린 열은 건너뛴다. 우리는 검토된 태그가 바로 검증 대상이므로, 대신 매 실행마다 모든 텍스트 열을 표본 추출할 것을 권장한다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Measure a detector against labelled spans
A column-level answer is enough to classify support.tickets.body, but not to mask it: masking replaces each span of personal data in each text, and a span the detector misses stays readable. To know what a detector misses, compare its spans with the truth, the labelled spans of every ticket and review. Two matching rules are standard, and lib.pii applies both.
Exact matching, from the CoNLL-2003 shared task, counts an entity "correct only if it is an exact match of the corresponding entity in the data file" (Tjong Kim Sang and De Meulder, 2003). Overlap matching asks only for one shared character, with a label of its type, which the SemEval-2013 task on drug interactions calls "Type matching" (Segura-Bedmar et al., 2013). Neither credits a span hidden under another label, so lib.pii adds a third measure, coverage: whether every character of a labelled span is inside some detection, whatever its label. Exact asks whether the detector labelled the whole span right, overlap whether it noticed the span as what it is, and coverage whether masking its detections hides the span.
lib.pii.score computes exact and overlap matching as recall, the share of labelled spans of each type that a detector finds with a label of that type, and precision, the share of each label's detections that match a labelled span. lib.pii.coverage gives the share of each type's spans covered. score also counts the texts in which every labelled character falls inside some detection, the texts that masking would leave with nothing personal. The Text Anonymization Benchmark protects a person only when all of their mentions are masked (Pilán et al., 2022), and the platform counts whole texts, which is stricter. Recall comes first: Presidio's evaluation guide says it often matters more than precision, because missing personal data is the error to avoid (Presidio docs).
Presidio's defaults, scored against the labelled spans of every ticket and review, give the baseline the rest of the chapter improves on:
1"""Runs Presidio with its defaults over every ticket and review, and scores its detections against the truth's labelled2spans. The prose rests on the recall and precision lines, the count of texts left with nothing personal, and the two3date lines that test the truth's limits."""4import re56from lib import pii78labelled = pii.truth()9print("texts:", f"{sum(len(pii.texts(source)) for source in pii.SOURCES):,}")10print("labelled spans:", f"{len(labelled):,}")1112detector = pii.detector() # Presidio's defaults: every recognizer, threshold 013found = pii.find(detector)14# scoring needs a label of the span's type; coverage doesn't15scores, covered = pii.score(found, labelled), pii.coverage(found, labelled)16print("detections:", f"{len(found):,}")1718for entity, (n, exact, overlap) in scores.recall.items():19 print(f"{entity}, labelled: {n:,}")20 print(f"{entity}, recall, exact: {exact / n:.2f}")21 print(f"{entity}, recall, overlap: {overlap / n:.2f}")22 print(f"{entity}, covered, any label: {covered[entity][1] / n:.2f}") # covered[entity] is (spans, covered), so n is the same count2324for label, (n, exact, overlap) in scores.precision.items(): # precision is per label: what each detection was called25 print(f"{label}, detections: {n:,}")26 print(f"{label}, precision, exact: {exact / n:.2f}")2728# every bank number and driver's licence detection is a false alarm: none matches a labelled span29assert all(scores.precision[entity][1] == 0 for entity in ("US_BANK_NUMBER", "US_DRIVER_LICENSE"))30print("precision of all detections, exact:", f"{pii.precision_overall(scores):.2f}")31print("texts holding personal data:", f"{scores.texts:,}")32print("texts with every labelled span masked:", f"{scores.masked:,}")3334# The truth's limits. The generator builds every text from a few sentence templates, and writes every date of birth35# one way. Presidio's date patterns don't read every way of writing one.36print("sentence templates the texts are built from:", len(pii.templates()))37births = [pii.text_of(s) for s in labelled if s.entity == "DATE_TIME"] # the truth's dates are the dates of birth: dates of a problem are unlabelled38written_ymd = sum(re.fullmatch(r"\d{4}-\d\d-\d\d", d) is not None for d in births) # year first, the generator's one form39assert written_ymd == len(births) # so every date of birth is written this way40print("dates of birth written like 1985-03-12:", written_ymd)41for written in ("1985-03-12", "12 March 1985"):42 text = f"My date of birth on the account is wrong, it should be {written}." # a sentence in the generator's shape43 print(f"dates found in a date of birth written {written}:", sum(e == "DATE_TIME" for _, _, e, _ in pii.detect(detector, text)))
python measure-pattern-recognizers.pytexts: 5,000
labelled spans: 4,292
detections: 3,902
CREDIT_CARD, labelled: 213
CREDIT_CARD, recall, exact: 0.71
CREDIT_CARD, recall, overlap: 0.71
CREDIT_CARD, covered, any label: 0.89
DATE_TIME, labelled: 196
DATE_TIME, recall, exact: 1.00
DATE_TIME, recall, overlap: 1.00
DATE_TIME, covered, any label: 1.00
EMAIL_ADDRESS, labelled: 543
EMAIL_ADDRESS, recall, exact: 1.00
EMAIL_ADDRESS, recall, overlap: 1.00
EMAIL_ADDRESS, covered, any label: 1.00
IBAN_CODE, labelled: 236
IBAN_CODE, recall, exact: 1.00
IBAN_CODE, recall, overlap: 1.00
IBAN_CODE, covered, any label: 1.00
LOCATION, labelled: 210
LOCATION, recall, exact: 0.00
LOCATION, recall, overlap: 0.00
LOCATION, covered, any label: 0.00
PERSON, labelled: 2,455
PERSON, recall, exact: 0.00
PERSON, recall, overlap: 0.00
PERSON, covered, any label: 0.00
PHONE_NUMBER, labelled: 439
PHONE_NUMBER, recall, exact: 0.78
PHONE_NUMBER, recall, overlap: 0.79
PHONE_NUMBER, covered, any label: 0.91
CREDIT_CARD, detections: 151
CREDIT_CARD, precision, exact: 1.00
DATE_TIME, detections: 421
DATE_TIME, precision, exact: 0.47
EMAIL_ADDRESS, detections: 543
EMAIL_ADDRESS, precision, exact: 1.00
IBAN_CODE, detections: 236
IBAN_CODE, precision, exact: 1.00
MEDICAL_LICENSE, detections: 18
MEDICAL_LICENSE, precision, exact: 0.00
PHONE_NUMBER, detections: 384
PHONE_NUMBER, precision, exact: 0.89
UK_NHS, detections: 8
UK_NHS, precision, exact: 0.00
URL, detections: 751
URL, precision, exact: 0.00
US_BANK_NUMBER, detections: 689
US_BANK_NUMBER, precision, exact: 0.00
US_DRIVER_LICENSE, detections: 683
US_DRIVER_LICENSE, precision, exact: 0.00
US_ITIN, detections: 2
US_ITIN, precision, exact: 0.00
US_PASSPORT, detections: 8
US_PASSPORT, precision, exact: 0.00
US_SSN, detections: 8
US_SSN, precision, exact: 0.00
precision of all detections, exact: 0.38
texts holding personal data: 2,586
texts with every labelled span masked: 425
sentence templates the texts are built from: 30
dates of birth written like 1985-03-12: 196
dates found in a date of birth written 1985-03-12: 1
dates found in a date of birth written 12 March 1985: 0
| Entity type | Labelled | Recall, exact | Recall, overlap | Covered |
|---|---|---|---|---|
PERSON |
2,455 | 0.00 | 0.00 | 0.00 |
EMAIL_ADDRESS |
543 | 1.00 | 1.00 | 1.00 |
PHONE_NUMBER |
439 | 0.78 | 0.79 | 0.91 |
IBAN_CODE |
236 | 1.00 | 1.00 | 1.00 |
CREDIT_CARD |
213 | 0.71 | 0.71 | 0.89 |
LOCATION |
210 | 0.00 | 0.00 | 0.00 |
DATE_TIME |
196 | 1.00 | 1.00 | 1.00 |
The defaults find every email, IBAN and date of birth, most phone and card numbers, and no names or street addresses at all, which is most of what the tickets hold. Most of what they flag is wrong: 0.38 of all detections match a labelled span. The 689 US bank numbers, 683 driver's licences and 751 URLs are all wrong. Phone, card and invoice numbers, all runs of digits, come out as bank numbers and licences, and pieces of emails and a few addresses come out as URLs. A wrong label still hides the digits, so phone and card numbers are covered more often than they're found.
Masking would leave personal data in most texts that hold it. Only 425 of the 2,586 texts would come out with nothing personal left.
The truth has its own limits, because the text is generated. Every ticket and review is built from 30 sentence templates in prep/generate.py, so each hard case is one sentence frame, and all 196 dates of birth are written like 1985-03-12. The date recall of 1.00 therefore measures one format: in the generator's own sentence, Presidio finds 1 date when the date of birth is written 1985-03-12, and 0 when it's written 12 March 1985. A measure on this text tells you how a detector behaves on these shapes, not how it will do on yours.
Working rule: Measure every detector against labelled spans before it masks anything: recall per entity type by exact and by overlap matching, both with a label of the right type, coverage whatever the label, precision per label, and the share of texts with nothing personal left. Weigh recall first. Before you trust a detector on your text, label a few hundred real records by hand and measure on those too.
인식기를 정답에 맞춰 조정한다
Presidio의 기본 설정은 약한 일치를 모두 반환한다. FAQ는 두 가지 해결책을 제시한다. 하나는 수락 임곗값을 바꾸는 것이고, 다른 하나는 데이터 세트에 필요하지 않은 인식기를 제거하는 것이다. 또한 FAQ는 오탐과 미탐 사이에 상충 관계가 있음을 언급한다(FAQ). Presidio 2.2.364는 인식기별, 엔터티별 점수 임곗값을 추가했으며, 분석기의 기본값이 대체값으로 쓰인다(변경 기록, 릴리스 노트). 학습자가 여러 나라에 사는 플랫폼에서는 전화번호 인식기의 설정 하나가 더 중요하다. 이 인식기가 읽는 번호 체계의 지역이다(소스).
governance/pii.toml은 플랫폼의 텍스트에 들어 있을 수 있는 식별자에 해당하는 6개의 인식기를 유지하며, 각 인식기에 고유한 임곗값을 둔다. 전화번호 지역은 설정값이 아니다. lib.pii.countries이 crm.learners에서 학습자가 사는 나라를 읽어 오므로, 지역은 데이터를 따른다. 네 가지 설정을 같은 정답에 대해 채점한다.
1"""Scores four detector settings against the same truth: Presidio's defaults, one threshold for every recognizer, the kept2recognizers with their own thresholds, and those with phone numbers read in the learners' countries. The prose rests on3the precision, phone recall, coverage and fully masked texts of each setting."""4from lib import pii56labelled = pii.truth()7print("recognizers kept:", len(pii.THRESHOLDS)) # read from governance/pii.toml8print("recognizers kept, with their thresholds:", ", ".join(f"{name} {t}" for name, t in pii.THRESHOLDS.items()))9print("phone regions, Presidio's default:", ", ".join(pii.PhoneRecognizer.DEFAULT_SUPPORTED_REGIONS)) # Presidio's own list10print("phone regions, the learners' countries:", ", ".join(pii.countries())) # read from crm.learners1112# In the order the prose builds them: the defaults, one threshold, the kept recognizers, then their regions.13settings = {14 "Presidio's defaults": pii.detector(),15 "one threshold of 0.5": pii.detector(default_threshold=0.5),16 "kept recognizers, own thresholds": pii.detector(pii.THRESHOLDS),17 "kept recognizers, learners' phone regions": pii.platform_detector(),18}19for name, detector in settings.items():20 found = pii.find(detector)21 scores, covered = pii.score(found, labelled), pii.coverage(found, labelled)22 n, exact, _ = scores.recall["PHONE_NUMBER"] # the table reads exact recall, so the overlap count is left out23 print(f"{name}, detections: {len(found):,}")24 print(f"{name}, precision, exact: {pii.precision_overall(scores):.2f}")25 print(f"{name}, phone recall, exact: {exact / n:.2f}")26 # coverage takes a detection under any label27 for entity, short in (("PHONE_NUMBER", "phones"), ("CREDIT_CARD", "cards")):28 print(f"{name}, {short} covered, any label: {covered[entity][1] / covered[entity][0]:.2f}")29 print(f"{name}, texts with every labelled span masked: {scores.masked:,}")
python tune-pattern-recognizers.pyrecognizers kept: 6
recognizers kept, with their thresholds: EmailRecognizer 0.5, PhoneRecognizer 0.4, CreditCardRecognizer 0.5, IbanRecognizer 0.5, DateRecognizer 0.5, IpRecognizer 0.5
phone regions, Presidio's default: US, GB, DE, FR, IL, IN, CA, BR
phone regions, the learners' countries: AU, BR, CA, DE, ES, FR, GB, IN, JP, MX, PL, US
Presidio's defaults, detections: 3,902
Presidio's defaults, precision, exact: 0.38
Presidio's defaults, phone recall, exact: 0.78
Presidio's defaults, phones covered, any label: 0.91
Presidio's defaults, cards covered, any label: 0.89
Presidio's defaults, texts with every labelled span masked: 425
one threshold of 0.5, detections: 1,903
one threshold of 0.5, precision, exact: 0.59
one threshold of 0.5, phone recall, exact: 0.00
one threshold of 0.5, phones covered, any label: 0.01
one threshold of 0.5, cards covered, any label: 0.71
one threshold of 0.5, texts with every labelled span masked: 284
kept recognizers, own thresholds, detections: 1,510
kept recognizers, own thresholds, precision, exact: 0.97
kept recognizers, own thresholds, phone recall, exact: 0.78
kept recognizers, own thresholds, phones covered, any label: 0.78
kept recognizers, own thresholds, cards covered, any label: 0.72
kept recognizers, own thresholds, texts with every labelled span masked: 394
kept recognizers, learners' phone regions, detections: 1,539
kept recognizers, learners' phone regions, precision, exact: 0.97
kept recognizers, learners' phone regions, phone recall, exact: 0.84
kept recognizers, learners' phone regions, phones covered, any label: 0.84
kept recognizers, learners' phone regions, cards covered, any label: 0.72
kept recognizers, learners' phone regions, texts with every labelled span masked: 405
| 측정값 | 기본 설정 | 단일 임곗값 0.5 | 유지한 인식기 | 학습자 지역 추가 |
|---|---|---|---|---|
| 탐지 수 | 3,902 | 1,903 | 1,510 | 1,539 |
| 정밀도, 정확 일치 | 0.38 | 0.59 | 0.97 | 0.97 |
| 전화번호 재현율, 정확 일치 | 0.78 | 0.00 | 0.78 | 0.84 |
| 포괄된 전화번호 | 0.91 | 0.01 | 0.78 | 0.84 |
| 포괄된 카드 번호 | 0.89 | 0.71 | 0.72 | 0.72 |
| 완전히 마스킹된 텍스트 | 425 | 284 | 394 | 405 |
뻔한 해결책은 실패한다. 임곗값을 0.5 하나로 두면 약한 일치가 사라지지만 전화번호도 모두 함께 사라진다. 문맥 단어가 점수를 올려 주지 않으면 전화번호의 점수는 0.40이기 때문이다. 6개의 인식기만 유지하고 각각에 고유한 임곗값을 두며 전화번호 임곗값을 전화번호가 받는 점수에 맞추면, 정밀도가 0.97로 오르고 전화번호 재현율은 그대로 유지된다. Presidio의 기본 지역 대신 학습자의 지역에서 전화번호를 읽으면 전화번호 재현율이 0.84로 오른다.
한 가지 수치는 나쁜 방향으로 움직인다. 완전히 마스킹된 텍스트가 기본 설정일 때보다 줄어든다. 제거한 인식기들은 일부 전화번호와 카드 번호를 잘못된 레이블로 마스킹하고 있었는데, 그래도 숫자는 가려진다. 그 인식기들이 없으면 포괄 비율이 전화번호는 0.91에서 0.78으로, 카드 번호는 0.89에서 0.72로 떨어진다. 우리는 정밀도가 그만한 값어치가 있다고 본다. 탐지 결과는 검토자가 읽는 증거인데, 기본 설정에서는 티켓과 리뷰에 689건의 은행 번호와 683건의 운전면허 번호가 들어 있고 그중 실제 번호는 하나도 없기 때문이다. governance/pii.toml은 두 엔터티 모두 분류에 대응시키지 않으므로 제안은 달라지지 않지만, 각각은 누군가 읽고 기각해야 하는 발견 사항이다. 미국 인식기를 제거하는 것은 플랫폼의 텍스트에 대한 주장이기도 하다. 정답에는 미국 은행 번호나 면허 번호가 없다. 플랫폼이 언젠가 그런 번호를 요구하면 이 주장과 측정을 바꿔야 한다.
작업 규칙: Presidio의 기본 임곗값 0으로 실행하지 않으며, 모든 인식기에 임곗값 하나를 쓰지도 않는다. 텍스트에 들어 있을 수 있는 식별자의 인식기만 유지하고, 각 인식기의 임곗값을 그 인식기 자체의 점수로 정하며, 전화번호는 사용자가 사는 지역에서 읽되 그 지역을 데이터에서 도출한다. 이 모두를 코드에 넣고, 바뀔 때마다 다시 측정한다.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
See where patterns fail
The tuned recognizers score well on totals, and totals hide where they fail. A pattern detector fails in three ways: there's no pattern for the thing at all, its pattern or its validation rejects the value, or it matches something that only looks personal. The platform's tickets plant cases of each kind, as its data card says: names typed in lowercase, initials, first names that are ordinary words, phone numbers typed without separators, and invoice numbers, order numbers and dates of a problem, which the truth leaves unlabelled. Count each case with the platform's detector. A case counts as caught when some detection overlaps it, whatever its label:
1"""Counts each hard case the tickets and reviews plant, and each look-alike number or date the platform's detector flags,2then checks the card and phone numbers it misses. The prose rests on the caught counts for names and invoice numbers,3and on the misses' lengths, first digits and validity."""4from collections import Counter56from lib import pii7from lib.platform import load89labelled = pii.truth()10found = pii.find(pii.platform_detector())1112cases = pii.hard_cases(labelled) # a case is caught when some detection overlaps it, whatever its label13for case, spans in cases.items():14 print(f"{case}, labelled: {len(spans)}")15 print(f"{case}, caught: {pii.caught(spans, found)}")16print("hard cases of a name, caught:", pii.caught([s for spans in cases.values() for s in spans if s.entity == "PERSON"], found))1718# Look-alikes are numbers and dates that aren't personal data, so any detection on one is a false alarm.19for kind, spans in pii.lookalikes().items():20 print(f"{kind}, in the tickets: {len(spans)}")21 print(f"{kind}, flagged as personal data: {pii.caught(spans, found)}")2223# Card numbers the card pattern misses: too short or long for its 13 to 17 digits, or a first digit it doesn't accept.24# Out of range, the length is the reason; in range, the first digit is, since the pattern accepts only some of them.25missed = pii.missed([s for s in labelled if s.entity == "CREDIT_CARD"], found, {"CREDIT_CARD"})26why = lambda digits: f"{len(digits)} digits" if not 13 <= len(digits) <= 17 else f"{len(digits)} digits starting with {digits[0]}"27print("CREDIT_CARD, missed:", len(missed))28for reason, n in Counter(why(pii.text_of(s)) for s in missed).most_common():29 print(f"CREDIT_CARD, missed, {reason}: {n}")30print("IBAN_CODE, missed:", len(pii.missed([s for s in labelled if s.entity == "IBAN_CODE"], found, {"IBAN_CODE"})))3132# Are the phone numbers the patterns miss valid where the ticket's learner lives? Phones appear only in tickets,33# so each phone's learner comes from its ticket.34learner_of = {row["ticket_id"]: row["learner_id"] for row in load("support.tickets").to_pylist()}35country_of = {row["learner_id"]: row["country"] for row in load("crm.learners").to_pylist()}36phones = pii.missed([s for s in labelled if s.entity == "PHONE_NUMBER"], found, {"PHONE_NUMBER"})37print("phone numbers missed:", len(phones))38print("of them, invalid in the learner's country:",39 sum(not pii.valid_phone(pii.text_of(s), country_of[learner_of[s.row_id]]) for s in phones))
python pattern-hard-cases.pynames in lowercase, labelled: 211
names in lowercase, caught: 0
initial and surname, labelled: 163
initial and surname, caught: 0
first names that are words, labelled: 187
first names that are words, caught: 0
instructors named in reviews, labelled: 798
instructors named in reviews, caught: 0
phones typed as digits only, labelled: 257
phones typed as digits only, caught: 220
hard cases of a name, caught: 0
invoice numbers, in the tickets: 224
invoice numbers, flagged as personal data: 31
order numbers, in the tickets: 212
order numbers, flagged as personal data: 0
dates of a problem, in the tickets: 225
dates of a problem, flagged as personal data: 0
CREDIT_CARD, missed: 62
CREDIT_CARD, missed, 19 digits: 23
CREDIT_CARD, missed, 12 digits: 22
CREDIT_CARD, missed, 16 digits starting with 2: 10
CREDIT_CARD, missed, 15 digits starting with 2: 7
IBAN_CODE, missed: 0
phone numbers missed: 65
of them, invalid in the learner's country: 63
Names are the gap, since no pattern matches a name. Across the names in lowercase, the initials with a surname, the first names that are ordinary words and the instructors named in reviews, the patterns catch 0 of them. Phones typed as digits only do well, 220 of 257. The price is the look-alikes: 31 of the 224 invoice numbers are flagged as personal data.
The platform's detector flags 0 of the 212 order numbers and 0 of the 225 dates of a problem. With Presidio's defaults, 12/03 was a date at 0.10, and the date recognizer returned 421 dates where the truth labels 196. The threshold in governance/pii.toml keeps such weak matches out.
The card pattern's shape causes the card misses. It accepts 13 to 17 digits starting with 1, 3, 4, 50 to 55 or 6, but no 13-digit number starting with 1 (source), and every card number in the data passes the Luhn checksum, so validation rejects none. Of the 62 card numbers it misses, 23 have 19 digits and 22 have 12, and the rest start with 2, which the pattern doesn't accept.
Validation causes the other misses. The IBAN recognizer checks each country's format, length and check digits (source). Every labelled IBAN passes, so it misses 0, but a learner who mistypes one digit writes a value it rejects. The phone recognizer accepts only numbers that python-phonenumbers calls valid (source), and the generator doesn't always write those. Of the 65 phone numbers the detector misses, 63 aren't valid in the country the ticket's learner lives in, so most of the phone misses are the generator's, not the detector's.
Working rule: Read a detector's misses by case, not by total, and keep a list of the cases your text holds. Use patterns for what has a checkable shape, such as emails, IBANs, card numbers, phone numbers and dates, and never for names or addresses. Keep the checksums and numbering plans, because they're what makes the patterns precise. Check the card pattern against the card numbers your payment provider issues, and the date patterns against the ways your users write dates.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Compare a PII model
The patterns have nothing for names or addresses, and those are most of the personal data in the platform's text: 2,455 names and 210 addresses among the 4,292 labelled spans. Finding them takes a model that reads context, not a general named-entity model. The authors of the SPY dataset define PII as the entities that can identify, contact or locate a specific individual, and found that dedicated named-entity models have limits in PII contexts (Savkin et al., 2025). We use a PII model, one trained for personally identifiable information: OpenAI's Privacy Filter, an Apache-2.0 model with eight fixed labels, such as private_person, private_address and account_number (model card).
The labels are the model's own. OpenAI's card says its label policies can't be set while the model runs (OpenAI, 2026, section 4.2). We chose it over GLiNER2-PII, the runner-up among open PII models, for those fixed labels. GLiNER2-PII takes the labels to look for at inference time, along with a threshold (model card), so a recording of it would have to pin both. Privacy Filter has 1.5 billion parameters, 50 million of them active, and runs on a CPU.
The book's examples never run a model, so they read a recording of its output. The recording, data/recorded/pii-entities/, which this chapter and chapter 9 use, holds every span the model found in every ticket and review, with its offsets, its text and the model's label. recordings/pii_entities.py made it on a CPU with the model's reference decoder, opf (code), which chooses the best sequence of labels for a whole text, so most spans come back in one piece rather than token by token. It used the default operating point, with all six transition biases at 0 (calibration file); the model can also be set "for improved recall" (section 2.3.2). The manifest pins the model's revision and the SHA-256 of its weights, so anyone can record it again and compare, and lib.pii.recorded refuses spans that don't match the text at their offsets. The recording keeps the model's labels, and the example maps them to the truth's types, so a different mapping never needs a new recording. Score the model, the platform's patterns and both together, masking whatever either finds:
1"""Scores the recorded spans of OpenAI's Privacy Filter against the truth, alone and beside the platform's patterns. The2prose rests on the recall lines for names, addresses, phones and cards, the precision lines, and the texts left with3nothing personal."""4from lib import pii56# Privacy Filter's eight labels, mapped to the truth's entity types. A label the truth has no type for matches nothing.7MATCHES = {"private_person": {"PERSON"}, "private_email": {"EMAIL_ADDRESS"}, "private_phone": {"PHONE_NUMBER"},8 "private_address": {"LOCATION"}, "account_number": {"CREDIT_CARD", "IBAN_CODE"},9 "private_date": {"DATE_TIME"}, "private_url": set(), "secret": set()}1011model, manifest = pii.recorded("pii-entities") # the model's spans, from a recording: no model runs here12print("model:", manifest["model"], "decoded with", manifest["settings"]["decode_mode"])13labelled = pii.truth()14detectors = {"patterns": pii.find(pii.platform_detector()), "model": model}15# both lists joined: a span either one finds is masked, and a span both find is simply counted twice16detectors["both"] = detectors["patterns"] + model1718for name, found in detectors.items():19 scores = pii.score(found, labelled, MATCHES) # Presidio's labels aren't in MATCHES, so match themselves20 print(f"{name}, detections: {len(found):,}")21 for entity, (n, exact, overlap) in scores.recall.items():22 print(f"{name}, {entity}, recall, exact: {exact / n:.2f}")23 print(f"{name}, {entity}, recall, overlap: {overlap / n:.2f}")24 print(f"{name}, precision, exact: {pii.precision_overall(scores):.2f}")25 print(f"{name}, texts with every labelled span masked: {scores.masked:,} of {scores.texts:,}")2627# Precision by the model's own label: a label that's right for one truth type can still be wrong for another28for label, (n, exact, overlap) in pii.score(model, labelled, MATCHES).precision.items():29 print(f"model, {label}, detections: {n:,}")30 print(f"model, {label}, precision, exact: {exact / n:.2f}")31 print(f"model, {label}, precision, overlap: {overlap / n:.2f}")
python privacy-filter-spans.pymodel: openai/privacy-filter decoded with viterbi
patterns, detections: 1,539
patterns, CREDIT_CARD, recall, exact: 0.71
patterns, CREDIT_CARD, recall, overlap: 0.71
patterns, DATE_TIME, recall, exact: 1.00
patterns, DATE_TIME, recall, overlap: 1.00
patterns, EMAIL_ADDRESS, recall, exact: 1.00
patterns, EMAIL_ADDRESS, recall, overlap: 1.00
patterns, IBAN_CODE, recall, exact: 1.00
patterns, IBAN_CODE, recall, overlap: 1.00
patterns, LOCATION, recall, exact: 0.00
patterns, LOCATION, recall, overlap: 0.00
patterns, PERSON, recall, exact: 0.00
patterns, PERSON, recall, overlap: 0.00
patterns, PHONE_NUMBER, recall, exact: 0.84
patterns, PHONE_NUMBER, recall, overlap: 0.85
patterns, precision, exact: 0.97
patterns, texts with every labelled span masked: 405 of 2,586
model, detections: 4,424
model, CREDIT_CARD, recall, exact: 0.69
model, CREDIT_CARD, recall, overlap: 0.69
model, DATE_TIME, recall, exact: 0.97
model, DATE_TIME, recall, overlap: 0.97
model, EMAIL_ADDRESS, recall, exact: 1.00
model, EMAIL_ADDRESS, recall, overlap: 1.00
model, IBAN_CODE, recall, exact: 0.99
model, IBAN_CODE, recall, overlap: 1.00
model, LOCATION, recall, exact: 0.67
model, LOCATION, recall, overlap: 0.79
model, PERSON, recall, exact: 0.84
model, PERSON, recall, overlap: 0.84
model, PHONE_NUMBER, recall, exact: 0.94
model, PHONE_NUMBER, recall, overlap: 0.94
model, precision, exact: 0.84
model, texts with every labelled span masked: 2,095 of 2,586
both, detections: 5,963
both, CREDIT_CARD, recall, exact: 0.92
both, CREDIT_CARD, recall, overlap: 0.92
both, DATE_TIME, recall, exact: 1.00
both, DATE_TIME, recall, overlap: 1.00
both, EMAIL_ADDRESS, recall, exact: 1.00
both, EMAIL_ADDRESS, recall, overlap: 1.00
both, IBAN_CODE, recall, exact: 1.00
both, IBAN_CODE, recall, overlap: 1.00
both, LOCATION, recall, exact: 0.67
both, LOCATION, recall, overlap: 0.79
both, PERSON, recall, exact: 0.84
both, PERSON, recall, overlap: 0.84
both, PHONE_NUMBER, recall, exact: 0.99
both, PHONE_NUMBER, recall, overlap: 0.99
both, precision, exact: 0.88
both, texts with every labelled span masked: 2,120 of 2,586
model, account_number, detections: 746
model, account_number, precision, exact: 0.51
model, account_number, precision, overlap: 0.51
model, private_address, detections: 177
model, private_address, precision, exact: 0.79
model, private_address, precision, overlap: 1.00
model, private_date, detections: 384
model, private_date, precision, exact: 0.50
model, private_date, precision, overlap: 0.50
model, private_email, detections: 543
model, private_email, precision, exact: 1.00
model, private_email, precision, overlap: 1.00
model, private_person, detections: 2,110
model, private_person, precision, exact: 0.98
model, private_person, precision, overlap: 0.98
model, private_phone, detections: 464
model, private_phone, precision, exact: 0.89
model, private_phone, precision, overlap: 0.89
| Measure | Patterns | Model | Both |
|---|---|---|---|
Recall, exact, PERSON |
0.00 | 0.84 | 0.84 |
Recall, exact, LOCATION |
0.00 | 0.67 | 0.67 |
Recall, exact, PHONE_NUMBER |
0.84 | 0.94 | 0.99 |
Recall, exact, CREDIT_CARD |
0.71 | 0.69 | 0.92 |
Recall, exact, IBAN_CODE |
1.00 | 0.99 | 1.00 |
| Precision, exact | 0.97 | 0.84 | 0.88 |
| Texts fully masked | 405 of 2,586 | 2,095 of 2,586 | 2,120 of 2,586 |
The model finds most of the names and addresses the patterns can't, and the two detectors complement each other on card numbers. The model and the patterns find about the same share of cards, 0.69 and 0.71, but not the same cards, so together they find 0.92. A later section shows why the model's share is low.
The model pays in precision, 0.84 overall, and only 0.51 for account_number and 0.50 for private_date. That partly measures a difference in label policy. The truth leaves invoice numbers and dates of a problem unlabelled, and OpenAI's card says appropriate label policies can differ (section 4.2).
Together, the two detectors leave 2,120 of 2,586 texts with nothing personal in them, against 405 of 2,586 for the patterns alone.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Read the model's misses
OpenAI's card says where the model fails: it misses rare names, names from other naming traditions, initials, names wrapped in titles and domain-specific identifiers, and it over-masks public entities, locations and common nouns when the local context is unclear (OpenAI, 2026, section 4.3). Check that list against the hard cases the tickets plant:
1"""Counts how many of each hard case the model's recorded spans catch, and reads the names it misses by case and length.2The prose rests on the missed-name counts and on the first lowercase ticket that both detectors miss."""3from collections import Counter45from lib import pii67model, _ = pii.recorded("pii-entities")8labelled = pii.truth()9cases = pii.hard_cases(labelled)1011for case, spans in cases.items():12 print(f"{case}, labelled: {len(spans)}")13 print(f"{case}, caught by the model: {pii.caught(spans, model)}")14for kind, spans in pii.lookalikes().items():15 print(f"{kind}, flagged by the model: {pii.caught(spans, model)}")1617# The names the model misses, by case: read them before trusting a recall figure.18missed = pii.missed([s for s in labelled if s.entity == "PERSON"], model, {"private_person"})19print("names the model misses:", len(missed))20for case in ("first names that are words", "instructors named in reviews"):21 often = Counter(pii.text_of(s) for s in pii.missed(cases[case], model, {"private_person"})).most_common(3)22 print(f"{case}, missed most often:", ", ".join(name for name, _ in often))2324# A first name on its own says less than a full name: how many of the misses are one word?25print("missed names that are one word:", sum(len(pii.text_of(s).split()) == 1 for s in missed))26print("missed names of two words or more:", sum(len(pii.text_of(s).split()) > 1 for s in missed))27# islower() counts only names typed entirely in lowercase, so a name with a capital letter, such as "Jane", doesn't.28print("of them, typed in lowercase:", sum(len(pii.text_of(s).split()) > 1 and pii.text_of(s).islower() for s in missed))2930# A lowercase name that neither the model nor the platform's patterns cover any part of, in the first ticket with one.31# The model's spans and the patterns' together: a name is missed only when both miss it. The lowest ticket ID is taken,32# so the same ticket is printed on every run.33first = min(pii.missed(cases["names in lowercase"], model + pii.find(pii.platform_detector())), key=lambda s: s.row_id)34print("first ticket with a lowercase name both detectors miss:", first.row_id)35print("the name, as typed:", pii.text_of(first))
python privacy-filter-hard-cases.pynames in lowercase, labelled: 211
names in lowercase, caught by the model: 167
initial and surname, labelled: 163
initial and surname, caught by the model: 162
first names that are words, labelled: 187
first names that are words, caught by the model: 128
instructors named in reviews, labelled: 798
instructors named in reviews, caught by the model: 535
phones typed as digits only, labelled: 257
phones typed as digits only, caught by the model: 256
invoice numbers, flagged by the model: 224
order numbers, flagged by the model: 125
dates of a problem, flagged by the model: 188
names the model misses: 387
first names that are words, missed most often: Summer, Will, Hope
instructors named in reviews, missed most often: Marie, Lori, Augustin
missed names that are one word: 337
missed names of two words or more: 50
of them, typed in lowercase: 43
first ticket with a lowercase name both detectors miss: 19
the name, as typed: juan fonseca
The misses are first names that are also words, and an instructor's first name alone in a review. The model catches 128 of the 187 first names that are words, and 535 of the 798 instructors' names.
Initials aren't the problem: the model catches 162 of 163 initials with a surname, and lowercase names are mostly caught, 167 of 211.
Most often it misses the words Summer, Will, Hope and the instructors' names Marie, Lori, Augustin. The model also flags far more look-alikes than the tuned patterns: 224 invoice numbers, 125 order numbers and 188 dates of a problem, against 31, 0 and 0.
How much the misses matter depends on their length. Of the 387 names the model misses, 337 are a single word, a first name on its own, and 50 are longer, 43 of those typed in lowercase. The authors of the Text Anonymization Benchmark draw the same distinction: missing a direct identifier, such as a full name, is much more harmful than missing a quasi-identifier (Pilán et al., 2022). A first name alone still needs masking, since a review carries its learner's ID, and an instructor's first name in a review of their own course points at one person.
The card recommends testing the model in your own domain against local policy references before production use (section 4.5). It also lists education among the settings that need extra caution (section 4.4), and calls the model "a redaction and data minimization aid, not an anonymization, compliance, or a safety guarantee" (section 4.1).
Working rule: Mask free text with a PII model and the tuned patterns together, whatever either finds: the model for names and addresses, the patterns' checksums for card numbers and IBANs. Record the model's output with its revision, weights and operating point pinned, keep its own labels, and map them in code. Measure it on your own text, by hard case, before use and after every model change; if it misses too much, try an operating point that leans toward recall and record again. Treat masked text as reduced, never as anonymous.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Count what a wrong label costs
The comparison with the patterns scored the model by exact and overlap matching, which both need a label of the span's type. Masking doesn't read the label: a card number the model labels private_phone is still hidden. Classification does, because governance/pii.toml maps each entity type to a classification: a phone number to personal, a card number to sensitive. Measure the model's coverage beside its recall, and the labels it gives the platform's card numbers, phone numbers and addresses:
1"""Measures the model's coverage of card numbers, phone numbers and addresses whatever their label, and counts the labels2that would misclassify them. The prose rests on the coverage lines, the mislabelled counts and the uncovered addresses."""3from lib import pii45# What the model's labels cost. A span is covered when every character of it is inside some detection, whatever its6# label: that's what masking needs. Exact and overlap matching also need a label of the span's type.7model, _ = pii.recorded("pii-entities")8labelled = pii.truth()9covered = pii.coverage(model, labelled)10for entity in ("CREDIT_CARD", "PHONE_NUMBER", "LOCATION"):11 n, fully = covered[entity]12 print(f"{entity}, covered by the model, any label: {fully / n:.2f}")1314spans = {entity: [s for s in labelled if s.entity == entity] for entity in ("CREDIT_CARD", "PHONE_NUMBER", "LOCATION")}15print("card numbers labelled private_phone:", pii.caught(spans["CREDIT_CARD"], model, {"private_phone"}))16print("phone numbers labelled account_number:", pii.caught(spans["PHONE_NUMBER"], model, {"account_number"}))1718# For each address, the labels of the detections that overlap it, and the characters no detection covers.19on = pii.labels_on(spans["LOCATION"], model)20print("addresses with no detection at all:", sum(not labels for labels in on))21print("addresses read, at least in part, as a person:", sum("private_person" in labels for labels in on))22# A person label and no address label: the address was read as a name, which overlap matching can't credit.23print("of them, with no address label:", sum({"private_person"} <= labels and "private_address" not in labels for labels in on))24left = [pii.uncovered(address, model) for address in spans["LOCATION"]]25print("addresses not fully covered:", sum(bool(chars) for chars in left))26print("of them, with only spaces and punctuation left:", sum(bool(chars) and not any(c.isalnum() for c in chars) for chars in left))
python privacy-filter-labels.pyCREDIT_CARD, covered by the model, any label: 0.87
PHONE_NUMBER, covered by the model, any label: 1.00
LOCATION, covered by the model, any label: 0.67
card numbers labelled private_phone: 40
phone numbers labelled account_number: 27
addresses with no detection at all: 24
addresses read, at least in part, as a person: 28
of them, with no address label: 20
addresses not fully covered: 70
of them, with only spaces and punctuation left: 21
A wrong label costs classification more than it costs masking. The model covers 0.87 of card numbers, but labels 40 of the 213 as phone numbers, so its recall for cards is only 0.69. If the platform classified columns from the model's labels, a column of those card numbers would be proposed personal, not sensitive. That's what the patterns add: their Luhn check keeps card numbers labelled as cards.
The mistake also runs the other way, toward the stricter side. The model labels 27 phone numbers account_number, which maps to card numbers and IBANs, so its phone coverage is 1.00 and its recall 0.94.
Addresses show what each measure counts. The model's exact recall for them is 0.67 and its overlap recall 0.79, and its coverage is 0.67. The model finds nothing at all in 24 addresses.
The model reads part of 28 addresses as a person, since Faker builds many street names from surnames, as the platform's data card notes. It gives 20 of those no address label at all, so overlap matching misses them too. Other addresses come back in pieces with the right label, such as a street and a town split at a comma. Overlap matching credits those, and coverage doesn't. Of the 70 addresses not fully covered, 21 have every letter and digit masked and only a comma or a space left. Coverage is the measure masking needs, and it's strict.
Working rule: Report coverage, whatever the label, beside recall with the right label: masking needs the first, classification the second. Read which labels a detector gives each type of span, and keep a recognizer with a checksum for every identifier whose label decides a classification, such as card numbers.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
Classify again when the schema changes
A classification describes a column at one moment, and chapter 2 showed how schemas move under it. So the question is when to classify again, not on every schema ID: in PyIceberg, editing a column's description gives the schema a new ID with no new column. What needs classifying is a field ID, the number Iceberg gives each column, that the metadata has never seen. Iceberg gives every added column a new one, and a renamed column keeps its ID and so its reviewed classification. Values can also change without a schema change, so the samples of text columns need a rerun on a schedule too. Vendors promise this as a service: Databricks says its classification scans incrementally and classifies new data with no manual setup (Databricks docs), a claim we don't test.
The support team wants a number to call learners back on for tickets opened by phone, so it adds a column called callback_number to support.tickets. The change comes in four steps: edit a description, add the column, fill it from the learners' phone numbers, then classify only the field IDs that governance/metadata.toml lacks:
1"""Edits the tickets table twice, first a column's description and then a new column, and after each change counts the field2IDs the metadata lacks. The prose rests on the schema IDs, the field counts and the proposal for the new column."""3from pyiceberg.types import StringType45from lib import metadata, pii6from lib.catalog import fill_column, open_catalog7from lib.platform import load8from lib.warehouse import connect910catalog = open_catalog(["support.tickets"])11tickets = catalog.load_table("support.tickets")12reviewed = metadata.load()["support.tickets"].columns # field ID -> the column as people reviewed it13# A field ID the reviewed metadata doesn't have: Iceberg gives each added column a new one.14unclassified = lambda: {f.field_id for f in tickets.schema().fields if f.field_id not in reviewed}15print("schema ID at the start:", tickets.schema().schema_id)1617with tickets.update_schema() as update: # a doc edit: a new schema, no new column18 update.update_column("body", doc="What the learner wrote, as they wrote it.")19print("schema ID after a doc edit:", tickets.schema().schema_id)20print("fields to classify after the doc edit:", len(unclassified()))2122# The support team starts storing the number to call back for tickets opened by phone: the learner's phone number.23with tickets.update_schema() as update:24 update.add_column("callback_number", StringType(), doc="Where to call the learner back.")25phone_of = {row["learner_id"]: row["phone"] for row in load("crm.learners").to_pylist()}26# tickets from other channels have no callback number, so their value is None27fill_column(tickets, "callback_number", lambda row: phone_of[row["learner_id"]] if row["channel"] == "phone" else None)28print("schema ID after adding a column:", tickets.schema().schema_id)29print("fields to classify after adding a column:", len(unclassified()))3031# only= keeps the proposals to the field IDs the metadata lacks.32for f in pii.classify(catalog, connect(catalog), pii.platform_detector(), only={"support.tickets": unclassified()}):33 found = ", ".join(f"{entity} in {n} of {f.sampled}" for entity, n in f.found.items())34 print(f"{f.table}.{f.column}, field ID: {f.field_id}")35 print(f"{f.table}.{f.column}, entity its name suggests: {f.named}")36 print(f"{f.table}.{f.column}, found in its sampled values: {found}")37 print(f"{f.table}.{f.column}, proposed classification: {f.classification}")
python reclassify-new-columns.pyschema ID at the start: 0
schema ID after a doc edit: 1
fields to classify after the doc edit: 0
schema ID after adding a column: 2
fields to classify after adding a column: 1
support.tickets.callback_number, field ID: 9
support.tickets.callback_number, entity its name suggests: None
support.tickets.callback_number, found in its sampled values: PHONE_NUMBER in 154 of 173
support.tickets.callback_number, proposed classification: personal
A doc edit moves the schema to a new ID and leaves 0 fields to classify, since it adds no column. Adding the column moves the schema to 2, with 1 field to classify: field 9.
No name rule matches callback_number, since it contains none of the words the phone rule looks for, which is how a name rule misses a column nobody wrote it for. Its values settle it. The sample finds PHONE_NUMBER in 154 of 173, and the proposal is personal.
The pull request that adds the column is where its classification belongs, beside the metadata entry it needs: chapter 10 fails that pull request until the column has one. A proposal from values needs a column that holds values, so this one comes from the scheduled sample.
Working rule: Classify every field ID the metadata doesn't have, in the pull request that adds it, and don't let it merge without a reviewed classification. Key the trigger on field IDs, never on schema IDs. Sample every text column again on a schedule too, because values change without a schema change.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
What goes wrong in production
A detector's worst failure is a miss, and its output always looks like a list of findings: what it misses never appears in it:
- Phone numbers read as bank numbers. With Presidio's defaults, every run of 8 to 17 digits is a US bank number, and so is every phone number typed without separators. The evidence for the ticket table lists 689 bank numbers that aren't there, a report of "bank numbers in tickets" sends someone to the wrong team, and mapping that entity to a classification would raise columns for the wrong reason. Keep only the recognizers your text needs, each with its own threshold, as the tuning section does, and count detections per label: a label with hundreds of hits and no labelled spans is a false positive.
- Names in free text missed. Patterns find no names, and the model misses some lowercase names, first names that are also words, and first names standing alone. Ticket 19 still says "this is juan fonseca, i still can't log in" after both detectors have run, and passes every check that counts detections. You notice only by measuring recall against labelled text, per hard case, and by reading a sample of masked output.
- Personal data in a column called
notes. No name rule marks it, and a reviewer who never read it classifies itinternal. Sample every text column's values, and treat free-text columns as personal until a sample says otherwise. - A column more sensitive than its metadata. Learners paste card numbers into tickets, so
support.tickets.bodyholdssensitivedata under apersonalclassification. Compare every proposal with the reviewed metadata, and route a stricter one to the table's owner. - A detector that reaches the network.
AnalyzerEngine()downloads a model, and tldextract fetches the Public Suffix List. Offline, the pipeline fails; online, the same text can be classified differently on two machines. A setting read at import holds only if it's made before anything imports the library. Configure the engine, pin what it would fetch, check that the setting held, and run the detector's tests with the network off. - Truth shaped by its generator. The platform's text comes from 30 sentence templates filled by Faker, which writes every date of birth one way, phone numbers that aren't valid where the learner lives, and street names built from people's names. Read the misses before trusting a recall figure, and measure again on labelled samples of your own text.
이 부분은 아직 번역되지 않았으므로 영어로 표시된다.
What this chapter decided
- Columns are classified from their name, their type and their values. Name rules run on every column; text columns are sampled; the strictest evidence wins, with
internalas the floor, and only a person setspublic. - Samples are reproducible: 200 non-null values per text column, by a seeded reservoir sample on one DuckDB thread.
- Presidio runs offline, on
NoOpNlpEngine, with 6 recognizers, each with its own threshold, phone regions read from the learners' countries, and tldextract's own copy of the Public Suffix List, checked whenlib.piiloads. - Detectors are measured against labelled spans: recall per entity type by exact and overlap matching with a label of the right type, coverage whatever the label, precision per label and texts left with nothing personal, with recall weighed first.
- A proposal stricter than the metadata is a finding:
support.tickets.bodygoes toteam-supportassensitive, andbilling.payments.card_numbertoteam-billing, with the evidence that it holds full card numbers. - Free text is masked with two detectors: OpenAI's Privacy Filter, recorded at a pinned revision and operating point, for names and addresses, and the tuned patterns for card numbers, IBANs and phone numbers, whose checksums keep card numbers labelled as cards. Masking hides whatever either finds; neither is trusted to make text anonymous.
- No example prints a whole phone number, card number or IBAN:
lib.pii.maskedkeeps the last four digits. - Reclassification follows field IDs: every field the metadata lacks is classified in the pull request that adds it, and text columns are sampled again on a schedule. A detector is only as good as the measurement behind it, and what it misses never appears in its output. The measurement is what tells a team which of its classifications it can rely on, and which it has to check by hand.
이 판의 이 장
2026.11.31판예제 통과 2026년 11월 19이 장에서 11개 예제개 중 11개가 통과한다
영어판 2026.11.31판을 번역한 것이다.