어휘 검색

읽는 데 약 66분분 걸린다, 12절에 13개 예제이(가) 있음.

2026.11.10판 예제 통과 2026년 11월 19 132개 예제 중 132개 통과 변경 사항

AI로 번역함. 영어 원문을 사람이 검증함. 우리가 책을 만드는 방식

영어판 2026.11.10판을 번역한 것이다. 원문 읽기

이 장을 마치면 어휘 검색을 처음부터 만들 수 있다. 상품 텍스트를 용어로 바꾸고, 용어를 색인하고, TF-IDF와 BM25로 상품에 점수를 매겨 bm25s 라이브러리의 점수와 하나하나 일치시킨다. 또한 텍스트 처리와 BM25 매개변수에서 어떤 변경을 유지할 가치가 있는지, 학습 질의로 선택하고 테스트 질의로 확인하여 판단할 수 있다.

어휘 검색(lexical retrieval)은 질의와 공유하는 단어를 기준으로 상품을 찾고 순위를 매긴다. 쇼핑객이 상점의 가구와 생활용품을 검색할 때 결과의 순위를 정하는 것은 bm25s 라이브러리를 통한 BM25이다. 쇼핑객이 7qt slow cooker 또는 pool floats를 입력하면 1장의 BM25가 내놓는 결과는 부실하다. 테스트 질의에서의 평균 nDCG@10인 0.725이 그 사실을 보여 주지만 이유까지 알려 주지는 않는다. 1장에서는 BM25를 bm25s의 기본값으로 쓰는 블랙박스로 다루었기 때문이다. 이 장에서는 그 상자를 연다. 상점의 상품 12,000개를 대상으로 각 단계를 직접 다시 구현하여 bm25s와 대조하고, 이어서 한 번에 한 단계씩 바꾸면서 각 변경을 상점의 판정에 비추어 측정한다.

각 변경은 50개의 학습 질의로 선택하고 30개의 테스트 질의로 확인한다. 테스트 결과를 읽기 전에, 이 장은 테스트 질의에서 수행할 비교를 미리 정해 둔다. 비교는 다섯 가지이며, 각각 변경이 없는 같은 검색과 대조한다.

  1. 어간 추출
  2. 학습에서 가장 좋은 k1와 b를 bm25s의 기본값과 비교(튜닝)
  3. 학습에서 가장 좋은 상품명 가중치(필드)
  4. 필드로서의 상품 속성(속성)
  5. 학습에서 가장 좋은 구문 가산점(구문)

계획된 비교가 다섯 가지이므로 이 책의 규칙에 따라 기준을 p-값 0.01로 정한다. 대응 순열 검정의 p-값이 이 기준보다 작으면 향상이 입증된 것으로 본다. 각 비교는 대응 t-검정과 99% 부트스트랩 구간도 함께 보고하며, 각각이 순열 검정과 일치하는지도 밝힌다.

제품 텍스트를 용어로 나누기

마지막으로 변경된 판: 2026.11.8판

검색은 원시 문자열을 그대로 비교하지 않는다. 질의와 모든 제품의 텍스트는 먼저 같은 단계를 거친다. 텍스트를 조각으로 나누면 토큰이 된다. 소문자로 바꾸면 Chair와 chair가 하나의 토큰이 된다. the나 with처럼 매우 흔한 단어인 불용어를 제거하면 검색이 비교하는 용어가 남고, 카탈로그에 있는 모든 용어가 어휘를 이룬다. 제품은 질의와 공유하는 용어로만 질의와 일치할 수 있으므로, 이 단계들은 점수 계산이 시작되기 전에 무엇이 일치할 수 있는지를 결정한다.

스토어의 BM25를 실행하는 bm25s는 여기서 세 가지를 선택한다. 첫째, 소문자로 바꾼다. 둘째, 글자, 숫자, 밑줄로 이루어진 단어 문자가 두 개 이상 이어진 구간을 모두 유지하며, 이때 정규식 (?u)\b\w\w+\b을 사용한다. bm25s의 docstring은 이를 scikit-learn의 CountVectorizer 방식이라고 설명한다. 셋째, 영어 불용어 33개를 제거하며, 여기에는 no, not, with, for가 들어 있다. 어간 추출기를 넘겨주지 않으면 어간 추출은 하지 않는다(bm25s tokenizer, stop words). 단계를 하나씩 살펴보려면 직접 손으로 작성하고, 모든 제품에 대해 그 결과를 bm25s.tokenize와 비교하고, 학습 질의 7qt slow cooker를 거치게 해 본다.

tokenize-product-text.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1 통과
"""Splits product text into terms by hand, checks the split against bm25s on every product, then shows what the store's sizes and a negation turn into. The prose's point rests on the line that counts products tokenized differently, and on the lines that count products with the term 7qt and products whose text says 7 qt."""import reimport bm25sfrom bm25s.stopwords import STOPWORDS_ENfrom lib.lexical import LexicalSearchfrom lib.store import load_products, load_queries, product_text, qrels# two or more word characters (letters, digits or underscores): a single character, such as the 7 in 7 qt, is no tokenTOKEN = re.compile(r"(?u)\b\w\w+\b")STOPWORDS = set(STOPWORDS_EN)               # the 33 English stop words bm25s removesdef tokenize(text, pattern=TOKEN):    """Lowercase, split into tokens, and drop stop words."""    return [token for token in pattern.findall(text.lower()) if token not in STOPWORDS]products, queries = load_products(), load_queries()texts = product_text(products).tolist()by_hand = [tokenize(text) for text in texts]# bm25s's own split, with stopwords="en": the same 33 words as STOPWORDS abovewith_bm25s = bm25s.tokenize(texts, stopwords="en", return_ids=False, show_progress=False)print("products tokenized differently:", sum(mine != theirs for mine, theirs in zip(by_hand, with_bm25s)))print()query = queries.at[62, "query"]                                    # a train queryprint("query 62:", query, "->", tokenize(query))print("product 1:", products.at[1, "name"], "->", tokenize(products.at[1, "name"]))print("products with the term 7qt:", sum("7qt" in terms for terms in by_hand))# a 7 on its own: the lookbehind skips 27 qt and 3.7 qt, which are other sizesprint("products whose text says 7 qt:", sum(re.search(r"(?<![\d.])7 qt\b", text.lower()) is not None for text in texts))grades = qrels("train")[62]print("exact products judged for query 62:", sum(grade == 2 for grade in grades.values()))top_10 = LexicalSearch(products).search(query, k=10).index                # chapter 1's BM25print("exact products in bm25's top 10:", sum(grades.get(product_id) == 2 for product_id in top_10))digits_apart = re.compile(r"(?u)\d+|[^\W\d_]+")                    # keeps single digits; 7qt -> 7, qtchanged = queries["query"].map(lambda text: tokenize(text) != tokenize(text, digits_apart))for split in ["train", "test"]:    print(f"{split} queries whose terms change with digits apart:", changed[queries["split"] == split].sum())print()print("probe:", "rug not wool", "->", tokenize("rug not wool"))      # written for this chapter# whole words, so note and nothing don't countprint("queries with not or no:", queries["query"].str.contains(r"\b(?:not|no)\b").sum())
출력python tokenize-product-text.py
products tokenized differently: 0

query 62: 7qt slow cooker -> ['7qt', 'slow', 'cooker']
product 1: all-clad 7 qt . slow cooker -> ['all', 'clad', 'qt', 'slow', 'cooker']
products with the term 7qt: 0
products whose text says 7 qt: 8
exact products judged for query 62: 9
exact products in bm25's top 10: 1
train queries whose terms change with digits apart: 4
test queries whose terms change with digits apart: 0

probe: rug not wool -> ['rug', 'wool']
queries with not or no: 0

직접 작성한 토크나이저와 bm25s가 다르게 나눈 제품의 수는 0이므로, 문제는 둘 중 어느 하나에 있지 않고 둘이 공유하는 규칙에 있다. 7qt slow cooker를 입력하는 쇼핑객은 7쿼트 조리기를 원하며, 7qt은 실제로 하나의 토큰으로 남는다. 그러나 이 토큰과 일치하는 제품은 0개이다. 제품 텍스트는 크기를 띄어 써서 8개 제품에서 7 qt로 적고, 토크나이저는 한 글자인 7을 버리기 때문이다. 그래서 크기는 일치의 양쪽 어디에서도 순위에 아무 역할을 하지 못하고, BM25는 7쿼트 조리기를 다른 슬로 쿠커와 구별하지 못한다. 심사자가 이 질의에 exact로 표시한 제품 9개 가운데 BM25의 상위 10개에 들어 있는 것은 1개이다.

한 자리 숫자를 유지하고 숫자를 단위와 분리하는 패턴 (?u)\d+|[^\W\d_]+은 학습 질의 4개와 테스트 질의 0개의 용어를 바꾸므로, 스토어의 판정으로는 이 패턴에 대해 말할 수 있는 것이 거의 없다.

불용어 목록도 같은 방식으로 실패하며, bm25s는 오류를 내지 않는다. 탐침 질의인 rug not wool를 예로 들어 보자. 탐침 질의는 스토어의 질의가 아니라 어떤 동작을 시험하려고 이 장을 위해 작성한 질의이다. 이 질의는 rug wool가 되므로, 양모 러그는 두 용어 모두와 일치한다. 판정으로는 부정이 들어 있는 질의에서만 이 피해를 측정할 수 있는데, 스토어의 질의 중 not 또는 no을 포함한 것은 0개이다. 검색 시스템은 긴 불용어 목록에서 짧은 목록이나 목록 없음으로 옮겨 왔다. 일부 질의에서는 불용어가 의미를 담고 있기 때문이기도 하고, 불용어를 유지하는 비용이 낮아졌기 때문이기도 하다(Manning et al., 2008).

작업 규칙: 점수 계산을 조정하기 전에 질의와 제품이 어떤 용어가 되는지 출력해 보고, 둘을 같은 토크나이저로 처리한다. 스토어처럼 bm25s의 기본값에서 시작한다. 쇼핑객이 크기, 수량, 모델 번호를 입력한다면 위의 패턴처럼 한 자리 숫자를 유지하고 숫자를 단위와 분리하는 토큰 패턴을 사용한다. 부정어를 입력한다면 불용어 목록을 끈다. 각각은 검색을 바꾸는 일이므로, 유지하기 전에 학습 질의로 측정하고 테스트 질의로 확인한다.

역색인 만들기

마지막으로 변경된 판: 2026.11.10판

이 용어들로 이루어진 질의를 채점하려면, 검색은 각 용어를 포함하는 제품을 알아야 한다. 질의마다 모든 제품의 텍스트를 읽는 방식은 작은 표본에서는 통하지만, 수백만 개의 제품을 담은 카탈로그에서는 통하지 않는다. 역색인(inverted index)은 텍스트를 뒤집어, 각 용어마다 그 용어를 포함하는 제품의 목록을 보관한다. 이 목록을 그 용어의 포스팅(postings)이라고 한다(Manning et al., 2008). 포스팅은 각 제품과 함께 용어 빈도(term frequency), 즉 그 용어가 해당 제품의 텍스트에 나타나는 횟수를 저장한다. 한 용어의 포스팅에 들어 있는 제품의 수는 그 용어의 문서 빈도(document frequency)이다. 검색 엔진은 검색 대상 항목을 문서라고 부르며, 이 책에서 문서는 제품이다. 질의는 자신이 가진 용어의 포스팅만 읽는다.

스토어의 색인은 각 용어를 제품 ID와 용어 빈도의 사전에 대응시킨다. 또한 BM25에 필요한 제품별 길이를 용어 수로 함께 보관한다. 색인을 만들려면 앞 절의 토크나이저로 모든 제품의 텍스트를 한 번 훑으면 된다. 그런 다음 코드는 학습 질의 7qt slow cooker의 용어를 찾아본다.

inverted-index-by-hand.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Builds the store's inverted index by hand from each product's tokenized text, then looks up a train query's terms in it. The prose's point rests on the lines that count products with every query term and with any query term, and on the line for the product code."""from collections import Counterfrom lib.lexical_by_hand import tokenizefrom lib.store import load_products, load_queries, product_textpostings = {}       # {term: {product_id: how many times the term occurs in that product}}lengths = {}        # {product_id: number of terms}for product_id, text in product_text(load_products()).items():    terms = tokenize(text)    lengths[product_id] = len(terms)    for term, count in Counter(terms).items():        postings.setdefault(term, {})[product_id] = countprint(f"vocabulary: {len(postings):,} terms")print(f"average product length: {sum(lengths.values()) / len(lengths):.1f} terms")# (product ID, term frequency) pairs, starting from product 1print("postings for cooker, first 3:", list(postings["cooker"].items())[:3])print()query = load_queries().at[62, "query"]for term in tokenize(query):    print(f"products with {term}:", len(postings.get(term, {})))found = [set(postings.get(term, {})) for term in tokenize(query)]print("products with any query term:", len(set.union(*found)))print("products with every query term:", len(set.intersection(*found)))print("products with w003096809:", len(postings.get("w003096809", {})))    # the code in query 205
출력python inverted-index-by-hand.py
vocabulary: 20,393 terms
average product length: 50.3 terms
postings for cooker, first 3: [(1, 3), (2, 2), (115, 1)]

products with 7qt: 0
products with slow: 157
products with cooker: 141
products with any query term: 158
products with every query term: 0
products with w003096809: 0

스토어의 제품 12,000개는 20,393 terms을 사용하며, 제품 텍스트의 평균 길이는 50.3 terms이다. cooker의 포스팅에 있는 각 쌍은 제품 ID와 용어 빈도이다. 첫 번째는 토크나이저 절의 7 qt 슬로 쿠커인 제품 1이다.

7qt slow cooker의 경우 포스팅에는 slow가 있는 제품이 157개, cooker가 있는 제품이 141개, 7qt이 있는 제품이 0개 나열되며, 이들을 합하면 158개의 제품을 포괄한다. 이 질의를 채점할 때 읽어야 하는 것은 이 제품들뿐이다. 세 용어를 모두 가진 제품은 0개인데, 7qt을 가진 제품이 없기 때문이다. 따라서 질의의 모든 용어를 포함하는 제품만 반환하는 불리언 AND 질의는 이 질의에 아무것도 반환하지 않는다. BM25처럼 용어 중 하나라도 포함하는 모든 제품을 채점하는 검색은 여전히 슬로 쿠커를 반환하며, 어느 것을 먼저 보일지는 점수가 정한다.

작업 규칙: 질의의 결과가 이상해 보이면 채점 방식을 바꾸기 전에 색인에서 그 질의의 용어를 먼저 찾아보라. 7qt처럼 포스팅이 없는 용어는 채점기가 무엇이든 어떤 제품에도 도움이 되지 못하므로, 대신 토크나이저나 질의를 고쳐야 한다. 쇼핑객이 입력하는 질의에서는 용어를 모두 포함하는 제품이 아니라 하나라도 포함하는 제품을 일치시켜라. 그렇지 않으면 일치하지 않는 단어 하나 때문에 결과가 비어 버리기 때문이다. 서비스는 라이브러리의 색인으로 제공하고, 이와 같은 작은 색인은 용어가 무엇과 일치하는지 확인하는 용도로 따로 두어라.

TF-IDF로 용어에 가중치 주기

마지막으로 변경된 판: 2026.11.10판

모든 일치가 같은 가치를 지니지는 않는다. 가중치를 매기는 데에는 두 가지 아이디어가 쓰인다. 하나는 어떤 용어가 상품 텍스트에 자주 나타날수록 그 상품이 그 용어에 관한 것일 가능성이 높다는 것으로, 이것이 용어 빈도(term frequency, tf)이다. 다른 하나는 드문 용어에서의 일치가 흔한 용어에서의 일치보다 더 많은 것을 말해 준다는 것이다. Karen Spärck Jones는 용어를 포함하는 문서의 수에 따라 가중치를 매기자고 제안했으며, 그 이유는 "덜 빈번하고 더 구체적인 용어에서의 일치가 빈번한 용어에서의 일치보다 더 큰 가치를 지니기" 때문이다(Spärck Jones, 1972; 무료 재인쇄본). 이 가중치가 역문서 빈도(inverse document frequency, IDF)가 되었다. IDF는 상품 수를 용어의 문서 빈도로 나눈 값의 로그이며, 드문 용어에서는 높고 흔한 용어에서는 낮다(Manning et al., 2008). TF-IDF는 각 질의 용어의 tf와 IDF를 곱한 뒤 그 값을 모두 더한다(Manning et al., 2008).

idf(t)       = ln(N / df(t))                        N products, df(t) of them contain t
tf-idf(q, d) = sum over query terms t of  tf(t, d) × idf(t)

furniture은(는) 이 상점의 상품에서 흔한 용어이다. TF-IDF는 역색인을 기반으로 한다. 코드는 furniture의 IDF와 train 질의 laundry basket with wheels에 포함된 각 용어의 IDF를 보여 주고, 이어서 그 질의에 대한 TF-IDF의 상위 3개 결과를 각 상품의 점수 및 용어 빈도와 함께 보여 준다. 마지막으로 50개의 train 질의 전체를 TF-IDF와 1장에서 실행한 BM25로 채점한다.

tf-idf-by-hand.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Computes TF-IDF over the inverted index, shows the IDF of each term of one train query and TF-IDF's top 3 for it, then scores every train query with TF-IDF and with BM25. The prose's point rests on the train nDCG@10 and judged@10 lines, and on the line that counts the queries where BM25 scores higher or lower."""import mathimport pandas as pdfrom lib.lexical import LexicalSearchfrom lib.lexical_by_hand import InvertedIndex, tokenizefrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, product_text, qrelsproducts, train, judgments = load_products(), load_queries("train"), qrels("train")index = InvertedIndex(product_text(products))n = len(index.lengths)                                      # every product, the N in the IDF formuladef tf_idf(query):    """TF-IDF scores of the products that contain a query term, best first."""    scores = {}    for term in tokenize(query):        postings = index.postings.get(term, {})        # a term no product contains has no postings, so it adds nothing to any score        idf = math.log(n / len(postings)) if postings else 0.0        for product_id, tf in postings.items():            scores[product_id] = scores.get(product_id, 0.0) + tf * idf    # stable: tied products keep the order in which they were first scored    return pd.Series(scores, dtype=float).sort_values(ascending=False, kind="stable")# train query 36, by its ID; judgments grade 2, 1 and 0 as exact, partial and irrelevantquery, labels = train.at[36, "query"], {2: "exact", 1: "partial", 0: "irrelevant"}for term in ["furniture"] + tokenize(query):    print(f"idf of {term}: {math.log(n / len(index.postings[term])):.3f}")print(f"\ntf-idf top 3 for {query!r}:")for rank, (product_id, score) in enumerate(tf_idf(query).head(3).items(), start=1):    counts = ", ".join(f"{term} {index.postings[term].get(product_id, 0)}" for term in tokenize(query))    print(f"  {rank}  {score:4.1f}  {labels[judgments[36][product_id]]:<8} {counts:<29} {products.at[product_id, 'name'][:22]}")print()# the top 100 of each scorer on every train query; bm25 is chapter 1's, from lib.lexicalruns = {"tf-idf": {query_id: list(tf_idf(text).index[:100]) for query_id, text in train["query"].items()},        "bm25": LexicalSearch(products).run(train)}results = {name: evaluate(run, judgments) for name, run in runs.items()}        # one row per train queryfor name, frame in results.items():    print(f"{name} train ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{name} train judged@10: {frame['judged@10'].mean():.3f}")# rounded to three decimals, so a difference too small to show counts as a tiechange = (results["bm25"]["ndcg@10"] - results["tf-idf"]["ndcg@10"]).round(3)print(f"train queries where bm25 scores higher, lower: {(change > 0).sum()}, {(change < 0).sum()}")
출력python tf-idf-by-hand.py
idf of furniture: 0.919
idf of laundry: 3.985
idf of basket: 3.925
idf of wheels: 4.839

tf-idf top 3 for 'laundry basket with wheels':
  1  51.3  partial  laundry 4, basket 9, wheels 0 dirty clothes basket ,
  2  45.4  exact    laundry 6, basket 3, wheels 2 laundry sorter basket
  3  39.7  partial  laundry 7, basket 3, wheels 0 weave laundry basket

tf-idf train ndcg@10: 0.579
tf-idf train judged@10: 0.836
bm25 train ndcg@10: 0.695
bm25 train judged@10: 0.930
train queries where bm25 scores higher, lower: 36, 10

TF-IDF의 첫 번째 상품은 부분 일치이다. 이 바구니는 wheels을(를) 전혀 언급하지 않지만 basket을(를) 그 아래의 두 상품보다 더 많이 반복한다. IDF가 이런 결과를 만든다. furniture의 IDF는 0.919이고, 질의 용어의 IDF는 laundry이(가) 3.985, basket이(가) 3.925, wheels이(가) 4.839로, 더 드물고 서로 비슷하다. IDF가 이처럼 비슷하면 점수에서 보듯 반복 횟수가 wheels의 부재를 압도한다. 세 상품 중 wheels을(를) 언급하는 유일한 상품인 정확히 일치하는 상품은 두 번째에 놓인다. TF-IDF에서는 용어 빈도에 상한이 없으므로, 어떤 상품은 단어를 반복하는 것만으로 다른 상품보다 높은 점수를 받을 수 있다.

train 질의 전체에서 TF-IDF의 평균 nDCG@10은 0.579이고 BM25는 0.695이며, BM25의 점수가 더 높은 질의와 더 낮은 질의는 각각 36, 10개이다. TF-IDF의 judged@10은 0.836이고 BM25는 0.930이다. 어떤 실행의 judged@10이 더 낮으면, 그 격차의 일부는 더 나쁜 상품 때문이 아니라 아무도 판정하지 않은 상품 때문일 수 있다. train 질의에서의 이 비교는 어떤 채점기를 기반으로 삼을지 알려 주며, 이 책은 그 근거를 test 질의를 위해 남겨 둔다.

작업 규칙: 어떤 어휘 기반 채점기에서든 IDF로 용어에 가중치를 매긴다. 드문 용어에서의 일치가 더 나은 증거이기 때문이다. 원시 TF-IDF로 상품의 순위를 매기지는 않는다. 반복된 용어가 더할 수 있는 값에 상한을 두고 긴 텍스트를 보정해야 하며, BM25가 바로 이를 수행한다. TF-IDF는 각 용어의 tf와 idf를 읽어 낼 수 있는 순위 설명용으로 남겨 둔다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Rebuild BM25 by hand to match bm25s

마지막으로 변경된 판: 2026.11.8판

BM25 keeps IDF and changes how term frequency counts, in two ways. With saturation, each repeat of a term adds less than the one before, and no number of repeats can push the term's contribution past a ceiling (Robertson and Zaragoza, 2009). The parameter k1 sets how quickly a term approaches it. With length normalization, a long text, which has more chances to contain any term, gets its term frequencies discounted. The parameter b sets how much: "b = 1 will perform full document-length normalisation, while b = 0 will switch normalisation off" (Robertson and Zaragoza). In the form bm25s computes by default (bm25s scoring):

idf(t)     = ln(1 + (N − df(t) + 0.5) / (df(t) + 0.5))
bm25(q, d) = sum over query terms t of
             idf(t) × tf(t, d) / (tf(t, d) + k1 × (1 − b + b × length(d) / average length))

Two details differ between textbooks and libraries. A common variant multiplies the fraction by k1 + 1, which grows every score by the same factor and so leaves the ranking unchanged (Robertson and Zaragoza, section 3.5.1). And Robertson's original IDF, without the 1 inside the logarithm, is negative for a term in more than half of the documents, so Lucene, an open-source search library, adds the 1 (Kamphuis et al., 2020). bm25s's default, method="lucene", is Lucene's formula computed with exact document lengths (bm25s README), where Lucene itself approximates them (Kamphuis et al.). "BM25" names a family of such variants, and a study of eight of them found no significant differences in effectiveness on three newswire collections (Kamphuis et al.).

Write the formula over the store's inverted index at bm25s's defaults, k1=1.5 and b=0.75, and check it against bm25s 0.3.16: first every product's score for each of the 80 queries, then the two rankings, by their top 100s and by nDCG@10:

bm25-by-hand.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Writes BM25 by hand over the inverted index at bm25s's defaults, and checks every score and every ranking against bm25s. The prose's point rests on the IDF lines for the word your, the largest score difference, the same-top-100 line and the test nDCG@10 lines."""import mathimport pandas as pdfrom lib.lexical import LexicalSearchfrom lib.lexical_by_hand import InvertedIndex, tokenizefrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, product_text, qrelsproducts, queries = load_products(), load_queries()index, library = InvertedIndex(product_text(products)), LexicalSearch(products)  # by hand, and bm25s as chapter 1 uses it# k1 and b at bm25s's defaultsn, k1, b = len(index.lengths), 1.5, 0.75def bm25(query):    scores = {}    for term in tokenize(query):        df = len(index.postings.get(term, {}))        # Lucene's IDF: the 1 in the logarithm keeps it positive, even for a term in most products        idf = math.log(1 + (n - df + 0.5) / (df + 0.5))        for product_id, tf in index.postings.get(term, {}).items():            # length normalization: a product of average length gets a factor of 1            length = 1 - b + b * index.lengths[product_id] / index.average_length            scores[product_id] = scores.get(product_id, 0.0) + idf * tf / (tf + k1 * length)    return pd.Series(scores, dtype=float)df = len(index.postings["your"])print(f"products with your: {df:,}")# Robertson's IDF has no 1 in the logarithm, so it goes negative for a term in over half the productsprint(f"robertson idf of your: {math.log((n - df + 0.5) / (df + 0.5)):.3f}")print(f"lucene idf of your: {math.log(1 + (n - df + 0.5) / (df + 0.5)):.3f}")for tf in [1, 2, 4, 9]:                                   # one term, in a product of average length    print(f"term weight at tf {tf}: {tf / (tf + k1):.3f}")# products that share no term with a query score 0, as bm25s gives themgaps = [(bm25(query).reindex(products.index, fill_value=0) - library.retriever.get_scores(tokenize(query))).abs().max()        for query in queries["query"]]print(f"\nlargest score difference from bm25s: {max(gaps):.0e}")print("bm25s score type:", library.retriever.get_scores(["chair"]).dtype)ranked = {query_id: list(bm25(query).sort_index().sort_values(ascending=False, kind="stable").index[:100])          for query_id, query in queries["query"].items()}         # equal scores in product_id orderlibrary_ranked = library.run(queries)                     # lib.lexical also orders equal scores by product_idprint("same top 100 for every query:", "yes" if ranked == library_ranked else "no")for split in ["train", "test"]:    for name, run in [("by hand", ranked), ("with bm25s", library_ranked)]:        print(f"{split} ndcg@10 {name}: {evaluate(run, qrels(split))['ndcg@10'].mean():.3f}")
출력python bm25-by-hand.py
products with your: 6,135
robertson idf of your: -0.045
lucene idf of your: 0.671
term weight at tf 1: 0.400
term weight at tf 2: 0.571
term weight at tf 4: 0.727
term weight at tf 9: 0.857

largest score difference from bm25s: 1e-06
bm25s score type: float32
same top 100 for every query: yes
train ndcg@10 by hand: 0.695
train ndcg@10 with bm25s: 0.695
test ndcg@10 by hand: 0.725
test ndcg@10 with bm25s: 0.725

The word your occurs in 6,135 of the 12,000 products. Robertson's original IDF gives it -0.045, and Lucene's gives it 0.671. A negative IDF would make a match on a word lower a product's score. The term weights show saturation: in a product of average length, one occurrence of a term earns 0.400 of its IDF, and nine earn 0.857.

The scores match. The largest difference over every product and query is 1e-06, at the precision of the float32 numbers bm25s returns. Matching scores don't settle a ranking on their own, though: where scores are equal, an implementation still has to choose which tied product comes first. Ours lists tied products by product ID. So does the book's shared BM25, lib.lexical from chapter 1, which takes every product's score from bm25s and then orders equal scores by product ID. Asked whether both give the same top 100 for every query, the example answers yes.

The rankings follow from the scores. On the train queries, the mean nDCG@10 is 0.695 by hand and 0.695 with bm25s; on the test queries, 0.725 and 0.725.

Working rule: Serve BM25 from a library such as bm25s, and keep a small version you wrote yourself to check it, because yours shows where every part of a score comes from. Compare two implementations by their scores first, then by their rankings. Before you compare rankings, break ties the same way in both, by product ID as the book's BM25 does, or tied products alone can make two correct implementations rank differently. When two BM25 systems rank differently, compare their tokenizers before their formulas. The study of variants held tokenization and stop words fixed, and notes that more mundane settings, such as the choice of stop words, often make larger differences than the formula (Kamphuis et al.).

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Stem terms so word forms match

마지막으로 변경된 판: 2026.11.8판

A shopper types pool floats, and a product says float or floating. To the tokenizer, those are different terms, so they don't match. Stemming cuts common endings off words, so that the forms of a word share one term, its stem. Martin Porter designed his suffix-stripping algorithm for retrieval, and in his test it reduced a vocabulary of 10,000 words to 6,370 stems, a cut of about one third (Porter, 1980). His page recommends its successor, the Snowball English or Porter2 stemmer (Porter), which stems slightly over 5% of words differently (Snowball). Stemming trades precision for recall. It raises recall while harming precision, and it helps some queries a lot while hurting others as much (Manning et al., 2008).

A crude stemmer takes a few lines. Write one by hand that cuts -ing, -es, -ed or -s when at least three letters remain, and compare it with the Snowball English stemmer from PyStemmer, which bm25s's tokenizer recommends (bm25s tokenizer) and the book pins at 3.1.0 (PyStemmer). Then count the products whose text contains a word, and those whose text contains its stem:

stemmer-merges-word-forms.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Compares a hand-written suffix stripper with the Snowball English stemmer on a few words, then counts the products whose text contains each word and its stem. The prose's point rests on the counts for floats and for molding, and on the vocabulary reduction line."""import Stemmerfrom lib.lexical_by_hand import InvertedIndexfrom lib.store import load_products, product_textdef strip_suffix(word):    """A stemmer by hand: cut the first ending that matches, if at least three letters are left."""    # the first ending that matches is cut, so the order matters: "es" is tried before "s"    for ending in ["ing", "es", "ed", "s"]:        # at least three letters must remain, so short words such as "is" keep their endings        if word.endswith(ending) and len(word) - len(ending) >= 3:            return word[: -len(ending)]    return wordstemmer = Stemmer.Stemmer("english")          # Snowball English, also called Porter2words = ["floats", "floating", "molding", "molded", "moldings", "shelves", "shelving", "shelf"]for word, stem in zip(words, stemmer.stemWords(words)):    print(f"{word}: snowball {stem}, by hand {strip_suffix(word)}")products = load_products()text = product_text(products)plain, stemmed = InvertedIndex(text), InvertedIndex(text, stem=True)print()for word in ["floats", "molding"]:    print(f"products with {word}:", len(plain.postings[word]))    print(f"products with the stem of {word}:", len(stemmed.postings[stemmer.stemWord(word)]))# every product with a word whose stem is float: float, floats, floating and any other formfloat_products = list(stemmed.postings["float"])print("products with the stem of floats that say floating:", len(plain.postings["floating"].keys() & set(float_products)))for product_class, count in products.loc[float_products, "product_class"].value_counts().head(2).items():    print(f"products with the stem of floats in {product_class}: {count}")print(f"vocabulary without stemming: {len(plain.postings):,} terms")print(f"vocabulary with stemming: {len(stemmed.postings):,} terms")print(f"vocabulary reduction: {1 - len(stemmed.postings) / len(plain.postings):.0%}")
출력python stemmer-merges-word-forms.py
floats: snowball float, by hand float
floating: snowball float, by hand float
molding: snowball mold, by hand mold
molded: snowball mold, by hand mold
moldings: snowball mold, by hand molding
shelves: snowball shelv, by hand shelv
shelving: snowball shelv, by hand shelv
shelf: snowball shelf, by hand shelf

products with floats: 11
products with the stem of floats: 288
products with molding: 123
products with the stem of molding: 262
products with the stem of floats that say floating: 244
products with the stem of floats in Wall Mounted Shelves: 57
products with the stem of floats in Desks: 36
vocabulary without stemming: 20,393 terms
vocabulary with stemming: 15,258 terms
vocabulary reduction: 25%

The hand-written stemmer agrees with Snowball on every word but moldings, where it stops after cutting -s, and Snowball's rules go on to mold. Both turn shelves into shelv and leave shelf alone: rules that cut endings don't know that shelves is the plural of shelf.

The counts show both sides of the trade. floats is in 11 products and its stem in 288, but 244 of those say floating. The two most common product classes among them are wall-mounted shelves, with 57, and desks, with 36. So the stem reaches floating shelves as well as pool floats, and only the judgments can say which effect wins (next section).

molding occurs in 123 and its stem in 262, because mold also stands for molded, moldings and mold itself, a word with more than one meaning. The vocabulary shrinks by 25%, from 20,393 terms to 15,258 terms. Porter's test used different text and an earlier algorithm, so they don't compare directly.

To serve stemmed BM25 from bm25s, pass the same stemmer to bm25s.tokenize for the products and for every query. Then check that bm25s, given the same stemmer, scores like the hand-built index with stemming, InvertedIndex(text, stem=True) from lib.lexical_by_hand:

bm25s-with-a-stemmer.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Builds bm25s with the same stemmer as the hand-built index, then checks every score on every query against it. The prose's point rests on the line that prints the largest score difference from bm25s."""import bm25simport Stemmerfrom lib.lexical_by_hand import InvertedIndexfrom lib.store import load_products, load_queries, product_textproducts, queries = load_products(), load_queries()texts = product_text(products)stemmer = Stemmer.Stemmer("english")library = bm25s.BM25()                                            # k1=1.5, b=0.75, as in chapter 1# products and queries must go through the same stop list and stemmer, or their terms won't matchlibrary.index(bm25s.tokenize(texts.tolist(), stopwords="en", stemmer=stemmer, show_progress=False),              show_progress=False)by_hand = InvertedIndex(texts, stem=True)gaps = []for query in queries["query"]:    tokens = bm25s.tokenize([query], stopwords="en", stemmer=stemmer, return_ids=False, show_progress=False)[0]    scores = library.get_scores(tokens)                           # every product, in catalog order    # search keeps only matching products, best first: reindex puts each in its catalog place, 0 for the rest    gaps.append(abs(by_hand.search(query, k=len(products)).reindex(products.index, fill_value=0) - scores).max())print("query terms for pool floats:", bm25s.tokenize(["pool floats"], stopwords="en", stemmer=stemmer,                                                     return_ids=False, show_progress=False)[0])print(f"largest score difference from bm25s with a stemmer: {max(gaps):.0e}")
출력python bm25s-with-a-stemmer.py
query terms for pool floats: ['pool', 'float']
largest score difference from bm25s with a stemmer: 2e-06

pool floats becomes pool and float, and the largest score difference over all products and queries is 2e-06, the same rounding as without stemming.

Working rule: Stem English product text and queries with the Snowball English stemmer, at indexing time and at query time alike: in bm25s, pass stemmer= to bm25s.tokenize on both sides. Read the products that a stem adds, as the floats counts do, before you trust a gain. When a stem merges words that your shoppers keep apart, such as molding and mold, keep a list of words you don't stem. For text in another language, use that language's stemmer.

테스트 쿼리로 어간 추출 확인하기

마지막으로 변경된 판: 2026.11.8판

어간 추출은 정밀도를 내주고 재현율을 얻는 방법이므로, 이것이 상점에 이득이 되는지는 판정만이 말해 줄 수 있다. 어간 추출을 적용하기로 한 선택은 50개의 훈련 쿼리에서 나온다. 이 비교는 이 장의 계획된 비교 다섯 가지 가운데 첫 번째이며, 30개의 테스트 쿼리로 수행한다. 다음으로 어간을 추출한 색인을 BM25 절의 어간을 추출하지 않은 색인과 비교한다.

measure-stemming.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Compares the stemmed index with the unstemmed one on train and test queries, and runs the planned test comparison for stemming. The prose's point rests on the test permutation p and t-test p lines, and on the line that reports the query with the largest gain."""from lib.lexical_by_hand import THRESHOLD, InvertedIndex, report_test_comparisonfrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, product_text, qrelstext = product_text(load_products())indexes = {"without stemming": InvertedIndex(text), "with stemming": InvertedIndex(text, stem=True)}scores = {split: {name: evaluate(index.run(load_queries(split)), qrels(split)) for name, index in indexes.items()}          for split in ["train", "test"]}for split in ["train", "test"]:    for name, frame in scores[split].items():        print(f"{split} ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}")        print(f"{split} judged@10 {name}: {frame['judged@10'].mean():.3f}")change = (scores["train"]["with stemming"]["ndcg@10"] - scores["train"]["without stemming"]["ndcg@10"]).round(3)print(f"train queries better, worse: {(change > 0).sum()}, {(change < 0).sum()}")    # for choosing, not evidencefor query_id in [410, 359]:                       # pool floats and french molding, train queries    for name, frame in scores["train"].items():        print(f"query {query_id} {name}: {frame.at[query_id, 'ndcg@10']:.3f}")print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:")after, before = scores["test"]["with stemming"]["ndcg@10"], scores["test"]["without stemming"]["ndcg@10"]# the planned test comparison: permutation test, t-test and bootstrap intervalreport_test_comparison(after, before)gain = after - beforeprint("test query with the largest gain:", gain.idxmax())print(f"its gain: {gain.max():.3f}")print(f"mean test gain without it: {gain.drop(gain.idxmax()).mean():.3f}")
출력python measure-stemming.py
train ndcg@10 without stemming: 0.695
train judged@10 without stemming: 0.930
train ndcg@10 with stemming: 0.725
train judged@10 with stemming: 0.950
test ndcg@10 without stemming: 0.725
test judged@10 without stemming: 0.927
test ndcg@10 with stemming: 0.774
test judged@10 with stemming: 0.950
train queries better, worse: 24, 14
query 410 without stemming: 0.360
query 410 with stemming: 0.921
query 359 without stemming: 0.961
query 359 with stemming: 0.809

the planned test comparison, threshold 0.01:
mean difference: 0.050
wins, losses, ties: 16, 6, 8
permutation p: 0.005
established at 0.01: yes
t-test p: 0.037
t-test agrees: no
99% interval: 0.016 to 0.162
interval agrees: yes
test query with the largest gain: 133
its gain: 0.635
mean test gain without it: 0.029

훈련 쿼리에서 어간 추출은 평균 nDCG@10을 0.695에서 0.725로, judged@10을 0.930에서 0.950로 올린다. 결과가 좋아진 훈련 쿼리와 나빠진 훈련 쿼리의 수는 24, 14이다. pool floats는 0.360에서 0.921로, french molding는 0.961에서 0.809로 바뀐다. 이 훈련 수치는 어간 추출을 선택하는 근거일 뿐, 어간 추출이 도움이 된다는 증거는 아니다.

테스트 쿼리에서 어간 추출의 이득은 0.01 수준에서 성립하지만, t-검정은 이에 동의하지 않는다(계획된 비교). 평균 nDCG@10은 0.725에서 0.774로 오르고, 평균 차이는 0.050이다. Judged@10도 0.927에서 0.950로 오르므로, 이 이득의 일부는 어간을 추출하지 않은 실행에서 판정되지 않은 채 남은 상품에서 나왔을 수 있다.

쿼리 하나가 두 검정이 갈리는 이유를 설명한다. 쿼리 133, outdoor light fixtures은 0.635만큼 이득을 보며, 이 쿼리를 제외하면 평균 이득은 0.029이다. t-검정은 차이가 대략 정규분포를 따른다고 가정하므로, 큰 이득 하나가 차이의 퍼짐을 키워 t를 낮춘다. 순열 검정은 차이의 분포 형태에 아무것도 가정하지 않는다(3장). 이 쿼리는 또한 어간을 추출하지 않은 상위 10개가 1장에서 판정되지 않은 전구로 가득 차 있던 것으로 확인된 바로 그 쿼리이다.

작업 규칙: 텍스트 처리 변경은 훈련 쿼리에서 고르고, 계획된 비교가 성립할 때만 유지한다. 순열 검정과 t-검정이 일치하지 않으면 쿼리별 차이 가운데 가장 큰 것들을 읽고, 평균 옆에 가장 큰 이득을 제외한 이득을 함께 보고한다. 어간 추출은 성립하므로, 어휘 검색을 구축할 때는 어간을 추출한다. 이 책에서 공유하는 BM25는 이후 장의 결과를 서로 비교할 수 있도록 어간을 추출하지 않은 채로 둔다(결정). 이득의 크기는 불안정한 것으로 취급한다.

학습 질의로 BM25의 매개변수 조정하기

마지막으로 변경된 판: 2026.11.8판

BM25의 두 매개변수에는 하나로 정해진 올바른 값이 없으며, 라이브러리마다 기본값도 다르다. bm25s는 k1=1.5와 b=0.75를 사용한다(bm25s 소스). Lucene은 k1=1.2와 b=0.75을 사용한다(Lucene BM25Similarity). Kamphuis et al. (2020)은 연구용 툴킷인 Anserini의 기본값으로 k1=0.9와 b=0.4를 보고한다. Robertson and Zaragoza (2009)는 측정값이 아닌 경험칙을 제시한다. b는 0.5에서 0.8 사이, k1는 1.2에서 2 사이의 값이 많은 상황에서 대체로 좋지만, 최적값은 문서와 질의의 종류에 따라 달라진다는 것이다. 따라서 자신의 학습 질의로 매개변수를 조정하고, 그 결과를 테스트 질의로 확인한다.

역색인은 점수가 아니라 용어 빈도와 길이를 저장하므로, k1와 b는 색인을 다시 만들지 않고도 검색할 때마다 바꿀 수 있다. bm25s는 색인을 만들 때 모든 점수를 계산하며(Lù, 2024), 이 덕분에 검색이 빠르지만 k1나 b를 바꿀 때마다 새 색인이 필요하다. 어간 추출 색인에 대한 스윕은 25개의 설정으로 이루어진 격자를 50개의 학습 질의에 시험하여 가장 좋은 설정을 고른다. 그런 다음 그 설정을 기본값과 30개의 테스트 질의에서 비교한다. 이것이 임계값 0.01에서 하는 두 번째 계획된 비교다.

실행하기 전에 예측해 보자. 기본값에서 어간 추출을 적용한 BM25는 테스트 질의에서 0.774를 기록한다. 학습 질의에서 찾은 최적 설정은 여기에 얼마를 더할까?

sweep-k1-and-b.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Sweeps k1 and b over the stemmed index on train queries, picks the best setting, and checks it against the defaults on test queries. The prose's point rests on the train best and grid-edge lines, and on the test permutation p line."""import pandas as pdfrom lib.lexical_by_hand import InvertedIndex, report_test_comparisonfrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, product_text, qrelsindex = InvertedIndex(product_text(load_products()), stem=True)     # stemmed terms, k1 and b set per searchtrain, test = load_queries("train"), load_queries("test")# the grid: five values of k1 and five of b, with the defaults 1.5 and 0.75 among themk1_values, b_values = [0.5, 0.9, 1.2, 1.5, 2.0], [0.0, 0.25, 0.5, 0.75, 1.0]def scores(queries, split, k1, b):    """One row per query, with BM25 at these parameters."""    return evaluate(index.run(queries, k1=k1, b=b), qrels(split))# one column per b, one row per k1grid = pd.DataFrame({b: {k1: scores(train, "train", k1, b)["ndcg@10"].mean() for k1 in k1_values} for b in b_values})print("train ndcg@10, k1 down and b across:")print(grid.round(3).to_string())# the (k1, b) pair with the highest mean nDCG@10 on trainbest_k1, best_b = grid.stack().idxmax()print("settings in the grid:", grid.size)print(f"best k1 on train: {best_k1}")print(f"best b on train: {best_b}")# a best value at an end of the grid can mean a better one lies beyond itedge = best_k1 in (k1_values[0], k1_values[-1]) or best_b in (b_values[0], b_values[-1])print("best setting on the grid's edge:", "yes" if edge else "no")print(f"train ndcg@10 at the train best: {grid.at[best_k1, best_b]:.3f}")print(f"train ndcg@10 at k1 1.5, b 0.75: {grid.at[1.5, 0.75]:.3f}")print(f"train gain of the best over the defaults: {grid.at[best_k1, best_b] - grid.at[1.5, 0.75]:.3f}")print(f"lowest train ndcg@10 in the grid: {grid.min().min():.3f}")print()runs = {"at k1 1.5, b 0.75": scores(test, "test", 1.5, 0.75), "at the train best": scores(test, "test", best_k1, best_b)}for name, frame in runs.items():    print(f"test ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}")    print(f"test judged@10 {name}: {frame['judged@10'].mean():.3f}")# the planned test comparison: permutation test, t-test and bootstrap intervalreport_test_comparison(runs["at the train best"]["ndcg@10"], runs["at k1 1.5, b 0.75"]["ndcg@10"])
출력python sweep-k1-and-b.py
train ndcg@10, k1 down and b across:
      0.00   0.25   0.50   0.75   1.00
0.5  0.707  0.710  0.716  0.720  0.720
0.9  0.711  0.711  0.716  0.722  0.720
1.2  0.709  0.711  0.715  0.723  0.720
1.5  0.709  0.714  0.719  0.725  0.715
2.0  0.714  0.720  0.726  0.729  0.713
settings in the grid: 25
best k1 on train: 2.0
best b on train: 0.75
best setting on the grid's edge: yes
train ndcg@10 at the train best: 0.729
train ndcg@10 at k1 1.5, b 0.75: 0.725
train gain of the best over the defaults: 0.004
lowest train ndcg@10 in the grid: 0.707

test ndcg@10 at k1 1.5, b 0.75: 0.774
test judged@10 at k1 1.5, b 0.75: 0.950
test ndcg@10 at the train best: 0.775
test judged@10 at the train best: 0.953
mean difference: 0.001
wins, losses, ties: 7, 6, 17
permutation p: 0.757
established at 0.01: no
t-test p: 0.754
t-test agrees: yes
99% interval: -0.008 to 0.008
interval agrees: yes

학습 질의에서 격자 전체는 0.707에서 0.729 사이에 놓인다. 최적 설정인 k1 = 2.0, b = 0.75는 0.729를 기록하며, 기본값에서는 0.725를 기록한다. 예제는 최적 설정이 격자의 가장자리에 있는지도 보고하는데, 그 값은 yes이다. 더 넓은 격자는 더 높은 학습 점수를 찾을 수도 있지만, 격자 전체가 이만큼 가까운 만큼 달라지는 것은 거의 없을 것이다.

조정은 테스트 질의에서 확립된 향상을 보이지 않는다. 최적 설정은 0.775를 기록하고 기본값은 0.774를 기록하며, judged@10은 각각 0.953와 0.950이고, 그 차이는 확립되지 않는다(계획된 비교들).

몇 백분의 일 정도의 향상을 예측했다면, 학습 격자를 보건대 그럴 가능성은 낮다. 학습 질의에서 최적 설정은 기본값보다 0.004 높은데, 이 값조차 낙관적인 추정이다. 모든 설정 중에서 우연히 이 질의들에 가장 잘 맞은 설정이기 때문이며, 다른 질의에서는 이보다 적은 향상을 예상해야 한다.

작업 규칙: 라이브러리의 기본값에서 시작한다. 필드나 어간 추출처럼 색인하는 대상을 바꿀 때 학습 질의로 k1와 b를 조정하고, 계획된 비교가 확립될 때만 조정된 설정을 유지한다. 이 저장소의 조정된 설정은 확립되지 않았으므로, 저장소는 k1=1.5와 b=0.75를 유지한다.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Weight the product name above the description

마지막으로 변경된 판: 2026.11.10판

A product's text has parts, called fields: the store's text fields are name, class, category and description. They aren't equal evidence. A query word in the name says more about what the product is than the same word somewhere in a description. There are two ways to weight fields. The first that comes to mind scores each field with BM25 and adds the scores. Robertson and Zaragoza (2009, section 3.6) argue for the opposite order, "first streams, then terms": combine each term's weighted frequencies across the fields first, and then saturate the total evidence for that term. Combining a term's weighted frequencies across fields before saturating the total is BM25F. Their rule implies that adding per-field scores lets a term that matches in two fields escape saturation, because each field saturates on its own. With whole-number weights, simple BM25F gives the same result as plain BM25 on a text in which each field is repeated as many times as its weight (Robertson and Zaragoza, section 3.6.4), so you can try it with any BM25 library.

The store's search indexes one field, the name, class, category and description joined. The first try is the one that comes to mind: on the 50 train queries with stemming, the name and the other three fields are indexed apart, scored apart and added. Indexed apart, each field also gets its own IDF and its own average length, so this changes more than saturation. By Robertson and Zaragoza's argument, we expect it to do worse. On train queries, the result informs a choice and proves nothing:

fields-scored-apart.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Scores the name and the other fields as separate BM25 indexes and adds the scores, then compares the result with one field on train queries. The prose's point rests on the two train nDCG@10 lines and the line that counts the queries that get better and worse."""from collections import Counterfrom lib.lexical_by_hand import InvertedIndexfrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, product_text, qrelsproducts, train = load_products(), load_queries("train")others = products["product_class"] + ". " + products["category"] + ". " + products["description"]one_field = InvertedIndex(product_text(products), stem=True)                 # name, class, category, descriptionname, rest = InvertedIndex(products["name"], stem=True), InvertedIndex(others, stem=True)   # own IDF and lengthssummed = {}for query_id, query in train["query"].items():    scores = Counter(name.bm25(query)) + Counter(rest.bm25(query))           # each field saturates on its own    # the top 100, as in every other run    summed[query_id] = [product_id for product_id, _ in scores.most_common(100)]runs = {"one field": one_field.run(train), "fields scored apart": summed}scores = {label: evaluate(run, qrels("train")) for label, run in runs.items()}for label, frame in scores.items():    print(f"{label}, train ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{label}, train judged@10: {frame['judged@10'].mean():.3f}")change = (scores["fields scored apart"]["ndcg@10"] - scores["one field"]["ndcg@10"]).round(3)print(f"train queries better, worse with fields scored apart: {(change > 0).sum()}, {(change < 0).sum()}")
출력python fields-scored-apart.py
one field, train ndcg@10: 0.725
one field, train judged@10: 0.950
fields scored apart, train ndcg@10: 0.686
fields scored apart, train judged@10: 0.970
train queries better, worse with fields scored apart: 15, 24

Scored apart, the fields give a train nDCG@10 of 0.686 against 0.725 for one field, with judged@10 of 0.970 and 0.950. The train queries that get better and worse number 15, 24. That is enough to choose one field over this design, and not enough to say why: saturation, IDF and length normalization all changed at once.

A second try uses BM25F with whole-number weights: the name counted 1, 2, 3 or 5 times in the indexed text. It picks the weight with the best train score, then compares it with one field on the 30 test queries: the third planned comparison.

repeat-the-name-field.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Repeats the product name in the indexed text, once per extra weight, and picks the weight with the best train nDCG@10 for the test comparison. The prose's point rests on the best-weight line and the test permutation p line."""from lib.lexical_by_hand import InvertedIndex, report_test_comparisonfrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, product_text, qrelsproducts, train, test = load_products(), load_queries("train"), load_queries("test")# product_text already holds the name once, so weight - 1 more copies make it weight in allindexes = {weight: InvertedIndex((products["name"] + ". ") * (weight - 1) + product_text(products), stem=True)           for weight in [1, 2, 3, 5]}            # BM25F with whole-number weights: the name counted weight timestrain_scores = {weight: evaluate(index.run(train), qrels("train")) for weight, index in indexes.items()}for weight, frame in train_scores.items():    print(f"name weight {weight}, train ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"name weight {weight}, train judged@10: {frame['judged@10'].mean():.3f}")# on a tie, max keeps the first weight, the smaller onebest = max(train_scores, key=lambda weight: train_scores[weight]["ndcg@10"].mean())print("best name weight on train:", best)print()runs = {"one field": indexes[1].run(test), "best name weight": indexes[best].run(test)}scores = {label: evaluate(run, qrels("test")) for label, run in runs.items()}for label, frame in scores.items():    print(f"{label}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"{label}, test judged@10: {frame['judged@10'].mean():.3f}")# the planned test comparison: permutation test, t-test and bootstrap intervalreport_test_comparison(scores["best name weight"]["ndcg@10"], scores["one field"]["ndcg@10"])
출력python repeat-the-name-field.py
name weight 1, train ndcg@10: 0.725
name weight 1, train judged@10: 0.950
name weight 2, train ndcg@10: 0.734
name weight 2, train judged@10: 0.968
name weight 3, train ndcg@10: 0.729
name weight 3, train judged@10: 0.970
name weight 5, train ndcg@10: 0.722
name weight 5, train judged@10: 0.972
best name weight on train: 2

one field, test ndcg@10: 0.774
one field, test judged@10: 0.950
best name weight, test ndcg@10: 0.786
best name weight, test judged@10: 0.970
mean difference: 0.012
wins, losses, ties: 15, 9, 6
permutation p: 0.149
established at 0.01: no
t-test p: 0.153
t-test agrees: yes
99% interval: -0.009 to 0.032
interval agrees: yes

On train, the best name weight is 2, which scores 0.734 against 0.725 for one field, with judged@10 of 0.968 and 0.950.

Repeating the name shows no established gain on test: the best name weight scores 0.786 against 0.774, with judged@10 of 0.970 and 0.950, and the gain isn't established (the planned comparisons). Neither the t-test nor the interval disagrees.

Working rule: Don't add up BM25 scores from separate fields. To weight a field, combine term frequencies before saturation, with BM25F or by repeating the field. Tune the weight on train queries, and keep it only if its planned comparison is established. For the store, one combined field stays.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Index product attributes as a field

마지막으로 변경된 판: 2026.11.10판

Shoppers often add a modifier, a word that narrows what they want: a color, a material, a size. The WANDS judges mark a product partial when it matches the thing asked for but not its modifiers. Product names and descriptions don't always state a modifier, but a catalog often keeps facts about a product as structured attributes, such as color: clear or material: glass. The store's products carry a few from WANDS: color, material, style, finish, shape, size, width, height and depth (examples/data/wands/README.md). Indexed as one more field, attributes give a modifier something to match. They also add terms to every product that has them, so whether they help is a question for the judgments.

In chapter 1, BM25's results for clear shower curtain couldn't show which curtains were clear. The example adds each product's attributes, names and values, to its indexed text, as a field with weight 1, combined before saturation as the previous section recommends. It counts the exact products for that test query that contain clear, with and without attributes. It measures this one candidate, fixed in advance, on the 50 train queries and checks it on the 30 test queries: the fourth planned comparison. Being planned, it goes to test whatever train shows.

add-attributes-field.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Adds each product's attributes, names and values, to its indexed text as one more field, and checks that single candidate on train and test. The prose's point rests on the two lines that count exact products with clear, and on the test permutation p line."""from lib.lexical_by_hand import InvertedIndex, report_test_comparison, tokenizefrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, product_text, qrelsproducts = load_products()text = product_text(products)with_attributes = text + ". " + products["attributes"]      # names and values, one more field, weight 1exact = [product_id for product_id, grade in qrels("test")[246].items() if grade == 2]    # clear shower curtainprint("exact products for query 246:", len(exact))print("exact, with clear in their text:", sum("clear" in tokenize(text[product_id]) for product_id in exact))print("exact, with clear in text or attributes:", sum("clear" in tokenize(with_attributes[product_id]) for product_id in exact))# a product whose attributes say clear and whose text does not, to show what the field addsfound = next(product_id for product_id in exact             if "clear" in tokenize(products.at[product_id, "attributes"]) and "clear" not in tokenize(text[product_id]))print(f"for example: {products.at[found, 'name']} -> {products.at[found, 'attributes']}")names = {"color", "material", "style", "finish", "shape", "size", "width", "height", "depth"}       # the attribute namesprint("queries that contain an attribute name:", load_queries()["query"].map(lambda query: bool(names & set(tokenize(query)))).sum())print("products whose attributes name size:", sum("size" in tokenize(value) for value in products["attributes"]))indexes = {"one field": InvertedIndex(text, stem=True), "with attributes": InvertedIndex(with_attributes, stem=True)}scores = {split: {label: evaluate(index.run(load_queries(split)), qrels(split)) for label, index in indexes.items()}          for split in ["train", "test"]}           # one candidate, fixed in advance: tuned on nothingfor split in ["train", "test"]:    print()    for label, frame in scores[split].items():        print(f"{split} ndcg@10, {label}: {frame['ndcg@10'].mean():.3f}")        print(f"{split} judged@10, {label}: {frame['judged@10'].mean():.3f}")print()for query_id in [246, 58]:                       # clear shower curtain, medium size chandelier: test queries    for label, frame in scores["test"].items():        print(f"query {query_id} ndcg@10, {label}: {frame.at[query_id, 'ndcg@10']:.3f}")# the planned test comparison: permutation test, t-test and bootstrap intervalreport_test_comparison(scores["test"]["with attributes"]["ndcg@10"], scores["test"]["one field"]["ndcg@10"])
출력python add-attributes-field.py
exact products for query 246: 12
exact, with clear in their text: 4
exact, with clear in text or attributes: 8
for example: wynsum vinyl single shower curtain -> color: clear; width: 72; height: 84
queries that contain an attribute name: 1
products whose attributes name size: 950

train ndcg@10, one field: 0.725
train judged@10, one field: 0.950
train ndcg@10, with attributes: 0.717
train judged@10, with attributes: 0.948

test ndcg@10, one field: 0.774
test judged@10, one field: 0.950
test ndcg@10, with attributes: 0.787
test judged@10, with attributes: 0.963

query 246 ndcg@10, one field: 0.714
query 246 ndcg@10, with attributes: 0.844
query 58 ndcg@10, one field: 0.731
query 58 ndcg@10, with attributes: 0.667
mean difference: 0.013
wins, losses, ties: 8, 10, 12
permutation p: 0.303
established at 0.01: no
t-test p: 0.281
t-test agrees: yes
99% interval: -0.009 to 0.053
interval agrees: yes

For clear shower curtain, the attributes do what they should: 4 of the 12 exact products say clear in their text, and 8 do once attributes count, such as the vinyl curtain whose attributes say color: clear. For this query, nDCG@10 goes from 0.714 to 0.844. Indexing the attribute names has a cost, though: size now matches the 950 products whose attributes give a size. The store has 1 query that contains an attribute name, the test query medium size chandelier, and it goes from 0.731 to 0.667.

On train, judged@10 goes from 0.950 to 0.948, and nDCG@10 from 0.725 to 0.717. On test, judged@10 goes from 0.950 to 0.963, and nDCG@10 from 0.774 to 0.787, but the permutation test doesn't establish the gain (the planned comparisons).

Working rule: Keep structured attributes, and read the queries they help, but don't add them to the BM25 text on the strength of a few queries. Index the values without the names, so that a query word such as size doesn't match every product with a size, and keep the field only if its planned comparison is established. For the store, the attributes stay out of the BM25 text. When a modifier names an attribute value exactly, such as a color, a filter or a ranking feature can use it without adding its words to every product's text.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Match phrases with term positions

마지막으로 변경된 판: 2026.11.8판

BM25 scores a bag of words: it counts which terms a product contains and how often, never their order. To BM25, barn door and door barn are the same query. A phrase match needs the terms next to each other and in order, and an index of counts can't check that. A positional index stores, for each term and product, the positions where the term occurs, so that a phrase becomes a check on positions (Manning et al., 2008). bm25s precomputes one score for each pair of product and term (Lù, 2024). Its index keeps no positions (bm25s source), so phrases need an index of their own.

A positional index over the store's product names comes first, with stemmed terms as in the stemming section. Positions count terms after stop words are removed, so door for laundry holds the adjacent terms door and laundri: a phrase can match across a removed stop word. Take barn door, two words from one of the store's queries: assert that BM25 can't tell its two orders apart, and count the names with both words against those with the phrase:

phrase-positions-by-hand.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Builds a positional index over the product names, asserts that BM25 scores both word orders alike, and lists the names that have both words without the phrase. The prose's point rests on the two count lines, names with barn and door and names with the phrase barn door."""from lib.lexical_by_hand import InvertedIndex, tokenizefrom lib.store import load_products, product_textproducts = load_products()positions = {}            # {term: {product_id: [where the term occurs in the name, counted after stop words go]}}for product_id, name in products["name"].items():    # positions count the terms left after stop words are removed, so a phrase can span one    for place, term in enumerate(tokenize(name, stem=True)):        positions.setdefault(term, {}).setdefault(product_id, []).append(place)def has_phrase(product_id, terms):    """True if the terms occur next to each other, in this order, in the product's name."""    def places(term):        return positions.get(term, {}).get(product_id, [])    # each term has to sit at the next position after the one before it    return any(all(start + offset in places(term) for offset, term in enumerate(terms)) for start in places(terms[0]))index = InvertedIndex(product_text(products), stem=True)# BM25 counts words, not their order, so both orders must score alikeassert index.bm25("barn door") == index.bm25("door barn")both = set(positions["barn"]) & set(positions["door"])print("names with barn and door:", len(both))print("names with the phrase barn door:", sum(has_phrase(product_id, ["barn", "door"]) for product_id in both))for product_id in sorted(product_id for product_id in both if not has_phrase(product_id, ["barn", "door"])):    print("  ", products.at[product_id, "name"])
출력python phrase-positions-by-hand.py
names with barn and door: 71
names with the phrase barn door: 66
   low ceiling single bypass double door barn hardware kit
   artemas barn owl welcome non-slip outdoor door mat
   vintage rustic wooden door of old barn duvet cover set
   barnes entrance scraper non-slip outdoor door mat
   glass barn 60 '' w x 76 '' h single sliding frameless shower door

BM25 gives barn door and door barn the same scores, as the code asserts. Of the 71 names that contain both barn and door, 66 contain the phrase. The names without it show what a phrase check catches and what it costs. Two door mats and a duvet cover match both words but aren't barn doors, and one of the mats matches only because barnes stems to barn. The bypass hardware kit, on the other hand, is hardware for barn doors, and the glass sliding shower door is a barn-style door. A strict phrase filter would remove both.

A bonus is gentler than a filter: a fixed amount added to a product's BM25 score for each pair of adjacent query terms that appears, in order, in its name. The bonus example uses PositionalIndex from lib.lexical_by_hand, which holds the same positions, and it picks the bonus on the 50 train queries, then checks it on the 30 test queries: the last planned comparison.

phrase-bonus-train-and-test.py Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · pystemmer 3.1.0 · scipy 1.18.1 통과
"""Adds a bonus for each adjacent pair of query terms that appears in order in a product's name, picks the bonus on train and checks it on test. The prose's point rests on the test permutation p line and the test nDCG@10 lines."""from lib.lexical_by_hand import InvertedIndex, PositionalIndex, report_test_comparison, tokenizefrom lib.metrics import evaluatefrom lib.store import load_products, load_queries, product_text, qrelsproducts = load_products()index = InvertedIndex(product_text(products), stem=True)names = PositionalIndex(products["name"], stem=True)      # the positions phrase-positions-by-hand buildsdef with_bonus(split, bonus):    """BM25's top 100, plus the bonus for each pair of adjacent query terms found, in order, in the name."""    run = {}    for query_id, query in load_queries(split)["query"].items():        terms, scores = tokenize(query, stem=True), index.search(query)        pairs = list(zip(terms, terms[1:]))        # only BM25's top 100 get a bonus, so it reorders them and adds no product        scores += [bonus * sum(names.has_phrase(product_id, list(pair)) for pair in pairs) for product_id in scores.index]        run[query_id] = list(scores.sort_values(ascending=False, kind="stable").index)    return evaluate(run, qrels(split))train_scores = {bonus: with_bonus("train", bonus) for bonus in [0, 0.5, 1, 2]}for bonus, frame in train_scores.items():    print(f"bonus {bonus}, train ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"bonus {bonus}, train judged@10: {frame['judged@10'].mean():.3f}")best = max(train_scores, key=lambda bonus: train_scores[bonus]["ndcg@10"].mean())print("best bonus on train:", best)print()# the defaults, with no bonus, and the best bonus on traintest_scores = {bonus: with_bonus("test", bonus) for bonus in [0, best]}for bonus, frame in test_scores.items():    print(f"bonus {bonus}, test ndcg@10: {frame['ndcg@10'].mean():.3f}")    print(f"bonus {bonus}, test judged@10: {frame['judged@10'].mean():.3f}")# the planned test comparison: permutation test, t-test and bootstrap intervalreport_test_comparison(test_scores[best]["ndcg@10"], test_scores[0]["ndcg@10"])
출력python phrase-bonus-train-and-test.py
bonus 0, train ndcg@10: 0.725
bonus 0, train judged@10: 0.950
bonus 0.5, train ndcg@10: 0.727
bonus 0.5, train judged@10: 0.956
bonus 1, train ndcg@10: 0.725
bonus 1, train judged@10: 0.964
bonus 2, train ndcg@10: 0.716
bonus 2, train judged@10: 0.968
best bonus on train: 0.5

bonus 0, test ndcg@10: 0.774
bonus 0, test judged@10: 0.950
bonus 0.5, test ndcg@10: 0.776
bonus 0.5, test judged@10: 0.950
mean difference: 0.002
wins, losses, ties: 6, 7, 17
permutation p: 0.758
established at 0.01: no
t-test p: 0.769
t-test agrees: yes
99% interval: -0.012 to 0.015
interval agrees: yes

On train, the best bonus is 0.5, which scores 0.727 against 0.725 without one, with judged@10 of 0.956 and 0.950.

The best bonus scores 0.776 against 0.774, with judged@10 of 0.950 and 0.950, and the gain isn't established (the planned comparisons).

Working rule: Don't add a phrase bonus to every query by default, and keep one only if its planned comparison is established. Build positions when shoppers quote phrases, or when word order changes what they want, as between table lamp and lamp table. Prefer a bonus that you measure on train queries to a filter, because a filter removes products like the hardware kit.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

What goes wrong in production

마지막으로 변경된 판: 2026.10.7판

The index and the queries drift apart. Every step from the tokenizer to the stemmer must run the same way on products and on queries. Stem the queries but not the products, and a query term such as shelv finds no postings. Nothing warns you: bm25s leaves out query tokens that aren't in its vocabulary (bm25s source). Change text processing for both sides in one release, and rebuild the whole index.

Sizes, units and codes don't survive tokenizing. 7qt and 7 qt never meet, as the tokenizer section shows. Train query 205 ends in a product code, w003096809, that appears in no product's text in the sample: its postings hold 0 products, so no lexical scorer can match it. A misspelled term such as the ligth in ligth bulb can only match products with the same misspelling, and chapter 8 builds a corrector and shows what it still gets wrong.

IDF depends on the catalog. Every IDF in this chapter comes from the sample, which holds every product judged for the 80 queries plus randomly chosen unjudged ones (examples/data/wands/README.md). On the full catalog, document frequencies differ, and so do IDFs and scores. Recompute them whenever you rebuild the index.

A BM25 score has no fixed scale. Each query term adds at most its own IDF, so the highest possible score depends on which terms the query holds, and a score of 6 can be high for one query and low for another. Don't keep results above a fixed score threshold; cut the list by rank.

Stop words and stemming change meaning on a few queries. The store's judged queries have no negations, but shoppers type them, and the default stop list removes not and no. Stemming hurts some queries badly, as french molding shows in the stemming check. The judgments can't warn you about queries they don't include.

The test queries wear out. This chapter makes five planned comparisons on the same test queries, and the Bonferroni threshold of 0.01 keeps the chance of any false positive among them at 5% or less when none of the changes does anything. It can't count what nobody planned: every extra setting tried on test, and every look at a test query to explain a result, is another comparison. Confirm stemming's gain on queries and judgments that none of these choices has seen.

Working rule: Keep a set of probe queries for negations, sizes and codes, and read their results after every change to text processing. Store the tokenizer's and stemmer's settings with the index they built, log the terms that each query becomes, and watch for queries whose terms have no postings.

이 부분은 아직 번역되지 않았으므로 영어로 표시된다.

Decisions this chapter made

마지막으로 변경된 판: 2026.11.8판

Each setting here was picked on the train queries. Before reading any test result, the chapter fixed five planned comparisons on the test queries, read with chapter 3's harness: stemming and the attributes field as single candidates fixed in advance, and the best setting on train for k1 and b, the name weight and the phrase bonus. The attributes field went to test whatever train showed, because it was planned. Only an established difference changes a decision. Each column compares nDCG@10 on the test queries with the change against the same search without it, at the threshold of 0.01:

Measure Stemming Best k1 and b Best name weight Attributes field Best phrase bonus
Mean difference 0.050 0.001 0.012 0.013 0.002
99% interval 0.016 to 0.162 -0.008 to 0.008 -0.009 to 0.032 -0.009 to 0.053 -0.012 to 0.015
Permutation p 0.005 0.757 0.149 0.303 0.758
t-test p 0.037 0.754 0.153 0.281 0.769
Wins, losses, ties 16, 6, 8 7, 6, 17 15, 9, 6 8, 10, 12 6, 7, 17
Established yes no no no no
t-test agrees no yes yes yes yes
Interval agrees yes yes yes yes yes
  • Tokenizer: bm25s's defaults. Lowercasing, tokens of two or more word characters (letters, digits or underscores), and its 33 English stop words, rebuilt by hand and matched on every product. They lose single-digit sizes and negations. Too few judged queries could measure either: a size-aware pattern changes 4 train queries and 0 test queries, and 0 queries hold a negation. The defaults stay until probe queries or new judgments can.
  • Stemming: Snowball English from PyStemmer, on products and queries. Its test gain, from 0.725 to 0.774, is established by the permutation test, though the t-test disagrees and one query carries much of it. The book's shared baseline, lib.lexical from chapter 1, stays at bm25s's defaults without stemming, so that every chapter's results stay comparable. This chapter's stemmed BM25 is InvertedIndex(text, stem=True) in lib.lexical_by_hand, or bm25s with stemmer=.
  • Scoring: BM25 as bm25s computes it by default, Lucene's formula with exact lengths, k1=1.5 and b=0.75, with equal scores ordered by product ID. The best setting on train isn't established on test, so the defaults stay.
  • Fields: one field that joins name, class, category and description. Per-field scores, added, score lower on train. The best name weight on train and the attributes field aren't established on test.
  • Phrases: no phrase bonus. Positions over product names are cheap to build, but the best bonus on train isn't established on test.
  • Implementation: a by-hand index for experiments, a library for serving. The by-hand index lets k1 and b change without rebuilding, and bm25s precomputes scores so that searches are fast. On the same test queries, semantic retrieval with the recorded Snowflake/snowflake-arctic-embed-m-v1.5 embeddings scores 0.830 (chapter 1).
  • What the chapter showed: BM25 can be rebuilt by hand and matched score for score. Of the choices made on the train queries, only stemming held up on test, and even it rests on a few queries.

이 판의 이 장

2026.11.10판예제 통과 2026년 11월 19이 장에서 13개 예제개 중 13개가 통과한다

영어판 2026.11.10판을 번역한 것이다.

이 절 인용

형식

각 형식은 이 번역이 따르는 영문 원서를 출처로 표기한다.

모든 형식은 이 판을 명시한다. 링크는 현재 판을 열며, 여기서 절의 제목은 본문이 마지막으로 바뀐 시점을 알려 준다. 접근 날짜와 검색 날짜는 이 자료를 여는 날짜다.

이 페이지의 내용

11장 중 5장: 어휘 검색

  1. 제품 텍스트를 용어로 나누기
  2. 역색인 만들기
  3. TF-IDF로 용어에 가중치 주기
  4. Rebuild BM25 by hand to match bm25s
  5. Stem terms so word forms match
  6. 테스트 쿼리로 어간 추출 확인하기
  7. 학습 질의로 BM25의 매개변수 조정하기
  8. Weight the product name above the description
  9. Index product attributes as a field
  10. Match phrases with term positions
  11. What goes wrong in production
  12. Decisions this chapter made

맨 위로 돌아가기전체 장