文章目录
- 一、相关性和打分简介
- 二、TF-IDF得分计算公式
- 三、BM25(Best Matching 25)
- 四、使用explain查看TF-IDF
- 五、通过Boosting控制相关度
一、相关性和打分简介
举个例子来说明:
-
假设有一个电商网站,用户在搜索框中输入了关键词"手机",然后触发了搜索操作。Elasticsearch会根据用户的查询,在索引中找到所有包含"手机"的文档,并按照相关性对这些文档进行打分。
-
相关性评分的目的是确定搜索结果的质量和排序。相关性评分越高,表示搜索结果与用户查询的匹配程度越好。
-
例如,对于一个包含"手机"关键词的文档,如果它在标题、描述和其他字段中多次出现"手机",那么它的相关性评分可能会很高。而对于一个只在描述中出现一次"手机"的文档,它的相关性评分可能会较低。
-
在Elasticsearch 5.0版本之前,Elasticsearch使用的是TF-IDF(Term Frequency-Inverse Document Frequency)算法来进行相关性判断和打分。
-
TF-IDF算法是一种经典的信息检索算法,它考虑了词频(Term Frequency)和逆文档频率(Inverse Document Frequency)
词频表示词在文档中的出现次数
逆文档频率表示词在整个文档集合中的普遍程度。 -
根据TF-IDF算法,搜索词在文档中出现的次数越多,这个词对文档的相关性贡献越大。但是,如果这个词在整个文档集合中出现的次数越多,它对文档的相关性贡献越小,因为它在整个集合中普遍存在,不足以区分文档的重要性。
-
从Elasticsearch 5.0版本开始,默认使用了BM25(Best Matching 25)算法来进行相关性判断和打分。BM25算法在计算相关性得分时,考虑了文档的长度和搜索词的位置等因素。
-
BM25算法相对于TF-IDF算法更为先进和准确,它在实际应用中表现更好。所以,对于Elasticsearch 5.0及以后的版本,推荐使用BM25算法来进行相关性判断和打分。
二、TF-IDF得分计算公式
TF-IDF(Term Frequency-Inverse Document Frequency)得分计算公式是一种用于衡量词语在文档集合中重要性的指标。其计算公式如下:
TF-IDF = TF * IDF
其中,TF(Term Frequency)表示词语在文档中的频率,可以通过以下公式计算:
TF = (词语在文档中出现的次数) / (文档中总词语数)
IDF(Inverse Document Frequency)表示逆文档频率,可以通过以下公式计算:
IDF = log((文档集合中文档总数) / (包含词语的文档数))
TF-IDF得分越高,表示词语在文档集合中越重要。它的核心思想是:当一个词语在某个文档中频繁出现(高TF值),同时在其他文档中较少出现(高IDF值)时,该词语对于该文档的重要性较高。
TF-IDF常用于信息检索、文本分类、关键词提取等自然语言处理任务中。
三、BM25(Best Matching 25)
BM25(Best Matching 25)算法可以被视为对TF-IDF算法的改进和扩展。BM25算法是一种用于信息检索的评分算法,相较于TF-IDF算法,它在一些方面进行了改进,以提高检索结果的质量。
主要的改进包括以下几个方面:
-
考虑文档长度:TF-IDF算法仅仅考虑了词频,对文档长度没有考虑。而BM25算法引入了文档长度因子,可以更好地处理文档长度不同的情况。
-
调整词频饱和度:TF-IDF算法中,词频的值会随着出现次数的增加而线性增长,容易导致过度强调高频词语。而BM25算法使用了对数函数来调整词频的饱和度,使得高频词语的权重增长趋于饱和。
-
引入文档频率饱和度:BM25算法引入了文档频率的饱和度因子,用于调整文档频率的影响。这可以避免过度强调出现在大多数文档中的常见词语。
-
综上所述,BM25算法在TF-IDF算法的基础上进行了改进,更加准确地评估了词语在文档集合中的重要性,提高了信息检索的效果。
四、使用explain查看TF-IDF
通过使用Elasticsearch的Explain API,你可以查看特定查询的TF-IDF得分。以下是一个示例,展示如何使用Explain API来计算TF-IDF得分:
- 首先,在Kibana的Dev Tools中执行以下命令,创建一个新的索引并索引一些文档数据:
PUT my_index
{"mappings": {"properties": {"text": {"type": "text"}}}
}POST my_index/_doc
{"text": "This is the first document."
}POST my_index/_doc
{"text": "This document is the second document."
}POST my_index/_doc
{"text": "And this is the third one."
}POST my_index/_doc
{"text": "Is this the first document?"
}
执行如下:
2. 然后,使用Explain API来查看TF-IDF得分。在Kibana的Dev Tools中执行以下命令:
GET my_index/_search
{"explain": true,"query": {"match": {"text": "this is the first document"}}
}
以上命令将返回查询结果,并在每个匹配的文档中提供一个_explanation字段,其中包含有关得分计算的详细信息,包括TF-IDF得分。
请注意,Explain API返回的_explanation字段中包含了与得分相关的详细信息,如词频、文档频率、字段长度等。你可以通过解析_explanation字段来提取和分析TF-IDF得分。
3. explain内容如下:
#! Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.
{"took" : 16,"timed_out" : false,"_shards" : {"total" : 1,"successful" : 1,"skipped" : 0,"failed" : 0},"hits" : {"total" : {"value" : 4,"relation" : "eq"},"max_score" : 1.4186639,"hits" : [{"_shard" : "[my_index][0]","_node" : "NkTFrxzGQuOf4zelToSeKQ","_index" : "my_index","_type" : "_doc","_id" : "08FDOokBF8lAViln2DRj","_score" : 1.4186639,"_source" : {"text" : "This is the first document."},"_explanation" : {"value" : 1.4186639,"description" : "sum of:","details" : [{"value" : 0.10943023,"description" : "weight(text:this in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10943023,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.10943023,"description" : "weight(text:is in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10943023,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.10943023,"description" : "weight(text:the in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10943023,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.7199211,"description" : "weight(text:first in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.7199211,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.6931472,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 2,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.3704521,"description" : "weight(text:document in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.3704521,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.35667494,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 3,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]}]}},{"_shard" : "[my_index][0]","_node" : "NkTFrxzGQuOf4zelToSeKQ","_index" : "my_index","_type" : "_doc","_id" : "1sFEOokBF8lAVilnADS4","_score" : 1.4186639,"_source" : {"text" : "Is this the first document?"},"_explanation" : {"value" : 1.4186639,"description" : "sum of:","details" : [{"value" : 0.10943023,"description" : "weight(text:this in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10943023,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.10943023,"description" : "weight(text:is in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10943023,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.10943023,"description" : "weight(text:the in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10943023,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.7199211,"description" : "weight(text:first in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.7199211,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.6931472,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 2,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.3704521,"description" : "weight(text:document in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.3704521,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.35667494,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 3,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.472103,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 5.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]}]}},{"_shard" : "[my_index][0]","_node" : "NkTFrxzGQuOf4zelToSeKQ","_index" : "my_index","_type" : "_doc","_id" : "1MFDOokBF8lAViln6TSq","_score" : 0.78294927,"_source" : {"text" : "This document is the second document."},"_explanation" : {"value" : 0.78294927,"description" : "sum of:","details" : [{"value" : 0.10158265,"description" : "weight(text:this in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10158265,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.43824703,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 6.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.10158265,"description" : "weight(text:is in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10158265,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.43824703,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 6.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.10158265,"description" : "weight(text:the in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10158265,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.43824703,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 6.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.47820133,"description" : "weight(text:document in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.47820133,"description" : "score(freq=2.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.35667494,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 3,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.6094183,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 2.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 6.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]}]}},{"_shard" : "[my_index][0]","_node" : "NkTFrxzGQuOf4zelToSeKQ","_index" : "my_index","_type" : "_doc","_id" : "1cFDOokBF8lAViln9jS-","_score" : 0.30474794,"_source" : {"text" : "And this is the third one."},"_explanation" : {"value" : 0.30474794,"description" : "sum of:","details" : [{"value" : 0.10158265,"description" : "weight(text:this in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10158265,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.43824703,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 6.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.10158265,"description" : "weight(text:is in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10158265,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.43824703,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 6.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]},{"value" : 0.10158265,"description" : "weight(text:the in 0) [PerFieldSimilarity], result of:","details" : [{"value" : 0.10158265,"description" : "score(freq=1.0), computed as boost * idf * tf from:","details" : [{"value" : 2.2,"description" : "boost","details" : [ ]},{"value" : 0.105360515,"description" : "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:","details" : [{"value" : 4,"description" : "n, number of documents containing term","details" : [ ]},{"value" : 4,"description" : "N, total number of documents with field","details" : [ ]}]},{"value" : 0.43824703,"description" : "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:","details" : [{"value" : 1.0,"description" : "freq, occurrences of term within document","details" : [ ]},{"value" : 1.2,"description" : "k1, term saturation parameter","details" : [ ]},{"value" : 0.75,"description" : "b, length normalization parameter","details" : [ ]},{"value" : 6.0,"description" : "dl, length of field","details" : [ ]},{"value" : 5.5,"description" : "avgdl, average length of field","details" : [ ]}]}]}]}]}}]}
}
五、通过Boosting控制相关度
在搜索引擎和信息检索领域中,Boosting可以用来提升或降低文档的相关度得分,以便更好地满足用户的查询需求。
Boosting可以在不同的层次上进行,包括查询级别的Boosting和文档级别的Boosting。
1、查询级别的Boosting:在查询级别上,可以使用Boosting来提高或降低特定查询的相关度得分。这种Boosting通常通过设置查询子句的权重或使用特定的查询类型来实现。例如,可以使用更高的权重来突出某些关键词,或使用更复杂的查询类型(如布尔查询、模糊查询)来调整相关度得分。
2、文档级别的Boosting:在文档级别上,可以使用Boosting来提高或降低特定文档的相关度得分。这种Boosting通常通过为文档定义一个额外的因子或属性来实现。例如,可以为某些文档设置更高的权重,以便它们在搜索结果中排名更靠前,或者可以为某些文档设置更低的权重,以便它们在搜索结果中排名更靠后。
通过使用Boosting技术,可以根据具体需求和优先级来调整搜索结果的相关度得分,从而提供更好的用户体验和更精确的搜索结果。
以下是一个使用boost参数的示例:
1、首先,使用Kibana的Dev Tools或者其他方式创建一个索引,并添加一些文档。例如,创建一个名为my_index的索引,并添加一些包含title和content字段的文档:
PUT my_index
{"mappings": {"properties": {"title": { "type": "text" },"content": { "type": "text" }}}
}POST my_index/_doc
{"title": "Elasticsearch is powerful","content": "Elasticsearch is a highly scalable and distributed search engine"
}POST my_index/_doc
{"title": "Kibana is a visualization tool","content": "Kibana is used to explore and visualize data stored in Elasticsearch"
}
2、然后,执行一个带有boost参数的查询来调整字段的相关度得分。例如,执行一个查询,将title字段的相关度得分提高两倍:
GET my_index/_search
{"query": {"match": {"title": {"query": "Elasticsearch","boost": 2}}}
}
在上述查询中,我们使用了match查询来搜索包含关键词"Elasticsearch"的文档。
通过在match查询中设置boost参数为2,我们将title字段的相关度得分提高了两倍。
这意味着包含关键词"Elasticsearch"的文档在结果中会更加相关。
- 当boost大于1时,提高字段的相关度得分,使其在结果排序中具有更高的权重。
- 当0 < boost < 1时,降低字段的相关度得分,使其在结果排序中具有较低的权重。
- 当boost小于0时,会为字段贡献负分,可能导致字段在结果排序中的得分变为负数。