How Does Elasticsearch Handle Full-Text Search and Relevance Scoring?

Elasticsearch's full-text search capability relies on the Inverted Index, which breaks down document content into tokens and establishes a mapping from tokens to document lists. This structure transforms search operations from linear scans into O(1) complexity index queries.

1.1 词项分词与分析

When documents are indexed, Elasticsearch processes the text using the Analyzer:

• Tokenizer: Splits text into tokens (e.g., the standard tokenizer processes "Elasticsearch" as a single token).
• Filter: Applies filters (e.g., lowercase converts text to lowercase, stop removes stop words). For example, the analyzer configuration is as follows:

json
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop"]
        }
      }
    }
  }
}

1.2 倒排索引结构

The inverted index stores a mapping from token -> document ID list. For example:

• Token "Elasticsearch" -> Documents [1, 3]
• Token "search" -> Documents [2, 3, 4]

This structure supports efficient queries: when a user inputs a query term, Elasticsearch only scans the document lists containing that token, not all documents.

二、相关性评分：BM25 算法的核心作用

Elasticsearch defaults to using the BM25 (Best Match 25) algorithm for calculating relevance scores. This algorithm is a probabilistic model that considers term frequency, document length, and collection size.

2.1 BM25 算法详解

The BM25 scoring formula is:

$$ \text{score} = \frac{k_1 \times \text{tf} \times \log\left(\frac{N - n}{n + 1}\right)}{\text{tf} + k_1} $$

Where:

• tf: Term frequency (number of occurrences in the document).
• N: Total number of documents.
• n: Number of documents containing the term.
• k_1: Tunable parameter (default 1.2, affecting term frequency weighting).

Elasticsearch controls the number of matching tokens via index.search.max_expansions to avoid excessive expansion.

2.2 与 TF-IDF 的对比

• TF-IDF: An earlier method that only considers term frequency and inverse document frequency, ignoring document length.

• BM25: Superior because it introduces document length normalization (doc_length and avg_field_length), reducing penalties for long documents. For example:

  • Document length = 100, avg_field_length = 50, then the weight is higher.
  • Elasticsearch defaults to enabling bm25, and you can adjust the default field via index.query.default_field.

三、实践：代码示例与优化策略

3.1 创建索引与执行搜索

The following example demonstrates how to implement full-text search using the REST API:

Create Index (enabling custom analyzer):

json
PUT /products
{
  "settings": {
    "analysis": {
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "porter_stem"]
        }
      }
    }
  }
}

Index Document:

json
PUT /products/_doc/1
{
  "title": "Elasticsearch Introduction",
  "description": "Practical guide for distributed search engines."
}

Execute Search (using match query):

json
GET /products/_search
{
  "query": {
    "match": {
      "description": "search"
    }
  }
}

The result includes a score field, for example:

json
{
  "hits": {
    "hits": [
      {
        "_score": 0.65,
        "_id": "1",
        "_source": { ... }
      }
    ]
  }
}

3.2 优化相关性评分

• Adjust k_1 Parameter: Use index.search.max_expansions to limit the number of matching tokens (default 25) to avoid performance degradation.
• Use Field Data: Ensure search fields are of text type (e.g., ""type": "text"") rather than keyword.
• Enable explain API: Analyze scoring details:

json
GET /products/_explain/1?explain=true
{
  "query": {
    "match": {
      "description": "Elasticsearch"
    }
  }
}

• Optimize Index: Regularly use refresh strategy to reduce latency, or optimize merge policies via index.merge.policy.

Practical Advice: In production environments, monitor scoring changes using the _search API's explain parameter. For example, when users query ""Elasticsearch"", check if the score is reasonable due to document length normalization. For high-traffic scenarios, use index.query.default_field to specify the default search field for consistency.

四、结论

Elasticsearch efficiently handles full-text search through inverted indexing and the BM25 algorithm. Its relevance scoring mechanism requires adjustment based on business needs. Developers should focus on:

• Understanding BM25 Parameter Impact (e.g., k_1 and b).
• Validating with Code Examples: Test scoring during development using match queries.
• Continuous Optimization: Monitor index.search.max_expansions and document length to ensure search performance.

Mastering these technical points significantly enhances search experience. Elasticsearch's flexibility makes it suitable for log analysis, e-commerce search, etc. We recommend using Kibana Dev Tools for practical verification. Ultimately, relevance scoring is not just a technical issue but a key to user experience—careful design ensures search results truly meet user needs.

​