1. Introduction: Why Near Real-time Search is Needed?
In the IT field, near real-time search typically refers to a delay of approximately 1 second from data ingestion to searchability. Traditional databases like relational systems prioritize transactional guarantees but often sacrifice query speed; Elasticsearch achieves sub-second response times by integrating the Lucene engine with a distributed architecture while maintaining high reliability. For instance, in e-commerce product search, users expect immediate results after initiating a query, and near real-time search significantly enhances user experience. Elasticsearch's near real-time mechanism is its core competitive advantage over other search libraries, derived from its optimized indexing design.
2. Core Mechanisms of Elasticsearch Near Real-time Search
1. Lucene's Inverted Index and Sharding
Elasticsearch is built on Apache Lucene, and its search capabilities rely on the inverted index (Inverted Index). When data is ingested, Elasticsearch breaks each document into terms and establishes a mapping from terms to document IDs. To enable horizontal scaling, data is distributed across multiple shards (Shard), each maintaining its own index. The key to near real-time search is that the indexing process is non-atomic and occurs in stages, ensuring data is quickly available after ingestion.
2. Translog and Commit Mechanism
Elasticsearch ensures data persistence through translog (transaction log). When a write request arrives:
- The document is first written to the in-memory index (index buffer) in memory.
- Simultaneously, data is recorded to the translog file (persistent storage) for recovery upon service restart.
- When the translog reaches a certain size or time interval, Elasticsearch triggers a commit operation, writing the in-memory index to disk.
The core of near real-time search is: after ingestion, data can be immediately queried in the in-memory index, but it must wait for the translog to sync to disk. By default, Elasticsearch uses refresh interval (refresh interval) to control the conversion from in-memory index to disk, typically set to 1 second. This allows data to be searchable within 1 second after ingestion, achieving near real-time performance.
3. Role of Refresh Interval
refresh interval is a critical parameter governing near real-time behavior. The default value is 1 second, meaning Elasticsearch refreshes the in-memory index to disk every 1 second. The refresh process:
- Writes the in-memory index to a disk copy (called a segment).
- Generates a new searchable index for queries.
Why is it near real-time? Data is immediately searchable in the in-memory index after ingestion, so queries can return new data. However, strictly speaking, data is searchable in the in-memory index while disk synchronization occurs as a background operation to ensure reliability. If faster response is required, the refresh interval can be reduced (e.g., to 0.5 seconds), but performance must be balanced: frequent refreshes increase I/O load, potentially impacting write throughput.
4. Data Flow in Practice
The typical data flow during ingestion to Elasticsearch:
- The client sends a request to a coordinating node.
- The node distributes data to primary and replica shards.
- Data is written to the in-memory index and translog.
- Every 1 second, refresh triggers, writing the in-memory index to disk.
- Query requests can immediately utilize the in-memory index and return results.